Page 1 of 1

List of Accessibility usage : Control UI / UI Event

Posted: 08 Dec 2017 17:29
by Desmanto
Part of the modified quote from Google's email to dev.
uses the Accessibility API for a responsible, innovative purpose that isn’t related to accessibility
how your app benefits users
As we know enabling accessibility in Automagic allow us to use trigger UI Event and Action Control UI. These are ones of the main reason I switched from tasker to automagic. I have used it to do many things which really help my daily activities; even though I am not disabled, at least not physically. Well, if you consider laziness as one of the disability, then maybe I fit to that category. :) Accessibility has really make my life much easier, doing things in ways those I've never imagined before.

These are the things I have done using Control UI and UI Event in Automagic :
01. Sending whatsapp chat : Even though Whatsapp has API for it, the last send button still need to be pressed manually. Control UI can press the send button. The flow has been expanded to auto reply. It will detect the incoming sender and message using notification, then if it fit the condition (certain sender/message), it will auto reply them. Kinda like auto responder, useful for meeting, driving, vacation. It can be used to send message at scheduled time, such as wishing birthday, sharing our current location (google maps) to someone or reminding them for upcoming schedule (appointment, important meeting or other)
02. Printing a web page to pdf using Google Chrome for Android. So far, the best way to do it is using Google Chrome. Using Control UI we can automate the whole process and save the final web page into PDF.
03. Querying Chrome URL address bar. I have gesture mapped to execute the flow, querying Chrome URL address bar. So far, only https url can be copied by tapping the https and long press on the address. Using Control UI, I can query the text in chrome's url, regardless of the http. And the queried url don't have http in front of it; so it can be used properly. Since Control UI has the script ability, i just make 2 lines of script to check the if it doesn't has http, it will append one. The url copied can be pasted anywhere, or maybe used again in another flow. Such as, drawing another W gesture, will share the link directly to Whatsapp where I only need to pick the recepient. Or can be used to open in another browser, downloader, send to PC via eventghost (to open the copied url directly at PC's Chrome; must faster than waiting for the sync).
04. Multiclipboard : While there are a lot useful clipboard manager app out there; for me It is better to create my own clipboard manager using Automagic. Why wasting another app resources if all can be done by using Automagic alone? Part of the flow requires to paste the clipboard when tapping on the widget. So without Control UI, I can only tap to copy only, need additional long press and tap paste. Using Control UI, I can integrate the paste() function directly, much faster
05. Extending Obsqr : While automagic has action Scan barcode; it can detect several kinds of code. Sometimes it can misdetect the QR code as barcode, when the text get longer (more that 1.000 char). So I use Obsqr for scanning long QR code. The problem is the app always detect link and only allow open link, but not copy it. So I use UI event coupled with Control UI to detect the scanned text and copy it if it is a link. The flow can be expanded to log each scanned QR and give it proper timestamp, save it as csv or maybe sent it somewhere else.
06. Stitchcraft auto start. To take a new long screenshot, several tap is needed. I automated all those tap to single tap shortcut to flow in automagic.
7. Downloading multiple images. When using chrome to download multiple images, I have to long press and click "Download Image". Doing it 10-20 times can be boring. I just activate the flow, so everytime I long press the image, automagic will click the "Download Image". Saving the additional tap
08. Flowception. When creating flows in Automagic, I have several elements that I access frequently. Since Automagic doesn't have favourite element option yet, I made a flow to help me create the flow (thus flowception). Using Control UI, I navigate thru the element selection menu and select my favourite element when I select it in the overlay widget. Another similiar flow is to create multiple parallel expression, where I parse the trigger name and automagically created the expression to filter the trigger.
09. Logging open file for certain text. When testing app for certain text, I use UI event and Control UI to get the text from the opened apps, log it to file so I can check it later.
10. Automatic telnet client. I haven't found any working telnet binary which can be automated. So I simply use Telnet client, use Control UI to feed in the input of username password and the command needed. Then parse the output text and get the text I want, save it into a variable. The whole process is so fast, that it takes no longer than 2 seconds for the whole operation; which probably tooks more than 1 minute if I do it manually.
11. Automatic playstore update during certain time/network. Our internet provider gives us a bigger internet quota during midnight (started at 1 am). So to minimize the use of regular quota, we will prefer to update the apps only after that time. The problem is most of us have went to sleep already. So using Control UI, we managed to automate the update checking at that time, and update all as necessary. If there is a new permission, log it, continue the update but gives notification after finished.
12. Whatsapp group adding new number. How many times we join into a new whatsapp group with a lot of people we don't know yet? We have to select each of them and save them one by one. Imagine doing this for 30 or maybe 100 participants! Using Control UI, I can query the whole group number along with its name, prompt a new window to select which number to be added to the contact. After done, automatically creating the contact in the phonebook.
13. Numbering file. At times, i have several files need be renumbered. The file names are random and don't have any pattern, so need to be renumbered manually. Usually I have to long tap the file, rename, tap the input text, tap the begining of the text, type the number I need and tap OK. It is too long and very painful to do it for 10+ files. Using UI Event and Control UI, I can trigger the folder to be rename. Next, I only need to tap the file sequentially. Tap 1st file, it will be renamed to 01. filename.ext; 2nd file, 3rd file and so on. All renaming now require only single tap, instead of severals.
14. Eject OTG. My phone don't have notification to eject the USB OTG. I have to use Control UI to manually scroll to the location where I can tap the "unmount" button. I can do it all automatically, but I prefer to stop there and press unmount manually. SInce there is "format" button also nearby, I don't wanna risk pressing the wrong button.
15. Firewall shortcut. My phone has firewall built-in, but the menu is deep down several click and can't be invoke using activities. I use Control UI to slip thru the UI directly, Can be expanded to automatically scroll to find the last installed app and stop there.
16. Autofill in input dialog. I have web url intent received flow, where I redirect each url to corresponding app. During the redirection, when new domain/url type detected, I will be prompt to add a new choice. Control UI will automatically parse the domain only and set it to the input dialog box; so I don't need to retype the whole domain.
17. Refreshing widget only when in certain home screen page. These can be achieved for some certain launcher, such as Nova; which make announcement when it changes page. The same announcement event can be extended to detect alarm, fired alarm, snoozed, call etc.
18. Automating deposit purchase. One of my friend in forum need to automate the whole deposit purchase from PC. There is no program available for PC, so he has to send the request back to Phone. Since the app doesn't support automation or intent, he automated the whole process using Control UI to select the purchase nominal, fill in the phone number and send the request. Communication from PC to phone can be done via sms, local wifi, or other means.
19. Automatic camera capture. Not all phone has root function to emulate the camera button. Some camera app also don't react to camera button. So using Control UI, i can link trigger to press the capture button on any camera app. I usually paired it to other trigger method, such as HTTP request, voice command, timer, bluetooth headset media button. The concept can be expanded to create timelapse in any camera app, changing setting to certain profile even if the app doesn't support it.
20. Screensaver. Using component click and scroll, we can create the countdown timer for screensaver. If no component clicked/scrolled during certain duration, a screen saver will pop up and cover the whole screen.
21. Collecting data from Mi Fit. Sometimes i can forget that I haven't achieved my goal that day. So I made the reminder, so that it will open mi fit at certain time, connect to Mi band 2 and sync the steps. Then using Control UI to get the step count and give notification if the goal haven't achieved yet. This way, I won't missed the consecutive days goal.

There are probably still a lot of others usage I haven't put it. Maybe I will try to update it if I re-index the thread I have seen until now. Feel free to add part of your usage.

Re: List of Accessibility usage : Control UI / UI Event

Posted: 31 Aug 2018 03:07
by Bluscre
And there you have the problem. Even your first example is malicious because whatsapp's EULA states that only a user should send messages no machines or bots

Re: List of Accessibility usage : Control UI / UI Event

Posted: 10 Oct 2019 23:50
by icefox56
Can you share the 13th flow "Numbering file" . I have similar use case, but I'm not sure how to create a flow like this.

Re: List of Accessibility usage : Control UI / UI Event

Posted: 11 Oct 2019 17:28
by Desmanto
icefox56 wrote:
10 Oct 2019 23:50
Can you share the 13th flow "Numbering file" . I have similar use case, but I'm not sure how to create a flow like this.
That is old flow which only works with ES File Explorer. I don't use that anymore since it has gone rogue. I use MiXplorer now and it is different to rename file. I haven't need to use my renamer flow for a long time so I haven't modify it yet. But a quick change already can rename the quickly now, although in different way with previous version. I still need to work on the widget.