Page 1 of 2

Control UI Tutorial

Posted: 30 Mar 2018 16:31
by Desmanto
Control UI, The Most Powerful Accessibility feature from Automagic

Android Accessibility feature which was introduced in JB 4.3, was first designed to help disabled user. It still true until now. But the power user including Automagic users have found a new way to use this feature to make our life easier, even if we are not disabled. If you can think that laziness is one of the disability, then Control UI really fits in the context of helping people with laziness disability.
Accessibility feature allow the user to be assisted by the OS to click, scroll, or selecting certain UI Element. You can see various usage of this feature in my other thread : List of Accessibility usage : Control UI / UI Event

Accessibility in Automagic are used by 4 elements,
3 triggers : UI event, App Task Started, App Task Ended
1 action : Control UI.
In here, I am going to show more about the Control UI.

For Ex-tasker User
For you who came from tasker, I assume you must be familiar with plugin AutoInput. Control UI behaves almost exactly like AutoInput Action, except it is presented in script format. And the good news is Control UI is built-in inside Automagic, no need to install any additional plugin. All function that you can find in AutoInput, most of them are already built-in inside Automagic (not all under Control UI, but can be Screenshot, screen recording etc).

You must be thinking, how can control UI is an UI action, but presented using script? How is that possible? That used to be my question too. But after using it for a while, I found out Automagic Control UI is far more superior and flexible than any other Automation app implementation I've ever tested. It is because this sole feature that makes me convert from tasker to Automagic. (Although I found out more reason along the process). You will see later, how easy and flexible is it to create the script, even you have no scripting experience. (of course, having it will be an advantage)

Control UI require Accessibility services. You can turn in on from Automagic settings : Hamburger menu > Settings > Features & Permissions > Accessibility service > check > and you will be brought to android accessibility settings > Enable/activate it. Next, if you are using Android MM 6.0+ (or some LP 5.0/5.1 custom ROM) you also need draw over other apps permission too, to help you create the script easily. You can find this in your phone settings > manage app > Automagic Premium (App Info) > Draw over other apps > Yes. In some cases, you might need to enable the Modify system settings too. Some phone may put this option in different location, it is your part to find it and enable.

Just remember, that both these permission must be enabled first. Automagic will warn you if accessibility is not enabled yet. And if you don't see the red overlay button after you press show overlay button, that means the draw over other apps permission still not granted. On some custom ROM from certain vendor, accessibility service has been crippled or modified as such that It won't work anymore in Automagic. For those kind of ROM, you can try to change the Accessibility Feedback Type and see if it helps. If still cannot, then unfortunately there is no other workaround. Some other ROM may have built-in accessibility which conflict with Automagic. For those kind of ROM, you decide by yourself, which one is more worthy. As you can only use one of them only at one time.

Google intervention
After later you find out how powerful this accessibility, you will know why it can be used for malicious things as well. That's why google try to disable this feature in November 2017. There are a lot of outrage caused by this silent decision, you can read more at the thread. All technology always poses double edge function, it is up to the user to use it for their own benefit. So if you know you use this accessibility for your benefit and help you in your daily life, please support Automagic when Google try to remove this service. For mean time, we are in the unknown state, google not yet approved the continue use of Accessibility, nor to banned it.

Things to Note
1. Control UI function only available in Action Control UI script. You can't use any of those Control UI function in script or expression. But you can use any of the script function inside Control UI. So you can say that Control UI is Script powered up by accessibility.

2. Since every Android is designed differently, Control UI will behaves differently in each phone. I always say that I can't help you in creating the Control UI, because script that works in my phone, doesn't guarantee will be working too in your phone. Especially when we are crossing brand, ROM version, Custom ROM, or android version. Even the same phone model can be different if one is using MIUI ROM, while the other is using AOSP or LineAge OS. Heck, even the same phone using the same ROM can behave differently, because I can maybe change the DPI of my phone to makes my UI element become smaller; while the other use bigger one. Or maybe I have installed some additional app to change the layout and add new element in certain UI, while the other remain stock. The result is my Control UI script won't work even though we are using the same phone. Or even if I simply change the language to different one, the same Control UI at the same phone can fail to work properly. Thus this Control UI is your part to do it. I only can give you general tutorial on how to do it. That's why I always say I can't help much when it comes to Control UI, you have to do it by yourself.

3. The next thing is Control UI only works as far as accessibility allowed. In some secured app, accessibility is disabled. Any app that utilized webframe component (most browser) also will sandboxed its process from accessibility. So basically you can't use Control UI to click anything inside the webframe element. It probably can detect it, but it won't work when you use the script. But you can still click or interact with the element outside of the webframe (example Chrome's URL bar, 3 dot menu and all of setting). Anything that you can't click manually will remain unclickable. But sometimes you can still can grab its text or check its visibility.

4. Control UI cannot click on floating overlay windows. It will only react to the foreground app. So even if you have overlay covering the button, it still can click the button underneath the overlay. This can be useful or can be used maliciously. As you can spoof the button view, but still perfoming action beneath it.

5. Control UI only works when the screen is on and unlocked (of course). You cannot use it when screen off. So if you need to start the Control UI from screen off position, you have to add additonal turn on screen, unlock and then continue to the Control UI part. If you need it back off, you need close the flow with action lock screen again.

Table of content of this thread
This will be a comprehensive but not exhaustive tutorial. I will add more and more detail when it is possible (can't promise when). Look at this list to see what get covered in this thread. Each post is limited to 3 pictures, so I will maximized as far as I can.
1. Toggle Airplane mode
2. Toggle wifi tethering

Coming soon :
Checking input dialog multiple choice
Prefill expression multiple trigger
Prefill input dialog
Waiting for element to exists
Whatsapp send, individual or group
Clicking camera button, fetching long exposure time
getting chrome url
scroll finding certain text
combining scrolling text
multiple branching in single Control UI
emulating typing, ghost typing
search and replace text

Re: Control UI Tutorial

Posted: 30 Mar 2018 16:32
by Desmanto
Turning on Airplane Mode

Warning : Before you start this tutorial, I need to warn you to do this only at home. Don't try this tutorial when you are inside airplane or any place that restrict phone signal, as it will toggle your airplane mode on and off. Failing to follow this warning, may get you into trouble (fined by the airport officer or put into jail), risk your life, other's life or even the whole airplane. When you continue this tutorial, it means you have know about this warning. I am not responsible for any mis-use or risk caused by this tutorial.

Starting from JB 4.2, Android has been limiting toggling airplane mode by user app. This cause any automation app failed to toggle airplane mode, except when using root. But not everyone has the luxury to root the phone, there are so many reason not to do it or maybe there is not even an option to do it (locked bootloader). So our final hope to toggle it is using accessibility, which is Control UI part in Automagic. We need to navigate to system setting, where the Airplane toggle located and tap it to toggle it. Or we can use quick settings too. I am going to explain only the system setting one, as quick setting toggling is not working reliably in all android.

Most user may want to auto turn on/off airplane mode for certain occassion. Example maybe turn on airplane mode every night at certain hours, to prevent any disturbance. When using Control UI with these time trigger or shortcut trigger, we usually need to pair it with Launch app, or Start Activity. We want to go to the place where the airplane mode toggle located. This is different in each phone, so you have to find it by yourself.

Launch App
So create a flow, add action > new > Launch App. Launch app actually can launch app directly at its activity (class name).
1. Launch App.png
1. Launch App.png (248.17 KiB) Viewed 12581 times
At mine, the airplane toggle is located at settings, which is (most phone use this). Then, I need to find where the class which correspond to the airplane mode toggle location. You can guess and try, or you can use my Windows Recognizer flow to detect the class name. At my phone, the class name is Wireless & networks (.Settings$WirelessSettingsActivity).

After selecting, you can save the action. And try to execute the flow (or that action only). Your setting where airplane mode toggle located should open. If it is wrong, or it doesn't open at all, try another class name. Or maybe you need to confirm the app name and class name using my windows recognizer flow above. It is your part to find it.

Adding Control UI
After you have added the Launch app and it successly brings you to the airplane mode toggle location, you can now add Control UI action. From launch app, drag down the plus sign > Action > New > Control UI
2. Control UI.png
2. Control UI.png (156.81 KiB) Viewed 12581 times
Show Overlay Control
You should see the default screen of Control UI. Delete the default click("Help") and tap the Show Overlay Control to pop up the overlay. You should see something like mine, two red rectangles with text INFO and CLOSE. For Android 7.0+ Nougat above, you will see additional text GESTURE in the middle.
3. Overlay.png
3. Overlay.png (93.9 KiB) Viewed 12581 times
If you can't see the red overlay, it means you have set the permission properly, especially granting draw over other app permission to Automagic, See post 1.

Re: Control UI Tutorial

Posted: 30 Mar 2018 16:33
by Desmanto
Now here comes the fun part. You can press home to navigate manually to your airplane mode toggle location (leave the Control UI open). Or you can press save and go to the Launch app which you have created just now, open it and execute it manually; as it will bring you directly to the Airplane mode setting. Either way works, I prefer execute manually, as It is precisely as what will happen when you execute flow.

If you do it manually you can see something like this. Settings > More
4. Settings.png
4. Settings.png (89.19 KiB) Viewed 12580 times
But if I execute manually from the launch app, I will arrived immediately at the More windows (the right one), bypassing the first screen (left one). We only need the More windows, since Airplane mode is located here.

Since we are at the screen where we have the toggle already, we can now use the overlay to select the element. Tap the Info and you will see green rectangle bordering all available elements in the screen.
5. Click Airplane.png
5. Click Airplane.png (181.15 KiB) Viewed 12580 times
This green border denotes what elements are found and can be used or interacted to. In this case, I want to toggle the Airplane mode. I have 2 choice, clicking at the Airplane mode text or clicking at the toggle at the right of the Airplane mode. If I click on the toggle (right), it will show me several choice and there is no click("Airplane mode"); here. So we can't use the right one. There is clickById() which is preferred in certain situation, but we can't use that now (I will explain later). So we can only click at Airplane mode text and prompt the choices at the left one. Tap that click("Airplane mode"); to copy it to the clipboard.

You will see the toast message that "Action appended to clipboard content". Press back to close the green border overlay. We have the action copied already, we can go back to Automagic and open the same Control UI we have created just now. Paste the script there and Close the red Overlay. The result is the flow will be something like below.
6. Paste.png
6. Paste.png (152.46 KiB) Viewed 12580 times
So the flow only consist Launch app, launching immediately to the screen where Airplane mode toggle located and perfom the action of clicking the Airplane mode. This is a very basic Control UI usage, just to click single element. Try it out to see the result, execute the flow manually (remember the warning above, don't try it if you are inside airplane). You should see Automagic open the setting app where the airplane mode toggle is located, and automatically click that (toggle it).

If the Airplane didn't toggled, probably the Control UI is executed before the setting launched properly. You can try to add sleep(1000); (sleep 1 seconds) above. So your script become

Code: Select all

click("Airplane mode");
Try again, it should work now.

If it is still not working, probably your accessibility services got killed by the battery optimization. Try to disable automagic accessibility services. Restart and enable it again. This is mostly ROM's problem. You have to make sure Automagic is whitelisted from all battery optimization or task killer. Never use "Clear All", "Clean RAM", "Optimize Memory", "Kill all task" or similar button.

Re: Control UI Tutorial

Posted: 30 Mar 2018 16:35
by Desmanto
Toggling Wifi Hotspot

Need two clicks or more
So you have tried the Airplane mode and success. This airplane mode only require single control UI. But what if we need two clicks instead of one. So here I take another example for Wifi Hotspot (or tethering). In Android Oreo 8.0 above, the wifi tethering require extra workaround to make it work. If the workaround is not working for you, can use this Control UI to toggle it.

Actual Shortcut
At my phone, I actually have the direct activities to the tether setting (the toggle location).
Package Name :
Class Name :
You should use this when it is possible. We want to launch the direct activity as far as we can and minimize the use of Control UI when possible.

But in here I am going to use the same launch app from the airplane mode above. As it will demonstrate what happen when you need 2 or more clicks. If your phone also has similar layout with mine, you can use the similar concept to dig down the menu for the seconds clicks. I am going directly to the green border now, as the detail has been explained above.

First add action the same as the Airplane mode. So it will bring me directly to the More. Then tap the INFO to activate the UI selection. Tap on Tethering and tap click("Tethering");
7. Click Tethering.png
7. Click Tethering.png (142.24 KiB) Viewed 12548 times
After that, the script will be copied. Press back to close the green border overlay. Tap that Tethering to open the menu just like the 3rd windows (right). We need to toggle that Portable Wi-Fi hotspot.

So the same window, tap INFO > tap Portable Wi-Fi hotspot and tap click("Portable Wi-Fi hotspot"); It will copy and append the script again to the clipboard (not replace, but append).
8. Click Portable Wi-Fi hotspot.png
8. Click Portable Wi-Fi hotspot.png (201.87 KiB) Viewed 12548 times
Press back and and close the overlay. Go back to Automagic Control UI and paste the script. You can see that the two click action was separate by sleep(1000); This is sleep by default, to make sure the Control UI has enough time in between action. Save and close the red overlay.

Now you can try again the flow, execute it and see if it works. The first and second click will delay 1 seconds (because it has sleep(1000) in between). If you have problem, you can troubleshoot just like above.


Optimizing the sleep delay
So you have sleep(1000) by default. Can we reduce it so the script can perform faster? Yes. Depends on your phone spec and ROM, in most cases, you can reduce it to 200 ms without any problem. If the app support hardware acceleration, and your phone is very smooth (and you have turn off windows animation and transition), you can reduce it as low as 10 ms only! But beware, reducing it too low will cause your script to skip; because it doesn't have enough time to wait in between action. So you need an optimal value. For more detail on how much sleep you need between each action, you can read : How much sleep do I need?. I have explained along with the example and how to test for the best optimal sleep value.

In our case, we would try to reduce it to 200 ms first (try 500 ms if your the script skips). And then in most automation, we would like the toggling done and the app return to previous state before the toggling. If your script is only just like above, each execution will bring you to the toggle setting and stay there. It won't back to your current app. So we have to add back action to the script, depends on how deep we are inside the UI. For the first Airplane mode, we only open app once and perfom action there, so we can just add single back();

Code: Select all

click("Airplane mode");
After the airplane mode toggled, it will back to the previous app.

Back to original state
For the second tethering, we have dig down one level after the launch app, so we have to add back twice.

Code: Select all

click("Portable Wi-Fi hotspot");
The number of back needed is depends on how many levels you have dig down since the launch app. Even if you perform 10 clicks, but only dig down 3 times; you only need 4 back() only (3 dig down + first launch). But of course depends on how the app react to the back(), some may require extra back() again to confirm exit, some need only 2 back() to exit completely now matter how deep you have dig into the app. Please modify it depends on the app (that's why it is very phone specific).

All in One Go
You might wanna ask, how far you can do in one run of the Control UI script? How long the script can be appended? The answer is : practically unlimited, as long as you have the red overlay on. The script will keep be appended until you close the overlay. So if you are going to use it for another separated script, make sure you close it and tap Show Overlay Control again to reset the clipboard.

This is exactly what I love from Automagic's Control UI script. You can do all the things in one run only. For you who ever tested the AutoInput in tasker, you must already know that each Autoinput action requires you to go back to tasker. So if you have 10 actions, you have to switching back-thru for 10 times!!! While in Automagic, you can do them all in one run and switching back-thru only once! Once you get used to Automagic's Control UI, you will never go back to the other solution anymore.

Various available Control UI function
If you scroll over the Control UI documentation, you will see various of function you can use. For each of these I would roughly split them to 5 groups :
1. Generic button : such as back(), home(), recents()
2. Gesture (only in Android 7.0+) : toggleSplitScreen and touchGesture
3. UI interaction : click, longclick, check
4. Text interaction : cut, copy, paste
5. Keyboard/character input (require Automagic keyboard selected) : sendKey, sendText

From all of those, I only utilize probably 10 of them so far. The most used one of course are sleep(), back(), home(), click(), check()/uncheck(), scrollForward()/scrollBackward(), getText(0, setText2(), existElementById() and sendKey(). There are some rare occassion I would use another, but these are the most common ones I found useful till now. We have been using sleep(), back(), and click() so far. I will show you more later when possible.

For gesture function, it is actually very interesting, as you can simulate swipe. But unfortunately I only can test it in emulator. Keyboard input also a very interesting things to use, but so far I haven't found the practical use of it, except for special occassion (show off).

Which to function choose?
When you look over the overlay and presented with so many choices, you must be thinking which one should I choose for my script. Of course, usually it is very obvious already. If you need to click, choose the click ones. But there are 3 click version, so which should I use?

In most of the UI interaction groups, we have 3 types for the same function. Example for click(), we have 3 types of click function :

The same goes for other UI interaction; so in check(), you have check(pattern), check(x,y) and checkById(id).

In terms of usage, at any point, we should try to prioritize the use of clickById(id) first. If it is not possible, fall back to click(pattern) and last resort will be click(x,y).
1. clickById(id) will click on the element which has the Id as the name suggest. Android app UI element will have the element Id associated to it. If you point to this element Id, it doesn't matter if somehow the label change to something else, Control UI still can click that element. For example, if you change the language to Indonesian, the label of the "Airplane mode" will change "Mode pesawat". Using the same script above, it will now fail to toggle the Airplane mode, as it can't find any text labeled as "Airplane mode" anymore.

But here lies the problem. As you can see from the Airplane mode and tethering cases, both have the same element ID : "android:id/title". In fact, all the label from Data usage, Airplane mode, Cast screen, Tethering, VPN has the same element ID : "android:id/list". When multiple same element ID detected, and we use clickById("android:id/title"); only the first one from the top left corner will be clicked! (or interacted to, if you use other function than click). In this case, only Data usage will be clicked, no matter how many other elements with the same Id exist in the same window.
9. Click ById.png
9. Click ById.png (118.75 KiB) Viewed 12548 times
This is the problem when the UI element are in the list format as the one we have typically in setting or list of items. To solve this problem, we need a new function which can traverse through the index of the elements which has the same element ID; so in my case, something like clickListId("android:id/title", 0) to click the Data usage. But the function doesn't exist yet in Automagic 1.34.0. I have reported this to Martin, and we have to wait until it is implemented. (no guarantee yet that it will be implemented soon)

For mean time, when you see something like this, you have to fall back to the next function click(pattern), which is what we have done above. clickById(Id) will be mostly useful for element which has single separated function, such as Automagic yes no button in various elements will be
clickById("android:id/button1"); for yes
clickById("android:id/button2"); for no
or something like the back menu at the top left of my More Settings (the < sign) is
Use it when there is no duplicate element Id.

2. click(pattern) will be used when you can't use clickById() or when the element doesn't give you the choice for the clickById(). If you use this, it will search for that label from the top left of the screen and click any first label matches exactly as that. In most cases, UI element always has different label in the list, so this will work generally. It doesn't matter if somehow the upper element suddenly has longer text and move the label a little to the bottom, or the font size/screen dpi/resolution change. As long as the label still can be seen from the same window, it can be clicked by Control UI.

However there are some limitation :
- If there are two button has the same label in the list (duplicate item), only the first element from top left will be clicked.
- When you change the language/locale of the phone, even it is only for that app only, then this click(pattern) will fail to click.
- If the label is dynamic, where the text changes from time to time, you can't click that reliably.
- In a rare case of changing font size, dpi or screen resolution; there is possibility that the label will wrap wrongly and doesn't show up properly.
- the text must be perfectly match. Some label may have extra spaces or invisible special character.
This is why you only use this when the first clickById() can't be used. Use this also when there is no duplicated label, static label and when you don't change the language of the phone.

3. click(x,y) will be used at the very very very last resort when none above work. This will simulate click directly at point x and y, almost the same as execute root command input x y. The difference is click(x,y) is accessibilty action which can click thru webframe. While input x y in execute root command is direct hardware touch event, which can click thru even the most secured app or webframe. click(x,y) can bypass the weakness of the clickById(Id) and click(pattern) when there is duplicated element or duplicated label.

But you have to be careful when using this as it can fail from time to time. It is not Automagic fault that it doesn't perform click(x,y). But it is the element that now doesn't occupy the same pixel when you create the script. The best is it fails to perform the action and do nothing. The worst is it can mis-click wrong element, creating logic flaw, such as when you want to "cancel", but because the pixel has shifted, now you click "yes". This can result in havoc; can be you mis-transferred money to the wrong person or reply yes to trigger the nuke button. So you have been warned! I suggest not to use it all, find other option instead. But it is provided here as the accessibility allowed it and you probably still can find some good use from it.

The limitation, which include the limitation from click(pattern), added :
- the x,y point can move radically depends on the target app.
- if you change dpi, resolution, font size; then it will fail and worst is it can click wrong element (result in chaos)
- Any dynamic content of the windows will certainly fatal to this function, as all elements will change their location from time to time
- In some case when you change language, the label can be shorter or longer than expected, changing its x,y point
- Unlike clickById(id) which search for element Id or click(pattern) which search for element text; click(x,y) doesn't check against the element located at that point. So even if it is not the intended element, it will still click it. Thus you need to protect this by checking the surrounding element for certain elemend Id or text when it is possible to do so. (will be explained in the future).

That is why this is the last resort, and require extra check when you really have to use it. Use it for a very static window content that the position of the element will never change anymore even if you change the resolution, dpi, language or other variable. If the window contains dynamic content, you should use extra check on nearby element Id/text to ensure you are clicking on the correct x,y.

As most UI interaction function have almost similar 3 types for each action, the same rules of usage and limitation applied to most of them too. So you will still prefer to use checkById(Id) instead of check(pattern).

Re: Control UI Tutorial

Posted: 30 Mar 2018 16:36
by Desmanto
Ah, the above ones tooks tens of hours to complete. I have been typing this since yesterday and several days ago (while still doing other things too). It still cover probably 10% of the Control UI usage I have been using till now. That's why forgive me if I reserve all the first page posts for my future updates. Can't promise when I will update this, but surely and slowly I will put more and more info. :D

Autumn-Leigh has made the video tutorial for Control UI

Re: Control UI Tutorial

Posted: 30 Mar 2018 16:37
by Desmanto
Reserved for futures update. Sorry for taking up the space, but I need it.

Re: Control UI Tutorial

Posted: 30 Mar 2018 16:38
by Desmanto
Reserved for futures update. Sorry for taking up the space, but I need it.

Re: Control UI Tutorial

Posted: 30 Mar 2018 16:39
by Desmanto
Reserved for futures update. Sorry for taking up the space, but I need it.

Re: Control UI Tutorial

Posted: 30 Mar 2018 16:40
by Desmanto
Reserved for futures update. Sorry for taking up the space, but I need it.

Re: Control UI Tutorial

Posted: 30 Mar 2018 16:41
by Desmanto
Reserved for futures update. Sorry for taking up the space, but I need it.