Control UI Tutorial

General discussions about Automagic and automation in general

Moderator: Martin

User avatar
Autumn-Leigh
Posts: 67
Joined: 15 Mar 2018 19:23
Location: UK
Contact:

Re: Control UI Tutorial

Post by Autumn-Leigh » 30 Mar 2018 21:35

Desmanto wrote:Control UI, The Most Powerful Accessibility feature from Automagic

Prologue
Android Accessibility feature which was introduced in JB 4.3, was first designed to help disabled user. It still true until now. But the power user including Automagic users have found a new way to use this feature to make our life easier, even if we are not disabled. If you can think that laziness is one of the disability, then Control UI really fits in the context of helping people with laziness disability.
Accessibility feature allow the user to be assisted by the OS to click, scroll, or selecting certain UI Element. You can see various usage of this feature in my other thread : List of Accessibility usage : Control UI / UI Event

Accessibility in Automagic are used by 4 elements,
3 triggers : UI event, App Task Started, App Task Ended
1 action : Control UI.
In here, I am going to show more about the Control UI.

For Ex-tasker User
For you who came from tasker, I assume you must be familiar with plugin AutoInput. Control UI behaves almost exactly like AutoInput Action, except it is presented in script format. And the good news is Control UI is built-in inside Automagic, no need to install any additional plugin. All function that you can find in AutoInput, most of them are already built-in inside Automagic (not all under Control UI, but can be Screenshot, screen recording etc).

You must be thinking, how can control UI is an UI action, but presented using script? How is that possible? That used to be my question too. But after using it for a while, I found out Automagic Control UI is far more superior and flexible than any other Automation app implementation I've ever tested. It is because this sole feature that makes me convert from tasker to Automagic. (Although I found out more reason along the process). You will see later, how easy and flexible is it to create the script, even you have no scripting experience. (of course, having it will be an advantage)

Requirement
Control UI require Accessibility services. You can turn in on from Automagic settings : Hamburger menu > Settings > Features & Permissions > Accessibility service > check > and you will be brought to android accessibility settings > Enable/activate it. Next, if you are using Android MM 6.0+ (or some LP 5.0/5.1 custom ROM) you also need draw over other apps permission too, to help you create the script easily. You can find this in your phone settings > manage app > Automagic Premium (App Info) > Draw over other apps > Yes. In some cases, you might need to enable the Modify system settings too. Some phone may put this option in different location, it is your part to find it and enable.

Just remember, that both these permission must be enabled first. Automagic will warn you if accessibility is not enabled yet. And if you don't see the red overlay button after you press show overlay button, that means the draw over other apps permission still not granted. On some custom ROM from certain vendor, accessibility service has been crippled or modified as such that It won't work anymore in Automagic. For those kind of ROM, you can try to change the Accessibility Feedback Type and see if it helps. If still cannot, then unfortunately there is no other workaround. Some other ROM may have built-in accessibility which conflict with Automagic. For those kind of ROM, you decide by yourself, which one is more worthy. As you can only use one of them only at one time.

Google intervention
After later you find out how powerful this accessibility, you will know why it can be used for malicious things as well. That's why google try to disable this feature in November 2017. There are a lot of outrage caused by this silent decision, you can read more at the thread. All technology always poses double edge function, it is up to the user to use it for their own benefit. So if you know you use this accessibility for your benefit and help you in your daily life, please support Automagic when Google try to remove this service. For mean time, we are in the unknown state, google not yet approved the continue use of Accessibility, nor to banned it.

Things to Note
1. Control UI function only available in Action Control UI script. You can't use any of those Control UI function in script or expression. But you can use any of the script function inside Control UI. So you can say that Control UI is Script powered up by accessibility.

2. Since every Android is designed differently, Control UI will behaves differently in each phone. I always say that I can't help you in creating the Control UI, because script that works in my phone, doesn't guarantee will be working too in your phone. Especially when we are crossing brand, ROM version, Custom ROM, or android version. Even the same phone model can be different if one is using MIUI ROM, while the other is using AOSP or LineAge OS. Heck, even the same phone using the same ROM can behave differently, because I can maybe change the DPI of my phone to makes my UI element become smaller; while the other use bigger one. Or maybe I have installed some additional app to change the layout and add new element in certain UI, while the other remain stock. The result is my Control UI script won't work even though we are using the same phone. Or even if I simply change the language to different one, the same Control UI at the same phone can fail to work properly. Thus this Control UI is your part to do it. I only can give you general tutorial on how to do it. That's why I always say I can't help much when it comes to Control UI, you have to do it by yourself.

3. The next thing is Control UI only works as far as accessibility allowed. In some secured app, accessibility is disabled. Any app that utilized webframe component (most browser) also will sandboxed its process from accessibility. So basically you can't use Control UI to click anything inside the webframe element. It probably can detect it, but it won't work when you use the script. But you can still click or interact with the element outside of the webframe (example Chrome's URL bar, 3 dot menu and all of setting). Anything that you can't click manually will remain unclickable. But sometimes you can still can grab its text or check its visibility.

4. Control UI cannot click on floating overlay windows. It will only react to the foreground app. So even if you have overlay covering the button, it still can click the button underneath the overlay. This can be useful or can be used maliciously. As you can spoof the button view, but still perfoming action beneath it.

5. Control UI only works when the screen is on and unlocked (of course). You cannot use it when screen off. So if you need to start the Control UI from screen off position, you have to add additonal turn on screen, unlock and then continue to the Control UI part. If you need it back off, you need close the flow with action lock screen again.

Table of content of this thread
This will be a comprehensive but not exhaustive tutorial. I will add more and more detail when it is possible (can't promise when). Look at this list to see what get covered in this thread. Each post is limited to 3 pictures, so I will maximized as far as I can.
1. Toggle Airplane mode
2. Toggle wifi tethering

Coming soon :
Checking input dialog multiple choice
Prefill expression multiple trigger
Prefill input dialog
Waiting for element to exists
Whatsapp send, individual or group
Clicking camera button, fetching long exposure time
getting chrome url
scroll finding certain text
combining scrolling text
multiple branching in single Control UI
emulating typing, ghost typing
search and replace text
This is amazing! Thank you so much!
YouTube Channel - https://youtu.be/fui3RO9EmNU
If anyone wants a shoutout, or has a flow they want me to do, be it theirs or a new one, let me know :)

goodyog
Posts: 4
Joined: 29 Sep 2018 13:24

Re: Control UI Tutorial

Post by goodyog » 02 Oct 2018 12:02

Desmanto wrote:Reserved for futures update. Sorry for taking up the space, but I need it.
Hello, thanks a lot for the detailed explanation.

I still want to ask if any update has been able to clickById on a specific element that is not showing on windows (I.e, i have to scroll down to see it). Because when I use the clickById, it just gives me something like this(screenshot) instead of what I specifically clicked and it performs nothing. I hope my screenshot will explain will to you
Attachments
PicsArt_10-02-01.00.16.png
PicsArt_10-02-01.00.16.png (587.07 KiB) Viewed 6828 times

User avatar
Desmanto
Posts: 2709
Joined: 21 Jul 2017 17:50

Re: Control UI Tutorial

Post by Desmanto » 02 Oct 2018 16:39

It has been added in Automagic EAP 1.36.0, clickByIdAndIndex(). You will have to wait until it is released in public version (playstore). You can see example of the loop scroll here : viewtopic.php?f=5&t=7580
Even though we will have that function later, you still need to search the current screen whether it contains the label that you want to click. For that case, I have made a similar flow, to fix a small bug at my phone. It can be done in Automagic 1.35.0, without the need of the new function.

My phone Redmi Note 5, either at MIUI or custom ROM AOSP, including RR 6.2 Oreo 8.1; has the bug when connected to certain car bluetooth audio, it will default to AAC codec. Resulting in no sound transferred. Some vendor has the firmware update to fix this. But for those without update, the only way is to go to developer option and click that codec, change it back to SBC. The sound will play normal after that. Since it is a UI thing, we can fix it using Control UI. The original post is at here : https://www.kaskus.co.id/show_post/5b9f ... 4567/578/-

Basically the key point is to loop scroll until it found label you want (using the combination of while and scrollForward()), then click that. You can't use clickById() since it is a list. You can only use click(pattern).

Code: Select all

sle = 100;
sleep(sle);
while(!contains(getTextInActiveWindow(), "Denmark"))
{
  scrollForwardById("android:id/title");
  sleep(sle);
}
click("Denmark");
sleep(sle);
back();
In your case, you want the "Denmark", and I assume the list has the same element "android:id/title". If not, change to the element Id you have there. If you need extra click after "Denmark", you need to add below that (example click "yes" to confirm).
Index of Automagic useful thread List of my other useful posts (and others')
Xiaomi Redmi Note 5 (whyred), AOSP Extended v6.7 build 20200310 Official, Android Pie 9.0, Rooted.

lojzik
Posts: 9
Joined: 10 Sep 2018 08:03

Re: Control UI Tutorial

Post by lojzik » 02 Oct 2018 16:45

Is it possible to use Control UI on locked phone?

User avatar
Desmanto
Posts: 2709
Joined: 21 Jul 2017 17:50

Re: Control UI Tutorial

Post by Desmanto » 02 Oct 2018 17:38

Yes and no. No means Control UI won't do anything when the phone is locked. Yes means, you can do it by first use action to unlock the phone, do all the Control UI stuff and lock the phone back. The unlock/lock process can be tricky and won't work properly on all phones.
Index of Automagic useful thread List of my other useful posts (and others')
Xiaomi Redmi Note 5 (whyred), AOSP Extended v6.7 build 20200310 Official, Android Pie 9.0, Rooted.

lojzik
Posts: 9
Joined: 10 Sep 2018 08:03

Re: Control UI Tutorial

Post by lojzik » 02 Oct 2018 19:18

Is there some example how to unlock phone (without root)?

User avatar
Desmanto
Posts: 2709
Joined: 21 Jul 2017 17:50

Re: Control UI Tutorial

Post by Desmanto » 04 Oct 2018 15:57

lojzik wrote:Is there some example how to unlock phone (without root)?
You can try Set Keyguard State (no root) or Set Lock Pattern State (probably need root), set it to off. Use action turn on screen and dismiss Slide to unlock keyguard if there is. Do all the Control UI job until finish. Set the keyguard state back to on, and turn off the screen.

This heavily depends on the device and ROM you have. In close to AOSP ROM, it will most likely work without problem. But in devices with heavy customization from the vendor, such as Touchwiz, EMUI, MIUI, ColorOS, Funtouch OS, you will find some action doesn't work well. That's why I flash custom ROM RR 6.2 Oreo 8.1 on my Redmi Note 5 which supposed to have MIUI 9.5.17.0
Index of Automagic useful thread List of my other useful posts (and others')
Xiaomi Redmi Note 5 (whyred), AOSP Extended v6.7 build 20200310 Official, Android Pie 9.0, Rooted.

lojzik
Posts: 9
Joined: 10 Sep 2018 08:03

Re: Control UI Tutorial

Post by lojzik » 04 Oct 2018 19:15

I use lock pattern and Set Lock Pattern State needs root mandatory.
btw how can be "smartphone" smart without sufficient automation support, touch-fest instead of automation isn't smart.

User avatar
Desmanto
Posts: 2709
Joined: 21 Jul 2017 17:50

Re: Control UI Tutorial

Post by Desmanto » 05 Oct 2018 05:47

How about Set Keyguard state?

There must be a balance between automation and security. You don't want to open so loose that any app can lock/unlock the screen for the sake of automation. Hence secure function like this most likely need root in the newer android version. You know, google keep tightening up everything in each new version.

The other ugly alternative is to disable the built in lockscreen, don't use it at all. Then you create your own lockscreen solution using automagic's widget. The widget will cover the whole screen and only certain swipe can dismiss this widget, thus unlock the screen.
Index of Automagic useful thread List of my other useful posts (and others')
Xiaomi Redmi Note 5 (whyred), AOSP Extended v6.7 build 20200310 Official, Android Pie 9.0, Rooted.

Post Reply