Error when deleting files

Post your questions and help other users.

Moderator: Martin

vertigo
Posts: 147
Joined: 28 Oct 2018 00:28

Error when deleting files

Post by vertigo » 02 May 2020 23:48

I created a very basic flow (attached) to run when I turn the display off or when I run either of two image viewing apps. When it runs, its purpose is to simply move files from a couple directories on internal storage to external storage locations. The problem is, while it seems to work as expected in most respects, I often get an error that says, for example, "Error: Could not delete file /storage/emulated/0/DCIM/Camera/IMG_20200505_161920257.jpg." Every time I get this, when I check, the file has been moved properly, so I'm not sure why it's giving me the error.

Also, as a side note, the reason for the triggers is because that was the best way I could come up with to get it to move the files after I take a picture or screenshot and before I try looking at them. Of course, it's far from ideal, since it requires me to either turn my screen off and on, which is cumbersome (though I often take a picture then turn it off, then maybe a bit later turn it back on to take another picture, so it works well enough in those cases), or it waits until I've already opened the image app, which means I then have to wait for the files to be moved and the image app to show them. I'd appreciate any ideas of possible better triggers. My initial thought was a file monitor, to trigger when there are any changes to the directories on internal storage, but I was worried it would cause issues if it triggered while I'm taking pictures, making the pictures save slower and therefore impairing the ability to take multiple pictures quickly. I decided to test it just now, and not only was I right, but it causes AutoMagic's emergency stop to kick in when pressing and holding to take numerous rapid succession pictures, and the phone became practically unresponsive, so that's definitely not the way to go, at least not without making the flow much more complex and even then it seems iffy. But I did get the "could not delete" error that way as well, so it's not related to the trigger.
Attachments
flow_DCIM_Screenshots_SD_20200502_185932.xml
(2.4 KiB) Downloaded 807 times

User avatar
Desmanto
Posts: 2709
Joined: 21 Jul 2017 17:50

Re: Error when deleting files

Post by Desmanto » 03 May 2020 11:37

It is quite tricky to work with file observer, so far I only use one. If I use my variable logger flow to log all the events from the file observer trigger (ticking all option); a single HDR photo shot from GCam will result in 2000+ triggers. That surely will hit AES in no time. If I untick the first 2 options (File accessed and Modify), the triggers dropped to 60+. So first 2 options should be avoided if not needed.

As my test for my scrcpy file drop flow, the last event of the file observer is 4 - metadata changed (checked using my variable logger). I use that one as the trigger and it works flawlessly until now. I drop an image into whatsapp chat and immediately share it to that chat. You might need to test your camera app to find out what is the last event. I think 4- metadata changed should be quite universal. But I still enable that flow only when I use mirroring, so it won't mistrigger when I am working with file in that folder when not in mirroring mode.
Index of Automagic useful thread List of my other useful posts (and others')
Xiaomi Redmi Note 5 (whyred), AOSP Extended v6.7 build 20200310 Official, Android Pie 9.0, Rooted.

vertigo
Posts: 147
Joined: 28 Oct 2018 00:28

Re: Error when deleting files

Post by vertigo » 03 May 2020 18:08

Well, even if I could get the file observer to work "properly," there would still be the problem of it triggering after taking one photo and running while I'm possibly trying to take other photos. That's why the screen off or image viewer opening triggers are good, because they mean I'm done taking pictures, at least for the moment. They're just not great, because it means the photos often don't transfer as soon as I'd like, but there's probably going to be a trade off one way or the other, and I've accepted that. At some point I'll get a new phone with more internal storage and be able to just keep all photos there and not have to deal with this at all. Or the storage might be fast enough to handle simultaneously transferring files to SD while saving new photos.

My real issue here, and the reason for the thread, is the error it throws when moving the files. I'd like to figure out why it's doing that so I can hopefully fix it.

User avatar
Desmanto
Posts: 2709
Joined: 21 Jul 2017 17:50

Re: Error when deleting files

Post by Desmanto » 05 May 2020 12:15

The error possibly caused by some app still locking file preventing the move. Automagic produce the error, but still trying to move the file afteward and success. Maybe you need to add action init variable file list first before that. Then add exception debug dialog to see what is going on when the error happened.

Other prevention you can try is to check the file existence before moving, using existsFile(). But it requires loop and won't detect if the error happen within action move files. So better troubleshoot first.
Index of Automagic useful thread List of my other useful posts (and others')
Xiaomi Redmi Note 5 (whyred), AOSP Extended v6.7 build 20200310 Official, Android Pie 9.0, Rooted.

vertigo
Posts: 147
Joined: 28 Oct 2018 00:28

Re: Error when deleting files

Post by vertigo » 17 May 2020 10:52

I think I figured out the problem, or problems. It boils down to the same thing, which is that the flow is triggered while it's running, causing it to execute the action to move files while that action is already running, and that's why it errors saying it can't delete the file, because by the time the second running of the flow gets around to trying to move (and therefore delete) the file, it's been moved by the first running of the flow. I'm not entirely sure why that was happening before, when I was using display off and app task started triggers, since those would only trigger it once, but I assume I was sometimes turning the display off then back on then off, or maybe off then the Moto display would come on, then go off.

Anyways, I decided to try using the file observer trigger, since the others are just not ideal at all, and that's when I figured out the problem, because when I take a picture, it appears to create a temporary file, which was triggering the flow, then when the final jpg file is created the flow is triggered again. So I added a 3s sleep at the very beginning and set the flow to stop the current instance and execute the newer instance, so each time it's triggered, whether by a temp file then the real image file or by taking another picture, the process will be reset. So far this seems to have solved the problem, and it should take care of the concern I had about using the file observer trigger, which was that it might interfere with taking multiple pictures since the storage would be busy doing file transfers, by resetting the flow in such a scenario.

I am curious, though, what would happen if it was in the middle of a file move when another trigger is received. Since it's set to stop executing, would the move finish before it stops, would the move be canceled, or would it simply be cut off and leave me with corrupted files?

Also, just an FYI, the temporary file type appears to be a .met file and/or a .mp4 file. The .met file seems to possibly be more common, and searching for info about it shows it to be a P2P/emule file, though that's clearly not the case here. I did find a couple somewhat buried references to .met files on Android devices, but I didn't find any real info about it, so I have no idea what's actually going on there. And the .mp4 is confusing, too, since at least once I saw one (using the debug condition) even though I was only taking pictures, not videos.

User avatar
Desmanto
Posts: 2709
Joined: 21 Jul 2017 17:50

Re: Error when deleting files

Post by Desmanto » 17 May 2020 18:13

To protect from multiple quick trigger, use FEP - Skip + sleep at least 500 ms at the end of the flow. You can use your current method FEP - Stop with sleep several seconds, but if the event takes 3 seconds, you still have problem. And as you might be afraid of, we don't know what will happen if the flow get stopped during the moving. So better FEP - Skip.

To use file observer properly, it is better to log down the whole file writing process. (using variable logger) Most of the time the file writing end with metadata changed or unwriteable file closed. Use this as the trigger in file observer. You will reduce mis-trigger and can cut down the number of flow execution to check for false trigger. If the it always trigger twice (metadata changed) within 3 seconds, you can use the that option and protect the main flow using execution count 2 in 5 seconds (for safety).
Index of Automagic useful thread List of my other useful posts (and others')
Xiaomi Redmi Note 5 (whyred), AOSP Extended v6.7 build 20200310 Official, Android Pie 9.0, Rooted.

vertigo
Posts: 147
Joined: 28 Oct 2018 00:28

Re: Error when deleting files

Post by vertigo » 18 May 2020 04:26

Problem is I think stopping the current one and starting over is better than skipping, and here's why:

1) Take picture
2) Temp file (.met / .mp4) written
3) Flow triggered by temp file
4) Temp file saved to jpg file
5) Flow triggered by jpg file
6a - FEP = stop) Flow restarts and moves jpg file
6b - FEP = skip) Flow continues and attempts to move temp file, which either no longer exists, is removed during the move, or is moved which prevents it from being saved as jpg file

I'll try using the logger to see what I can figure out there.

vertigo
Posts: 147
Joined: 28 Oct 2018 00:28

Re: Error when deleting files

Post by vertigo » 18 May 2020 07:50

Ok, I tried logging to see what's going on with the file writing, but not sure how to do it. I tried setting the trigger as file observer with almost all options checked and leading that to a write to log action with text {trigger} and taking a picture, but I end up with dozens of pages in the log, far too much to be able to rally go through on the phone, and every trigger has all the file observer options, not just whatever one actually triggered it.

User avatar
Desmanto
Posts: 2709
Joined: 21 Jul 2017 17:50

Re: Error when deleting files

Post by Desmanto » 23 May 2020 19:38

Oh, there is the temp file problem as you mention. However if you pick the correct event, it should be only the final file that trigger the flow.

Don't tick every option, especially File accessed and Modify, as both these are triggered too often. Try to limit to metadata changed or unwriteable file close. Then if nothing found, try to add another option one by one.
Index of Automagic useful thread List of my other useful posts (and others')
Xiaomi Redmi Note 5 (whyred), AOSP Extended v6.7 build 20200310 Official, Android Pie 9.0, Rooted.

vertigo
Posts: 147
Joined: 28 Oct 2018 00:28

Re: Error when deleting files

Post by vertigo » 26 May 2020 10:29

I've been doing some testing with this, but haven't made much progress. First I set up a file observer trigger with all the items checked, but that didn't work because no matter what file system action caused the trigger, the trigger shown in the log was the same, showing all file system actions. So I created a separate trigger for each action, which showed me exactly what you said: file modified and, to a lesser extent, file accessed, occur a TON. So I've removed those to see what, besides those, is the last action to occur, but that's as far as I've made it so far, because I'm having another issue: whether or not the flow actually triggers is erratic, working sometimes but frequently not working. I have it monitoring the camera folder, and take a picture to trigger it, but after it triggered once, it stopped working. After a few attempts to get it to trigger without it working, I disabled the flow then re-enabled it, and it worked once again, but aaffter that it stopped working again, and so far I haven't been able to get it to trigger again, even after turning the flow off and on again.

And all of that is a bit besides the point, anyway, since the problem wasn't with triggering the flow, but with ensuring the move files action doesn't proceed until the file is done being written to so it's free to be deleted and therefore doesn't cause an error. The bottom line is that regardless of which action I use to trigger it, there will be many more actions on the file before it's done being modified. For example, even if I use the metadata or file closed action for the trigger, the file modified action occurs dozens of times after, so beginning the file move after a metadata or file closed action might still result in an error.

It seems to me the only solution is to use the solution I came up with, which is to wait a few seconds at the beginning of the flow to make sure there's enough time for the file to be closed before attempting to move it, and restarting the flow if it's triggered again, so if a picture is taken, causing the flow to start, then another picture is taken, it will restart and move both pictures instead of trying to move them both while the second one is still being written. And the problem with that is the question of whether that might cause data corruption if the flow is restarted in the middle of a file move. I tested it by taking several pictures with varying pauses between them, and they all appeared to be moved without any issue, but I'm concerned that it might just take just the right timing to cause an issue and lose a photo. It'd be nice if Martin could chime in on how Automagic behaves when in the middle of an action, particularly a file move, when it's triggered with FEP set to restart the flow on each trigger. And of course I'm open to other suggestions on how to solve the problem, not to mention possible solutions to the problem of the flow randomly not working/triggering. I really want to get this figured out, since at this point it's the only thing holding me up on finishing my big backup flow, after which I want to move on to the calendar event flow.

Post Reply