Cannot delete folder from' Delete folder' + various other small problems ...

[Willo3092]

I did some googling and it seems to be something to do with the background file eraser.

I just ended up putting a script in cron.hourly to delete any directory ending with .del

 

[simonc]

I think you get the .del.del.del thing if you go to look at deleted things in the bin and re-delete them before the background eraser has removed them.

 

[Willo3092]

I don't have the bin enabled at all though. When I delete stuff I just want it gone.
The pli forum suggests that background file eraser deletes things a chunk at a time and it can take a while.
I will have to have a look and see how ATV do it because that seems to work as expected when the trash is disabled.

 

[Willo3092]

Just tried it on ATV 7.0 with a directory with a few random files in it and it deleted fine.
It doesn't seem to mention file eraser like ViX does though.

2021-08-31 20:38:28+0100 [-] [InfoBarGenerics] Key: 398 (Break) KeyID='KEY_RED'.
2021-08-31 20:38:28+0100 [-] [ActionMap] Keymap 'ColorActions' -> Action 'red'.
2021-08-31 20:38:28+0100 [-] [Skin] Processing screen 'MessageBox', position=(240, 160), size=(800x400) for module 'MessageBox'.
2021-08-31 20:38:28+0100 [-] [Screen] Warning: Skin is missing element 'icon' in <class 'Screens.MessageBox.MessageBox'>('AAAA' contains 5 file(s) and 0 subfolders.
2021-08-31 20:38:28+0100 [-] Are you sure you want to delete?).
2021-08-31 20:38:28+0100 [-] [Pixmap] setPixmapNum(0) failed!  Defined pixmaps: [].
2021-08-31 20:38:28+0100 [-] [Skin] Processing screen 'MessageBox_summary' from list 'MessageBox_summary, SimpleSummary', position=(0, 0), size=(1x1) for module 'SimpleSummary'.
2021-08-31 20:38:28+0100 [-] [Screen] Showing screen '['MessageBox_summary', 'SimpleSummary']'.
2021-08-31 20:38:28+0100 [-] [Screen] Showing screen '['MessageBox']'.
[eRCDeviceInputDev] 1 160 1
2021-08-31 20:38:33+0100 [-] [InfoBarGenerics] Key: 352 (Make) KeyID='KEY_OK'.
2021-08-31 20:38:33+0100 [-] [ActionMap] Keymap 'MsgBoxActions' -> Action 'ok'.
2021-08-31 20:38:33+0100 [-] [Screen] Showing screen 'MovieSelectionSummary'.
2021-08-31 20:38:33+0100 [-] [Screen] Showing screen 'MovieSelection'.
2021-08-31 20:38:33+0100 [-] [DeleteFolderTask] files  /media/autofs/RECORDINGS/AAAA
2021-08-31 20:38:33+0100 [-] job Components.Task.Job name=Deleting files #tasks=1 completed with [] in None
[eRCDeviceInputDev] 0 160 1

This is the log from ViX 5.5 for exactly the same thing:

20:49:48.6296 [InfoBarGenerics] Key: 398 (Break) KeyID='KEY_RED' Binding='('RED',)'.
20:49:48.6299 [ActionMap] Keymap 'ColorActions' -> Action = 'red'.
20:49:48.6431 [Skin] Processing screen 'MessageBox', position=(0, 0), size=(1920 x 1080) for module 'MessageBox'.
20:49:48.6643 [Screen] Warning: Skin is missing element 'icon' in <class 'Screens.MessageBox.MessageBox'>(Do you really want to permanently delete 'AAAB'?).
20:49:48.6687 [Pixmap] setPixmapNum(0) failed! defined pixmaps: []
20:49:48.6699 [Skin] Processing screen 'MessageBox_summary' from list 'MessageBoxSummary, ScreenSummary, MessageBox_summary, SimpleSummary', position=(0, 0), size=(1 x 1) for module 'ScreenSummary'.
20:49:50.3378 [eInputDeviceInit] 1 160 (352) 1
20:49:50.3382 [eRCDeviceInputDev] emit: 1
20:49:50.3401 [InfoBarGenerics] Key: 352 (Make) KeyID='KEY_OK' Binding='('OK',)'.
20:49:50.3404 [ActionMap] Keymap 'MsgBoxActions' -> Action = 'ok'.
20:49:50.3625 [eThread] old thread joined 0
20:49:50.3628 [setIoPrio] best-effort level 7 ok
20:49:50.3629 [eBackgroundFileEraser] deleting '/media/autofs/RECORDINGS/AAAB.del'
20:49:50.3630 [eBackgroundFileEraser] removing /media/autofs/RECORDINGS/AAAB.del failed: Is a directory
20:49:50.5465 [eInputDeviceInit] 0 160 (352) 1

 

[simonc]

Might be related to the multiselect changes from earlier this year. I'll take a look.

 

[birdman]

Yes the .del.del.del.del thing is a, probably harmless, bug with the delayed deletion feature.

The descriptions so far imply that the background eraser won't delete a directory - ever.

 

[simonc]

Here's the smoking gun:

18:49:51.4987 [eBackgroundFileEraser] deleting '/media/hdd/movie/k.del'
18:49:51.4989 [eBackgroundFileEraser] removing /media/hdd/movie/k.del failed: Is a directory

I can't currently determine why ATV deletes directories successfully and Vix doesn't. There are only slight differences on Vix with the filename processing in the file_eraser.cpp, which is where this message originates.

Can anyone shed any light on why deleting a file is considered so heavyweight that it has to be done in a background thread with masses of complicated checks for the size of files being deleted? The initial commit for the background eraser says what it does, not why it's being added. I'd wondered if it was due to slow or unreliable network drives, but if that was the case, then renaming to postfix .del on the filename would be just as laggy as removing the file from the filesystem index. The size of the file being deleted shouldn't have any effect on how quick it is to delete the file, so I also don't understand why there's a delete speed in the movielist settings.

By comparison, on the internal HDD, deleting a file using Python's os.remove or an entire directory tree of hundreds of files using shutils.rmtree is consistently a sub-second operation.

 

[ccs]

... that's what I thought when I had a look at the code, a massively complicated process to slow down the deletion of (a number of) files.

Maybe there was an impact on performance once upon a time, but I can't really see why it should be such a problem.

I'd simplify it and see if there are still any issues.

 

[simonc]

Ah, the multiselect additions broke deleting directories when not using the trashcan. I'm not surprised I screwed it up: the old the MovieSelection delete was one of those functions that works out what to do and displays a message box, then calls back onto itself with a flag saying "do the thing I've just worked out what to do". Great idea, no need to write 2 functions when you could save a bit of typing.

Amusingly, directory deletion was previously completely bypassing the background delete job and just using rmtree, so if no-one has had problems with deleting directories permanently prior to the introduction of this bug, then the background eraser isn't doing anything useful.

 

[BrokenUnusableAccount]

I don't have the bin enabled at all though. When I delete stuff I just want it gone.
The pli forum suggests that background file eraser deletes things a chunk at a time and it can take a while.
I will have to have a look and see how ATV do it because that seems to work as expected when the trash is disabled.

Ah, the multiselect additions broke deleting directories when not using the trashcan. I'm not surprised I screwed it up: the old the MovieSelection delete was one of those functions that works out what to do and displays a message box, then calls back onto itself with a flag saying "do the thing I've just worked out what to do". Great idea, no need to write 2 functions when you could save a bit of typing.

Amusingly, directory deletion was previously completely bypassing the background delete job and just using rmtree, so if no-one has had problems with deleting directories permanently prior to the introduction of this bug, then the background eraser isn't doing anything useful.

Yes. I'm not convinced that deleting needs to be slowed down in most cases. Linux has enough buffers to cope if a large deletion takes up to maybe around half a second.
I would think the only case that might take longer would be on some SSDs that have rather dumb implementations of TRIM, if TRIMming is done as the deletions are done.
Or if you're doing dsecure erasing for some weird reason.

Linux EXT file systems aren't particularly slow at deleteing stuff are they?

 

[simonc]

The especially odd thing about the background deletion is that it does an ltrunc on each file, shortening it by 20MB at a time until it has zero length. I just can't see how this can possibly be any more efficient or light on CPU than just removing the file's index entry with unlink. This has the smells of someone looking for a problem to apply a clever solution to. Now, if they'd commented it with workaround for unlink being slow across networks/due to a bug in drivers on Miraclebox 123 then we'd know where we were.

 

[Valiant]

Is there a comment/warning somewhere about recordings in progress/about to start? Would this be the reason why the the files are not removed by rm or unlink?

 

[simonc]

You may be onto something; when you delete an in progress recording, it stops the timer, then tells the background deleter to remove the file. The recording code runs in a background thread, so it may not stop writing before the deleter starts. There's no useful comments to back this idea up though.

 

[simonc]

PLi reckon it's because deleting large files on slow media can cause recording stutters. Probably related to the way Linux's ext filesystem organises the chunks of a large file.

Found this on SuperUser

When deleting a file, the ext3 filesystem will actually zero out the block pointers in the inode. The larger the file, the more blocks, and the more block pointers, thus the delete operation takes longer on larger files than smaller ones.


This is different behavior than both ext2, which merely zeroes out the inode and leaves the blocks containing the block pointers intact (but marked as free) and ext4, which uses extents (and, since extents are a much more compact structure, has much better delete performance, that slows down based on how fragmented the file is, rather than how big it is).

 

[birdman]

Linux has enough buffers to cope if a large deletion takes up to maybe around half a second.

Deleting just removes the entries in the directory and frees the extents. It shouldn't take long, and not much I/O.

I would think the only case that might take longer would be on some SSDs that have rather dumb implementations of TRIM, if TRIMming is done as the deletions are done.

Only if you set the option, which isn't recommended. Better to run an fstrim once a week (which is effectively what MSWindows does too).

 

[BrokenUnusableAccount]

Deleting just removes the entries in the directory and frees the extents. It shouldn't take long, and not much I/O.
Only if you set the option, which isn't recommended. Better to run an fstrim once a week (which is effectively what MSWindows does too).

My Windoze 7 PCs definately do TRIM as they delete. I notice that deleting larges files from my SSD lights the disk access light much longer than deleteing them from an HDD does.

 

[birdman]

My Windoze 7 PCs definately do TRIM as they delete.

Well, they do scheduled TRIMs too.
Go to the disk/partition on file explorer, select Properties/Tools and click on Optimise. If you select to optimize an SSD it runs a TRIM on it. (My Windows partition currently says "OK (23 days since last retrim)". It's scheduled to run weekly (but I don't boot into Windows all that often, nor for very long).

If it's TRIMming on each deletion as well then it's just wasting time and effort.

"fsutil behavior query DisableDeleteNotify" does say that TRIM is enabled. Hmmm.....might help to explain some slow Windows actions. Might try changing it...

 

[BrokenUnusableAccount]

Well, they do scheduled TRIMs too.

Yes I think I remember seeing that Win 10 does scheduled TRIMs. But Win 7 doesn't, well not unless they hid it really well anyway.
I don't know if Win 10 does TRIMs as it goes as well as scheduled.
I assume a well designed modern SSD shouldn't slow anywhere near much as my old SATA ones do when TRIMming.