72 replies to this topic

[WanWizard]

Yes, you can just delete them.

 

What it does is it renames a file to ".del", then in a loop it slowly removes a part from it, so it makes it shorter, until it is all gone. If I read it correctly, it should erase by 20MB/s.

 

It would not be a bad idea to check when Enigma starts to see if there are ".del" files in the Trashcan, and of so, start the file eraser again.

[sjlouis]

I don't understand why it needs time to erase files. Perhaps it's not the same way with Linux but with Windows, the name of the file and its address are erased from the allocation table, then its place on the disk is free.

 

Yes it would be a good idea to check the trashcan when Enigma restarts .

[WanWizard]

FAT does that yes, in NTFS it works different, but similar.

 

ext3 and ext4 (without extents) use an indirect block mapping scheme, where it tracks each block used. This makes deleting large files very slow. If you would just use the OS "delete" command, Enigma would be blocked (and display a spinner) for as long as is needed to delete all these blocks. And this is because Enigma is not written to be multithreaded, things don't happen in parallel.

[sjlouis]

Ok. That's why I don't see it previously a few years ago when we have only SD files (<> HD files). There were smaller.

 

Does it explain it's so difficult to restore a deleted file with Linux while there are a lot of tools which work well with Windows?

[WanWizard]

Yes.

 

A file deleted in ext3 and ext4 is gone, as there is no reference to the blocks that made up the file, and sequence of blocks. With FAT, only the directory entry is deleted, but the file allocation table isn't, so if you can recover the starting block (which is still in the directory entry if not overwritten by a new file entry) you can recover the entire file.

[Pike_Bishop]

@Robinson,

 

 

Also, is there a command to move ALL files in the root of my HDD to HDD/movie folder?

 

you can use wildcards for that as example;

mv /media/hdd/*.* /media/hdd/movie

but in this case also the file epg.dat is moving to /media/hdd/movie and make sure that you didn't have directorys under /media/hdd with a dot in the name

as example action.thrill because otherwise they are also moved to /media/hdd/movie

 

 

regards

Biki3

Edited by Biki3, 30 January 2018 - 19:56.

[sjlouis]

Thank you WanWizard .

[Erik Slagter]

Additional note: space allocation using ext4 (recommended!) is done using "extends" which comprise large amounts of blocks. Files recorded usinge extends (and once you mounted the device as "ext4", all NEW files will have that), are removed remarkably quick.

[sjlouis]

Remarkably quick ? It takes several minutes to delete one movie and it can takes 20 minutes    to delete a few files.

[MiLo]

Additional note: space allocation using ext4 (recommended!) is done using "extends" which comprise large amounts of blocks. Files recorded usinge extends (and once you mounted the device as "ext4", all NEW files will have that), are removed remarkably quick.

Only on "native" ext4 partitions. If the disk was converted from ext3 to ext4, or if it's still ext2/3, it will not have the "extend" feature and files will be just as slow to delete.

[Erik Slagter]

That's not how I remember it, the extend feature should be one as soon as you mount as ext4, but of course, only for new files extends will be used. But quite simple to test I guess. I don't have file systems from ancient converted from ext3 to ext4, but maybe someone here does?

[sjlouis]

I have no old disk file but it's easy to take a "free" disk, format it in ext3, copy movies and delete them to see what you want to control . I don't know how we can see these extends.

[MiLo]

That's not how I remember it, the extend feature should be one as soon as you mount as ext4, but of course, only for new files extends will be used. But quite simple to test I guess. I don't have file systems from ancient converted from ext3 to ext4, but maybe someone here does?

 

I do. You're right, it still gets the "extent" feature. It's the lack of bigalloc probably the slows the delete down.

 

Filesystem features:

 

Converted disk:

dir_index filetype extent sparse_super large_file uninit_bg
 

"Native" ext4 disk:

has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize bigalloc

[Erik Slagter]

Bigalloc doesn't seem to make such a big difference and, at least where I am looking, it's said to be unsafe...

[MiLo]

It made quite a noticeable difference on the old 400MHz boxes, years ago when I introduced it. Probably with the newer "more than GHz" boxes the overhead isn't as noticable any more, I haven't done any measurements lately.

The major improvement is that it has to read less from the disk in order to write to it. Writing almost never stalls on the raw throughput, it always stalls on allocating new clusters which requires information to be read from disk, and contrary to just writing, you cannot continue until that read action returns the data you need.

Edited by MiLo, 9 February 2018 - 13:26.

[anudanan]

I´m using an old version of EMC on my boxes (not the new version with many new code for enigma functions, it works well with openpli 4.1 und 6.1) and for deleting files the EMC uses a different linux process to do that. The advantage is, that no stalling of enigma is there. I delete my movies with EMC, and the file delete goes quit. I have never  seen any interrupts in recordings or playbacks which are runs in parallel.

 

I think deleting files in a different linux process than enigma is a good way to do that. And with that you have no .DEL files on the HDD

Edited by anudanan, 9 February 2018 - 14:03.

[MiLo]

The reason you're not getting interrupts in recordings is that your filesystem is ext4. It's totally unrelated to EMC, to processes or to the alignment of verious planets.

The built-in delete uses a thread (light-weight process) to do the deleting. There's no advantage whatsoever in forking a process, the kernel does not treat them any different.

The ".del" files will disappear by themselves, since they're in the trashcan anyway.

[anudanan]

thanks for information. I´ve understand in this thread enigma makes a busy waiting unlink()

[Erik Slagter]

You might consider starting the deletion without renaming them, that way they would instantly become "invisable" (which in itself also has drawbacks).

[WanWizard]

That would be the case with an unlink, but that is not how it works.

 

Instead, it uses callbacks to truncate the file with a few blocks at the time, until it reaches the beginning, after which it calls unlink to delete the 0-byte file left.

 

And this is done because an unlink of a large file on ext3 takes an absolute age, producing a spinner...