196 replies to this topic

[Hump]

Hi MiLo,

thank you very much for your efforts in order to find the bottleneck of the ext4 driver module. Would like to test that option with nodelalloc, but I reformated my HDD to ext2 already. But I'm sure someone will do pick up and will conduct a long-run test. I'm anxious on what the result might be.

Again, thanks a million for your hard work! - You guys are THE best!

PS: After testing many HD recordings with ext2 and ext2 module, no freezer appeared at all.

[anudanan]

Milo:
would you like to set the nodealloc parm als default in your image in future?

Edited by anudanan, 5 February 2012 - 17:45.

[MiLo]

Even if you formatted your disk as ext2, using the ext4 driver with "nodealloc" should still have the desired effect (and perform better than ext2 too).

Edited by MiLo, 5 February 2012 - 18:10.

[MiLo]

would you like to set the nodealloc parm als default in your image in future?

It reduces performance considerably, so i'd prefer to find a way to solve it in user-space now that I have a good candidate for what is going on in the kernel. I was thinking along the lines of "fallocate" for example.

[Hump]

@MiLo, found some interesting conversation between Linus Torvalds and his folks talking about the ext3/4 filesystems and Journaling. Don't know, if you have read this already, so I'll post the Links:

http://www.h-online....xt4-740787.html

http://article.gmane...x.kernel/811654
http://article.gmane...x.kernel/811700
http://article.gmane...x.kernel/811699

http://www.linux-mag...Ext-3-und-Ext-4 (article in german)

[MiLo]

I've seen quite a lot of those - but those articles are mostly about integrity, not "weird" performance which is what's biting us now. "We" don't really care about that - all we want is that the disk remains usable after a power failure, and that you don't have to wait for a 25 minute fsck after that. "We" would rather lose that last 30 seconds of the recording and continue as quick as possible. The ext4 system fits that description a lot better than ext2, and probably ext3 too.

[MiLo]

Actually, I seriously think -though haven't tried it yet- that formatting your harddisk in NTFS will solve all these problems too. It probably outperforms ext4 too, if the ntfs-3g guys did their job well...

[MiLo]

Couldn't resist, and just tried. Apparently, ntfs-3g doesn't use the cache. Which gives nice consistent write times, but those semi-synchronous writes seem to cost tons of CPU and wall clock time. Though from the outside, it's running just fine, and no overruns when tormenting the disk... So basically, using NTFS won't be satisfactory

[anudanan]

two more questions to ;Milo around ext2/3/4
In the past I´ve choosen ext2 because I heared about some negativ aspects with journal of ext3. Is it right that a file delete stalls all other write because the delete is an atomic operation on ext3 journaling filesystem. So a delete of a movie file will produce some freeze on the recordings? I´ve never tested that but I seems logical for me. Is that the reason for the background delete parameter in your Image?
Do you know if ext4 solves that delete stalling problem. Is there also the need for background deleting parameter?

Do you think the ext4 performance with nodealloc parameter is to low for the box? Is it lower that the ext2 performance in the past? If not I think the ext4 performance with nodealloc would be ok for the boxes.

[MiLo]

ext4 with nodealloc is "faster" than ext2.

ext4 does not completely solve the stalling-on-delete problem, but improves on delete performance by a large factor. The stall is caused by heavy disk IO, not the journal. This is only the case for files that were created AFTER the ext4 conversion, deleting files that were written when the filesystem was ext3 or ext2 (deleting files on ext3 and ext2 is equally slow) is still as slow.

[Qu@rk]

@MiLo
Thanks a lot for your investigation.

I have tested with disabled journaling, writeback and I have much less "freezers".

I would like to test with default mount parameters and then activate "nodelalloc".

I would like to test it correclty:

Please, what are the default mount parameters (e.g. to set with tune2fs) in PLi 2.1 with ext4?
has_journal
journal_data_ordered

Edited by Qu@rk, 8 February 2012 - 16:19.

[Qu@rk]

I have restored the default mount parameters (I hope so: journal activated, data=ordered).
Are these the correct default parameters?

fsck
reboot

Then I set "nodelalloc" using tune2fs: tune2fs -o nodelalloc /dev/sda1

reboot
fsck

tune2fs -l /dev/sda1 lists the correct mount options.

But the mount command does not show "nodelalloc" as a flag for /dev/sda1

How can I check whether "nodelalloc" has been activated?

[MiLo]

Add a line like this to /etc/fstab (assuming the disk is /dev/sda1, you should use the UUID though):

/dev/sda1		    /media/hdd		   ext4	   defaults,nodelalloc,data=writeback  0  0

To test, just unmount and mount the disk with "mount /dev/sda1" (without options). Then type "mount" and you should see the options in effect.

[Qu@rk]

I have added to /etc/fstab

UUID=37408d8b-d8ba-4442-b158-88b5bbf25a22 /media/hdd ext4 defaults,nodelalloc,data=writeback 0 0

umount /dev/sda1
mount /dev/sda1

mount

rootfs on / type rootfs (rw)
ubi0:rootfs on / type ubifs (rw,relatime)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
tmpfs on /dev type tmpfs (rw,relatime,size=64k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
usbfs on /proc/bus/usb type usbfs (rw,relatime)
tmpfs on /var/volatile type tmpfs (rw,relatime)
/dev/sda1 on /media/hdd type ext4 (rw,relatime,barrier=1,data=writeback)


I can see, that the options were used because of the change from "ordered" to "writeback".
But no "nodelalloc".

What is PLi default "writeback" or "ordered"?
Journal is activated by default?


dmesg | grep EXT4

EXT4-fs (sda1): mounted filesystem with writeback data mode. Opts: nodelalloc,data=writeback

But mount does not show "nodelalloc".

Edited by Qu@rk, 8 February 2012 - 19:12.

[MiLo]

Strange... both ETs here show correct mount options.

For optimum better-than-ext2 performance, turn of the journaling. You can leave delalloc enabled in that case. The price you pay is that after power failure, you will need to run fsck to fix the system. That may take several minutes to complete. I've tested this, pulled the power plug while 4 HD recordings and a DD transfer were running. Booted fine, mounted okay, fsck took about 2 minutes and fixed some things.

For better brownout stability while preventing overruns, disable delalloc. You can use data=writeback and/ore commit=30 and/or barrier=0 if you want to squeeze out more performance, but this will cost you some stability, and you will lose more data and risk data corruption again.

I'm still not really convinced what's best. I don't care about losing recently written data on powerout, who cares about that 5 seconds of video when the box is down for at least a minute. But it should be able to survive power loss without corrupting the rest of the filesystem... Any takers? (btw, btrfs didn't run well on the ET, the system crashed after a few minutes of recording).

[anudanan]

Maybe the reason for different output of mount command (with nodelalloc and without nodelalloc) based on different mount binaries? I think in the e2fsprog package there is also a mount binary included. Is that different to the mount binary which in normally in you image? I think Quark has installed the e2fsprog tools to use tunefs

[Qu@rk]

Thanks MiLo, I will try, as soon, as I can clearify this mount option "nodelalloc" problem.

anudanan, very good idea!

During my tests, I have installed the packages e2fsprog and e2fsprog-tune2fs (to get blkid and tune2fs). I think e2fsprog-tune2fs would have been enough, but I did notice that to late.

If I uninstall e2fsprog-package, will I get back all the original binaries?
Or will I have to do a complete reinstall (oh my gosh) of my PLi 2.1 to get back the original PLi 2.1 state?

Edited by Qu@rk, 9 February 2012 - 07:27.

[MiLo]

If you uninstall it, you should get the originals back. "mount" and "blkid" are supplied by busybox.

Instead of e2fsprogs, only install e2fsprogs-tune2fs, that contains the binary you need. The e2fsprogs package draws in quite a lot of unneeded dependencies.

I'll change the image to include tune2fs by default on the systems with a newer kernel. Makes it easier for the harddisk menu to use it.

[Qu@rk]

Great! Thanks a lot for your support.

I will check it out (uninstall e2fsprogs) and report here (today afternoon or evening). CU

[Qu@rk]

I have uninstalled e2fsprogs and unfortunately no change.

I have downloaded busybox from feed and reinstalled it.

busybox mount still shows no "nodelalloc" flag.

I have disabled journaling (tune2fs -O ^has_journal /dev/sda1), did an fsck and reboot.

I don't know what's wrong.

Does your mount show the "nodelalloc" flag?

My settings:

/etc/fstab

rootfs			   /					auto	   defaults			  1  1
proc				 /proc				proc	   defaults			  0  0
devpts			   /dev/pts			 devpts	 mode=0620,gid=5	   0  0
usbfs				/proc/bus/usb		usbfs	  defaults			  0  0
tmpfs				/var/volatile		tmpfs	  defaults			  0  0
UUID=37408d8b-d8ba-4442-b158-88b5bbf25a22 /media/hdd ext4 defaults,nodelalloc,data=writeback 0 0

mount (or busybox mount)

rootfs on / type rootfs (rw)
ubi0:rootfs on / type ubifs (rw,relatime)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
tmpfs on /dev type tmpfs (rw,relatime,size=64k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
/dev/sda1 on /media/hdd type ext4 (rw,relatime,barrier=1)
usbfs on /proc/bus/usb type usbfs (rw,relatime)
tmpfs on /var/volatile type tmpfs (rw,relatime)

tune2fs -l /dev/sda1

tune2fs 1.41.14 (22-Dec-2010)
Filesystem volume name:   <none>
Last mounted on:		  /media/hdd
Filesystem UUID:		  37408d8b-d8ba-4442-b158-88b5bbf25a22
Filesystem magic number:  0xEF53
Filesystem revision #:	1 (dynamic)
Filesystem features:	  ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:		 signed_directory_hash
Default mount options:	journal_data_writeback nodelalloc
Filesystem state:		 not clean
Errors behavior:		  Continue
Filesystem OS type:	   Linux
Inode count:			  1907712
Block count:			  488361943
Reserved block count:	 0
Free blocks:			  131638341
Free inodes:			  1905775
First block:			  0
Block size:			   4096
Fragment size:			4096
Reserved GDT blocks:	  907
Blocks per group:		 32768
Fragments per group:	  32768
Inodes per group:		 128
Inode blocks per group:   8
Flex block group size:	16
Filesystem created:	   Fri Aug  5 17:11:48 2011
Last mount time:		  Thu Jan  1 01:00:03 1970
Last write time:		  Thu Feb  9 14:47:48 2012
Mount count:			  1
Maximum mount count:	  30
Last checked:			 Thu Feb  9 14:34:53 2012
Check interval:		   15552000 (6 months)
Next check after:		 Tue Aug  7 15:34:53 2012
Lifetime writes:		  3136 GB
Reserved blocks uid:	  0 (user root)
Reserved blocks gid:	  0 (group root)
First inode:			  11
Inode size:			   256
Required extra isize:	 28
Desired extra isize:	  28
Default directory hash:   half_md4
Directory Hash Seed:	  89cd731e-bce3-4f9a-ba08-2be6509aad34
Journal backup:		   inode blocks

dmesg | grep EXT4

EXT4-fs (sda1): mounted filesystem without journal. Opts: nodelalloc,data=writeback

Edited by Qu@rk, 9 February 2012 - 14:57.