196 replies to this topic

[pieterg]

No, it won't hurt, but I think we'd like rule them out.
Could you please re-install your sysctl.conf file, by doing the following:

opkg install kernel-params -force-reinstall

And reboot. If you have no increase or decrease in recording 'performance', we can forget about the sysctl settings.

I tested several HD recordings after deletion of sysctl.conf and could not find any differences in performance. Still no freezers with ext4-journaling off and writeback enabled.

I know, but the question is whether or not the deletion of sysctl.conf contributes to that or not.
Only one way to find out

[Hump]

I know, but the question is whether or not the deletion of sysctl.conf contributes to that or not.
Only one way to find out

Okay, I will test and report the results ASAP. Thanks for your efforts.

[MiLo]

The underlying question is: Why does a write() for 188k bytes, block-aligned, take almost 2 seconds to complete, while there is still 100MB RAM available to cache that data. What the hell is that kernel doing all that time?

It improves when you basically kill the journalling, but that still does not really solve the issue. Every now and then, a write() takes much longer than the others, sometimes just 10 ms instead of the usual 3ms, but every now and then, it's a LOT more, causing an overflow in that 1,5MB buffer.

[pieterg]

My guess is it might be the UBIFS writeback (which is running from the same pdflush process)

[MiLo]

It only "stalls" when there's a lot of hardddisk activity. There's no activity on the flash at that time (and besides, writing to the flash doesn't take that long either).

My guess so far is that, in order do fulfill the "write" request, the system needs to read something from the disk first (inode information or whatever). If this happens to occur at the same moment the journal is being flushed (which is every 5s), the "read" must wait for that call to complete first, before it can continue. This explains the long stall in the write call, there's plenty room in the cache, but system is still figuring out where to put the data first.

I'd rather solve this with kernel configurations than having to build a user-space buffer for the recording data (and then probably using DIRECT I/O to dump the buffer's contents). That would actually be quite similar to just setting the demux buffer size to something like 5MB so that it can cope with 5s delays.

It's a nice example of "realtime" behaviour: The system can sustain 50MB/s writing to disk, but you cannot get it to write 188k bytes in less than 1.5 seconds.

[Hump]

No, it won't hurt, but I think we'd like rule them out.
Could you please re-install your sysctl.conf file, by doing the following:

opkg install kernel-params -force-reinstall

And reboot. If you have no increase or decrease in recording 'performance', we can forget about the sysctl settings.

I tested several HD recordings after deletion of sysctl.conf and could not find any differences in performance. Still no freezers with ext4-journaling off and writeback enabled.

I know, but the question is whether or not the deletion of sysctl.conf contributes to that or not.
Only one way to find out

Okay, tested some HD recordings after reinstalling the sysctl.conf. No freezers at all., So deleting the sysctl.conf or not doesn't make a difference.

[pieterg]

ok, thanks for testing

[Hump]

Did you guys change anything lateley pertaining to ext4? I didn't had freezers with jornaling off/ writeback and all of a sudden, I see freezers (there is at least one freezer in each HD recording) in HD recordings again? I guess I have to format the HDD ext2 and run it with the ext2 driver, since this option (ext2) seems to be the only solution at the time.

[MiLo]

Nope, nothing changed.

Instead of going ext2, you could simply turn of journalling. That way, you get the faster block allocation of ext4 (and much faster "delete") without the penalty of journaling. You don't even need a reboot for that, just unmount the disk, "tune2fs -O ^has_journal /dev/sda" (I listed those somewhere on the forum) and mount it. This should outperform ext2 on all fronts.

Also, you can now increase the demux buffer to 4MB in the recording options. That'll more than double the amount of buffering (the penalty being that IF if overflows, you lose 4MB of data).

[Hump]

Thanks MiLo. I'm already running ext4 without journal and with writeback activated but fine,I will try and test with expanded demux buffer to 4MB first. If I still get freezers, then I go back to ext2 with the ext2 driver (I know I have to reinstall it first) in fstab. A lot of users reported that using ext2 fixed it all and they are recording freezer-free.

Thank you and thanks to the whole PLi Team for helping us out.

[Qu@rk]

I have the same problem like Hump. I have turned off journaling and turned writeback on.

dmesg | grep EXT4 reports:

EXT4-fs (sda1): mounted filesystem without journal. Opts: data=writeback,barrier=0,nobh,errors=remount-ro

recording resync buffer = OFF
recording demux buffer = 1 MB

as recommended.

I still have one or two freezers per recording, the same as Hump (last Update 27.01.2012).
I did an PLI update today. But if there was no change within the ext4 drivers, there should be no difference.

@MiLo
Is there any hope for a fix in the near future? I know ETA are nearly impossible. But to change to ext2 I would need to copy 1,4 TB to an external USB, format ext2 and restore again. That would mean 2 days of copy work.

There are so many reports of ext2 users, who do not have any problems. But they use the "older" ext2/3 kernel modules (manually installed).

Edited by Qu@rk, 31 January 2012 - 13:40.

[MiLo]

Set recording demux buffer to 4MB, that should help quite a bit, and the 9x00 can handle it.

[MiLo]

It's not the ext4 driver. Only a few people report these problems, and I for one can't reproduce it, no matter what I try, and my disk is a lot slower than yours. If it were just the driver, or just the kernel, then everyone would be having this problem. The "solutions" so far are just a shotgun approach: Just blast it with enough changes, and hopefully one of those changes will hide the underlying problem.

The actual problem is probably that the system needs to read something from disk in order to continue writing. That read action may block up to several seconds, even though there's hardly any real load on the IO subsystem. This happens in just very particular situations.

There is still no real known cause. None of the developers is able to reproduce it, anywhere.
Only thing I could think of to reproduce it is send me your harddisk. Until someone does something like that, the problem is not likely to be solved (other than by sheer accident) in the near future.

[anudanan]

An other idea for finding the reason of freeze. Is there a possible way to give some users with freeze an image or some modules with debug code which write debug output to a file if the code around the write calls or buffer management see some buffer overflow or big latency. If those users have freeze they can send to you the debuig file. In the debug file you can write many interessting infos which you need. I know that is not easy but I think it is a possible way.

[Hump]

Based on the positiv results and reports by many users, I finally changed the filesystem from ext4 to ext2 and that resolved it all. No more freezers are showing, what's all ever. Runs rock-stable so far.

Strange, I know, but hey it works.

Could you please reinsert the ext2 and ext3 Modules? I think that would make sense...

Edited by Hump, 2 February 2012 - 10:30.

[MiLo]

An other idea for finding the reason of freeze. Is there a possible way to give some users with freeze an image or some modules with debug code which write debug output to a file if the code around the write calls or buffer management see some buffer overflow or big latency. If those users have freeze they can send to you the debuig file. In the debug file you can write many interessting infos which you need. I know that is not easy but I think it is a possible way.

I'll prepare a "debug" version this weekend (hopefully), that may provide the insight.

After all, reason #1 to move to a more recent kernel was ext4 support, so it would be kind of strange to move two generations back to ext2 instead...

[anudanan]

cool, I´ve posted for some minutes that information inside et-view forum to find user for testing with debug image

[1701d]

I'll prepare a "debug" version this weekend (hopefully), that may provide the insight.

After all, reason #1 to move to a more recent kernel was ext4 support, so it would be kind of strange to move two generations back to ext2 instead...

Hi Milo,

if you prepare an image, I'm more then happy to try this and give you feedback.

Kind regards

Klaus

[MiLo]

Quick conclusion for those eager to know: If you want ext2 behavior, disable "delayed allocation" on the ext4 driver. This will make it perform slower, but more predictable (but still faster and more reliable than ext2). To do so, specify "nodelalloc" in the mount options, e.g.: "mount -o nodelalloc /dev/sda1 /media/hdd"

I've been doing some more tests myself this weekend. Re-formatted my disk as ext3, and used the different ext2/3/4 drivers to mount it with varying options. To test their "recording" behaviour, I start a recording on one HD channel, and then "torture" the system with "dd if=/dev/zero of=/media/hdd/blanks2 bs=188k count=10000" to dump about 1.8 GB on the disk, erase it, then dump it again. Enigma2 measures and writes the time it takes to write each 188k block to its output.

As for details:
With ext4 "fully functional", I see write times of ~5ms. Under torture, most aren't affected, but some are detained for 100..1000ms. This effect is larger when data=ordered is active, data=writeback tampers it. The ext2 and ext3 drivers show slightly larger overall write times, but when under stress, the load is spread more evenly amongst calls, so instead of a single call taking up 100ms, you now see 5 calls of 20ms, or something similar to that. This is much easier to cope with. Since the ext4 driver uses features not previously seen in ext2/3, shutting one of them down should bring back that old behavior. Indeed, "delalloc" showed to be the one. If I tell the ext4 driver to stop using delayed allocation, it behaves much more like the ext2/3 drivers when mounting an ext3 disk. Even better, when I converted the disk to ext4, and then added the "nodelalloc" option to the mount command, this effect remained in place. What I did notice is that the "torture" command now had to pay the price. It was writing significantly slower (from 50 to 30MB/s).
I have to check the documentation, but my understanding is that the system, instead of allocating blocks at each call to write(), it just collects them and allocates them in bigger chunks. This does help file transfer significantly, but on the typical load from recordings, causes the long delays on some of the calls when there are other threads performing IO as well.

[anudanan]

Great test you've made. One question around that.
Does only the parameter nodealloc solve the latency problem or is there a need for data=writeback instead of data=ordered.