Jump to content


Photo

ET9000:Hänger in HD Aufnahmen


  • Please log in to reply
196 replies to this topic

Re: ET9000:Hänger in HD Aufnahmen #41 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 26 January 2012 - 13:54


No, it won't hurt, but I think we'd like rule them out.
Could you please re-install your sysctl.conf file, by doing the following:

opkg install kernel-params -force-reinstall

And reboot. If you have no increase or decrease in recording 'performance', we can forget about the sysctl settings.


I tested several HD recordings after deletion of sysctl.conf and could not find any differences in performance. Still no freezers with ext4-journaling off and writeback enabled.


I know, but the question is whether or not the deletion of sysctl.conf contributes to that or not.
Only one way to find out ;)

Re: ET9000:Hänger in HD Aufnahmen #42 Hump

  • Senior Member
  • 88 posts

+3
Neutral

Posted 26 January 2012 - 13:58

I know, but the question is whether or not the deletion of sysctl.conf contributes to that or not.
Only one way to find out ;)


Okay, I will test and report the results ASAP. Thanks for your efforts.

Re: ET9000:Hänger in HD Aufnahmen #43 MiLo

  • PLi® Core member
  • 14,055 posts

+298
Excellent

Posted 26 January 2012 - 14:29

The underlying question is: Why does a write() for 188k bytes, block-aligned, take almost 2 seconds to complete, while there is still 100MB RAM available to cache that data. What the hell is that kernel doing all that time?

It improves when you basically kill the journalling, but that still does not really solve the issue. Every now and then, a write() takes much longer than the others, sometimes just 10 ms instead of the usual 3ms, but every now and then, it's a LOT more, causing an overflow in that 1,5MB buffer.
Real musicians never die - they just decompose

Re: ET9000:Hänger in HD Aufnahmen #44 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 26 January 2012 - 16:16

My guess is it might be the UBIFS writeback (which is running from the same pdflush process)

Re: ET9000:Hänger in HD Aufnahmen #45 MiLo

  • PLi® Core member
  • 14,055 posts

+298
Excellent

Posted 27 January 2012 - 12:00

It only "stalls" when there's a lot of hardddisk activity. There's no activity on the flash at that time (and besides, writing to the flash doesn't take that long either).

My guess so far is that, in order do fulfill the "write" request, the system needs to read something from the disk first (inode information or whatever). If this happens to occur at the same moment the journal is being flushed (which is every 5s), the "read" must wait for that call to complete first, before it can continue. This explains the long stall in the write call, there's plenty room in the cache, but system is still figuring out where to put the data first.

I'd rather solve this with kernel configurations than having to build a user-space buffer for the recording data (and then probably using DIRECT I/O to dump the buffer's contents). That would actually be quite similar to just setting the demux buffer size to something like 5MB so that it can cope with 5s delays.

It's a nice example of "realtime" behaviour: The system can sustain 50MB/s writing to disk, but you cannot get it to write 188k bytes in less than 1.5 seconds.
Real musicians never die - they just decompose

Re: ET9000:Hänger in HD Aufnahmen #46 Hump

  • Senior Member
  • 88 posts

+3
Neutral

Posted 27 January 2012 - 13:23



No, it won't hurt, but I think we'd like rule them out.
Could you please re-install your sysctl.conf file, by doing the following:

opkg install kernel-params -force-reinstall

And reboot. If you have no increase or decrease in recording 'performance', we can forget about the sysctl settings.


I tested several HD recordings after deletion of sysctl.conf and could not find any differences in performance. Still no freezers with ext4-journaling off and writeback enabled.


I know, but the question is whether or not the deletion of sysctl.conf contributes to that or not.
Only one way to find out ;)



Okay, tested some HD recordings after reinstalling the sysctl.conf. No freezers at all., So deleting the sysctl.conf or not doesn't make a difference.

Re: ET9000:Hänger in HD Aufnahmen #47 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 27 January 2012 - 20:59

ok, thanks for testing

Re: ET9000:Hänger in HD Aufnahmen #48 Hump

  • Senior Member
  • 88 posts

+3
Neutral

Posted 29 January 2012 - 15:35

Did you guys change anything lateley pertaining to ext4? I didn't had freezers with jornaling off/ writeback and all of a sudden, I see freezers (there is at least one freezer in each HD recording) in HD recordings again? I guess I have to format the HDD ext2 and run it with the ext2 driver, since this option (ext2) seems to be the only solution at the time.

Re: ET9000:Hänger in HD Aufnahmen #49 MiLo

  • PLi® Core member
  • 14,055 posts

+298
Excellent

Posted 29 January 2012 - 19:22

Nope, nothing changed.

Instead of going ext2, you could simply turn of journalling. That way, you get the faster block allocation of ext4 (and much faster "delete") without the penalty of journaling. You don't even need a reboot for that, just unmount the disk, "tune2fs -O ^has_journal /dev/sda" (I listed those somewhere on the forum) and mount it. This should outperform ext2 on all fronts.

Also, you can now increase the demux buffer to 4MB in the recording options. That'll more than double the amount of buffering (the penalty being that IF if overflows, you lose 4MB of data).
Real musicians never die - they just decompose

Re: ET9000:Hänger in HD Aufnahmen #50 Hump

  • Senior Member
  • 88 posts

+3
Neutral

Posted 29 January 2012 - 19:41

Thanks MiLo. I'm already running ext4 without journal and with writeback activated but fine,I will try and test with expanded demux buffer to 4MB first. If I still get freezers, then I go back to ext2 with the ext2 driver (I know I have to reinstall it first) in fstab. A lot of users reported that using ext2 fixed it all and they are recording freezer-free.

Thank you and thanks to the whole PLi Team for helping us out. :)

Re: ET9000:Hänger in HD Aufnahmen #51 Qu@rk

  • Senior Member
  • 192 posts

+2
Neutral

Posted 31 January 2012 - 13:36

I have the same problem like Hump. I have turned off journaling and turned writeback on.

dmesg | grep EXT4 reports:

EXT4-fs (sda1): mounted filesystem without journal. Opts: data=writeback,barrier=0,nobh,errors=remount-ro

recording resync buffer = OFF
recording demux buffer = 1 MB

as recommended.

I still have one or two freezers per recording, the same as Hump (last Update 27.01.2012).
I did an PLI update today. But if there was no change within the ext4 drivers, there should be no difference.

@MiLo
Is there any hope for a fix in the near future? I know ETA are nearly impossible. But to change to ext2 I would need to copy 1,4 TB to an external USB, format ext2 and restore again. That would mean 2 days of copy work.

There are so many reports of ext2 users, who do not have any problems. But they use the "older" ext2/3 kernel modules (manually installed).

Edited by Qu@rk, 31 January 2012 - 13:40.


Re: ET9000:Hänger in HD Aufnahmen #52 MiLo

  • PLi® Core member
  • 14,055 posts

+298
Excellent

Posted 31 January 2012 - 14:25

Set recording demux buffer to 4MB, that should help quite a bit, and the 9x00 can handle it.
Real musicians never die - they just decompose

Re: ET9000:Hänger in HD Aufnahmen #53 MiLo

  • PLi® Core member
  • 14,055 posts

+298
Excellent

Posted 31 January 2012 - 14:35

It's not the ext4 driver. Only a few people report these problems, and I for one can't reproduce it, no matter what I try, and my disk is a lot slower than yours. If it were just the driver, or just the kernel, then everyone would be having this problem. The "solutions" so far are just a shotgun approach: Just blast it with enough changes, and hopefully one of those changes will hide the underlying problem.

The actual problem is probably that the system needs to read something from disk in order to continue writing. That read action may block up to several seconds, even though there's hardly any real load on the IO subsystem. This happens in just very particular situations.

There is still no real known cause. None of the developers is able to reproduce it, anywhere.
Only thing I could think of to reproduce it is send me your harddisk. Until someone does something like that, the problem is not likely to be solved (other than by sheer accident) in the near future.
Real musicians never die - they just decompose

Re: ET9000:Hänger in HD Aufnahmen #54 anudanan

  • Senior Member
  • 1,185 posts

+16
Neutral

Posted 31 January 2012 - 21:51

An other idea for finding the reason of freeze. Is there a possible way to give some users with freeze an image or some modules with debug code which write debug output to a file if the code around the write calls or buffer management see some buffer overflow or big latency. If those users have freeze they can send to you the debuig file. In the debug file you can write many interessting infos which you need. I know that is not easy but I think it is a possible way.

Receiver:2 x Uno4k SE (PLI 7.3 rel), 1 x ET9200 (PLI 4.0), NAS: 2 x QNAP 410, TV: LG 65C8llla, LG 47LB570V, LG 42LM615S, Sound: Yamaha RX-v663, Teufel System 5 THX


Re: ET9000:Hänger in HD Aufnahmen #55 Hump

  • Senior Member
  • 88 posts

+3
Neutral

Posted 2 February 2012 - 10:28

Based on the positiv results and reports by many users, I finally changed the filesystem from ext4 to ext2 and that resolved it all. No more freezers are showing, what's all ever. Runs rock-stable so far.

Strange, I know, but hey it works.

Could you please reinsert the ext2 and ext3 Modules? I think that would make sense...

Edited by Hump, 2 February 2012 - 10:30.


Re: ET9000:Hänger in HD Aufnahmen #56 MiLo

  • PLi® Core member
  • 14,055 posts

+298
Excellent

Posted 2 February 2012 - 18:01

An other idea for finding the reason of freeze. Is there a possible way to give some users with freeze an image or some modules with debug code which write debug output to a file if the code around the write calls or buffer management see some buffer overflow or big latency. If those users have freeze they can send to you the debuig file. In the debug file you can write many interessting infos which you need. I know that is not easy but I think it is a possible way.


I'll prepare a "debug" version this weekend (hopefully), that may provide the insight.

After all, reason #1 to move to a more recent kernel was ext4 support, so it would be kind of strange to move two generations back to ext2 instead...
Real musicians never die - they just decompose

Re: ET9000:Hänger in HD Aufnahmen #57 anudanan

  • Senior Member
  • 1,185 posts

+16
Neutral

Posted 2 February 2012 - 23:23

cool, I´ve posted for some minutes that information inside et-view forum to find user for testing with debug image

Receiver:2 x Uno4k SE (PLI 7.3 rel), 1 x ET9200 (PLI 4.0), NAS: 2 x QNAP 410, TV: LG 65C8llla, LG 47LB570V, LG 42LM615S, Sound: Yamaha RX-v663, Teufel System 5 THX


Re: ET9000:Hänger in HD Aufnahmen #58 1701d

  • Senior Member
  • 38 posts

+3
Neutral

Posted 3 February 2012 - 08:08


I'll prepare a "debug" version this weekend (hopefully), that may provide the insight.

After all, reason #1 to move to a more recent kernel was ext4 support, so it would be kind of strange to move two generations back to ext2 instead...


Hi Milo,

if you prepare an image, I'm more then happy to try this and give you feedback.

Kind regards

Klaus

Re: ET9000:Hänger in HD Aufnahmen #59 MiLo

  • PLi® Core member
  • 14,055 posts

+298
Excellent

Posted 5 February 2012 - 16:15

Quick conclusion for those eager to know: If you want ext2 behavior, disable "delayed allocation" on the ext4 driver. This will make it perform slower, but more predictable (but still faster and more reliable than ext2). To do so, specify "nodelalloc" in the mount options, e.g.: "mount -o nodelalloc /dev/sda1 /media/hdd"

I've been doing some more tests myself this weekend. Re-formatted my disk as ext3, and used the different ext2/3/4 drivers to mount it with varying options. To test their "recording" behaviour, I start a recording on one HD channel, and then "torture" the system with "dd if=/dev/zero of=/media/hdd/blanks2 bs=188k count=10000" to dump about 1.8 GB on the disk, erase it, then dump it again. Enigma2 measures and writes the time it takes to write each 188k block to its output.

As for details:
With ext4 "fully functional", I see write times of ~5ms. Under torture, most aren't affected, but some are detained for 100..1000ms. This effect is larger when data=ordered is active, data=writeback tampers it. The ext2 and ext3 drivers show slightly larger overall write times, but when under stress, the load is spread more evenly amongst calls, so instead of a single call taking up 100ms, you now see 5 calls of 20ms, or something similar to that. This is much easier to cope with. Since the ext4 driver uses features not previously seen in ext2/3, shutting one of them down should bring back that old behavior. Indeed, "delalloc" showed to be the one. If I tell the ext4 driver to stop using delayed allocation, it behaves much more like the ext2/3 drivers when mounting an ext3 disk. Even better, when I converted the disk to ext4, and then added the "nodelalloc" option to the mount command, this effect remained in place. What I did notice is that the "torture" command now had to pay the price. It was writing significantly slower (from 50 to 30MB/s).
I have to check the documentation, but my understanding is that the system, instead of allocating blocks at each call to write(), it just collects them and allocates them in bigger chunks. This does help file transfer significantly, but on the typical load from recordings, causes the long delays on some of the calls when there are other threads performing IO as well.
Real musicians never die - they just decompose

Re: ET9000:Hänger in HD Aufnahmen #60 anudanan

  • Senior Member
  • 1,185 posts

+16
Neutral

Posted 5 February 2012 - 16:43

Great test you've made. One question around that.
Does only the parameter nodealloc solve the latency problem or is there a need for data=writeback instead of data=ordered.

Receiver:2 x Uno4k SE (PLI 7.3 rel), 1 x ET9200 (PLI 4.0), NAS: 2 x QNAP 410, TV: LG 65C8llla, LG 47LB570V, LG 42LM615S, Sound: Yamaha RX-v663, Teufel System 5 THX



2 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users