41 replies to this topic

[sirwio]

I do observe problem with stutter while streaming 720p channels. Eg. SVT1 HD from Canal Digital Sweden DVB-S2. Other SD and 1080i channels stream fine.

While the stutter occurs I have observed the following in the enigma2 log:

[eFilePushThreadRecorder] Warning: All write buffers busy
[eDVBRecordFileThread] cancelling
[eDVBRecordStreamThread] failed to cancel, killing all waiting IO
[eDVBRecordFileThread] poll: aio_return returned failure: Interrupted system call
[eDVBRecordFileThread] poll: aio_return returned failure: Interrupted system call
[eDVBRecordFileThread] poll: aio_return returned failure: Interrupted system call
[eDVBRecordFileThread] Waiting for I/O to complete

And when the streaming is aborted:

[eDVBRecordFileThread] buffer usage histogram (4 buffers of 188 kB)
	  1:	165
	  2:	137
	  3:	  2
	  4:	  2

The above can be observed while e.g. streaming from a client on the network using the following command:
wget http://vuultimo:8001...:E080000:0:0:0: -O - > /dev/null

The warnings do not appear if the stream is collected locally on the ultimo host.

I suspect that network issues as well as client performance make the send buffers overflow while streaming. Note that recording the same channels work fine without stutter on playback even if recorded to a nas attached to the network.

Notice that it appears as if only 4 buffers are used. What is the reason for this?

Looking at the sources of demux.cpp I notices that the calculation/use of number of buffers differ if the stream is saved to file (while recording) and when streamed.

While recording the amount is determined by the amount of available memory. For the ultimo this appears to be 20 buffers. But when streamed the amount is hardcoded to be 4.

Would it make sense to modify the code in demux.cpp to use the same number of buffers as while recording or at least have it default to a higher number?

[littlesat]

Network issues?

[sirwio]

I don't think its network issues.

xbmc tmp # ping -c 10 vuultimo
PING vuultimo.sirvio.home (192.168.0.108) 56(84) bytes of data.
64 bytes from vuultimo.sirvio.home (192.168.0.108): icmp_seq=1 ttl=64 time=0.159 ms
64 bytes from vuultimo.sirvio.home (192.168.0.108): icmp_seq=2 ttl=64 time=0.149 ms
64 bytes from vuultimo.sirvio.home (192.168.0.108): icmp_seq=3 ttl=64 time=0.358 ms
64 bytes from vuultimo.sirvio.home (192.168.0.108): icmp_seq=4 ttl=64 time=0.275 ms
64 bytes from vuultimo.sirvio.home (192.168.0.108): icmp_seq=5 ttl=64 time=0.153 ms
64 bytes from vuultimo.sirvio.home (192.168.0.108): icmp_seq=6 ttl=64 time=0.160 ms
64 bytes from vuultimo.sirvio.home (192.168.0.108): icmp_seq=7 ttl=64 time=0.176 ms
64 bytes from vuultimo.sirvio.home (192.168.0.108): icmp_seq=8 ttl=64 time=0.162 ms
64 bytes from vuultimo.sirvio.home (192.168.0.108): icmp_seq=9 ttl=64 time=0.180 ms
64 bytes from vuultimo.sirvio.home (192.168.0.108): icmp_seq=10 ttl=64 time=0.160 ms

--- vuultimo.sirvio.home ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 8994ms
rtt min/avg/max/mdev = 0.149/0.193/0.358/0.065 ms

xbmc tmp # traceroute vuultimo
traceroute to vuultimo (192.168.0.108), 30 hops max, 60 byte packets
1  vuultimo.sirvio.home (192.168.0.108)  0.162 ms  0.170 ms  0.187 ms

Downloading an recording using ftp yield a download speed of 8.9 MB/s.
While streaming a 720p channel wget rarely peaks more than 2 MB/s.

xbmc tmp # wget http://vuultimo:8001/1:0:19:57E:45:46:E080000:0:0:0:  -O - > /dev/null
--2013-02-08 13:36:28--  http://vuultimo:8001/1:0:19:57E:45:46:E080000:0:0:0:
Resolving vuultimo... 192.168.0.108
Connecting to vuultimo|192.168.0.108|:8001... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [video/mpeg]
Saving to: ‘STDOUT’

	[		<=>					  ] 9,052,408   1.94MB/s

[sirwio]

So I have made some tests increasing the buffer count while streaming from the default value using 4 buffers.

With 20 buffers there are still occasional hickups but much less frequently.

Bumped it to 40 buffers and it streamed fine without any stutter from two 720p channels concurrently. One to an xbmc client and the other dumping the stream to /dev/null using wget. I noticed though that the buffers where close to filling up since between 34 and 39 buffers where reported having hits when the streams where closed.

[eDVBRecordFileThread] buffer usage histogram (40 buffers of 188 kB)
	  0:	  1
	  1:	544
	  2:	453
	  3:	  7
	  4:	  7
	  5:	  6
	  6:	  8
	  7:	  6
	  8:	  6
	  9:	  6
	 10:	  5
	 11:	  6
	 12:	  7
	 13:	  7
	 14:	  7
	 15:	  7
	 16:	 10
	 17:	  7
	 18:	  7
	 19:	  5
	 20:	  6
	 21:	  6
	 22:	  4
	 23:	  4
	 24:	  4
	 25:	  5
	 26:	  5
	 27:	  1
	 28:	  1
	 29:	  2
	 30:	  2
	 31:	  2
	 32:	  2
	 33:	  1
	 34:	  2
	 35:	  2
	 36:	  1

Is increasing the buffer count a viable solution or do you pro's still think its a network related issue?

Anyone else having success streaming 720p channels without stutter from vu+ ultimo?

[spock]

Got the exact same problem. I can hardly even stream SD channels, it's like if 1 packet is lost, the stream never really recovers and keep stuttering and usually it eventually breaks up completely. It didn't used to be like this, some year ago I remember seeing in trac that they changed send buffer to be very small in e2 (or somewhere else in openpli), I believe thats when my problems started. Streaming HD appears to be more of a dream for me, even though I can easily stream 9-10MB/sec data for hours on end using other protocols. Over FTP/Samba I get ~10MB/sec (yes, 100mbit/s) consistently. Samba was worse in the beginning showing a max of around 4-5MB/sec, but fixed after adding the following to smb.conf global section:

socket options = TCP_NODELAY SO_RCVBUF=262144 SO_SNDBUF=262144 SO_KEEPALIVE

How do I change enigma2/streaming buffers? My network works pretty much flawlessly, but with wireless a single packet can end up in the void no matter how well configured the network is. I wonder if more buffers would alleviate/fix this problem.. And before someone says don't stream over wifi, I've seen the same happening over CAT5.

[spock]

I should add I use VLC and various iPhone/Android apps to stream, and they all behave the same.

Buffer size at max wherever possible but there is really no difference, perhaps it will work "ok" a little while longer. But that's in minutes.

Is it possible during a packet loss that the player/client who want's a packet resent, e2 has already dropped it from it's buffers so it no longer available, and as such the player/client gets out of sync with the stream? One would think that the player (especially VLC) would recover if that was the case, but it doesn't, it continues to have hickups/freezes until it eventually just displays garbage. Again, buffer size in player plays little to no role, and as I stated above, I've seen it happen over standard CAT5 ethernet too.

Edited by spock, 28 May 2013 - 17:29.

[WanWizard]

TCP buffering happens at the network layer, and the way TCP works is that it will keep the packet until an ACK is received from the other side. TCP is connection oriented, retransmissions will cover for any packet loss.

 

So it must be an issue in the higher layers (client or server).

[spock]

Thanks for replying,

I've actually had this problem annoy me for well over a year I think, but I never bothered to make a post/ticket about it (iirc). Few other people seem to have similar problems..

Could you point me in some direction as to getting around this problem? To increase e2 buffers I'd have to build it myself (which should be no problem for testing), as long as I know where to look in the code.

Edited by spock, 28 May 2013 - 17:48.

[WanWizard]

How exactly do you stream? From the webif?

 

I've just streamed an HD channel over a crappy wifi link, and in VLC, if you choose Tools, Codec Information, Statistics, I can see that I haven't lost a single packet, but as soon as I get pixilation, the input discarded/corrupted counter goes up.

 

If network is no issue, then the question is what corrupts the input? I ran a wireshark session in the background, and that doesn't show any error at the TCP level, even over a crappy wifi...

[spock]

I stream from webif using VLC, or similar from the tablets/phones.

Generally there will be either one lost packet (I think), or a too high ECM, causing a glitch. After this the stream will never recover for me, it gradually gets worse until it stalls. To describe it better, it starts with some pixelation/slight freeze, more and more of that until I just see a bunch of smeared out colors, together with stuttering/missing audio. This happens usually within 10 minutes, sometimes in 1 minute.. always within half an hour. So I click stop & play and continue watching, until next time.

My VU+ Ultimo is hooked up to an openwrt wifi wds repeater, running at 450Mbit/s stable link on 5GHz band, it's really solid for everything else I do. But I've tested straight ethernet too, which has less chances of seeing this problem, although the problem has shown up there too.

If it ain't happening for you, maybe it's the VU drivers....again.

edit: How did the original poster increase e2 buffer sizes? Would be interesting to try.

Edited by spock, 28 May 2013 - 20:25.

[WanWizard]

I don't know if it's different from your issue, but I can easily reproduce similar VLC behavior, which is why I picked the crappy wifi link (from downstairs, other side of the house).

 

I wanted to confirm it's not an issue at the TCP layer. That one dropped packet is always there is seems, I think it's the first one received. Buf after that none, and my wireshark sniff confirms that the network or TCP layer is not the issue, all packets sent by the box are received by VLC.

 

I'm more puzzeled about the incrementing "corrupted" counter. What does corrupt the packet payload, and why?

[spock]

You think devices between the client and server are corrupting the packet payload? Or maybe gstreamer, enigma2 or vu drivers? ... It's a hard thing to guess. I wouldn't even know where to begin.

Edited by spock, 28 May 2013 - 20:57.

[WanWizard]

I doubt that, because I have the same issue as you, and I can easily reproduce it, if I make the connection unreliable enough.

[spock]

So to summarize it, my connection is too poor to handle streams.

 

Since I really can't afford to pull ethernet all over the house (and I got pads, phones too that wants streaming), I am just basically fu**ed?

 

Recording to PC/HDD and streaming over wifi from there works atleast, so I'm not all out of options.

[WanWizard]

If you have the same over a wired connection, I sincerly doubt that the quality of the connection has anything to do with it. After all, there are no TCP errors and no dropped packets (except for the one). The only thing that can play a part on the network level is latency (and jitter).

 

You can test that simply from your box (easier then from the iPad), by doing a

ping -s 1492 <ip-address-of-the-ipad>

and check what the response times are, and what the variation is reponse times is.

 

I think the problem isn't that the data gets lost, the problem is that it arrives too late, causing the player to "stall"...

[spock]

First test from stb to pc (vlc). All tests done during streaming content.

 

 

ping -s 1492
........
1500 bytes from 192.168.199.201: seq=192 ttl=128 time=2.771 ms
1500 bytes from 192.168.199.201: seq=193 ttl=128 time=2.097 ms
1500 bytes from 192.168.199.201: seq=194 ttl=128 time=2.738 ms
1500 bytes from 192.168.199.201: seq=195 ttl=128 time=2.645 ms
1500 bytes from 192.168.199.201: seq=196 ttl=128 time=2.483 ms
1500 bytes from 192.168.199.201: seq=197 ttl=128 time=2.298 ms
1500 bytes from 192.168.199.201: seq=198 ttl=128 time=2.440 ms
1500 bytes from 192.168.199.201: seq=199 ttl=128 time=2.560 ms
1500 bytes from 192.168.199.201: seq=200 ttl=128 time=2.358 ms
^C
--- 192.168.199.201 ping statistics ---
201 packets transmitted, 201 packets received, 0% packet loss
round-trip min/avg/max = 2.095/2.730/10.425 ms

 

The ipad3 is a lot more jittery (power saving features on it's wifi?) It didn't matter whetever I was barely out of range or right next up to the access point.

 

1500 bytes from 192.168.199.227: seq=90 ttl=64 time=2.739 ms
1500 bytes from 192.168.199.227: seq=91 ttl=64 time=118.730 ms
1500 bytes from 192.168.199.227: seq=92 ttl=64 time=3.376 ms
1500 bytes from 192.168.199.227: seq=93 ttl=64 time=7.261 ms
1500 bytes from 192.168.199.227: seq=94 ttl=64 time=88.792 ms
1500 bytes from 192.168.199.227: seq=95 ttl=64 time=111.188 ms
1500 bytes from 192.168.199.227: seq=96 ttl=64 time=7.306 ms
1500 bytes from 192.168.199.227: seq=97 ttl=64 time=11.168 ms
1500 bytes from 192.168.199.227: seq=98 ttl=64 time=79.790 ms
1500 bytes from 192.168.199.227: seq=99 ttl=64 time=2.963 ms
1500 bytes from 192.168.199.227: seq=100 ttl=64 time=3.046 ms
1500 bytes from 192.168.199.227: seq=101 ttl=64 time=2.935 ms
1500 bytes from 192.168.199.227: seq=102 ttl=64 time=8.642 ms
^C
--- 192.168.199.227 ping statistics ---
103 packets transmitted, 103 packets received, 0% packet loss
round-trip min/avg/max = 2.585/38.659/137.710 ms

 

And to the android (which also is much more jittery, whetever up close to the access point or not):

 

1500 bytes from 192.168.199.127: seq=94 ttl=64 time=104.019 ms
1500 bytes from 192.168.199.127: seq=95 ttl=64 time=5.761 ms
1500 bytes from 192.168.199.127: seq=96 ttl=64 time=45.698 ms
1500 bytes from 192.168.199.127: seq=97 ttl=64 time=6.495 ms
1500 bytes from 192.168.199.127: seq=98 ttl=64 time=6.191 ms
1500 bytes from 192.168.199.127: seq=99 ttl=64 time=119.509 ms
1500 bytes from 192.168.199.127: seq=100 ttl=64 time=41.324 ms
1500 bytes from 192.168.199.127: seq=101 ttl=64 time=14.523 ms
^C
--- 192.168.199.127 ping statistics ---
102 packets transmitted, 102 packets received, 0% packet loss
round-trip min/avg/max = 4.009/46.016/121.589 ms

 

I cant see any flaws there, other then the jitter (but I suspect it's power saving features of the mobile devices, seeing how they happen either with good or poor wifi signal). Aside from that, mainly I want to be able to stream to the PC/VLC anyway.

 

Packets arriving out of order, thats really a typical wifi thing isn't it?

 

As soon as I get the toolchain ready I'll build openpli/e2 with more buffers to check the outcome.

 

Many thanks for helping btw.

Edited by spock, 29 May 2013 - 10:26.

[WanWizard]

The latency variations you notice are definitely contributing to the problem.

 

Packets should not arrive out of order in a local LAN, you might have that in a large network with multiple routing paths.

[spock]

Look at STB -> PC, there are no variations there.

 

Anyhow, network problem, case closed?

[WanWizard]

If you don't have a problem STB -> PC, then I would say "yes".

 

I don't see how you could tell the STB to send data faster then the client would need it, so it can be buffered client-side, to cover for latency issues, but perhaps a developer can tell you more...

[spock]

The STB -> PC link is over wireless bridge too, the same wireless link that the mobile devices uses. And again, when I pulled a CAT5 cable between the rooms, the same issue appeared, it just took more time for it to happen.

 

Unless a developer has some insights, I guess there is nothing to do. I'll record to PC/HDD and stream it from there to wireless clients instead, which works flawlessly.

 

As soon as I get openpli buildroot set up (what's up with all those git server timeouts?) I'll try adding more buffers and see if it helps.

 

 

Thanks again!