Jump to content


Photo

MovieCut sometimes slow


  • Please log in to reply
89 replies to this topic

Re: MovieCut sometimes slow #61 Erik Slagter

  • PLi® Core member
  • 46,969 posts

+542
Excellent

Posted 9 January 2018 - 19:56

root@solo4k:~# uptime
 19:53:22 up 32 min,  1 user,  load average: 1.70, 1.74, 1.53

This is on my solo4k (which is kind of similar).

You see the load is quite high (it has two cpu's, so 2.00 means all cpu's are busy all of the time). And it's doing nothing now, only watching TV. Enigma uses 2-3% and all other processes are even lower cpu usage.

 

So it's exactly like I said, some device driver, kernel process and/or interrupt handler are keeping the cpu busy. No relation to harddisk access. There isn't even a harddisk fitted in this receiver.


* Wavefrontier T90 with 28E/23E/19E/13E via SCR switches 2 x 2 x 6 user bands
I don't read PM -> if you have something to ask or to report, do it in the forum so others can benefit. I don't take freelance jobs.
Ik lees geen PM -> als je iets te vragen of te melden hebt, doe het op het forum, zodat anderen er ook wat aan hebben.


Re: MovieCut sometimes slow #62 anudanan

  • Senior Member
  • 1,185 posts

+16
Neutral

Posted 9 January 2018 - 19:57

Why do you think a higher nice for the linux process mcut and linux process mv (for moving files started from EMC) makes no sense to put more CPU power to enigma?


Receiver:2 x Uno4k SE (PLI 7.3 rel), 1 x ET9200 (PLI 4.0), NAS: 2 x QNAP 410, TV: LG 65C8llla, LG 47LB570V, LG 42LM615S, Sound: Yamaha RX-v663, Teufel System 5 THX


Re: MovieCut sometimes slow #63 anudanan

  • Senior Member
  • 1,185 posts

+16
Neutral

Posted 9 January 2018 - 20:03

my uno4kse shows

root@Schlafzimmer:~# uptime
 20:00:31 up  1:22,  load average: 0.11, 0.59, 1.49
 
and the ET9200 shows
root@Annabell:~# uptime
 20:02:42 up 8 days,  4:03,  load average: 0.13, 0.08, 0.06
root@Annabell:~#
 

Receiver:2 x Uno4k SE (PLI 7.3 rel), 1 x ET9200 (PLI 4.0), NAS: 2 x QNAP 410, TV: LG 65C8llla, LG 47LB570V, LG 42LM615S, Sound: Yamaha RX-v663, Teufel System 5 THX


Re: MovieCut sometimes slow #64 Erik Slagter

  • PLi® Core member
  • 46,969 posts

+542
Excellent

Posted 9 January 2018 - 20:04

Re-read my explanation. Or read on some operating system basics. Nice is RELATIVE it's not ABSOLUTE. As long as the cpu is not completely loaded most of the time it does not make a difference.

 

Even if the architecture would not be using DMA, the code that would copy the data from/to the device, is not running in a user process, so it's not sensitive to priorities anyway.

 

Having said that, please be my guest, go ahead and find out for yourself.


* Wavefrontier T90 with 28E/23E/19E/13E via SCR switches 2 x 2 x 6 user bands
I don't read PM -> if you have something to ask or to report, do it in the forum so others can benefit. I don't take freelance jobs.
Ik lees geen PM -> als je iets te vragen of te melden hebt, doe het op het forum, zodat anderen er ook wat aan hebben.


Re: MovieCut sometimes slow #65 Erik Slagter

  • PLi® Core member
  • 46,969 posts

+542
Excellent

Posted 9 January 2018 - 20:06

my uno4kse shows

root@Schlafzimmer:~# uptime
 20:00:31 up  1:22,  load average: 0.11, 0.59, 1.49
 
and the ET9200 shows
root@Annabell:~# uptime
 20:02:42 up 8 days,  4:03,  load average: 0.13, 0.08, 0.06
root@Annabell:~#

And what is your point?

 

My point is that the "cpu load" of a set top box can very well be quite high, without "anything" (visible) happening. So it doesn't say anything. On a "normal" Linux PC it would be surprising, not on a STB.


* Wavefrontier T90 with 28E/23E/19E/13E via SCR switches 2 x 2 x 6 user bands
I don't read PM -> if you have something to ask or to report, do it in the forum so others can benefit. I don't take freelance jobs.
Ik lees geen PM -> als je iets te vragen of te melden hebt, doe het op het forum, zodat anderen er ook wat aan hebben.


Re: MovieCut sometimes slow #66 anudanan

  • Senior Member
  • 1,185 posts

+16
Neutral

Posted 9 January 2018 - 22:30

Yes, I now that nice makes only sense if CPU has 0 percent idle free. But during my tesIs with parallel recordings, cutting movie and moving movie I see with the tool top very often 0 percent idle load.

 

Normally during copying files form drive to drive (60-90Mbyte/s write performance) the i/o load is not high (10-25%) and the idle load is big enough.

 

Only during my above tests the idle load is often 0. 

 

Do you think these high I/O load  has the reason in some busy waits for DVB?


Edited by anudanan, 9 January 2018 - 22:35.

Receiver:2 x Uno4k SE (PLI 7.3 rel), 1 x ET9200 (PLI 4.0), NAS: 2 x QNAP 410, TV: LG 65C8llla, LG 47LB570V, LG 42LM615S, Sound: Yamaha RX-v663, Teufel System 5 THX


Re: MovieCut sometimes slow #67 ccs

  • Senior Member
  • 229 posts

+7
Neutral

Posted 9 January 2018 - 22:47

uptime command:

 

A completely idle computer has a load average of 0. Each running process either using or waiting for CPU resources adds 1 to the load average


test


Re: MovieCut sometimes slow #68 Erik Slagter

  • PLi® Core member
  • 46,969 posts

+542
Excellent

Posted 10 January 2018 - 21:06

Even though that is completely true (also known as "the run queue"), it's barely relevant, because software tends to run in very short bursts between waiting for I/O.


* Wavefrontier T90 with 28E/23E/19E/13E via SCR switches 2 x 2 x 6 user bands
I don't read PM -> if you have something to ask or to report, do it in the forum so others can benefit. I don't take freelance jobs.
Ik lees geen PM -> als je iets te vragen of te melden hebt, doe het op het forum, zodat anderen er ook wat aan hebben.


Re: MovieCut sometimes slow #69 Erik Slagter

  • PLi® Core member
  • 46,969 posts

+542
Excellent

Posted 10 January 2018 - 21:09

Yes, I now that nice makes only sense if CPU has 0 percent idle free. But during my tesIs with parallel recordings, cutting movie and moving movie I see with the tool top very often 0 percent idle load.

 

Normally during copying files form drive to drive (60-90Mbyte/s write performance) the i/o load is not high (10-25%) and the idle load is big enough.

 

Only during my above tests the idle load is often 0. 

 

Do you think these high I/O load  has the reason in some busy waits for DVB?

If you have multiple processes accessing your harddisk make the harddisk slow, even though hdparm -t says it should be fast, it is reasonable to assume the problem is the seeking on the harddisk, looks like the actuator is slow or getting slow (old age). That is the only explanation I can come up with. You'd really should run smartctl -a first to determine any pending problems.


Edited by Erik Slagter, 10 January 2018 - 21:09.

* Wavefrontier T90 with 28E/23E/19E/13E via SCR switches 2 x 2 x 6 user bands
I don't read PM -> if you have something to ask or to report, do it in the forum so others can benefit. I don't take freelance jobs.
Ik lees geen PM -> als je iets te vragen of te melden hebt, doe het op het forum, zodat anderen er ook wat aan hebben.


Re: MovieCut sometimes slow #70 anudanan

  • Senior Member
  • 1,185 posts

+16
Neutral

Posted 11 January 2018 - 10:11

After many tests with nice and different clustersizes if have found from my point of view the true reason for the poor /O performance to/from the hdd if the box make many write and read operations to/from the drive

 

The actual I/O scheduler for block devices like drives has tree methods to schedule the orders to the disk

 

NOOP. easy fifo method

dreadline: different queues and sortings for matching the sector numbers and some deadlines to prevent starving of drive orders

cfq : faire schedule based am time slices for different processes. This is the default scheduler

 

you can see the setting with

 

cat /sys/block/sd*/queue/scheduler

 

I don´t know exactly why cfq doesn´t work so efficient for the enigma box application if I make many recording in parallel to mcut and mv/cp commands but fact is, the i/o load is between 80-100 percent,  if you use massiv write and read operation, but the write transferrate to the disk is often poor. The idle load often is zero Sometimes you see only some Mbyte/s write performance to the hole drive for all recordings if you make mcut on mv/cp in parallel to the recordings. Maybe the timeslice doesn´t match the real time requirement of recording jobs. The result are interrrupts in the recordings and a poor GUI responsibiltiy. That is not what I expected from a modern box

 

Now I´ve tested the method  deadline. Setting is possible on runtime with

 

echo deadline >/sys/block/sdx/queue/scheduler

sdx = sda,sdc,sdc ...

for each drive

 

The result is phantastic: The box doesn´t make interrupts in the recordings if other jobs make massiv read operations. The reaction of the GUI ist much more better than with cfq.

For example with the cfg schedule if you are in the movielist and you select a movie which is recording in background (many records are running), very often you see a spinner or it needs time to show the size of the file. Now with deadline scheduler it is very quick. No waitings. I´ve seen that waitings also on my old et9200 box, They also use cfq scheduler but there it works better., I don´t know why cfg works better an MIPS than on ARM.

 

I´ve tested deadline with 14 parallel recordings and a cutting queue with mcut for other movies and moving movies with mv command (from EMC). The recording jobs now write all there datablocks to the disk and mcut and mv/cp get the restpower of the box for disk I/O. The command top now shows very often higher idle loads for the two cores and the I/O load is in a expected range

 

The problem with cfq are on all images. I´ve tested VTI, openATV and openpli. All use the same 4.1.20 kernel for unokse.

 

Now with the solution deadline I´m very happy and the box works as I expected. Other users may testet it and if all are happy, it would be great if the openpli  team will integrate deadline to the default scheduler during kernel compilation or later with startscripts on runlevel 


Receiver:2 x Uno4k SE (PLI 7.3 rel), 1 x ET9200 (PLI 4.0), NAS: 2 x QNAP 410, TV: LG 65C8llla, LG 47LB570V, LG 42LM615S, Sound: Yamaha RX-v663, Teufel System 5 THX


Re: MovieCut sometimes slow #71 twol

  • Senior Member
  • 448 posts

+15
Neutral

Posted 11 January 2018 - 11:27

Lots on this in different posts using deadline to get consistent I/O

see https://blog.codeshi...heduler-tuning/

&

https://access.redha.../solutions/5427

 

Also don't know if this of interest ref Moviecut:

https://github.com/o...12ddc83060874ab


Edited by twol, 11 January 2018 - 11:32.

Gigablue Quad 4K & UE 4K, Vu+Uno4KSE, DM900
.........FBC Tuners:
------------------> GT-SAT unicable lnb to 1.5M dish(28.2E)
------------------> Gigablue unicable lnb to 80 cm dish(19.2E)

Octagon sf8008, AX HD61, Edision Osmio 4K+, Zgemma H9Combo using Legacy ports on multiswitches
Zgemma H9twin & Zgemma H9 C/S mode into Giga4K
 


Re: MovieCut sometimes slow #72 ccs

  • Senior Member
  • 229 posts

+7
Neutral

Posted 11 January 2018 - 11:50

My ET10K running ViX 5.1.004 (Kernel 4.10.6) has a sda scheduler setting of

 

noop [cfq]


Edited by ccs, 11 January 2018 - 11:54.

test


Re: MovieCut sometimes slow #73 Pr2

  • PLi® Contributor
  • 6,181 posts

+261
Excellent

Posted 11 January 2018 - 12:49

@anudanan,

 

Nice finding which is confirmed by the link provided by @twol.

 

I check also on my STB and they are also in cfq while my Ubuntu PC is using deadline.

 

Pr2


NO SUPPORT by PM, it is a forum make your question public so everybody can benefit from the question/answer.
If you think that my answer helps you, you can press the up arrow in bottom right of the answer.

Wanna help with OpenPLi Translation? Please read our Wiki Information for translators

Sat: Hotbird 13.0E, Astra 19.2E, Eutelsat5A 5.0W
VU+ Solo 4K: 2*DVB-S2 + 2*DVB-C/T/T2 (used in DVB-C) & Duo 4K: 2*DVB-S2X + DVB-C (FBC)

AB-Com: PULSe 4K 1*DVB-S2X (+ DVB-C/T/T2)
Edision OS Mio 4K: 1*DVB-S2X + 1*DVB-C/T/T2
 


Re: MovieCut sometimes slow #74 anudanan

  • Senior Member
  • 1,185 posts

+16
Neutral

Posted 11 January 2018 - 13:15

i‘ve seen also on dreambox that they support only noop and cdq. Nice that openpli supports deadline

Receiver:2 x Uno4k SE (PLI 7.3 rel), 1 x ET9200 (PLI 4.0), NAS: 2 x QNAP 410, TV: LG 65C8llla, LG 47LB570V, LG 42LM615S, Sound: Yamaha RX-v663, Teufel System 5 THX


Re: MovieCut sometimes slow #75 anudanan

  • Senior Member
  • 1,185 posts

+16
Neutral

Posted 11 January 2018 - 14:25

 

Also don't know if this of interest ref Moviecut:

https://github.com/o...12ddc83060874ab

 

I think this change moves mcut from a normal Linux Process to an Enigma job. 

 

Do anyone know the reason for that?


Receiver:2 x Uno4k SE (PLI 7.3 rel), 1 x ET9200 (PLI 4.0), NAS: 2 x QNAP 410, TV: LG 65C8llla, LG 47LB570V, LG 42LM615S, Sound: Yamaha RX-v663, Teufel System 5 THX


Re: MovieCut sometimes slow #76 twol

  • Senior Member
  • 448 posts

+15
Neutral

Posted 11 January 2018 - 15:00

i‘ve seen also on dreambox that they support only noop and cdq. Nice that openpli supports deadline


The arm boxes on ViX (therefore all OEA images) support deadline

Gigablue Quad 4K & UE 4K, Vu+Uno4KSE, DM900
.........FBC Tuners:
------------------> GT-SAT unicable lnb to 1.5M dish(28.2E)
------------------> Gigablue unicable lnb to 80 cm dish(19.2E)

Octagon sf8008, AX HD61, Edision Osmio 4K+, Zgemma H9Combo using Legacy ports on multiswitches
Zgemma H9twin & Zgemma H9 C/S mode into Giga4K
 


Re: MovieCut sometimes slow #77 anudanan

  • Senior Member
  • 1,185 posts

+16
Neutral

Posted 11 January 2018 - 16:12

I´ve found  one article which means to deadline

 

Benefits: 
- Nearly a real-time scheduler. 
- Excels in reducing latency of any given single I/O 
- Best scheduler for database access and queries. 
- Does quite well in benchmarks, most likely the best
- Like noop, a good scheduler for solid state/flash drives

 

https://androidmodgu...schedulers.html

 

I think realtime is the most important benefit for the receivers


Receiver:2 x Uno4k SE (PLI 7.3 rel), 1 x ET9200 (PLI 4.0), NAS: 2 x QNAP 410, TV: LG 65C8llla, LG 47LB570V, LG 42LM615S, Sound: Yamaha RX-v663, Teufel System 5 THX


Re: MovieCut sometimes slow #78 twol

  • Senior Member
  • 448 posts

+15
Neutral

Posted 11 January 2018 - 17:05

Also don't know if this of interest ref Moviecut:
https://github.com/o...12ddc83060874ab

 
I think this change moves mcut from a normal Linux Process to an Enigma job. 
 
Do anyone know the reason for that?
Afraid you will have to ask the author

Gigablue Quad 4K & UE 4K, Vu+Uno4KSE, DM900
.........FBC Tuners:
------------------> GT-SAT unicable lnb to 1.5M dish(28.2E)
------------------> Gigablue unicable lnb to 80 cm dish(19.2E)

Octagon sf8008, AX HD61, Edision Osmio 4K+, Zgemma H9Combo using Legacy ports on multiswitches
Zgemma H9twin & Zgemma H9 C/S mode into Giga4K
 


Re: MovieCut sometimes slow #79 MiLo

  • PLi® Core member
  • 14,055 posts

+298
Excellent

Posted 11 January 2018 - 19:18

The IO scheduler you really want is BFQ, but that's only available as a default scheduler in 4.14. It can be added to older kernels as a patch of sorts though.

If you look way back in posts, you may be able to find the posts where I evaluated the schedulers and other tunes for the filesystem to stabilize recording. Probably using an ET9000 and older stuff for testing. At that point, the CFQ was by far the best choice. The "deadline" only made things worse. CFQ yields optimal performance on rotating harddisks, and in real life (e.g. desktop) outperforms deadline in both latency and throughput, even when using SSD. That Ubuntu made it default was just sheer stupidity. That deadline performs well in benchmarks is of no concern to real users. It is NOT a real time scheduler, and never was intended to be so. It was only intended as a "simple" scheduler for constrained systems.
Real musicians never die - they just decompose

Re: MovieCut sometimes slow #80 MiLo

  • PLi® Core member
  • 14,055 posts

+298
Excellent

Posted 11 January 2018 - 19:22

Another major difference with the newer boxes is that they have way more RAM. The tuning settings for the cache - also dating from the pre-et9000 era - may be causing trouble now because they allocate way too much cache. I've been using similar settings to speed up disk access on my desktop, but the same settings caused lockups when applied to the much larger amount of memory available on the build server.
Real musicians never die - they just decompose


5 user(s) are reading this topic

0 members, 5 guests, 0 anonymous users