173 replies to this topic

[SpaceRat]

May I remind you that the hardware RTC is the smallest part of the problem?
The hardware RTC would help to preserve time across boots, especially full power off, yes.

It doesn't add the slightest benefit for getting and maintaining an accurate time in the first place though.
Even the most accurate masterpiece of Swiss made wristwatch will just show a wrong time permanently if all you have for adjustment is "the sun is highest at high noon" in winter.

What we have now in oe-a is:
1st step: Restore build or last shutdown time early on boot (Like in Raspbian, but as "save-rtc.sh" it has been in yocto forever too, it was just disabled to keep other ugly workarounds working)
2nd step: Restore FP pseudo-RTC if it was powered and contains a realistic time
3rd step: Sync with NTP (Has always been in yocto too, but was broken, distros added another NTP shot in enigma2.sh as a workaround, but for boot services it came too late)
4th step: Maintain sync using NTP or DVB time (With oscam apparently even disallowing this)

1st and 3rd are common practice anyways:
1st: Even systemd alone will advance the system time to systemd built time for early boot, the fake-hwclock in Raspbian just makes it a bit more accurate.
At that stage, you do not even have access to a real RTC yet, so it won't get better here (And isn't even needed to get better).
2nd: All OSes I checked use NTP time sync when networked, common Linux distros like Debian/Linux/MiNT as well as Windows and macOS.

2nd differs from machines with a battery backed up hardware RTC only in the way that it will be useless after power off, but the vast majority of users won't ever notice as the time from 1st is still good enough for any task currently and predictably being invoked at that stage on an E2 box (Remember a networked box will get a very accurate time only seconds later). And non networked boxes are just even more unlikely to do anything at this stage that requires a precise time here.

Only boxes that are operated
- non networked
and
- not connected to DVB
would benefit from an RTC here.


So the problem(s) aren't in 1st, 2nd or 3rd, they reside in the 4th point:
For boxes that are not networked, the only source for time is the DVB transponder time and that one is almost entirely discarded later.

E2 allows the box to sync with DVB exactly once, all later so-called "syncs" are only allowed to confirm the time or more or less slowly drift it (max. +/- 120 sec adjustment).
All higher diffs are discarded.

So E2 puts the emphasis on bad transponders, which are not really a problem for our mainly Central European audience, as Astra (No matter if 19.2°E, 23.5°E or 28.5°E) and Hotbird seem to deliver accurate times.
It even does so in a way that the box is allowed to pick the time from a bad transponder first and then discards the good times from good transponders later.

And finally we have oscam with CLOCKBREAK that ruins even the small adjustments ...

So as far as I can see, the main issues are dvbtime.cpp in E2 and the clock fix in oscam.
It's boxes that got misadjusted in E2 or not adjusted thanks to oscam that are missing recordings, not boxes being off a bit during boot, before E2 even started.

So the whole discussion about HW-RTC reminds me of some car mechanics discussing how to upgrade a car with a seat heating, while that car wouldn't start at all thanks to a broken engine.

I have checked the CLOCKFIX related changes in oscam and as far as I can see, one can define CLOCK_MONOTONIC, but I have no clue if that would help at all.

IMHO, as we recommend oscam over all closed-source CAMs, the CLOCKFIX issue there should be addressed first, as it would currently prevent any time adjustments in E2 or through NTP entirely (= show stopper).
As soon or in parallel while that gets fixed, one could try to find better algorithms for the DVB sync to help those users that operate their box without network.

From what I understand, the time should be slewed rather than stepped for a beginning, as stepping time could make several events either be missed (If stepped over) or to trigger far too fast (If stepping inside the pause between).

Second, in case there are known good transponders on any of the set up positions, why not have E2 forcibly start there? Just until we got a time, after that, E2 can switch to the user selected startup or last tuned station.
Once we got that, one could try to detect transponders reporting local time by just subtracting full half hours from the difference. A transponder reporting UTC+1 and 3800 sec difference is probably still right and we are 200 sec off after subtracting the 3600 sec for the one hour ...

Edited by SpaceRat, 31 December 2017 - 09:40.

[littlesat]

http://bastioninfote...-dongle-key.jsp
https://www.tindie.c...-raspberry-pi-/

Edited by littlesat, 31 December 2017 - 09:58.

[Erik Slagter]

We know them. They're not really cheap. Also at least the second one does not have a battery, so it's ultimately useless for this goal. And also note it has an ATMega328p on it (exactly one of my ideas to make something like this).

[littlesat]

But actually the box makers should put it on their boxes...
So the anti battery army could continue yelling...
At least when you Google for what is needed you can’t find
any ready to go plug and play ready to use solution. I’m afraid this is due to we have ntp servers...

Edited by littlesat, 31 December 2017 - 10:44.

[WanWizard]

We have to break down this issue into different tasks:

1. Get rid of the clock correction code in Oscam.
2. Get rid of the clock correction code in Enigma.
3. Find a means of setting the clock to the correct time at bootup.
4. Find a means of keeping the clock correct.

Imho currently issues 1 and 2 prevent proper solutions for 3 and 4, so we have to address those first (I propose after 6.1-release).

 

For 3, these are the options available:

1. Use transponder time (may not be correct) 
2. Use ntpdate or ntp -q (only when the box is networked)
3. Use an RTC device of some sort (currently not supported)
4. Enter the correct time at bootup (which is what you would do with a PC with empty RTC memory) 

For 4, these are the options available:

1. Use transponder time (will send the time all over the place if not correct)
2. Use ntpdate or ntp -q (only when the box is networked)
3. Use an RTC device of some sort (currently not supported)
4. Do nothing (and hope the hardware clock doesn't drift too much)

[littlesat]

We work with it as is for more the a decennia and I never got a clock problem as it is right now.:.

Edited by littlesat, 31 December 2017 - 16:13.

[WanWizard]

We work with it as is for more the a decennia and I never got a clock problem as it is right now.:.

 

That doesn't mean it is not a problem. It only means that of the (many thousands of) users, you are one without a problem.

[littlesat]

I understand... but why dies it suddenly pop-up? Since e1 from 2002 rately someone complaints.... I’m afraid oscam also playing with the clock is the biggest issue this simptom pop-ups....

[WanWizard]

The fact that Oscam decides to block changes does certainly play a part.

[SpaceRat]

We work with it as is for more the a decennia and I never got a clock problem as it is right now.:.

There has always been a problem, I presented links from multiple distros, but here again, just OpenPLi:
https://forums.openp...me-in-dreambox/
https://forums.openp...-time-and-date/
https://forums.openp...0-correct-time/
https://forums.openp...tem-time-again/
https://forums.openp...ong-clock-time/
https://forums.openp...-by-10-minutes/
https://forums.openp...-time-sync-ntp/
...

And while you might feel like users never use forum searches, but some actually do and some neither add a "me too!" to existing threads.
So they just shut down and power off the box and put it back on to get a correct time, as suggested in existing threads, without writing here.

Time syncing has always been a problem in E2

Edited by SpaceRat, 31 December 2017 - 17:01.

[littlesat]

I don’t say there is no issue... but the clock is wrong when...
1. When no transponder can be tuned (bad weather)
2. When the first transponder tuned ahs an incorrect time.
3. Oscam holds the clock.

For 1. We can use ntp or add a rtc. But ntp sprays and for rtc we need to have to add hardware
For 2. We could e.g. Blacklist bad transponders
For 3. Oscam needs to help.

Edited by littlesat, 31 December 2017 - 17:11.

[WanWizard]

3 is only an issue if 1 or 2 is the case.

 

And oscam can be addressed using a compile time switch in the recipe, see the post of Parasol.

[SpaceRat]

And oscam can be addressed using a compile time switch in the recipe, see the post of Parasol.

https://github.com/S...cd70861267b50ce

I will give it a test, but on the other hand I'm using NTP ... as far as I understand theparasol it's mainly to address bad transponder time syncs ...

[SpaceRat]

I decided to start at the beginning and finish at the end ...

https://github.com/O...igma2/pull/1192

It doesn't address all known issues, but only one.
Instead of stepping each and every time difference dvbtime.cpp finds, smaller ones are now slewed, that means the clock will gradually drift towards the desired time, rather than jumping to it.

That means this effect ...

19:31:09.9116 [   ] dvb/dvbtime.cpp:468 updateTime [eDVBLocalTimerHandler] Transponder time is 01.01.2018 19:31:11
19:31:09.9117 [   ] dvb/dvbtime.cpp:481 updateTime [eDVBLocalTimerHandler] diff is 2
19:31:09.9123 [   ] dvb/dvbtime.cpp:581 updateTime [eDVBLocalTimerHandler] set Linux Time
19:31:11.9127 [   ] dvb/dvbtime.cpp:592 updateTime [eDVBLocalTimerHandler] time after update is 19:31:11

(note the 2 seconds time warp from 19:31:09.9123 to 19:31:11.9123)

... is gone ...

16:56:44.5658 [   ] dvb/dvbtime.cpp:468 updateTime [eDVBLocalTimerHandler] Transponder time is 02.01.2018 16:56:42
16:56:44.6767 [   ] dvb/dvbtime.cpp:588 updateTime [eDVBLocalTimerHandler] slewing Linux Time by -02 seconds
16:56:44.6773 [   ] dvb/dvb.cpp:2415 getDemux [eDVBChannel] getDemux cap=00
16:56:44.6796 [   ] dvb/pmt.cpp:950 SDTScanEvent [eDVBServicePMTHandler] sdt update done!
16:56:44.8041 [   ] dvb/epgcache.cpp:1594 startEPG [eEPGCache] start caching events(1514908604)

(note that time appears to continue normally)

... the clock gets slowed down ultraslightly, so that it will eventually (in about 1-2hrs) have lost the 2 seconds it was ahead of the transponder.

So this fix should be enough though to make the CLOCKFIX in oscam obsolete.

[Erik Slagter]

Applied and thanks!

[SpaceRat]

The next step would be to make E2 always tune a good transponder first at start, even
- if it is set to start in standby
- if the startup channel is IPTV or HDMI-In or something else like that
- if the startup channel is on a bad transponder (The harder part)

E2 should initially shortly tune (invisible/pseudo recording) to a DVB (-S/-C/-T) channel with a good time (If it is configured for DVB of course, a box without coax obiously can't).

Part A:
On start shortly (just long enough for a time sync) tune the first tuneable DVB service, only then
- go to standby
or
- tune to preferred startup channel
(whatever is set).

A can be omitted if the startup channel is on DVB and the box is configured to not start in standby.


Part B:
Find - in the long run preferably on each and every satellite - at least one stable and time reliable transponder, mark it with <reftime>1</reftime> in satellites.xml and use that one for the first sync from part A.

Note:
I'm definitely not the right one to do this

[fcs001fcs]

Applied and thanks!

 

My 2 DM7020HDs (OpenPLi 4) had a 3 min error again today but my Vu+ Zero (OpenPLi 6) did not. My guess was it happen yesterday as the overnight recording was OK yesterday but 3 mins were cut off today.

 

Are these changes/corrections talked about here also being applied to OpenPLi 4?

[SpaceRat]

My 2 DM7020HDs (OpenPLi 4) had a 3 min error again today

3 min are outside of what E2 will correct ...
... that was part of this topic

E2 only corrects differences up to +/- 120 sec, it will consider everything beyond as time drift of the transponder resp. bad transponder time.

So far, no solution for this has been added. As there are really transponders that are off by some minutes (Found one with 5 minutes offset), applying larger corrections would make your clock go to wonderland.
E2 would need something like "reference transponders" with trustworthy times.

For a networked box, NTP remains to be the better choice.

but my Vu+ Zero (OpenPLi 6) did not. My guess was it happen yesterday as the overnight recording was OK yesterday but 3 mins were cut off today.

That's unavoidable if you don't add some grace period to record before and after the scheduled programme.

I use 5 min pre and 20 min post.
20min post are to cope with some politician/actor/unknown person dying and some "special news" being broadcasted, postponing scheduled programmes or live TV not staying within the schedule.

[WanWizard]

Lucky you. Some providers are all over the place, I've noticed CDS in NL and TVV in BE even inserting gaps in their EPG to deal with programmes running late.

 

In Belgium, 20min at the end of the evening is not unusual. In the UK, the BBC announcer apologises with "we are sorry this programme is later than billed" if it runs 5 minutes late...

[fcs001fcs]

For a networked box, NTP remains to be the better choice.

 

 

All my boxes are networked.

 

I tried the NTP plugin (Systemtime plugin) before and it did nothing to correct the issue. 

 

In my situation, with a 3 min lag between the DM7020HD with OpenPli 4 and correct time, how do I setup a NTP update that will actually correct the time on the DM7020HD?

 

 

BTW this is a recent issue as for many months/year I never had a time synchronisation issue with any of my STBs and their overnight recordings. I would say it changed during the last 3 months or so.