30 replies to this topic

[doglover]

There is one website I lost to abilty to update.  Tried to make contact with the administrator of that site, but no reply yet.  (send a message on 29/07/2018)

I asumed that the website was somehow off-line.

Now I am discovering, you can still download from it.  Problem...  (obsolete data)

Replace (search and replace) in the rytec.sources.xml file:

~rytecepg/epg_data

with something different, like

~rytecepg/epg_data234

So the importer cannot find the old data anymore.

 

Willy

 

PS: this website is run by OpenVIX -- Home of VIX Team

Edited by doglover, 13 August 2018 - 08:18.

[Abu Baniaz]

He is off on holiday.

Sent from my Moto G (5S) using Forum Fiend v1.3.3.

[Abu Baniaz]

Message sent, he will attempt to remote in and resolve the problem.

[doglover]

Still no cure.

 

@Abu Baniaz:  Can you acces the server?  If yes pls delete all .xz files from the epg folder.  This will eleminate any download of obsolete files.

 

Willy

[Abu Baniaz]

I don't have access.

[Pr2]

Is it solved now?

[doglover]

No.  Not yet.

 

Willy

[Frenske]

I guess it still isn't solved today. Most of the Dutch EPG isn't present tomorrow.

Is this a logical consequence of this issue?

[WanWizard]

A download server is selected at random, for every source you have enabled. There are 5 servers, so for every source you have a 20% chance it hits the OpenVIX server and you will miss the data for that source.

 

I'm thinking about a mechanism to deal with this, but I haven't found anything simple enough yet.

[Pr2]

Create a dummy empty file flag containing with the date and time when the last upload is successful.

 

LastUpdate.log

 

Containing for exemple:

 

20180809-14H30

 

EPGimport will first get this file LastUpdate.log and retrieve the date in it then compare it with the current date of the system. If the date is older then 2 days then switch to another mirror. If it is in the slot, then use that mirror.

If all mirrors return the same date then use the data files anyway, since the files often content 5 days EPG and this will cover the case when there was a problem on the upload side (grab failed and upload was not possible).

 

The time is given as informational only because every upload servers will get a different time so date is more important than time in this check. Or we should consider a timeslot for the check between the different mirrors, if time is taken into account.

 

Pr2

[WanWizard]

I was thinking along this line.

 

At the start of the import, fetch the last-update-timestamp from all download mirrors, sort the result, and dump all mirrors that have a timestamp that is out of bounds (for example, 2 days old is acceptable, 3 days isn't).

 

The last-update-timestamp should be a separate file that is created/generated by Willy's scripts and uploaded with the rest of the files, it should not be linked to the server. So you also don't have issues with variations between servers, it's either the same, or it isn't.

[doglover]

I have added a statement to the upload script:

 

echo $(date -I) > $EPGDIR/LastUpdate.log

 

And statements to upload the file to servers after the upload has succeeded.

 

The file contains just the date.  2018-08-10

 

Willy

[Pr2]

@WanWizard,

 

Your approach will poll every possible severs by every clients in the world so we will lose the random load-balancer advantage, there is no need for this when there is no problem. While my approach will not.

I will check the LastUpdate.log only on the already selected random mirror and then only switch to another one when needed.

 

But I have no idea about the complexity to implement this in EPGimport.

 

@Doglover,

 

Thanks to already have update your script so skilled python developers (which I am not) can start testing the proposed solution.

But Willy be sure to write the file only if all the above file transfer are successful otherwise this will be useless.

Exemple a server as only a few Kb free space, it won't have room to accept the data file but enough space to have the LastUpdate.log file, so the flag files will be there but the data files are not.

We must be sure that if the flag file is there this means that all the data are properly uploaded.

Edited by Pr2, 10 August 2018 - 14:04.

[doglover]

Checked the exit code of curl, and only if it is 0 for all the uploads it will write the LastUpdate.log file.

I have done this now only for the VU-plus website.  It can be xtended to the other websites also.

 

Only for the PLi website I need something different, as the upload is not performed with CURL.

 

Willy

[WanWizard]

Your approach will poll every possible severs by every clients in the world so we will lose the random load-balancer advantage, there is no need for this when there is no problem. While my approach will not.

I will check the LastUpdate.log only on the already selected random mirror and then only switch to another one when needed.

 

The problem with this is that you don't know the status of a server until you can compare it to the others. If you get "2018-08-10" back, does that mean you discard it (because it is not today)? Or does that mean there was no update today and that server does contain the latest version? Or perhaps that your box is in a timezone so for away from Willy's timezone that there is a day difference?

 

You can also not compare it to something you have locally. If your box has been in deep standby for a week, the fact that a servers reports a date more recent than your last download doesn't say anything about how current the data on the server is.

 

Polling the servers to check the status isn't a problem, it is not the number of hits, it is the volume per month that is a challenge for those servers.

Edited by WanWizard, 11 August 2018 - 12:33.

[WanWizard]

Checked the exit code of curl, and only if it is 0 for all the uploads it will write the LastUpdate.log file.

I have done this now only for the VU-plus website.  It can be xtended to the other websites also.

 

Only for the PLi website I need something different, as the upload is not performed with CURL.

 

Why do something separate, and not create the file locally and upload it with the rest?

[doglover]

 

Checked the exit code of curl, and only if it is 0 for all the uploads it will write the LastUpdate.log file.

I have done this now only for the VU-plus website.  It can be xtended to the other websites also.

 

Only for the PLi website I need something different, as the upload is not performed with CURL.

 

Why do something separate, and not create the file locally and upload it with the rest?

 

 

Of course this can always be done without a problem.  And it was my first approach.

I wanted to go a step further and check that the update happened and all the files are uploaded without an error.

 

Willy

[Pr2]

 

Checked the exit code of curl, and only if it is 0 for all the uploads it will write the LastUpdate.log file.

I have done this now only for the VU-plus website.  It can be xtended to the other websites also.

 

Only for the PLi website I need something different, as the upload is not performed with CURL.

 

Why do something separate, and not create the file locally and upload it with the rest?

 

 

Please read again my explanation, if you do it that way you cannot rely on the LastUpdate.log file, it will not cover a problem with upload of bigger files.

[Pr2]

 

Your approach will poll every possible severs by every clients in the world so we will lose the random load-balancer advantage, there is no need for this when there is no problem. While my approach will not.

I will check the LastUpdate.log only on the already selected random mirror and then only switch to another one when needed.

 

The problem with this is that you don't know the status of a server until you can compare it to the others. If you get "2018-08-10" back, does that mean you discard it (because it is not today)? Or does that mean there was no update today and that server does contain the latest version? Or perhaps that your box is in a timezone so for away from Willy's timezone that there is a day difference?

 

You can also not compare it to something you have locally. If your box has been in deep standby for a week, the fact that a servers reports a date more recent than your last download doesn't say anything about how current the data on the server is.

 

Polling the servers to check the status isn't a problem, it is not the number of hits, it is the volume per month that is a challenge for those servers.

 

 

1) I say that we must still consider the server if it is in a 2 days difference slot between the local STB date and the LastUpdate.log, this will cover the world timezone. Since in the world we only have 1 day difference, that's where the 2 days came from.

2) If Willy upload has a problem to upload to every server that last 3 days, then your LastUpdate.log on ALL the servers will be the same (exemple 2018-08-07) and then which conclusion can you take today we are the 2018-08-11, if you didn't check the local date of the STB then you will download and process an obsolete data file, so same as today.

3) Timezone difference is always between +12H -12H from you position on earth, so always in MAX 1 day difference  (the earth turn on itself in 24H). So 2 days to compare date is nice. 

4) I never take into consideration the last download date in the brainstorm, this information is useless

5) Using a STB which is not at the right date and time to download EPG is totally useless... since you will upload in memory EPG datas that are set for the right date and time for the program. If your STB is still in the past you will NEVER saw the EPG informations. So to use EPG your STB has to be at the right date and time.

(Remark: that's why OpenPLi should implement by default in its image the SystemTime plugin setup to use NTP server instead of TP time; so when starting a STB even without antenna connected the STB will always be at the right time). And if people use the EPGimport the box is connected to the Internet, people that are not connected to the Internet then use TP time and don't use EPGimport.

 

 

So you need to compare with the local STB date with the LastUpdate.log and not trust the server date in this file since you can't conclude nothing from it about the current real life date. Another argument is that you cannot unscramble channel when the date and time is not correct since it is also timebased. So you really can assume that the STB is at the right time.

 

Pr2

[WanWizard]

1. Ok.
2. That means that it should have a fallback in case all servers are rejected because of the +2 days rule.
3. Ok.
4. Indeed.
5. Correct for the first part, the rest is not relevant to this discussion.

If Willy can make sure all servers will have this file, I can see what I can do with the code. Will take some time, I'm travelling the next few weeks...