Jump to content


Photo

One download site off-line / Downloads affected


  • Please log in to reply
27 replies to this topic

#1 doglover

  • Rytec EPG Team
  • 11,743 posts

+327
Excellent

Posted 31 July 2018 - 12:46

There is one website I lost to abilty to update.  Tried to make contact with the administrator of that site, but no reply yet.  (send a message on 29/07/2018)

I asumed that the website was somehow off-line.

Now I am discovering, you can still download from it.  Problem...  (obsolete data)

Replace (search and replace) in the rytec.sources.xml file:


~rytecepg/epg_data

with something different, like


~rytecepg/epg_data234

So the importer cannot find the old data anymore.

 

Willy

 

PS: this website is run by OpenVIX -- Home of VIX Team


Edited by doglover, 13 August 2018 - 08:18.

~~Rytec Team~~
Xtrend ET6000 OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: One download site off-line / Downloads affected #2 Abu Baniaz

  • Senior Member
  • 637 posts

+15
Neutral

Posted 31 July 2018 - 13:05

He is off on holiday.

Sent from my Moto G (5S) using Forum Fiend v1.3.3.

Re: One download site off-line / Downloads affected #3 Abu Baniaz

  • Senior Member
  • 637 posts

+15
Neutral

Posted 31 July 2018 - 14:13

Message sent, he will attempt to remote in and resolve the problem.



Re: One download site off-line / Downloads affected #4 doglover

  • Rytec EPG Team
  • 11,743 posts

+327
Excellent

Posted 1 August 2018 - 16:34

Still no cure.

 

@Abu Baniaz:  Can you acces the server?  If yes pls delete all .xz files from the epg folder.  This will eleminate any download of obsolete files.

 

Willy


~~Rytec Team~~
Xtrend ET6000 OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: One download site off-line / Downloads affected #5 Abu Baniaz

  • Senior Member
  • 637 posts

+15
Neutral

Posted 1 August 2018 - 21:42

I don't have access.



Re: One download site off-line / Downloads affected #6 Pr2

  • PLi® Test Team
  • 2,540 posts

+98
Good

Posted 7 August 2018 - 16:53

Is it solved now?

NO SUPPORT by PM, it is a forum make your question public so everybody can benefit from the question/answer.

If you think that my answer helps you, you can press the up arrow in bottom right the answer.

 

Octagon SF-4008: Hotbird 13.0E, Astra 19.2E, Eutelsat5A 5.0W (+DVB-C)
VU+ Solo 4K: 2*DVB-S2 + 2*DVB-C/T/T2 (used in DVB-C)
Zgemma H5: 1*DVB-S2 + 1*DVB-C

Zgemma H7C: 1*DVB-S2 + 2*DVB-C


Re: One download site off-line / Downloads affected #7 doglover

  • Rytec EPG Team
  • 11,743 posts

+327
Excellent

Posted 8 August 2018 - 16:40

No.  Not yet.

 

Willy


~~Rytec Team~~
Xtrend ET6000 OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: One download site off-line / Downloads affected #8 Frenske

  • Forum Moderator
    PLi® Core member
  • 19,937 posts

+144
Excellent

Posted 10 August 2018 - 09:19

I guess it still isn't solved today. Most of the Dutch EPG isn't present tomorrow.

Is this a logical consequence of this issue?


Mijn schotel is een T90 met 10 LNB's. Daarnaast voor de fun nog een draaibaar systeem met een Triax TD 78.

Een Dreambox DM 800-SE Verder nog een een VU+ duo met 500Gb harddisk + een VU+ Uno. + ook nog een Xtrend ET4000 en een Vu+ Zero erbij. Inmiddels heb ik ook nog een VU+ Duo2 erbij, een Vu+ Solo 4K, een Formuler F4, Xtrend ET10000, ET8500, ET7500, Mut@nt 2400HD, Xsarius Fusion HD se en nog het e.e.a.

Voor centrale opslag van media gebruik ik een Qnap 219P met tweemaal 2 Tb harddisks.

-------------------------------------------------------------------------------------------
Many answers to your question can be found in our wiki: Just one click away from this "solutioncentre".

Als ik alles al wist hoefde ik ook niets te vragen. If I had all the knowledge I had no questions at all.

Re: One download site off-line / Downloads affected #9 WanWizard

  • Forum Moderator
    PLi® Core member
  • 40,802 posts

+641
Excellent

Posted 10 August 2018 - 10:48

A download server is selected at random, for every source you have enabled. There are 5 servers, so for every source you have a 20% chance it hits the OpenVIX server and you will miss the data for that source.

 

I'm thinking about a mechanism to deal with this, but I haven't found anything simple enough yet.


Many answers to your question can be found in our new and improved wiki.

Currently in active use: HD2400 (4xS2)VU+Duo2 (3xS2)VU+Zero, Edision OS mini+, ET10000 (4xS2)

For testing purposes: XP1000Formuler F1 (2xS2)Miraclebox Premium Micro (S2+C/T),  ET7500 (S2)ET8500 (S2), Zgemma H2.H (S2+C)Zgemma H5.2TC (S2+C/T), SAB TripleAlpha (S2+C/T), Galaxy 4K (FBC), VU Zero 4K, Edision OS nino 


Re: One download site off-line / Downloads affected #10 Pr2

  • PLi® Test Team
  • 2,540 posts

+98
Good

Posted 10 August 2018 - 11:15

Create a dummy empty file flag containing with the date and time when the last upload is successful.

 

LastUpdate.log

 

Containing for exemple:

 

20180809-14H30

 

EPGimport will first get this file LastUpdate.log and retrieve the date in it then compare it with the current date of the system. If the date is older then 2 days then switch to another mirror. If it is in the slot, then use that mirror.

If all mirrors return the same date then use the data files anyway, since the files often content 5 days EPG and this will cover the case when there was a problem on the upload side (grab failed and upload was not possible).

 

The time is given as informational only because every upload servers will get a different time so date is more important than time in this check. Or we should consider a timeslot for the check between the different mirrors, if time is taken into account.

 

Pr2


NO SUPPORT by PM, it is a forum make your question public so everybody can benefit from the question/answer.

If you think that my answer helps you, you can press the up arrow in bottom right the answer.

 

Octagon SF-4008: Hotbird 13.0E, Astra 19.2E, Eutelsat5A 5.0W (+DVB-C)
VU+ Solo 4K: 2*DVB-S2 + 2*DVB-C/T/T2 (used in DVB-C)
Zgemma H5: 1*DVB-S2 + 1*DVB-C

Zgemma H7C: 1*DVB-S2 + 2*DVB-C


Re: One download site off-line / Downloads affected #11 WanWizard

  • Forum Moderator
    PLi® Core member
  • 40,802 posts

+641
Excellent

Posted 10 August 2018 - 11:31

I was thinking along this line.

 

At the start of the import, fetch the last-update-timestamp from all download mirrors, sort the result, and dump all mirrors that have a timestamp that is out of bounds (for example, 2 days old is acceptable, 3 days isn't).

 

The last-update-timestamp should be a separate file that is created/generated by Willy's scripts and uploaded with the rest of the files, it should not be linked to the server. So you also don't have issues with variations between servers, it's either the same, or it isn't.


Many answers to your question can be found in our new and improved wiki.

Currently in active use: HD2400 (4xS2)VU+Duo2 (3xS2)VU+Zero, Edision OS mini+, ET10000 (4xS2)

For testing purposes: XP1000Formuler F1 (2xS2)Miraclebox Premium Micro (S2+C/T),  ET7500 (S2)ET8500 (S2), Zgemma H2.H (S2+C)Zgemma H5.2TC (S2+C/T), SAB TripleAlpha (S2+C/T), Galaxy 4K (FBC), VU Zero 4K, Edision OS nino 


Re: One download site off-line / Downloads affected #12 doglover

  • Rytec EPG Team
  • 11,743 posts

+327
Excellent

Posted 10 August 2018 - 13:32

I have added a statement to the upload script:

 

echo $(date -I) > $EPGDIR/LastUpdate.log

 

And statements to upload the file to servers after the upload has succeeded.

 

The file contains just the date.  2018-08-10

 

Willy


~~Rytec Team~~
Xtrend ET6000 OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: One download site off-line / Downloads affected #13 Pr2

  • PLi® Test Team
  • 2,540 posts

+98
Good

Posted 10 August 2018 - 14:00

@WanWizard,

 

Your approach will poll every possible severs by every clients in the world so we will lose the random load-balancer advantage, there is no need for this when there is no problem. While my approach will not.

I will check the LastUpdate.log only on the already selected random mirror and then only switch to another one when needed.

 

But I have no idea about the complexity to implement this in EPGimport.

 

@Doglover,

 

Thanks to already have update your script so skilled python developers (which I am not) can start testing the proposed solution.

But Willy be sure to write the file only if all the above file transfer are successful otherwise this will be useless.

Exemple a server as only a few Kb free space, it won't have room to accept the data file but enough space to have the LastUpdate.log file, so the flag files will be there but the data files are not.

We must be sure that if the flag file is there this means that all the data are properly uploaded.


Edited by Pr2, 10 August 2018 - 14:04.

NO SUPPORT by PM, it is a forum make your question public so everybody can benefit from the question/answer.

If you think that my answer helps you, you can press the up arrow in bottom right the answer.

 

Octagon SF-4008: Hotbird 13.0E, Astra 19.2E, Eutelsat5A 5.0W (+DVB-C)
VU+ Solo 4K: 2*DVB-S2 + 2*DVB-C/T/T2 (used in DVB-C)
Zgemma H5: 1*DVB-S2 + 1*DVB-C

Zgemma H7C: 1*DVB-S2 + 2*DVB-C


Re: One download site off-line / Downloads affected #14 doglover

  • Rytec EPG Team
  • 11,743 posts

+327
Excellent

Posted 10 August 2018 - 15:53

Checked the exit code of curl, and only if it is 0 for all the uploads it will write the LastUpdate.log file.

I have done this now only for the VU-plus website.  It can be xtended to the other websites also.

 

Only for the PLi website I need something different, as the upload is not performed with CURL.

 

Willy


~~Rytec Team~~
Xtrend ET6000 OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: One download site off-line / Downloads affected #15 WanWizard

  • Forum Moderator
    PLi® Core member
  • 40,802 posts

+641
Excellent

Posted 11 August 2018 - 12:16

Your approach will poll every possible severs by every clients in the world so we will lose the random load-balancer advantage, there is no need for this when there is no problem. While my approach will not.

I will check the LastUpdate.log only on the already selected random mirror and then only switch to another one when needed.

 

The problem with this is that you don't know the status of a server until you can compare it to the others. If you get "2018-08-10" back, does that mean you discard it (because it is not today)? Or does that mean there was no update today and that server does contain the latest version? Or perhaps that your box is in a timezone so for away from Willy's timezone that there is a day difference?

 

You can also not compare it to something you have locally. If your box has been in deep standby for a week, the fact that a servers reports a date more recent than your last download doesn't say anything about how current the data on the server is.

 

Polling the servers to check the status isn't a problem, it is not the number of hits, it is the volume per month that is a challenge for those servers.


Edited by WanWizard, 11 August 2018 - 12:33.

Many answers to your question can be found in our new and improved wiki.

Currently in active use: HD2400 (4xS2)VU+Duo2 (3xS2)VU+Zero, Edision OS mini+, ET10000 (4xS2)

For testing purposes: XP1000Formuler F1 (2xS2)Miraclebox Premium Micro (S2+C/T),  ET7500 (S2)ET8500 (S2), Zgemma H2.H (S2+C)Zgemma H5.2TC (S2+C/T), SAB TripleAlpha (S2+C/T), Galaxy 4K (FBC), VU Zero 4K, Edision OS nino 


Re: One download site off-line / Downloads affected #16 WanWizard

  • Forum Moderator
    PLi® Core member
  • 40,802 posts

+641
Excellent

Posted 11 August 2018 - 12:18

Checked the exit code of curl, and only if it is 0 for all the uploads it will write the LastUpdate.log file.

I have done this now only for the VU-plus website.  It can be xtended to the other websites also.

 

Only for the PLi website I need something different, as the upload is not performed with CURL.

 

Why do something separate, and not create the file locally and upload it with the rest?


Many answers to your question can be found in our new and improved wiki.

Currently in active use: HD2400 (4xS2)VU+Duo2 (3xS2)VU+Zero, Edision OS mini+, ET10000 (4xS2)

For testing purposes: XP1000Formuler F1 (2xS2)Miraclebox Premium Micro (S2+C/T),  ET7500 (S2)ET8500 (S2), Zgemma H2.H (S2+C)Zgemma H5.2TC (S2+C/T), SAB TripleAlpha (S2+C/T), Galaxy 4K (FBC), VU Zero 4K, Edision OS nino 


Re: One download site off-line / Downloads affected #17 doglover

  • Rytec EPG Team
  • 11,743 posts

+327
Excellent

Posted 11 August 2018 - 12:28

 

Checked the exit code of curl, and only if it is 0 for all the uploads it will write the LastUpdate.log file.

I have done this now only for the VU-plus website.  It can be xtended to the other websites also.

 

Only for the PLi website I need something different, as the upload is not performed with CURL.

 

Why do something separate, and not create the file locally and upload it with the rest?

 

 

Of course this can always be done without a problem.  And it was my first approach.

I wanted to go a step further and check that the update happened and all the files are uploaded without an error.

 

Willy


~~Rytec Team~~
Xtrend ET6000 OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: One download site off-line / Downloads affected #18 Pr2

  • PLi® Test Team
  • 2,540 posts

+98
Good

Posted 11 August 2018 - 13:54

 

Checked the exit code of curl, and only if it is 0 for all the uploads it will write the LastUpdate.log file.

I have done this now only for the VU-plus website.  It can be xtended to the other websites also.

 

Only for the PLi website I need something different, as the upload is not performed with CURL.

 

Why do something separate, and not create the file locally and upload it with the rest?

 

 

Please read again my explanation, if you do it that way you cannot rely on the LastUpdate.log file, it will not cover a problem with upload of bigger files.


NO SUPPORT by PM, it is a forum make your question public so everybody can benefit from the question/answer.

If you think that my answer helps you, you can press the up arrow in bottom right the answer.

 

Octagon SF-4008: Hotbird 13.0E, Astra 19.2E, Eutelsat5A 5.0W (+DVB-C)
VU+ Solo 4K: 2*DVB-S2 + 2*DVB-C/T/T2 (used in DVB-C)
Zgemma H5: 1*DVB-S2 + 1*DVB-C

Zgemma H7C: 1*DVB-S2 + 2*DVB-C


Re: One download site off-line / Downloads affected #19 Pr2

  • PLi® Test Team
  • 2,540 posts

+98
Good

Posted 11 August 2018 - 14:21

 

Your approach will poll every possible severs by every clients in the world so we will lose the random load-balancer advantage, there is no need for this when there is no problem. While my approach will not.

I will check the LastUpdate.log only on the already selected random mirror and then only switch to another one when needed.

 

The problem with this is that you don't know the status of a server until you can compare it to the others. If you get "2018-08-10" back, does that mean you discard it (because it is not today)? Or does that mean there was no update today and that server does contain the latest version? Or perhaps that your box is in a timezone so for away from Willy's timezone that there is a day difference?

 

You can also not compare it to something you have locally. If your box has been in deep standby for a week, the fact that a servers reports a date more recent than your last download doesn't say anything about how current the data on the server is.

 

Polling the servers to check the status isn't a problem, it is not the number of hits, it is the volume per month that is a challenge for those servers.

 

 

1) I say that we must still consider the server if it is in a 2 days difference slot between the local STB date and the LastUpdate.log, this will cover the world timezone. Since in the world we only have 1 day difference, that's where the 2 days came from.

2) If Willy upload has a problem to upload to every server that last 3 days, then your LastUpdate.log on ALL the servers will be the same (exemple 2018-08-07) and then which conclusion can you take today we are the 2018-08-11, if you didn't check the local date of the STB then you will download and process an obsolete data file, so same as today.

3) Timezone difference is always between +12H -12H from you position on earth, so always in MAX 1 day difference  (the earth turn on itself in 24H). So 2 days to compare date is nice. 

4) I never take into consideration the last download date in the brainstorm, this information is useless

5) Using a STB which is not at the right date and time to download EPG is totally useless... since you will upload in memory EPG datas that are set for the right date and time for the program. If your STB is still in the past you will NEVER saw the EPG informations. So to use EPG your STB has to be at the right date and time.

(Remark: that's why OpenPLi should implement by default in its image the SystemTime plugin setup to use NTP server instead of TP time; so when starting a STB even without antenna connected the STB will always be at the right time). And if people use the EPGimport the box is connected to the Internet, people that are not connected to the Internet then use TP time and don't use EPGimport.

 

 

So you need to compare with the local STB date with the LastUpdate.log and not trust the server date in this file since you can't conclude nothing from it about the current real life date. Another argument is that you cannot unscramble channel when the date and time is not correct since it is also timebased. So you really can assume that the STB is at the right time.

 

Pr2


NO SUPPORT by PM, it is a forum make your question public so everybody can benefit from the question/answer.

If you think that my answer helps you, you can press the up arrow in bottom right the answer.

 

Octagon SF-4008: Hotbird 13.0E, Astra 19.2E, Eutelsat5A 5.0W (+DVB-C)
VU+ Solo 4K: 2*DVB-S2 + 2*DVB-C/T/T2 (used in DVB-C)
Zgemma H5: 1*DVB-S2 + 1*DVB-C

Zgemma H7C: 1*DVB-S2 + 2*DVB-C


Re: One download site off-line / Downloads affected #20 WanWizard

  • Forum Moderator
    PLi® Core member
  • 40,802 posts

+641
Excellent

Posted 11 August 2018 - 14:38

  1. Ok.
  2. That means that it should have a fallback in case all servers are rejected because of the +2 days rule.
  3. Ok.
  4. Indeed.
  5. Correct for the first part, the rest is not relevant to this discussion. ;)

If Willy can make sure all servers will have this file, I can see what I can do with the code. Will take some time, I'm travelling the next few weeks...


Many answers to your question can be found in our new and improved wiki.

Currently in active use: HD2400 (4xS2)VU+Duo2 (3xS2)VU+Zero, Edision OS mini+, ET10000 (4xS2)

For testing purposes: XP1000Formuler F1 (2xS2)Miraclebox Premium Micro (S2+C/T),  ET7500 (S2)ET8500 (S2), Zgemma H2.H (S2+C)Zgemma H5.2TC (S2+C/T), SAB TripleAlpha (S2+C/T), Galaxy 4K (FBC), VU Zero 4K, Edision OS nino 





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users