41 replies to this topic

[WanWizard]

What is the exact update needed?

[wortellica]

Without removing something, after sofware update epg work,s on all 3 boxes ?

[doglover]

I guess an update of the sources.  In the sources file remove the mirror in question.

Have not done this, and hesitate to do it, since the website would probably be online again after next weekend.  (I am in contact with the webmaster about this)

 

Another fix that should be applied is cater for this situation in the XMLTVimport plug-in.  When an unzipped file is received, the file should be skipped an another mirror tried.

Only I do not have a clue on how todo this.  If somebody with some python knowledge could have look to the plug-in and see if it this can be achieved.

 

The Github is here:  https://github.com/O...ons-xmltvimport

 

Willy

[doglover]

Without removing something, after sofware update epg work,s on all 3 boxes ?

 

 

This is just luck.  The plug-in has not tried to download from the mirror in question.

There are 5 mirrors and the plug-in just selects one at random.

 

Willy

[wortellica]

Ah ok, thank you for the explanation!

[Pr2]

Just my 2 cents when such problem occurs and the solution is to change the source file, I think it must be changed in the source package so we say end-user just to online update there box. This is typically what a packaging system is for.

 

We cannot ask every single user to edit a file and then when the site is back online nobody will put it back in, that's for sure.

 

It this is done centrally this will fix the issue once for all for everybody and you keep control on what every user use has source. And for people that update very often this problem will even not be mentioned at all.

[Q-BiC]

I guess an update of the sources.  In the sources file remove the mirror in question.

Have not done this, and hesitate to do it, since the website would probably be online again after next weekend.  (I am in contact with the webmaster about this)

 

Another fix that should be applied is cater for this situation in the XMLTVimport plug-in.  When an unzipped file is received, the file should be skipped an another mirror tried.

Only I do not have a clue on how todo this.  If somebody with some python knowledge could have look to the plug-in and see if it this can be achieved.

 

The Github is here:  https://github.com/O...ons-xmltvimport

 

Willy

 

 

Just a wild guess (very little Python knowledge and the code isn't really well documented either):

EPGImport.py, line 192, afterChannelDownload

After checking for file size, also check if the contents is actually xml. On failure, it'll go to channelDownloadFail which looks like it's trying another mirror.

(EDIT: there will need to be an appropriate check for epg.dat files too of course)

Edited by Q-BiC, 8 March 2017 - 17:17.

[WanWizard]

The correct fix is indeed to make sure it tries another mirror. We'll have to look into this.

[doglover]

Just my 2 cents when such problem occurs and the solution is to change the source file, I think it must be changed in the source package so we say end-user just to online update there box. This is typically what a packaging system is for.

 

We cannot ask every single user to edit a file and then when the site is back online nobody will put it back in, that's for sure.

 

It this is done centrally this will fix the issue once for all for everybody and you keep control on what every user use has source. And for people that update very often this problem will even not be mentioned at all.

 

Yes, you are absolutely correct.  However, changing the sources or programming a fix takes time.

So the immediate and work around solution was to edit the sources file. 

 

Willy

[doglover]

 

I guess an update of the sources.  In the sources file remove the mirror in question.

Have not done this, and hesitate to do it, since the website would probably be online again after next weekend.  (I am in contact with the webmaster about this)

 

Another fix that should be applied is cater for this situation in the XMLTVimport plug-in.  When an unzipped file is received, the file should be skipped an another mirror tried.

Only I do not have a clue on how todo this.  If somebody with some python knowledge could have look to the plug-in and see if it this can be achieved.

 

The Github is here:  https://github.com/O...ons-xmltvimport

 

Willy

 

 

Just a wild guess (very little Python knowledge and the code isn't really well documented either):

EPGImport.py, line 192, afterChannelDownload

After checking for file size, also check if the contents is actually xml. On failure, it'll go to channelDownloadFail which looks like it's trying another mirror.

(EDIT: there will need to be an appropriate check for epg.dat files too of course)

 

 

Is there not a check that can be applied to see if you get an gzipped file as a return?

All files on the mirrors are gzipped

 

One problem:  

At the moment the plug-in can import at the moment also xml files (or unzipped files)  And for testing I would like this to remain.

This can be overcome by skipping the gzipped check when the URL does not contain the .gz extension, or the file is a local file (i.e. not downloaded from the net)

 

Willy

Edited by doglover, 8 March 2017 - 19:02.

[Pr2]

Hi,

 

Looking at source code, when gzip operators are called error are not handled.

So I think that surrounding the gzip with a try:  except  should do the trick.

 

Pr2

[WanWizard]

You mean something like:

if filename.endswith('.gz'):
	try:
		self.fd = gzip.open(filename, 'rb')
	except Exception, e:
		self.downloadFail(e)

assuming that downloadFail will trigger the download of the next in line...

[Pr2]

Hi WanWizard,

 

Yes indeed that's exactly what I was thinking of.

So corrupted .gz file will trigger a new download from another mirror if it is the goal of self.downloadFail(e)

 

Pr2

 

@ALL,

 

An easiest work-around for this problem, is to edit the file:   /etc/hosts and add this line at the end:

 

127.0.0.1       www.xmltvepg.nl

 

Or for people that are not familiar with file editing in telnet you can perform once:

 

cat "127.0.0.1       www.xmltvepg.nl" >> /etc/hosts

 

Pr2

[WanWizard]

I'll see if I can test the changes.

[Q-BiC]

I'm having the same issue. But not only for UK. Benelux is also empty. 0 events.

The EPG has been acting up for a few days now (UK sometimes missing, and Benelux showing wrong events), but when I got home this evening, the whole EPG was empty and manually rescanning doesn't help.

 

Don't know if it's related to the mirror issues, but I'm seeing the same issues of wrong events on Belgian channels again. For example the Epg I'm seeing for the channel VIER for Sunday evening, is actually for Monday evening... (for example, the program De Mol at 8:35PM is on Mondays, but the EPG shows it on Sunday.)

[WanWizard]

Turned out not to be that easy, as the downloaded file is not processed in that script. But I found a solution, should be in tomorrows image.

 

In testing here, it now correctly iterates over the defined URL's until either the list is exhausted, or a valid file is downloaded.

[Pr2]

Great to heard that you find a solution.  I have tested on my side too but the first doesn't work since gzip.open unfortunately doesn't check if it is a valid gz file so since a wrong gz file is downloaded but that the file exist, it always work and so doesn't raise any exception. 

[WanWizard]

Yeah, as I said it turned out to be more complicated.

 

That code only opens the file, you actually don't get the error until you start reading from it, which happens two classes down, in gen_xmltv.py . So I added a dummy read to this code for force the error before trying to parse the xml...

 

See https://github.com/O...c6e732587be8946

[Pr2]

Hi,

 

Thanks but your change is incomplete because the gzip.open occurs twice in the source code, once for the XMLTV and once for the EPG.DAT so you need to use the same trick twice in the EPGImport.py

 

See here:

 

https://github.com/O...GImport.py#L143

 

Regards,

 

Pr2

[WanWizard]

True, didn't think about epg.dat, never use it. I'll modify that too.

 

edit: it isn't that simple, the actual opening and processing happens in a second thread, so that needs to be able to single back that an iteration is needed. Not sure my basic Python is up to the task here...