110 replies to this topic

[Eragon]

Unfortunately yesterday night there were the FIFA WorldCup qualifier matches on TV and I couldn't test the import due to my family seizing the Solo4K....

And as my Solo4K is currently running OpenVix 4.2 cause of the EPG problem, for testing I need to reflash OpenPLI and find some spare time (mine and Solo4k's!)

Will try as soon as possible.

 

[Dimitrij]

Problem only solo4k?

Use swap for test.

[littlesat]

Does someone has any idea since when this error did occur. Was it always on PLi's Solo4K image?

[Erik Slagter]

There has been some "work" on multi-byte encodings lately. I wouldn't be surprised that either the xml input contains invalid multi-byte sequences or there is a bug in the multi-byte processing or both. I am seeing invalid UTF-8 sequences in EPG quite frequently, so that is surely a part of the problem.

[doglover]

There has been some "work" on multi-byte encodings lately. I wouldn't be surprised that either the xml input contains invalid multi-byte sequences or there is a bug in the multi-byte processing or both. I am seeing invalid UTF-8 sequences in EPG quite frequently, so that is surely a part of the problem.

 

 

Problem is, the EPG is gathered from many different sources, and not everybody adheres to the same standard.

We can correct most errors (if not all).  But for this to happens, we will need at least a channel which has the problem, in order to locate the website (and grabber) which needs adjusting.

At the moment, I cannot identify even the channel which causes the problem.  So let alone correct the grabber for it.  And when not knowing where you are looking for, making changes is a gamble.

Moreover, the mipsel variants of enigma2 does not have a problem with this.  And since I have only mipsel receivers, I can not even test this.

 

We will try always to get it correct, but it is not easy if we do not even know were to look.

 

So Eric, why not report the invalid UTF-8 sequences when you see them.  (Look up the service ref of the channel, because this helps me to identify the exact channel and attached EPG)

Also make a screen shot of the error.  Which helps me finding the error.  (Trying to find a single occurrence in a 30 MB file is not easy)

 

Willy

Edited by doglover, 6 September 2016 - 10:22.

[Pr2]

Hi,

 

My Solo4K reports that the last valid import date from 1st September morning.

 

I updating on a (quite) regular basis my image, I even perform a fresh install (USB flash) to be sure not to have some kind of corruption.

 

Pr2

[Erik Slagter]

Willy, your fetching script should do some input validity checking. If the source encoding is supposed to be ASCII, it can't have any characters < 32 (except linefeed) and any characters > 127. If the source encoding is UTF-8, it should adhere to UTF-8 multibyte sequence requirements (see UTF-8 wikipedia page, for instance, all bytes in a multibyte sequence should have bit 7 set, some bits can only be set in the sequence start byte and some bits can only be set in the following bytes). This could spot sources that advert their encoding to be UTF-8, but having their actual encoding windows-1252 (which is quite common, unfortunately), because the people who make it, don't have a clue of encodings.

[doglover]

I am doing this now several years.  And now just suddenly this should be a problem.  It never was in the past.

Sometimes strangely formatted chars appeared yes.  But these never caused a crash.

 

Now they do.  Sorry.... Then I should be to blame for this.  A bit short reasoning don't you think

 

Willy

Edited by doglover, 6 September 2016 - 12:28.

[Pr2]

Hi,

 

Please don't blame anybody we are facing a problem that can highlight a problem that we have in code for years and only cause crash now on Solo4K (would be interesting to have OpenPLi compiled for another ARM based boxes and see if we have the same crash). Or it is the VU+ Solo4K driver that have problems.  The crash is so "hard" that we saw nothing in the log nor in dmesg.

 

We just miss some debugging to find exactly where the problem is both in the EPG plugin and in OpenPLi I think.  

 

You all did a great job and I try on my side to do my best to help you troubleshooting this problem.  I can understand how hard it is for Doglover to find where the problem is since he cannot reproduce the problem himself and I can also understand how hard it is to properly grab website to get the EPG (I also do some scripting and discovered that some pages mixed up different charset and don't respect anything).  When there is a problem with some charset Doglover always fix them very quickly.  

 

Pr2

[Erik Slagter]

Willy, I am not blaming you. It's the source you use that mess up the encodings. And for that, it would preferrable to at least be able to detect when it's wrong, before it messes up EPG on our receivers, be it a crash or simply garbage characters.

[doglover]

I can try something like this:

iconv -f utf-8 -t utf-8 -c file.txt

will clean up your UTF-8 file, skipping all the invalid characters.

-f is the source format
-t the target format
-c skips any invalid sequence

Will start experimenting with this.

 

Willy

[Pr2]

Hi,

 

I have done some more testing.

 

I have disabled EIT EPG and EIT now/next.

I have play with delete current EPG  yes/no

 

It always crash, but the events number is different on each try for the same file (BeNeLux).

 

Finally I load the epg.dat files (BeNeLux and France) and there it works.

 

So is it the same API that is called when loading .XML file into memory and epg.dat file from XMLTV import? 

 

Small tip for people facing the same problem and willing to load the epg.dat file instead of the XML one, you need to set:  delete current EPG to NO when loading several epg.dat otherwise it is only the latest loaded epg.dat that remains in memory.  I will create a dedicated post with this improvement suggestion.

 

Pr2

[doglover]

The UK, French, Benelux, Italian and German files are pushed though iconv now.

This will eliminate any non utf-8 characters which might reside in the file.

 

This will not mean that unicode chars will not occur anymore.  It only guaranties that the files will be conform to the utf-8 codepage.  Non-compliance chars, which could be potentially problematic, are eliminated.

 

Willy

[Pr2]

Hi,

 

If it is already online for Benelux then this was not the problem. 

i just import Benelux XMLTV and Solo4K still crash.

 

Pr2

[doglover]

I do not think you got the new files yet.

The iconv tretament was put in place this morning and the resultant files put in the upload queue.  Just checked and the were uploaded with the automatic uploaded at 12h10.

 

I would be surprised if this is the problem.  But you never know.

 

Willy

[Erik Slagter]

I can try something like this:

iconv -f utf-8 -t utf-8 -c file.txt

will clean up your UTF-8 file, skipping all the invalid characters.

-f is the source format
-t the target format
-c skips any invalid sequence

Will start experimenting with this.

 

Willy

If the source is actually UTF-8, yes I'd really appreciate that. If you know the source is not really UTF-8 but something else (the dreaded windows-1252 for example), then it won't help much, the garbage will be removed and not replaced by the proper code points.

[doglover]

The files are really utf-8.  I checked that over and over and are very conscientious to define it as utf-8 when writing a new grabber  (due to many problems in the past)

But this does not mean that from some websites garbage could be collected.  The windows-1252 code page is famous for that.  Websites exist which have a mixture of code-pages used onto on one webpage.

 

Also some websites even have their index pages in utf-8, But the detail pages are suddenly in windows-1252.  I know to be aware for this.  But it could result in garbage collection.

But most of the time this will show up very rapidly.  It is not one or 2 errors.  But I get very strange sentences.

 

Willy

Edited by doglover, 7 September 2016 - 15:35.

[Pr2]

Online OpenPLi update, restart the box and today import at 14H30 also crash the box, just to let you know.

Edited by Pr2, 8 September 2016 - 13:43.

[littlesat]

Until we cannot find the cause, it cannot be solved, so the crash might continue....

 

Again... Is this an "all time" issue -or- did it appear after a specific date?

[doglover]

From what I can compile from the previous posts the last succesfull import has been on sept 1, 2016.

But I do not know from when the update before this date was.  (Not my receiver)

 

 

Willy