110 replies to this topic

[betacentauri]

I have added a debug message here:
https://github.com/O...cache.cpp#L2908

I can see this debug message in the log.

Then e2 swtiches back to here: https://github.com/O...cache.cpp#L3023 (eEPGCache::importEvents). I think numberOfEvents is 1 (I cannot check that, because as said with a single debug message the problem disappears). So it only executes: "Py_END_ALLOW_THREADS;" and then goes back to Python code. But a debug message in python code is not shown.

 

Ok, I try printf.

[betacentauri]

Ok, printf helps. A printf at the end of the procedure importEvents is shown in the log.

A log in python here https://github.com/O...GImport.py#L228 so right after the importEvents call is not shown. So crash seems to happen in swig code. I have no idea how to debug that.

 

I also added my code to show backtraces. The backtrace shows:

<  9273.324> Backtrace:
<  9273.324> /tmp/enigma2(_Z17handleFatalSignaliP9siginfo_tPv) [0x694A8]
<  9273.324> /lib/libc.so.6(__default_rt_sa_restorer) [0xB652D6C0]

Anyone have an idea how to continue?

Could it be a problem with the Py_BEGIN_ALLOW_THREADS ?

[Eragon]

Excuse in advance my possibly silly observations...

 

but if the import problem is only occurring on the Solo4K ARM based receiver, and the XMLTVimport plugin source is the same for the MIPSEL based receivers, this issue might reside in the ARM source compiler or in something related in some way to the ARM system binaries...

Edited by Eragon, 9 September 2016 - 23:39.

[littlesat]

Does the hd51 also has this issue? When not there must be something in the vu's bps or drivers...

Edited by littlesat, 10 September 2016 - 07:02.

[doglover]

Does the hd51 also has this issue? When not there must be something in the vu's bps or drivers...

 

Good question:  Is it ARM related or VU+ related?

 

Problem with the Germany3 test file, I earlier created is the fact that the entry which causes the crash is now in the past, and wont probably any-more be imported.

Therefor I created a new testfile, from which I am sure it generates the fault.  I splattered a few entries around it from which I know they are too long and create the crash.

Also the file is 14 days long with a faulty entry at the end, so it can be used for some time.

 

In order to use this testfile, add the following source to the rytec,sources.xml file:

<source type="gen_xmltv" channels="http://testepg.better-than.tv/rytec.channels.xml.gz">
<description>Test Crash</description>
<url>http://epgalfasite.dyndns.tv/testwebgrab.xml.gz</url>
</source>

Willy

Edited by doglover, 10 September 2016 - 08:26.

[Erik Slagter]

IMS has done some work on the EPG import (generic) lately. His patches aren't yet applied, so he is not to blame

 

But from the patches, you can see a description can be 4k max. It's implemented using an ugly static char array of 4k. Which could work if the boundaries were always respected. But in practise it's just too easy to forget it at just one point.

 

Of course, a std::string should have been used and there would have been no problem at all. One of the IMS patches exactly does that. So if someone applies his patches, it may be enough (probably me ).

 

I have a theory about why it manifests on the solo4k and possibly the hd51 as well. I have been told the mips based SoC's have a very limited MMU. Maybe the arm based just have a more advanced MMU that can assign memory in a more fine-grained way.

[doglover]

4k max.

 

OK.  The crash occurred on a long description 3852 chars long.  Together with some formatting chars, this could just put it over the limit.

That at least gives me a guideline. 

Any grabber I have to revise will have this limit build in the future.  (I do not like to add it rucksichloss to every grabber without testing each one)

 

BTW:  A description this long is ridiculous, and nobody will ever read it.

 

Willy

[Erik Slagter]

Of course it will need to be fixed in enigma, anyway.

[betacentauri]

Char array means bytes and if you use UTF8 you can have more than 1 byte for each character. So this really might be the reason for the crash.
I'll try IMS patch later (perhaps tomorrow).

[Erik Slagter]

No the char array is used as storage for bytes, not for chars, if you know what I mean. If that was the problem, transition to a std::string wouldn't help either

[littlesat]

4k for one desciption.... Sounds like overkill... When you read a lot of epg your ram fills up dramatically......

[Eragon]

BTW in OpenVix 4.2 the EPGimport plugin doesn't give the error importing the same XML source, so wouldn't it be useful to check which are the  differences between XMLTVimport and EPGimport source codes?

[Dimitrij]

use swap

[doglover]

4k for one desciption.... Sounds like overkill... When you read a lot of epg your ram fills up dramatically......

 

Completely agree.  I did not intend it, it to be so much  (was not even aware of it).  But some websites give that much text for some shows. 

Teaches us to be more careful.

 

It still needs fixing though in EPGimport and/or enigma2.  No enlarging is needed nor wanted.  Just do not crash onto it.

I will build a upperlimit in the XMLTV generating software, but it will not be overnight.  As I stated above, every grabber I have to revise - or do some tests with it - will have the restriction build into it.

 

Willy

Edited by doglover, 10 September 2016 - 09:45.

[Rob van der Does]

use swap

For the Solo 4k???

[littlesat]

Time to move it to a compressed database strategy...
But still the > 4K bug on solo4k (arm?) should be solved...

[Dimitrij]

 

use swap

For the Solo 4k???

 

This simply test.

No answer?

Edited by Dimitrij, 10 September 2016 - 17:38.

[Rob van der Does]

Sorry, no 4k here.

[Pr2]

@Dimitrij,

 

There is plenty of memory available on the Solo4K when the crash occurred.  So swap won't fix anything since RAM remains available. Moreover if it was a RAM issue it will not be a specific long description that will make crash the box but the loading of the whole data set.

 

The problem is now clearly identified and easily reproducible, so please stop asking to create swap there is no need for swap on Solo4K.

 

Doglover can now create a file with only one event and the Solo4K will crash, so what swap will does more?

 

OpenPLi devs should have on there side a HD51 and try to reproduce the bug on it, it seems that no end-user with HD51 are following this thread so we still don't know if HD51 has the same issue or not.

 

If mut@nt want to offered me a HD51 I will be pleased to try to reproduce the bug myself. 

 

Pr2

Edited by Pr2, 10 September 2016 - 18:08.

[betacentauri]

I have tested with old Germany3 test file and it still crashes.

 

And unfortunately both patches from IMS (he has 2 different versions), which replace the char array with a string, don't fix the problem. Box still crashes