203 replies to this topic

[pieterg]

Can anyone tell me if ENABLE_PRIVATE_EPG is on by default in the openpli image?

yes, it is defined in epgcache.h
In fact we enabled all epg sources.

I think instead of using the fix above we only need to modify the eventData constructor

And then generate new utf-8 encoded EIT sections?
I think it might be better to first rewrite every location where users rely on getting EIT sections from the epgcache.
Then, we are free to store the data in any which format we want.

[awx]

Can anyone tell me if ENABLE_PRIVATE_EPG is on by default in the openpli image?

yes, it is defined in epgcache.h
In fact we enabled all epg sources.

I think instead of using the fix above we only need to modify the eventData constructor

And then generate new utf-8 encoded EIT sections?
I think it might be better to first rewrite every location where users rely on getting EIT sections from the epgcache.
Then, we are free to store the data in any which format we want.

Rewriting every place seems like a big task, is anyone currently working on that?
So I think my original patch to fix searching should be included for the timing being because who know how long rewriting all of that will take.

[pieterg]

nobody is working on that.
I think there is only a small number of 'external' epgcache users, relying on the EIT format.
The only one I can think of is recording (store .eit file) / playback (inject .eit data into now/next parser)

[awx]

nobody is working on that.
I think there is only a small number of 'external' epgcache users, relying on the EIT format.
The only one I can think of is recording (store .eit file) / playback (inject .eit data into now/next parser)

I think my proposed change for modifing eventData would not hurt anyone relying on the EIT format.
Since I would have the eventData constructor, rebuild the short event descriptor with the converted UTF8 string before storing it.
This way we could fix the text globally with out requiring a much larger change.

This then makes text work properly and we have time to rewrite epgcache properly.

[pieterg]

I think my proposed change for modifing eventData would not hurt anyone relying on the EIT format.
Since I would have the eventData constructor, rebuild the short event descriptor with the converted UTF8 string before storing it.

Yes, indeed that's why I was asking whether you would regenerate the EIT section.
I did not get that 100% from your initial proposal.

But yes, this method should be fine, external epgcache users will be able to use raw EIT if they wish.

[awx]

I think my proposed change for modifing eventData would not hurt anyone relying on the EIT format.
Since I would have the eventData constructor, rebuild the short event descriptor with the converted UTF8 string before storing it.

Yes, indeed that's why I was asking whether you would regenerate the EIT section.
I did not get that 100% from your initial proposal.

But yes, this method should be fine, external epgcache users will be able to use raw EIT if they wish.

Ah, sorry about that.
Thats good news then I will try to make a change like that for the time being.
Thanks for the help.

[awx]

I think my proposed change for modifing eventData would not hurt anyone relying on the EIT format.
Since I would have the eventData constructor, rebuild the short event descriptor with the converted UTF8 string before storing it.

Yes, indeed that's why I was asking whether you would regenerate the EIT section.
I did not get that 100% from your initial proposal.

But yes, this method should be fine, external epgcache users will be able to use raw EIT if they wish.

Here is the final patch, this will fix title searching and in turn any other place that we try to do a text conversion on the short event descriptor without the tsidonid.

Tested by doing the following
init 4
rm /media/hdd/epg.dat
enigma2
wait until some data comes in
check searching on special chars
shut down enigma2
start enigma2
check searching again ( sanity check, since we didn't change epg.dat storage but it shouldnt be broken)

All tests passed for me.

Attached Files

•  patch.tar.gz   2.24KB   8 downloads

[awx]

I think my proposed change for modifing eventData would not hurt anyone relying on the EIT format.
Since I would have the eventData constructor, rebuild the short event descriptor with the converted UTF8 string before storing it.

Yes, indeed that's why I was asking whether you would regenerate the EIT section.
I did not get that 100% from your initial proposal.

But yes, this method should be fine, external epgcache users will be able to use raw EIT if they wish.

Here is the final patch, this will fix title searching and in turn any other place that we try to do a text conversion on the short event descriptor without the tsidonid.

Tested by doing the following
init 4
rm /media/hdd/epg.dat
enigma2
wait until some data comes in
check searching on special chars
shut down enigma2
start enigma2
check searching again ( sanity check, since we didn't change epg.dat storage but it shouldnt be broken)

All tests passed for me.

Updating the previous patch, the last patch was fine, but incremented the pointer incorrectly which broke other descriptors.
This patch is updated to fix that, so everything should work now.

Attached Files

•  patch.tar.gz   2.24KB   10 downloads

[pieterg]

thanks, looking fine at first glance (not tested). I'll commit it, and we'll see what happens

[awx]

thanks, looking fine at first glance (not tested). I'll commit it, and we'll see what happens

You're welcome
Just a heads up, the change wont break anything if you leave your old epg.dat file in place, but if you do that title searching will not work correctly as the old titles in the cache file have not been converted.

[Robinson]

I can confirm that if I use Polish special characters, the search returns 0-results or rather refuses to do the search.
Does this patch correct EPG search in WebInterface?

[awx]

I can confirm that if I use Polish special characters, the search returns 0-results or rather refuses to do the search.
Does this patch correct EPG search in WebInterface?

This patch, patches the reading in of the eventData.
It takes the data and converts the data to utf8 before storing it, so functions that read the epg data out still need to use convertDVBUTF8 but the data is then converted properly.
I am not sure how the WebInterface does its search, but I would think its using one of the standard calls to epgcache, in which case it will fix that.

However I have seen some plugins decoding utf-8 and re-encoding to an iso standard, which will cause the search to not work properly.
All searches should be done in utf-8 with utf-8 encoded strings.

[awx]

I can confirm that if I use Polish special characters, the search returns 0-results or rather refuses to do the search.
Does this patch correct EPG search in WebInterface?

This patch, patches the reading in of the eventData.
It takes the data and converts the data to utf8 before storing it, so functions that read the epg data out still need to use convertDVBUTF8 but the data is then converted properly.
I am not sure how the WebInterface does its search, but I would think its using one of the standard calls to epgcache, in which case it will fix that.

However I have seen some plugins decoding utf-8 and re-encoding to an iso standard, which will cause the search to not work properly.
All searches should be done in utf-8 with utf-8 encoded strings.

Additional note, plugins were decoding and encoding to other character maps to work around the text searching issue, which infact still had the possiblity of not returning all results because of the nature of the bug.

[awx]

There seems to be a memory leak, I will hopefully fix that later today.

[awx]

There seems to be a memory leak, I will hopefully fix that later today.

Maybe spoke too soon, I could be wrong about this and there may be no memory leak, at least not in this area.

[awx]

Could you include these patches into the repo

I have refactored the code to use one constructor and prevent code duplication, also
I believe its possible to have bad descriptors coming in and have add a few sanity checks there before we load the data.

Attached Files

•  patches.tar.gz   1.94KB   6 downloads

[pieterg]

Did you have a look at this report?
http://openpli.org/f...__fromsearch__1

Any idea what might have broken multi-language epg?

[awx]

Did you have a look at this report?
http://openpli.org/f...__fromsearch__1

Any idea what might have broken multi-language epg?

I will take a look and see if I can figure out what is going on.
All though I am a little perplexed by the issue at the moment.

[ims]

Could be there problem with len function and then can begin problems ?

missing event title: Poločas štěstí, Pro pamětníky...

eDebug(">>> eventNameUTF8: %s<<< %d ", eventNameUTF8.c_str(), eventNameLen );

>>> eventNameUTF8: Poločas štěstí, Pro pamětníky...<<< 38

but when i count manually all chars, size is 36 only

[awx]

Could be there problem with len function and then can begin problems ?

missing event title: Poločas štěstí, Pro pamětníky...

eDebug(">>> eventNameUTF8: %s<<< %d ", eventNameUTF8.c_str(), eventNameLen );

>>> eventNameUTF8: Poločas štěstí, Pro pamětníky...<<< 38

but when i count manually all chars, size is 36 only

You use the wrong length in your example, eventNameLen, is the original length of the title, eventNameUTF8len is what you want