Jump to content


Photo

eepgcache and title searching


  • Please log in to reply
203 replies to this topic

Re: eepgcache and title searching #21 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 12 January 2012 - 10:40

Can anyone tell me if ENABLE_PRIVATE_EPG is on by default in the openpli image?


yes, it is defined in epgcache.h
In fact we enabled all epg sources.

I think instead of using the fix above we only need to modify the eventData constructor


And then generate new utf-8 encoded EIT sections?
I think it might be better to first rewrite every location where users rely on getting EIT sections from the epgcache.
Then, we are free to store the data in any which format we want.

Re: eepgcache and title searching #22 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 12 January 2012 - 10:44


Can anyone tell me if ENABLE_PRIVATE_EPG is on by default in the openpli image?


yes, it is defined in epgcache.h
In fact we enabled all epg sources.

I think instead of using the fix above we only need to modify the eventData constructor


And then generate new utf-8 encoded EIT sections?
I think it might be better to first rewrite every location where users rely on getting EIT sections from the epgcache.
Then, we are free to store the data in any which format we want.

Rewriting every place seems like a big task, is anyone currently working on that?
So I think my original patch to fix searching should be included for the timing being because who know how long rewriting all of that will take.

Re: eepgcache and title searching #23 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 12 January 2012 - 12:04

nobody is working on that.
I think there is only a small number of 'external' epgcache users, relying on the EIT format.
The only one I can think of is recording (store .eit file) / playback (inject .eit data into now/next parser)

Re: eepgcache and title searching #24 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 12 January 2012 - 12:10

nobody is working on that.
I think there is only a small number of 'external' epgcache users, relying on the EIT format.
The only one I can think of is recording (store .eit file) / playback (inject .eit data into now/next parser)

I think my proposed change for modifing eventData would not hurt anyone relying on the EIT format.
Since I would have the eventData constructor, rebuild the short event descriptor with the converted UTF8 string before storing it.
This way we could fix the text globally with out requiring a much larger change.

This then makes text work properly and we have time to rewrite epgcache properly.

Re: eepgcache and title searching #25 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 12 January 2012 - 12:42

I think my proposed change for modifing eventData would not hurt anyone relying on the EIT format.
Since I would have the eventData constructor, rebuild the short event descriptor with the converted UTF8 string before storing it.


Yes, indeed that's why I was asking whether you would regenerate the EIT section.
I did not get that 100% from your initial proposal.

But yes, this method should be fine, external epgcache users will be able to use raw EIT if they wish.

Re: eepgcache and title searching #26 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 12 January 2012 - 12:47


I think my proposed change for modifing eventData would not hurt anyone relying on the EIT format.
Since I would have the eventData constructor, rebuild the short event descriptor with the converted UTF8 string before storing it.


Yes, indeed that's why I was asking whether you would regenerate the EIT section.
I did not get that 100% from your initial proposal.

But yes, this method should be fine, external epgcache users will be able to use raw EIT if they wish.

Ah, sorry about that.
Thats good news then I will try to make a change like that for the time being.
Thanks for the help.

Re: eepgcache and title searching #27 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 15 January 2012 - 12:02


I think my proposed change for modifing eventData would not hurt anyone relying on the EIT format.
Since I would have the eventData constructor, rebuild the short event descriptor with the converted UTF8 string before storing it.


Yes, indeed that's why I was asking whether you would regenerate the EIT section.
I did not get that 100% from your initial proposal.

But yes, this method should be fine, external epgcache users will be able to use raw EIT if they wish.

Here is the final patch, this will fix title searching and in turn any other place that we try to do a text conversion on the short event descriptor without the tsidonid.

Tested by doing the following
init 4
rm /media/hdd/epg.dat
enigma2
wait until some data comes in
check searching on special chars
shut down enigma2
start enigma2
check searching again ( sanity check, since we didn't change epg.dat storage but it shouldnt be broken)

All tests passed for me.

Attached Files



Re: eepgcache and title searching #28 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 15 January 2012 - 13:53



I think my proposed change for modifing eventData would not hurt anyone relying on the EIT format.
Since I would have the eventData constructor, rebuild the short event descriptor with the converted UTF8 string before storing it.


Yes, indeed that's why I was asking whether you would regenerate the EIT section.
I did not get that 100% from your initial proposal.

But yes, this method should be fine, external epgcache users will be able to use raw EIT if they wish.

Here is the final patch, this will fix title searching and in turn any other place that we try to do a text conversion on the short event descriptor without the tsidonid.

Tested by doing the following
init 4
rm /media/hdd/epg.dat
enigma2
wait until some data comes in
check searching on special chars
shut down enigma2
start enigma2
check searching again ( sanity check, since we didn't change epg.dat storage but it shouldnt be broken)

All tests passed for me.

Updating the previous patch, the last patch was fine, but incremented the pointer incorrectly which broke other descriptors.
This patch is updated to fix that, so everything should work now.

Attached Files



Re: eepgcache and title searching #29 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 16 January 2012 - 11:12

thanks, looking fine at first glance (not tested). I'll commit it, and we'll see what happens ;)

Re: eepgcache and title searching #30 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 16 January 2012 - 11:37

thanks, looking fine at first glance (not tested). I'll commit it, and we'll see what happens ;)

You're welcome
Just a heads up, the change wont break anything if you leave your old epg.dat file in place, but if you do that title searching will not work correctly as the old titles in the cache file have not been converted.

Re: eepgcache and title searching #31 Robinson

  • Senior Member
  • 2,621 posts

+30
Good

Posted 16 January 2012 - 11:40

I can confirm that if I use Polish special characters, the search returns 0-results or rather refuses to do the search.
Does this patch correct EPG search in WebInterface?

ET9000, OpenPLi 4.0, 13E, 19E

HD51, OpenPLi 6.2, 75E - 30W


Re: eepgcache and title searching #32 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 16 January 2012 - 11:48

I can confirm that if I use Polish special characters, the search returns 0-results or rather refuses to do the search.
Does this patch correct EPG search in WebInterface?

This patch, patches the reading in of the eventData.
It takes the data and converts the data to utf8 before storing it, so functions that read the epg data out still need to use convertDVBUTF8 but the data is then converted properly.
I am not sure how the WebInterface does its search, but I would think its using one of the standard calls to epgcache, in which case it will fix that.

However I have seen some plugins decoding utf-8 and re-encoding to an iso standard, which will cause the search to not work properly.
All searches should be done in utf-8 with utf-8 encoded strings.

Re: eepgcache and title searching #33 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 16 January 2012 - 11:50


I can confirm that if I use Polish special characters, the search returns 0-results or rather refuses to do the search.
Does this patch correct EPG search in WebInterface?

This patch, patches the reading in of the eventData.
It takes the data and converts the data to utf8 before storing it, so functions that read the epg data out still need to use convertDVBUTF8 but the data is then converted properly.
I am not sure how the WebInterface does its search, but I would think its using one of the standard calls to epgcache, in which case it will fix that.

However I have seen some plugins decoding utf-8 and re-encoding to an iso standard, which will cause the search to not work properly.
All searches should be done in utf-8 with utf-8 encoded strings.

Additional note, plugins were decoding and encoding to other character maps to work around the text searching issue, which infact still had the possiblity of not returning all results because of the nature of the bug.

Re: eepgcache and title searching #34 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 17 January 2012 - 07:21

There seems to be a memory leak, I will hopefully fix that later today.

Re: eepgcache and title searching #35 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 17 January 2012 - 07:36

There seems to be a memory leak, I will hopefully fix that later today.

Maybe spoke too soon, I could be wrong about this and there may be no memory leak, at least not in this area.

Re: eepgcache and title searching #36 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 21 January 2012 - 09:21

Could you include these patches into the repo

I have refactored the code to use one constructor and prevent code duplication, also
I believe its possible to have bad descriptors coming in and have add a few sanity checks there before we load the data.

Attached Files



Re: eepgcache and title searching #37 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 21 January 2012 - 11:25

Did you have a look at this report?
http://openpli.org/f...__fromsearch__1

Any idea what might have broken multi-language epg?

Re: eepgcache and title searching #38 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 21 January 2012 - 11:37

Did you have a look at this report?
http://openpli.org/f...__fromsearch__1

Any idea what might have broken multi-language epg?

I will take a look and see if I can figure out what is going on.
All though I am a little perplexed by the issue at the moment.

Re: eepgcache and title searching #39 ims

  • PLi® Core member
  • 13,781 posts

+214
Excellent

Posted 21 January 2012 - 12:40

Could be there problem with len function and then can begin problems ?

missing event title: Poločas štěstí, Pro pamětníky...

eDebug(">>> eventNameUTF8: %s<<< %d ", eventNameUTF8.c_str(), eventNameLen );

>>> eventNameUTF8: Poločas štěstí, Pro pamětníky...<<< 38

but when i count manually all chars, size is 36 only
Kdo nic nedělá, nic nezkazí!

Re: eepgcache and title searching #40 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 21 January 2012 - 12:45

Could be there problem with len function and then can begin problems ?

missing event title: Poločas štěstí, Pro pamětníky...

eDebug(">>> eventNameUTF8: %s<<< %d ", eventNameUTF8.c_str(), eventNameLen );

>>> eventNameUTF8: Poločas štěstí, Pro pamětníky...<<< 38

but when i count manually all chars, size is 36 only

You use the wrong length in your example, eventNameLen, is the original length of the title, eventNameUTF8len is what you want


1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users