104 replies to this topic

[littlesat]

And in addition we need a good clear algorythm and to remove all these.... This is not easy... Clear overlaps with equal start time ans lengthnare fine, but what if they do overlap and start time and duration is not equal... What should you delete?

Then I suggest it is a much better work-a-round to exclude multiple epg sources.... Or something else (work with prio's and once xml is read do not read epg from transponders for that specific channel etc....)

I have now idea about how to create a good algorythm right now to compromise with the issues here....

Edited by littlesat, 6 January 2015 - 00:09.

[Lost in Space]

Then I suggest it is a much better work-a-round to exclude multiple epg sources.... Or something else (work with prio's and once xml is read do not read epg from transponders for that specific channel etc....)

 

 

You maybe will not like my reply, but this is now exactly the way it works in DreamOS

[WanWizard]

And what does that help us?

 

If you know so well how it works there, why not donate the code to OpenPLi? Or send in a patch, OpenPLi is open source (unlike DreamOS, of which I am very sure it contains GPL code, so it violates the license).

 

If you can't be constructive, please go somewhere else. Like they say overhere "we can miss you like a toothache".

[jeanclaude]

looks to me that "newer" EPG data should always replace existing EPG data.

while reading a new EPG source, look for a break in the channel/date sequence. When this occurs, first delete all existing data for the new channel/date, and then insert the new information until the next channel/date sequence break.

I wouldn't make too much fuss about it, with individual entries and starting/ending times, too complex. Just scratch the whole day and replace with the newest data.

[doglover]

looks to me that "newer" EPG data should always replace existing EPG data.

while reading a new EPG source, look for a break in the channel/date sequence. When this occurs, first delete all existing data for the new channel/date, and then insert the new information until the next channel/date sequence break.

I wouldn't make too much fuss about it, with individual entries and starting/ending times, too complex. Just scratch the whole day and replace with the newest data.

 

I agree with this.

 

Willy

[littlesat]

looks to me that "newer" EPG data should always replace existing EPG data.

while reading a new EPG source, look for a break in the channel/date sequence. When this occurs, first delete all existing data for the new channel/date, and then insert the new information until the next channel/date sequence break.

I wouldn't make too much fuss about it, with individual entries and starting/ending times, too complex. Just scratch the whole day and replace with the newest data.\

This is not a good idea as it will not work and will deliver complains about incomplete EPGs....  This idea will have many side effects... 

 

I still think this

 

(work with prio's and once xml is read do not read epg from transponders for that specific channel etc....)

 

is the best way to go...

 

The issue is EPG is coming from the transponder and from the WWW.... so you read two lists.... so you should keep it simple and solve it at the source. When you have complete EPG via the WWW you do not need it from the transponder at all...

 

I hope the community will offer a solution....

Edited by littlesat, 6 January 2015 - 14:20.

[doglover]

This is not a good idea as it will not work and will deliver complains about incomplete EPGs....  This idea will have many side effects...

 

 

Do not agree.  One solution - work around - we employ is delete all EPG and reimport.

Using this method we do not see holes and other "side effects"

 

And for the option of detecting changes in schedule:

The grabber I am using is faced with the same problem.  I have a cache.  When grabbing again the cache is checked against the new info on the web.  When there is difference in time or title, the details are grabbed again (because this is mostly opening a new webpage).  When there is no difference the old info from the cache is retained and no new webpage is opened - saves on grabtime.

 

I get in my log then this info:

   epg correction :  CHANGED show corrected,
      show with ---- start = 9/01/2015 6:19:00 stop = 9/01/2015 6:55:00 title = Baby's Eerste Liedjes en meer
      Replaces ----- start = 9/01/2015 5:57:00 stop = 9/01/2015 6:21:00 title = Baby's Eerste Liedjes en meer
      Replaces ----- start = 9/01/2015 6:21:00 stop = 9/01/2015 6:56:00 title = Baby's Eerste Liedjes en meer
   epg correction :  CHANGED show corrected,
      show with ---- start = 9/01/2015 6:55:00 stop = 9/01/2015 7:30:00 title = Kinderen en huisdieren en meer
      Replaces ----- start = 9/01/2015 6:21:00 stop = 9/01/2015 6:56:00 title = Baby's Eerste Liedjes en meer
      Replaces ----- start = 9/01/2015 6:56:00 stop = 9/01/2015 7:30:00 title = Kinderen en huisdieren en meer
   epg correction :  CHANGED show corrected,
      show with ---- start = 9/01/2015 7:30:00 stop = 9/01/2015 8:08:00 title = Mixed up Mary
      Replaces ----- start = 9/01/2015 7:30:00 stop = 9/01/2015 8:06:00 title = Meneer Slak en meer
      Replaces ----- start = 9/01/2015 8:06:00 stop = 9/01/2015 8:39:00 title = Bim en Bam in de dierenwereld en meer
   epg correction :  CHANGED show corrected,
      show with ---- start = 9/01/2015 8:08:00 stop = 9/01/2015 8:40:00 title = Bim en Bam in de dierenwereld en meer
      Replaces ----- start = 9/01/2015 8:06:00 stop = 9/01/2015 8:39:00 title = Bim en Bam in de dierenwereld en meer
      Replaces ----- start = 9/01/2015 8:39:00 stop = 9/01/2015 9:05:00 title = Tiny Beats en meer

So it can be done.

 

Willy

[doglover]

Another example:

   epg correction :  CHANGED show corrected,
      show with ---- start = 6/01/2015 12:25:00 stop = 6/01/2015 13:15:00 title = Witness
      Replaces ----- start = 6/01/2015 12:25:00 stop = 6/01/2015 13:20:00 title = Witness
   epg correction :  CHANGED show corrected,
      show with ---- start = 6/01/2015 13:15:00 stop = 6/01/2015 14:05:00 title = Apocalypse
      Replaces ----- start = 6/01/2015 12:25:00 stop = 6/01/2015 13:20:00 title = Witness
      Replaces ----- start = 6/01/2015 13:20:00 stop = 6/01/2015 14:15:00 title = Apocalypse
   epg correction :  CHANGED show corrected,
      show with ---- start = 6/01/2015 14:05:00 stop = 6/01/2015 14:55:00 title = Ultimate Airport Dubai
      Replaces ----- start = 6/01/2015 13:20:00 stop = 6/01/2015 14:15:00 title = Apocalypse
      Replaces ----- start = 6/01/2015 14:15:00 stop = 6/01/2015 15:10:00 title = Ultimate Airport Dubai
   epg correction :  CHANGED show corrected,
      show with ---- start = 6/01/2015 14:55:00 stop = 6/01/2015 15:20:00 title = Seconds from Disaster
      Replaces ----- start = 6/01/2015 14:15:00 stop = 6/01/2015 15:10:00 title = Ultimate Airport Dubai
      Replaces ----- start = 6/01/2015 15:10:00 stop = 6/01/2015 16:05:00 title = Air Crash Investigation
   epg correction :  CHANGED show corrected,
      show with ---- start = 6/01/2015 15:20:00 stop = 6/01/2015 16:10:00 title = Air Crash Investigation
      Replaces ----- start = 6/01/2015 15:10:00 stop = 6/01/2015 16:05:00 title = Air Crash Investigation
      Replaces ----- start = 6/01/2015 16:05:00 stop = 6/01/2015 17:00:00 title = Hell on the Highway
   epg correction :  CHANGED show corrected,
      show with ---- start = 6/01/2015 16:10:00 stop = 6/01/2015 17:00:00 title = Highway Thru Hell
      Replaces ----- start = 6/01/2015 16:05:00 stop = 6/01/2015 17:00:00 title = Hell on the Highway
   epg correction :  CHANGED show corrected,
      show with ---- start = 6/01/2015 22:00:00 stop = 6/01/2015 22:55:00 title = Wicked Tuna
      Replaces ----- start = 6/01/2015 22:00:00 stop = 6/01/2015 23:00:00 title = Remote Survival
   epg correction :  CHANGED show corrected,
      show with ---- start = 6/01/2015 22:55:00 stop = 6/01/2015 23:50:00 title = Ultimate Airport Dubai
      Replaces ----- start = 6/01/2015 22:00:00 stop = 6/01/2015 23:00:00 title = Remote Survival
      Replaces ----- start = 6/01/2015 23:00:00 stop = 6/01/2015 23:55:00 title = Survive the Tribe
   epg correction :  CHANGED show corrected,
      show with ---- start = 6/01/2015 23:50:00 stop = 7/01/2015 0:40:00 title = Megafactories
      Replaces ----- start = 6/01/2015 23:00:00 stop = 6/01/2015 23:55:00 title = Survive the Tribe
      Replaces ----- start = 6/01/2015 23:55:00 stop = 7/01/2015 0:50:00 title = Megafactories
   epg correction :  CHANGED show corrected,
      show with ---- start = 7/01/2015 0:40:00 stop = 7/01/2015 1:30:00 title = Yukon Gold
      Replaces ----- start = 6/01/2015 23:55:00 stop = 7/01/2015 0:50:00 title = Megafactories
      Replaces ----- start = 7/01/2015 0:50:00 stop = 7/01/2015 1:45:00 title = Yukon Gold
   epg correction :  CHANGED show corrected,
      show with ---- start = 7/01/2015 1:30:00 stop = 7/01/2015 2:15:00 title = Wicked Tuna
      Replaces ----- start = 7/01/2015 0:50:00 stop = 7/01/2015 1:45:00 title = Yukon Gold
      Replaces ----- start = 7/01/2015 1:45:00 stop = 7/01/2015 2:35:00 title = Remote Survival
   epg correction :  CHANGED show corrected,
      show with ---- start = 7/01/2015 2:15:00 stop = 7/01/2015 3:00:00 title = Inside Area 51's Secrets
      Replaces ----- start = 7/01/2015 1:45:00 stop = 7/01/2015 2:35:00 title = Remote Survival
      Replaces ----- start = 7/01/2015 2:35:00 stop = 7/01/2015 3:20:00 title = Inside Area 51's Secrets

Willy

[jort38]

If I use info on a timer there is also some changelog.
Looks like the log Willy showed.

[Rob van der Does]

Do not agree.  One solution - work around - we employ is delete all EPG and reimport.
Using this method we do not see holes and other "side effects"

But in real life their are more sources that supply data to one box: XMLTV/Cross(for Open-TV)/EIT/poss. others; so deleting all previous data is not an option.

[MiLo]

I think the EPG cache already tracks multiple sources. It just doesn't appear to handle things correctly. It can also handle multiple languages, I don't know if it stores them all but shows only one, or just drops other languages in favor of the preferred one.

What it needs is a source prioritization, probably the same as in use for languages (you can pick a preferred language in the settings).
When overlap occurs, the cache should make a decision based on priority. It would make sense to give higher prio to the imported data.

[littlesat]

Do not agree.  One solution - work around - we employ is delete all EPG and reimport.

 

This is not the solution...  This is hiding the symptom...

 

What it needs is a source prioritization, 

And probably we only need to add a 'flag' to the EPG files which is set when the source is coming from an external source... when something new is coming in which is coming from an internal source then an overlap check can check if there already exists something from an external source in the cache and if so drop the EPG from the internal source...

Edited by littlesat, 6 January 2015 - 15:44.

[pluizebol]

If I understand things correctly then the easiest solution for those who use only XMLTV is to a daily recurring clear of the EPG.dat at a programmable time (e.g. every day at 08:30).

 

I noticed that, when I do a manual clear of EPG.dat, the XMLTV EPG is cleared and a new import is done automatically.

No more double events are showing up that way.

 

Hence my question :

How can I realize that automated daily recurring clear of the EPG.dat (e.g. every day at 08:30) ?

That would solve the problem for the XMLTV users, or am I mistaking ?

[doglover]

But in real life their are more sources that supply data to one box: XMLTV/Cross(for Open-TV)/EIT/poss. others; so deleting all previous data is not an option.

 

 

In fact XMLTV and Crossepg use the same source.  The XMLTV files.  Even Crossepg XEPGDB files are just our XMLTV files which are repacked in another format which imports quicker.

Croosepg can also read the info from OpenTV (sat).

The others sources are from the sat and providers themselves.

 

Quote

Do not agree.  One solution - work around - we employ is delete all EPG and reimport.

 

This is not the solution...  This is hiding the symptom...

 

 

Agreed it is not the solution.  As a matter of fact it is ugly.  A brute force method.  But I do not have any other solution at the moment.

But the point I am trying to make is, that when importing XMLTV (XMLTV import or Crossepg) EPG data, no problems are observed with holes in the EPG or other effects.

 

So a selective erasure of the EPG as suggested by jeanclaude should work, for at least those who use the XMLTV import or Crossepg - including OpenTV on Crossepg.

 

looks to me that "newer" EPG data should always replace existing EPG data.

while reading a new EPG source, look for a break in the channel/date sequence. When this occurs, first delete all existing data for the new channel/date, and then insert the new information until the next channel/date sequence break.

I wouldn't make too much fuss about it, with individual entries and starting/ending times, too complex. Just scratch the whole day and replace with the newest data.

 

 

Willy

Edited by doglover, 6 January 2015 - 16:07.

[littlesat]

If I understand things correctly then the easiest solution for those who use only XMLTV is to a daily recurring clear of the EPG.dat at a programmable time (e.g. every day at 08:30).

This will only work when you clear the EPG after you download the EGM from XMLTV....

[Rob van der Does]

But in real life their are more sources that supply data to one box: XMLTV/Cross(for Open-TV)/EIT/poss. others; so deleting all previous data is not an option.

 

 

In fact XMLTV and Crossepg use the same source. The XMLTV files. Even Crossepg XEPGDB files are just our XMLTV files which are repacked in another format which imports quicker.

Croosepg can also read the info from OpenTV (sat).

I was talking about using Cross for Open-TV.

[pluizebol]

 

If I understand things correctly then the easiest solution for those who use only XMLTV is to a daily recurring clear of the EPG.dat at a programmable time (e.g. every day at 08:30).

This will only work when you clear the EPG after you download the EGM from XMLTV....

 

 

Sorry for another dumb question ... but what is the EGM ?

[jeanclaude]

I don't really see why someone should have multiple EPG sources active at any time - this is surely asking for problems.

OK, the "delete & reload" method should not be used when a "live" EPG datastream is used, otherwise the box will be doing nothing else but delete & replace it's EPG contents.

but it is feasable for those of us who download the EPG data "in batch" from time to time

(as long as no better algorithm is available for updating the EPG)

 

pluizebol : the easiest solution (and dirtiest) is to use "cron". I have created a file "/etc/cron/inittab" with the following 3 lines :

45 7 * * * init 4

46 7 * * * rm /tmp/epg.dat

47 7 * * * init 3

 

this will stop enigma2 daily at 7:45 am, delete the EPG file at 7:46 am, and restart enigma2 at 7:47 am. My EPG is then downloaded daily at 8:10 am.

it's crude, but it works. only snag is that if there is no EPG in the download, then all EPG info will be empty on the screen, but this happens very rarely.

Edited by jeanclaude, 6 January 2015 - 16:50.

[doglover]

 

But in real life their are more sources that supply data to one box: XMLTV/Cross(for Open-TV)/EIT/poss. others; so deleting all previous data is not an option.

 

 

In fact XMLTV and Crossepg use the same source. The XMLTV files. Even Crossepg XEPGDB files are just our XMLTV files which are repacked in another format which imports quicker.

Croosepg can also read the info from OpenTV (sat).

I was talking about using Cross for Open-TV.

 

 

This is exactly the same problem.

 

Crossepg read the OpenTV data from the provider (sat) and stores it in its own database.

This database is then used to pump the EPG in enigma in the same way as the XMLTVimporter uses.  So the OpenTV EPG becomes actually also an "external source" for EPG.

 

Willy

[Rob van der Does]

I don't really see why someone should have multiple EPG sources active at any time - this is surely asking for problems.

There is no other way atm.
I have XMLTV for a number of packages, and Cross's Open-TV for the UK. For other services, not covered by one of those sources, I have EIT enabled. And I can't selectively disable EIT for those services covered by XMLTV.