93 replies to this topic

[doglover]

Tried it on my PC with the Offline importer:

C:\EPG\python>C:\EPG\python\OfflineImport.py  D:\EPG\python\test.sources.xml
[EPGImport] oudeis patch not detected, using epg.dat instead.
[EPGImport] failed to load C implementation, sorry
[EPGImport] nextImport, source= Test Benelux
[EPGImport] afterDownload V:\epg\BExmltv.xml.xz
Traceback (most recent call last):
  File "C:\EPG\python\OfflineImport.py", line 68, in <module>
    importFrom(epgimport, xml)
  File "C:\EPG\python\OfflineImport.py", line 54, in importFrom
    epgimport.beginImport(longDescUntil = time.time() + (5*24*3600))
  File "C:\EPG\python\EPGImport.py", line 113, in beginImport
    self.nextImport()
  File "C:\EPG\python\EPGImport.py", line 122, in nextImport
    self.fetchUrl(self.source.url)
  File "C:\EPG\python\EPGImport.py", line 128, in fetchUrl
    self.afterDownload(None, filename, deleteFile=False)
  File "C:\EPG\python\EPGImport.py", line 188, in afterDownload
    from backports import lzma
ImportError: No module named backports

Willy

[doglover]

On the receiver:  (OpenPLI 6.0 RC)

 

ImportError: No module named backports

 

This could be Chinese to me.

 

Willy

Edited by doglover, 23 July 2017 - 13:17.

[WanWizard]

It's missing the python "backports" package. It doesn't seem to be in the feeds as well. And it probably also needs a dependency to liblzma.

 

So it's not production ready yet.

[doglover]

From tomorrow on some xz compressed files will be available on one downloadsite.

 

Willy

[MiLo]

Did some testing with the information you sent me, and fixed that downloaded files wouldn't work.

Tomorrow's develop build should work...

[MiLo]

On the receiver:  (OpenPLI 6.0 RC)
ImportError: No module named backports

Yeah, it'd only work on a "Develop" build for now. It needs the "python-lzma" backport Python module which in turn requires the "xz" library.

[WanWizard]

Ok, so that will be available in the image as of OpenPLi 6.1.

 

And in other images that use the latest importer code, and have updated their recipe dependencies.

[MiLo]

The OE environment requires these two changes:

https://github.com/O...807887ceb1cdca3
https://github.com/O...18f751e2fb916d3

They can probably even be put into openpli-4 if the need arises.

The xmltv-rytec sources package determines which files are to be downloaded, so it's possible to keep two versions of that alive for the transition period.

[WanWizard]

I was thinking about timing.

 

Because at some point, Willy has to change the references in the index files from gz to xz.

 

If the clients are not ready for that, all hell will break loose, so we have to make sure our production images are ready when that happens, or some fallback mechanism has to be designed (either via multiple references in the index, or by a "if not xz then try gz" download attempt).

[Pr2]

Hi,

 

Just my two cents on the subject, some channels EPG are spread out into multiple dedicated files. So downloading both belgian and french will have duplicates entries.

My idea is the following:

 

Create one file with EPG per TV channel, we already have a unique identifier for them, example: TF1.fr, France2.fr

So create a downloadable file:  TF1.fr.xz (or .gz) that will be cached locally on the STB and reuse it when we have multiple time the same channels on different SAT and or Cable provider needed for EPG.

 

So the source.xml file can be cleaned up to only mention the servers without any specific big file, the channels.xml maintained by Doglover is downloaded with only the channel ID and service reference.

 

So instead of downloading a big files to get EPG  for only a few channels that we can receive (for exemple to get the EPG for BBC while I don't need the other english channels) we pick-up the channels that we really needs in the  channels.xml file.

channels.xml file should then be probably extended to support the timezone difference for the channels, has possible tag in the channels.xml

 

So when people use the option to only load EPG for favorites, it will only download the EPG data that he really needs and not those big files. This option is today only useful to save STB memory but it doesn't save any bandwidth.

 

The drawback is that the server will get many more small files request to download.

 

But indeed, as it was already discussed for the OpenPLi update, setting up a kind of torrent distribution method can also be an effective solution. I have 3 STBs and I download EPG on all of them daily. 

 

Another idea: can we imagine to include this unique reference for EPG for channel in the userbouquet file? So no more needs to maintain individual channels.xml file.

Like we have #description tag we can imagine to  create a #epgid tag. So for TF1 I can mention:  #epgid TF1.fr

 

Pr2

[WanWizard]

My initial idea was to import all generated data into a database with service reference;start;end;title as key, with duplicate info detection, and timestamps on each record. This would allow you do create delta files between the data generated on time X and X-1, per service reference. 

 

And then instead of importing "a file with channels", you can (as you also suggest) have a single index list with the service references of the channels for which EPG is available, and an optional alias (so you can say EPG for ref A is in file B, for duplicates), and on the bx you could tailor the import further, for example limit to channels in a bouquet.

 

You would also need an index of the delta's available on the server, so the box could, based on it's last import timestamp, determine which delta's it should fetch. And if no last timestamp is available, or it's older than the available delta's, the option to fetch all data should be there as well, for example after you have flashed the box, and no EPG is present at all.

 

But I have no idea of the processing time and power required to do this, to that would have to be looked at.

 

Local EPG distribution is something we are thinking about at the Enigma level, so that instead of having to duplicate EPG, you just have one box that serves as "EPG server", a bit similar to the fallback tuner mechanism.

[littlesat]

Or discussie with oe-a images if they will also setup feeds for epg

Edited by littlesat, 23 July 2017 - 18:46.

[WanWizard]

There are already quite a few mirrors, my server is only one of many.

[Pr2]

Using service reference is a bad idea because it is too restrictive and you can have duplicate service reference especially for DVB-C users.

And service reference is hard to maintain and subject to change, ask Doglover of all the changes he has to perform each time french Canal changes its channels order.

And they sometimes simply perform service reference swap, how will you handle this?

 

And which service reference will you give to IPTV channels?  How will you let know which reference you assign to it?

 

The current naming convention is, I think, a quite good one.

Even if we can improve it by defining a real naming convention, something similar to the one used for the picon by name.

Edited by Pr2, 23 July 2017 - 20:06.

[WanWizard]

You need to map the schedule to the service ref somewhere. How do you now create the mapping with a DVB-C channel? And with an IPTV channel? So that is not really relevant for this disucssion, that work will never change.

 

I think you misunderstood me, the only challenge is coming up with an algorithm to calculate a unique hash for every channel being grabbed, so a correct diff can be generated. I haven't discussed anything else yet.

 

Btw, mapping by name is a very bad idea, exactly as the picon system shows you. It is full of logic to deal with it's inadequacies. 

[doglover]

Create one file with EPG per TV channel, we already have a unique identifier for them, example: TF1.fr, France2.fr

So create a downloadable file:  TF1.fr.xz (or .gz) that will be cached locally on the STB and reuse it when we have multiple time the same channels on different SAT and or Cable provider needed for EPG.

 

 

This not workable.  At the moment there are 2000 channels.  This will results in 2000 grabber config files to maintain.  If something has to be altered or maintained, this will mean finding the correct config file correcting it, finding the next config file which uses the same grabber, correct that and so on.  I do not know from the top of my hat which channels are grabbed with which grabber.   So if one EPG changes, I have to adapt the grabber, and sometimes the config file has to be changed as well.  So I have go through all 2000 config files to find which uses that grabber.  Now I have only 20 or so config files, and that is already difficult enough.

If you want a file for each channel, it will not be by me.

 

My initial idea was to import all generated data into a database with service reference;start;end;title as key, with duplicate info detection, and timestamps on each record. This would allow you do create delta files between the data generated on time X and X-1, per service reference.

 

And then instead of importing "a file with channels", you can (as you also suggest) have a single index list with the service references of the channels for which EPG is available, and an optional alias (so you can say EPG for ref A is in file B, for duplicates), and on the bx you could tailor the import further, for example limit to channels in a bouquet.

 

You would also need an index of the delta's available on the server, so the box could, based on it's last import timestamp, determine which delta's it should fetch. And if no last timestamp is available, or it's older than the available delta's, the option to fetch all data should be there as well, for example after you have flashed the box, and no EPG is present at all.

 

 

Not a good idea.  EPG is constantly changing.  It is not because once you have the data, it remains correct.  Changes are constantly applied each day.  You want the renew the info ideally constantly.  This impossible, and we ended up with a daily update, which also takes in account the changes which have been occurred since the last update.

 

Furthermore a file per day sound good.  However i cannot even produce those at the moment.

 

Using a database for the EPG is good idea for enigma2.  But has not to do with providing the EPG data.

You have to fill the database with data.  The data comes from EIT (or something else from the signal) or is imported.  If it is imported you end up with the same problems.
 

Using service reference is a bad idea because it is too restrictive and you can have duplicate service reference especially for DVB-C users.

And service reference is hard to maintain and subject to change, ask Doglover of all the changes he has to perform each time french Canal changes its channels order.

And they sometimes simply perform service reference swap, how will you handle this?

 

The mapping list (channels.xml file) is a real pain in the behind.  Problem, I do not know any other method of doing this reliably.

The list is changed almost daily.  And up to now it was a painstaking job.  Especially when a provider start reusing old service refs for another channel.  Finding the old mapping is difficult.

Although I recently started using excel (actually LibreOffice) for maintaining the mapping list.  I applied some logic on the spreadsheet, which raises an alarm when I enter a service ref which already exists.

That is the reason for the new format of the channels file.

 

The current naming convention is, I think, a quite good one.

Even if we can improve it by defining a real naming convention, something similar to the one used for the picon by name.

 

 

Using the name for assigning EPG is bound for a disaster.

f.i. Comedy Central is present with many different providers.  They all have a different schedule.  Or people want the EPG in their own language.

The picon for Comedy Central is the same for the French, Dutch, German, British, Italian, etc.. versions.  The EPG is not.

 

Willy

Edited by doglover, 24 July 2017 - 12:35.

[doglover]

What i could do i making smaller but more files.

 

f.i.

 

UK file split up in:

 

Freesat

Other FTA  (mostly Asian, Religious and shopping)

Sky

Sky-Sport

Sky-Movies

Cable only

 

This is then based on the sat.  And immediate the cable user will not have the same split, and start downloading files from which they need only 2 channels.

 

Do this for most of the packages.

 

Willy

[WanWizard]

Willy,

 

I think you've misunderstood me.

 

With what I had in mind (and it isn't more than that), you could even grab with a single config, as files like they are created now will be no longer relevant. Whatever is created by the grabber (either 1 or 1000 files) is imported into that database.

 

When the database import process runs on the server, it can compare every item of grabbed programme information with what is in the database. It might be new, it might be an update, or there might be a database record but nothing in the import (which is a delete). Whatever it is, the database is updated with it, and with the update an update timestamp is written.

 

Based on that timestamp, you can generate delta files (all programme data updated since time x). So if you have 7 days of data for a channel, the delta file could contain 1 update for today, nothing for Tuesday and Wednesday, 2 additions and 1 delete for Thursday, nothing for Friday and Saturday, and all entries for Sunday (as that is a new day compared to yesterday's run). Since these are delta files, you need to keep several days of it (in this example at least 7) online, so that you can also update a box that doesn't have had regular updates.

 

The server can also generate an "all channels" list, if needed ordered by language, provider, whatever, in a nested structure. This file can be used on the box to allow the user to create the mapping between the service refs there is EPG data for, and the users local service refs. This has to happen with a GUI that needs to be designed. The GUI can also have features to autmatically map, for example based on the current map files you already maintain for the different providers. This is all something to be worked out.

 

After completing the mapping locally, the box now has a local file with service ref's of all channels it wants EPG data of. This file can optionally contain an alias (so you can say "this service ref for a stream requires the EPG from this cable service ref). 

 

This is btw quite similar to how the EPG import GUI in Enigma1 used to work.

 

When the importer starts, it loops over this file, and

- for each entry, it will fire a download request containing the service ref and the last update timestamp: http://example.org?s...&t=89873125537 

- the server will respond with a data structure (xml, json), that lists the files the box needs to download

- the box downloads each file, and imports it

 

This way

- the box will only download EPG for the channels the user has defined she/he wants EPG data for.

- the box will only download every programme description exactly once.

- the load of downloading and processing is spread over time, as it isn't one big file, but several small files.

 

Only in the case of a complete reload (for example after a flash), a bit more data then needed is downloaded, as the delta's can contain updates so for those programme's you load a description more than once. But I think that in terms of volume that is acceptable.

 

Thinking very out of the box: We could even think of having that GUI send feedback to the server with regards to the mapping. So if the user says "this local service ref entry needs EPG from channel A instead of B", or "this channel no longer exists", the GUI can report it back, so the users can maintain the mapping list, instead of you.

[doglover]

OK.

 

But this is out of my league.  I can fill your database, but that is it. (and already almost a day job)

 

Willy

[WanWizard]

I didn't say you have to do it all. You're doing a great job grabbing, and I have some idea of the enormous amount of work involved.

 

At the moment I see this more as a brainstorm session, to pitch idea's back and forth.

 

Once we have an idea that could possibly work, we need to figure out the who, where, when and what. We will be needing developers (server side, and python for the plugin), hardware (that database has to do a lot of processing), a new distribution mechanism (you don't want to have to manually ftp the resulting files to all mirrors), etc.