93 replies to this topic

[WanWizard]

Willy,

 

Would you mind contacting the admin's of all XMLTV EPG mirrors, and ask how much data they (roughtly) process in a month?

 

Over June 2017, my server has send out 18.8TB of EPG data, and that is with over a third of the requests denied (requests from Kodi and other non-enigma platforms).

 

This seems to become serious business, and I think if we want to be able to sustain this growth, we need to think about a more efficient data distribution system, instead of sending large files with repeated and largely unneeded information on every request. If we do that, we probably also have to discuss this with developers from CrossEPG, and maybe other enigma tools out there.

 

Or do you also want to cater for non-Enigma platforms like Kodi (in which case we have to discuss any new format with them as well).

 

Let me know your thoughts on this.

[doglover]

I can produce smaller files with less info.  The number of files will however grow.  Problem is how to split up the different files.

And it will be a nightmare to go through the transition.  The grab files have to be manually sorted.

Also lets take the example of the French file:

 

This file contains the French channels and the Belgian channels from the French speaking part of the country.  If I split this out, everybody which want the Belgian channels which are in the Canalsat package have to download the Belgian package as well.  Etc. etc..

 

One thing I can do straight away is produce a news file.  This section is included in almost every other file, and thus make these file larger.  Then I could remove these files from all the other files.

But probably everybody will select the news section, and so downloading it.  The result would be zero, except more requests.

 

However this will not help. Due to the Kodi and IPTV users which have something like 2000 channels in their bouquet and want EPG for all the channels.

 

In the ideal world, I would like only to provide the EPG for enigma2 receivers.  But I do not know how to achieve that.

 

Willy

[WanWizard]

It's not something that has to happen now, but something that has to be considered.

 

Alternatively we create a new importer, and use your existing data to convert into a different file format for OpenPLi users. In that case nothing will change for you, and for non-OpenPLi users, or users that don't use our importer (but use Cross for example).

 

This is probably the most transparent for you, and for the users, as there is no transition period for them.

[WanWizard]

In the ideal world, I would like only to provide the EPG for enigma2 receivers.  But I do not know how to achieve that.

 

Difficult.

 

We thought about password protect the data, so you have to use the xmltvimporter and enter your username and password in the config. But as the code is open source, everyone can implement this feature, so it doesn't archieve anything.

[doglover]

I could compress the files in xz format.

 

File sizes:

 

Benelux file: gz:5751 kb xz:3444kb

UK file: gz:6362 kb xz:3173 kb

 

There is a 40 - 50 % savings in traffic.

 

Willy

[doglover]

The smallest files I could achieve with compression is with

 

xz - Ultra compression  (Extreme in linux version)

an almost the smae file size with

7x - Ultra compression

 

f.i the Benelux file became: 3216kb

 

Willy

Edited by doglover, 21 July 2017 - 15:57.

[doglover]

In each case the importer does not understand the xz format:

C:\EPG\python>C:\EPG\python\OfflineImport.py  D:\EPG\python\test.sources.xml
[EPGImport] oudeis patch not detected, using epg.dat instead.
[EPGImport] failed to load C implementation, sorry
[EPGImport] nextImport, source= Test Benelux
[EPGImport] afterDownload V:\epg\BExmltv.xml.xz
[EPGImport] afterChannelDownload None
[EPGImport] Parsing channels from 'V:\epg\rytec.channels.xml.xz'
Traceback (most recent call last):
  File "C:\EPG\python\OfflineImport.py", line 68, in <module>
    importFrom(epgimport, xml)
  File "C:\EPG\python\OfflineImport.py", line 54, in importFrom
    epgimport.beginImport(longDescUntil = time.time() + (5*24*3600))
  File "C:\EPG\python\EPGImport.py", line 111, in beginImport
    self.nextImport()
  File "C:\EPG\python\EPGImport.py", line 120, in nextImport
    self.fetchUrl(self.source.url)
  File "C:\EPG\python\EPGImport.py", line 126, in fetchUrl
    self.afterDownload(None, filename, deleteFile=False)
  File "C:\EPG\python\EPGImport.py", line 191, in afterDownload
    self.afterChannelDownload(None, None)
  File "C:\EPG\python\EPGImport.py", line 210, in afterChannelDownload
    self.iterator = self.createIterator(filename)
  File "C:\EPG\python\EPGImport.py", line 129, in createIterator
    self.source.channels.update(self.channelFilter, filename)
  File "C:\EPG\python\EPGConfig.py", line 77, in update
    self.parse(filterCallback, self.urls[0])
  File "C:\EPG\python\EPGConfig.py", line 58, in parse
    for event, elem in iterparse(self.openStream(downloadedFile)):
  File "<string>", line 91, in next
cElementTree.ParseError: not well-formed (invalid token): line 1, column 0

Willy

[Abu Baniaz]

I dont see why you have Virgin and Freesat EPG.

[doglover]

I dont see why you have Virgin and Freesat EPG.

This is a bad example.

They all use the same XMLTV (data) file.  In this case only the channels file (links between the actual channel and the EPG) is different.

Willy

 

In the past I had to split off the Virgin channels, as some references resembled too much references present on the sat.  And in this case the crossepg complained too much.

Edited by doglover, 21 July 2017 - 17:35.

[littlesat]

Why not distribute it over a torrent system? Or usenet?

[Lost in Space]

Blockchain it - then you are in full control

[doglover]

Blockchain it?

you have lost me here. Care to explain?

Willy

[WanWizard]

A funny note, without any useful contribution, as usual.

 

There is no reason to use a blockchain for EPG data, and I don't see a reason for using a torrent system either (that will use part of the end-users upstream, and some don't have a lot).

 

I'll have a good think about it.

[Abu Baniaz]

 

I dont see why you have Virgin and Freesat EPG.

This is a bad example.

They all use the same XMLTV (data) file.  In this case only the channels file (links between the actual channel and the EPG) is different.

Willy

 

In the past I had to split off the Virgin channels, as some references resembled too much references present on the sat.  And in this case the crossepg complained too much.

 

Sorry, I did not phrase that properly.

 

The EPG data is on the signal stream. People will start using it if they cant get the EPG using Rytec.

[doglover]

A dutch mirror reports this:

 

Ik houd tegenwoordig zelf niet echt statistieken bij en mijn hosting provider ook niet, maar de netwerk interface geeft aan dat er in de 73 dagen uptime zo'n 22,9 Terabyte is verstuurd. Dus grofweg 9,5TB in een maand, ongeveer de helft van zijn server.

 

Kleinere bestanden helpt alleen als de clients niet doorgaan met het opnieuw downloaden voor de dagen die ze al eens hebben verwerkt. Dus als ze 7 dagen vooruit zouden downloaden, wordt de 7e dag in principe nog steeds 7x keer gedownload. Dan zouden de clients en de data slimmer moeten worden waarbij bijvoorbeeld alleen een dag opnieuw zouden downloaden als er daadwerkelijk iets veranderd (of nog beter zou zijn dat ze alleen de verschillen krijgen).

 

Comment:

 

Splitting the files in daily chunks, would not help.   Since each day the daily files are checked for changes.  And there are always changes.  So each day all daily files are changed, and even if the importer is so smart only to download changed files, it will still download the full list, since all files are changed every day.

 

Willy

[doglover]

French Mirror:

 

 

From a quick eye I can see that my server has around 60-80 TB of traffic / month.
Attached you will find the screenshot of the last 24hrs.

With an average rate of 200-250 Mbps the results are 65-80 TB / month.
Is OK for me as I have unlimited plan on my dedicated servers located in OVH France.
My server has 1Gbps interface with 500Mbps guaranteed bandwidth,
The true is that is a lot of data but it dosn't bothers me a lot as things goes smooth and users are happy having EPG's.

As for your idea to find another way to distribute EPG's instead of sending those large files, a possible solution is a database where the EPG's will be stored and the users will download epg's only for the requested channels. If you have a MAG device (mag 250, mag 254, etc) you will see that uses this method. If OpenPLI developers can simulate this, then this might be the solution.

 

 

Willy

 

As you can see in the screenshot there are a lot of people who do not read forums.  Otherwise they would know that downloading from midnight until let's say 08h00 will only yield yesterday's file.

Attached Files

•  Screenshot_2.jpg   56.97KB   5 downloads

[MiLo]

In each case the importer does not understand the xz format:

We could teach it that though. Either using the lzma backport Python module, or just spawning a subprocess and passing the data through busybox' xz decompressor.

(In case you hadn't noticed, lzma and xz are basically the same thing)

[doglover]

 

In each case the importer does not understand the xz format:

We could teach it that though. Either using the lzma backport Python module, or just spawning a subprocess and passing the data through busybox' xz decompressor.

(In case you hadn't noticed, lzma and xz are basically the same thing)

 

 

If you can do so, it would be appreciated.  It is not a miracle thing, but it helps.

Pls. leave the existing import of gz and xml files intact.

 

Willy

Edited by doglover, 22 July 2017 - 17:17.

[MiLo]

Did a quick implementation and added a recipe to the "develop" branch. If you could test it that would be nice...

[doglover]

How do I install it?

Sorry about that.  I can compile on a linux machine, but I never know what I am getting.

 

But I see on the Github it is just a python file you have changed.

Question is it enough I replace the python file on my receiver?  Or do I need to install something more?

 

Willy