80 replies to this topic

[doglover]

EPGimport has the option of excluding channels from the import of EPG.

This is done by copying the Service Ref to the file ignore.conf in /etc/epgimport.

 

By doing so.  This channel will not have EPG imported.

 

There is however a different situation where you do not want EPG imported.

f.i. this is defined in the channels file:

<!-- DK --> <!-- 0.8W --> <channel id=" DiscoveryHD.dk "> 1:0:19:1006:29:46:E080000:0:0:0: </channel><!--  Discovery HD Showcase  -->
<!-- FI --> <!-- 0.8W --> <channel id=" DiscoveryHD.fi "> 1:0:19:1006:29:46:E080000:0:0:0: </channel><!--  Discovery HD Showcase  -->
<!-- HRV --> <!-- 0.8W --> <channel id=" DiscoveryHDShowcase.rs "> 1:0:19:1006:29:46:E080000:0:0:0: </channel><!--  Discovery HD Showcase  -->
<!-- HU --> <!-- 0.8W --> <channel id=" DiscoveryHDShowcase.hu "> 1:0:19:1006:29:46:E080000:0:0:0: </channel><!--  Discovery HD Showcase  -->
<!-- NO --> <!-- 0.8W --> <channel id=" DiscoveryChannel.no "> 1:0:19:1006:29:46:E080000:0:0:0: </channel><!--  Discovery HD Showcase  -->
<!-- SE --> <!-- 0.8W --> <channel id=" DiscoveryHDshowcase.se "> 1:0:19:1006:29:46:E080000:0:0:0: </channel><!--  Discovery HD Showcase  -->
<!-- SVN --> <!-- 0.8W --> <channel id=" DiscoveryHD.svn "> 1:0:19:1006:29:46:E080000:0:0:0: </channel><!--  HD Discovery Shovcase  -->

All the same service ref, but different language EPG attached.

You can avoid this problem not to import the files which contain this conflicting EPG.

 

I would ask if it is possible to program into the EPGimport plug-in a simular filter as above, but in which you define that you do not want.f.i.

DiscoveryHD.dk
DiscoveryHD.fi
DiscoveryHDShowcase.rs
DiscoveryHDShowcase.hu
DiscoveryHDshowcase.se
DiscoveryHD.svn

to be imported.  So ignoring this list when importing EPG.

And only import  DiscoveryChannel.no.  This being the only one not listed in the ignore list.

 

I think it is enough that the list is manually added.  No software has to be written, to create the list.

Edited by doglover, 14 March 2021 - 09:31.

[jeepcook]

Hello Doglover,

 

That's the solution I was looking for. I did a test with BabyTV. Here are the excluded ServiceRefs in my ignore.conf.

 

By this way only BabyTV.fr has te be in the epg but it doesn't work, the result is I have the pl epg, so the last one grab by epgimport.

 

<!-- 13.0E --><channel id="BabyTV.pl">1:0:1:421D:2F44:13E:820000:0:0:0:</channel><!-- Baby TV -->
<!-- 13.0E --><channel id="BabyTV.nl">1:0:1:421D:2F44:13E:820000:0:0:0:</channel><!-- Baby TV -->
 

 

Edit: Test to add channels in ignore.conf via epgimport menu, the result is the file contains only the ServiceRef, not the epg id. For my example, the content of ignore.conf is now:

1:0:1:421D:2F44:13E:820000:0:0:0:

 

So every languages are ignored

 

 

 

Edited by jeepcook, 16 March 2021 - 08:47.

[doglover]

That is true.  If you place the service ref in ignore.conf, no EPG is imported.  Never....

 

So in order to avoid this, I have written a script (just finished it) for a workaround.

I will publish it today or tomorrow, once I have written the instructions on how to use it.

Because if not use properly, I will get questions why no EPG is imported for half the channels.

[jeepcook]

That is true.  If you place the service ref in ignore.conf, no EPG is imported.  Never....

 

So in order to avoid this, I have written a script (just finished it) for a workaround.

I will publish it today or tomorrow, once I have written the instructions on how to use it.

Because if not use properly, I will get questions why no EPG is imported for half the channels.

Great news. Thanks

[doglover]

I have written a script to achieve the above.

I still prefer that this should be included in the EPGimport plug-in itself.

However, I am not capable of including it.

 

As a workaround this script.

The source and instructions are here:

https://github.com/doglover3920/EPGimport-Sources

(Click on Code and choose Download ZIP)

Edited by doglover, 17 March 2021 - 13:59.

[WanWizard]

That is true.  If you place the service ref in ignore.conf, no EPG is imported.  Never....

 

Does that mean it prints

Serviceref is in ignore list: ...

or

Invalid serviceref string; ....

or

Serviceref not in bouquets: ....

to the log on every service ref processed?

[doglover]

Placed for the matter of demo 1:0:19:1B1D:802:2:11A0000:0:0:0: in the ignore.conf file

This the result in the log:

[EPGImport] Downloading: http://rytecepg.dyndns.tv/epg_data/rytecBE_VL_Basic.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://rytecepg.dyndns.tv/epg_data/rytecBE_VL_Basic.xz
[EPGImport] checkValidServer rejected server download error for: http://rytecepg.dyndns.tv/epg_data
[EPGImport] download failed: checkValidServer reject the server
[EPGImport] Attempting alternative URL
[EPGImport] Downloading: http://epgspot.com/rytec_epg/rytecBE_VL_Basic.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://epgspot.com/rytec_epg/rytecBE_VL_Basic.xz
[EPGImport] afterDownload /tmp/epgimport.xz
[EPGImport] unlink /tmp/epgimport.xz
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/custom.channels.xml'
[EPGImport] Parsing channels from '/etc/epgimport/filtered.channels.xml'
[EPGImport] Parsing channels from '/etc/epgimport/filtered.channels.xml'
Serviceref is in ignore list: 1:0:19:1B1D:802:2:11A0000:0:0:0:
[XMLTVConverter] Enumerating event information
Unknown channel:  eurosport1.be
Unknown channel:  mtv.be
Unknown channel:  fox.be
Unknown channel:  tlc.be
Unknown channel:  elevenproleague2nl.be
Unknown channel:  elevenproleague3nl.be
Unknown channel:  actua.be
Unknown channel:  studio100tv.be
Unknown channel:  avsgent.be
Unknown channel:  stories.be
Unknown channel:  njam.be
Unknown channel:  eurosport2.be
Unknown channel:  nicktoons.be
Unknown channel:  q-music.be
Unknown channel:  plattelandstv.be
Unknown channel:  kanaalz.be
Unknown channel:  playmorekicks.be
Unknown channel:  playmorecinema.be
Unknown channel:  playtime.be
Unknown channel:  playmoreblack.be
Unknown channel:  playsportsgolf.be
Unknown channel:  playsportshd1.be
Unknown channel:  playsportshd3.be
Unknown channel:  playsportshd2.be
Unknown channel:  playsportshd4.be
Unknown channel:  playsportshd5.be
Unknown channel:  playsportshd6.be
Unknown channel:  playsportshd7.be
Unknown channel:  playsportshd8.be
Unknown channel:  discoveryvlaanderen.be
Unknown channel:  disneyvl.be
Unknown channel:  nickelodeonspike.be
Unknown channel:  vlaamsparlement.tv
[EPGImport] ### thread is ready ### Events: 4091
[EPGImport] imported 4091 events
[EPGImport] Save last import date and count event
[EPGImport] Run check deep standby after import
[EPGImport] #### Finished ####

Thus:  Serviceref is in ignore list: ...

 

This eliminates every import of EPG to that channel.  Also from your custom.channels.xml

Which should not happen. 

 

The ignore list should only have effect on the files downloaded from the internet. 

Not on your local files which your in full control of.  So you can take care there of what is in it. 

 

My script takes care of that, and more.

[WanWizard]

Ok, so the problem is not that the filter doesn't work, but that it works too well?

 

As far as I can see, the plugin just iterates over the available/defined sources, it has no idea what is "custom" and what is not. What is "custom", and how to detect a source is "custom"?

[doglover]

As far as I can see, the plugin just iterates over the available/defined sources, it has no idea what is "custom" and what is not. What is "custom", and how to detect a source is "custom"?

 

If I knew that....

Ideally it should only filter channels file which are downloaded from the internet.

Files which have in the sources file a local path or no path (so no http:) can be considered local custom files.

So also the special channels file custom.channels.xml.

 

The idea behind this, if want something special and create a separate file for it and put it on your receiver, this should be treated as such.  You are considered to know what you are doing.

 

BTW:  the ignore which is currently available is not able to filter out EPGrefs (like DiscoveryHD.se).  It can only eliminate service refs.  (Tested it....)

Edited by doglover, 17 March 2021 - 16:27.

[WanWizard]

The code doesn't have the concept of "downloaded from the internet".

 

It just iterates over local files (that may or may not have been downloaded before). So at that point, all have a local path.

 

So this might prove to be a challenge...

[doglover]

Too bad.  But I was just trying to find a way of achieving the ignore what it needed.

 

Even the current ignore could be made to achieve what I want if you make one the following modifications:

 

1. Limit the ignore to the file downloaded files from the internet, as I suggested before. 

But this is maybe not achieve-able, as you pointed out.

 

2. Extend the ignore not only to the service refs (f.i 1:0:19:1B1D:802:2:11A0000:0:0:0:) but also to EPGrefs (f.i. nicktoons.be, een.be, NPO1.nl etc...)

This ignore can be done on everything.

I have no idea if this is possible.

[WanWizard]

It might need more than a simple change, I'm not ruling anything out, but it will not be for tomorrow...

 

Can you define what exactly do you mean with "custom"? Can you profile example files that can be tested with?

[doglover]

In the hope not to confuse you more.  I made some imports to clarify my points:

The log is this (deleted a bunch of unknown channels)

[EPGImport] nextImport, source= News Channels (xz)
[EPGImport] Downloading: http://epg.vuplus-community.net/rytecNWS.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://epg.vuplus-community.net/rytecNWS.xz
[EPGImport] afterDownload /tmp/epgimport.xz
[EPGImport] unlink /tmp/epgimport.xz
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/custom.channels.xml'
[EPGImport] Parsing channels from '/etc/epgimport/testing.channels.xml'
[EPGImport] Parsing channels from '/etc/epgimport/testing.channels.xml'
Serviceref is in ignore list: 1:0:19:1B1D:802:2:11A0000:0:0:0:
[XMLTVConverter] Enumerating event information
Unknown channel:  tv5mondemaghreb-orient.fr
Unknown channel:  deutschewellede.de
Unknown channel:  skynwsarabia.nws
Unknown channel:  trtturk.tr
[EPGImport] ### thread is ready ### Events: 14409
[EPGImport] nextImport, source= Vlaanderen - Basis (xz)
[EPGImport] Downloading: http://rytecepg.wanwizard.eu/rytecBE_VL_Basic.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://rytecepg.wanwizard.eu/rytecBE_VL_Basic.xz
[EPGImport] afterDownload /tmp/epgimport.xz
[EPGImport] unlink /tmp/epgimport.xz
[EPGImport] Downloading: http://epg.vuplus-community.net/rytec.channels.xml.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://epg.vuplus-community.net/rytec.channels.xml.xz
[EPGImport] afterChannelDownload /tmp/epgimport.xz
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/custom.channels.xml'
[EPGImport] Parsing channels from 'rytec.channels.xml.xz'
[EPGImport] Parsing channels from 'rytec.channels.xml.xz'
Serviceref is in ignore list: 1:0:19:1B1D:802:2:11A0000:0:0:0:
[XMLTVConverter] Enumerating event information
Unknown channel:  disneyvl.be
Unknown channel:  nickelodeonspike.be
Unknown channel:  vlaamsparlement.tv
[EPGImport] ### thread is ready ### Events: 18451
[EPGImport] nextImport, source= Koala 7 dias XMLTV
[EPGImport] Downloading: http://tropical.jungle-team.online/epg/koala7.xml.gz to local path: /tmp/epgimport.gz
[EPGImport] Not cheching the server since nocheck is set for it: http://tropical.jungle-team.online/epg/koala7.xml.gz
[EPGImport] afterDownload /tmp/epgimport.gz
[EPGImport] unlink /tmp/epgimport.gz
[EPGImport] Downloading: http://tropical.jungle-team.online/epg/koala_epg.channels.xml.gz to local path: /tmp/epgimport.gz
[EPGImport] Not cheching the server since nocheck is set for it: http://tropical.jungle-team.online/epg/koala_epg.channels.xml.gz
[EPGImport] afterChannelDownload /tmp/epgimport.gz
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/custom.channels.xml'
[EPGImport] Parsing channels from 'http://tropical.jungle-team.online/epg/koala_epg.channels.xml.gz'
[EPGImport] Parsing channels from 'http://tropical.jungle-team.online/epg/koala_epg.channels.xml.gz'
Invalid serviceref string: 4097:0:1:525:0:0:0:0:0:0:http://.m3u88
[XMLTVConverter] Enumerating event information
Unknown channel:  m. laliga 10
Unknown channel:  m. deportes1

First import:  News channels.

I have set here the channels file to a local file (testing.channels.xml).  And as you can see nothing s downloaded.  The file is used as such.

Also the custom.channels.xml (a local file) is being parsed.

This files should be left alone.

[EPGImport] afterChannelDownload None

 

Second Import:  Vlaanderen - Basis (xz)

The source is used as set in the rytec.channels.xml file.  Thus downloading the file from the net.

You can see this in the log:

[EPGImport] Downloading: http://epg.vuplus-community.net/rytec.channels.xml.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://epg.vuplus-community.net/rytec.channels.xml.xz
[EPGImport] afterChannelDownload /tmp/epgimport.xz

This is candidate for the Ignore filter.  The file is downloaded in the /tmp/ folder as epgimport.gz.

In order to apply the filter, why not just filter this file just after the download.  The file is zipped.  So unzip it., use the grep to filter it.  Rezip it (or use the filtered file as such.)

Something, like this:

grep -v -f "/etc/epgimport/FILTERpattern.txt" "/tmp/rytec.channels.xml" > "/etc/epgimport/filtered.channels.xml"

Third Import: Koala 7 dias XMLTV

 

Again a channels file is downloaded from the net.  Also a candidate for the filer.

[EPGImport] Downloading: http://tropical.jungle-team.online/epg/koala_epg.channels.xml.gz to local path: /tmp/epgimport.gz
[EPGImport] Not cheching the server since nocheck is set for it: http://tropical.jungle-team.online/epg/koala_epg.channels.xml.gz
[EPGImport] afterChannelDownload /tmp/epgimport.gz

Again the downloaded channels file is in /tmp/epgimport.gz

So the same as above.

 

 

Now once you have downloaded these files they stay in the system, and will not be downloaded again the same day.  So no filtering is needed here, as it has been already been done just after the download.

 

I have added as an example my custom.channels.xml file and an example of the FILTERpattern.txt file.

Attached Files

•  FILTERpattern.txt   269bytes   3 downloads
•  custom.channels.xml   30.78KB   1 downloads

Edited by doglover, 19 March 2021 - 09:10.

[Pr2]

There is a plugin called epgimportfilter in the feed isn't it doing what you are requiring here?

[jeepcook]

There is a plugin called epgimportfilter in the feed isn't it doing what you are requiring here?

Hello, can't find it in oe-a repositories.

[stro11]

Hi,

 

The plugin epgimportfilter is not used anymore because this function is in the EPG Importer now.

This is the Description of the plugin: description="Filter EPGImport data for selected bouquets"

And here is the source (last update 2014): https://github.com/A...EPGImportFilter

 

regards

stro11

Edited by stro11, 21 March 2021 - 15:20.

[WanWizard]

Better remove it from the feeds then?

[Pr2]

Be careful that EPGImport used in OpenPLi is not the one used in OE-A images. OE-A is using XMLTV-import and it is not exactly the same plugin.

[stro11]

Incase of my Octagon sf8008 with PLi8.0 i have the option in the EPG Importer.

 

 EPG_Import.png   436.11KB   3 downloads

 

 

[Pr2]

Yes we have it too, but I just warm you that EPG Import and XMLTV import are not exactly the same plugin.