80 replies to this topic

[stro11]

The XMLTV version has the same option.

 

 XMLTV_Import.png   1.73MB   6 downloads

 

Here is the source: XMLTV-Import/plugin.py at master · oe-alliance/XMLTV-Import · GitHub

 

Maybe the plugin epgimportfilter is/was needed for PLi 4.0

 

regards

stro11

 

[Pr2]

Hi,

 

Here is a trial to implement the feature requested by Doglover.

 

Forewords

 

The filtering is only possible on the channel id and is based on regular expression (regex). I will not fully explain here what the regex syntax is but just a few remarks;

If you want to fully exclude all the EPG for all the channel id from france you need to use this syntax:

 

.*\.fr

 

. replace any characters

* means that any characters can repeat anytime

\. is to match the .  remember . (alone) means any character for the regex.

fr is "fr"

 

You can also specify a specific channel id:

 

Discovery\.dk

 

This will exclude Discovery.dk  is you didn't mention \. then the regex will match Discovery(1 time any character)dk

But most of the time simply specifying Discover.dk should not cause trouble but in case of doubt and if you want to be sure to have a perfect match use the \. notation.

 

The filtering is not case sensitive (so the whole filter that you will set will be switched to lower case because it is what is also done for the channel id matching.

So Discovery\.dk is the same as DiscoVery\.DK so you don't have to care about the exact case of the channel id name.

 

So filtering on anything else than the channel id is not foreseen (it is not possible to filter on partial service reference nor on comments). Why?  Because the channels.xml file is processed in a true XML way meaning that every element is retrieved independently so this filter will is focusing on the channel id and nothing else (filter by reference is already implemented into EPGImport).

 

One line per filter a OR operation will be performed between every lines.

 

 

To test go into this folder on your STB:

 

/usr/lib/enigma2/python/Plugins/Extensions/EPGImport

 

There make a copy of the existing: EPGConfig.py

cd /usr/lib/enigma2/python/Plugins/Extensions/EPGImport
cp EPGConfig.py EPGConfig.py.ori

 

Then delete the EPGConfig.pyo file (optional).

 

 

Download this file:

 

https://raw.githubus...rt/EPGConfig.py

 

Then transfer it to your STB into: /usr/lib/enigma2/python/Plugins/Extensions/EPGImport

 

Then you need to create a filter file:  

 

/etc/epgimport/channel_id_filter.conf

 

With the filter that you want:

 

.*\.fr
.*\.be
discovery\.dk

 

This will exclude all the channel id ending with .fr and .be and also exclude discovery.dk

 

Reboot your box. You only need to reboot your box once to have it creating the EPGConfig.pyo file.

Later on you simply need to edit and change the /etc/epgimport/channel_id_filter.conf

[jeepcook]

Hello Pr2,


Thanks for this change.


Tried it,

- for information EPGConfig.pyo is not created after reboot but after executing epgimport plugin and apply changes (green save button).

- applied some filters and everything seems to be fine


An advantage of your method is that we can filter channel ids available in rytec epg sources files and not yet added to rytec.channels.xml (hope I'm clear).


Is this change will be committed to oe-a branch ?


Thanks

[doglover]

Not exactly what I want.

 

EPGlog:

[EPGImport] autostart (0) occured at 1617601868.57
[EPGImport] autostart (0) occured at 1617601870.03
[EPGImport] WakeUpTime now set to -1 (now=1617601870)
[EPGImport] Selected source:  ['Vlaanderen - Basis (xz)']
sourcesDone():  False None
[EPGImport] nextImport, source= Vlaanderen - Basis (xz)
[EPGImport] Downloading: http://epg.vuplus-community.net/rytecBE_VL_Basic.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://epg.vuplus-community.net/rytecBE_VL_Basic.xz
[EPGImport] afterDownload /tmp/epgimport.xz
[EPGImport] unlink /tmp/epgimport.xz
[EPGImport] Downloading: http://epg.vuplus-community.net/rytec.channels.xml.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://epg.vuplus-community.net/rytec.channels.xml.xz
[EPGImport] afterChannelDownload /tmp/epgimport.xz
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/custom.channels.xml'
[EPGImport] Parsing channels from 'rytec.channels.xml.xz'
[EPGImport] INFO : final regex een\.be|play4\.be|ketnet\.be|een\.be compiled successfully.
[EPGImport] INFO : skipping play4.be due to channel_id_filter.conf
[EPGImport] INFO : skipping ketnet.be due to channel_id_filter.conf
[EPGImport] Parsing channels from 'rytec.channels.xml.xz'
[EPGImport] INFO : final regex een\.be|play4\.be|ketnet\.be|een\.be compiled successfully.
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping ketnet.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping ketnet.be due to channel_id_filter.conf
[EPGImport] INFO : skipping play4.be due to channel_id_filter.conf
[EPGImport] INFO : skipping play4.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping ketnet.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping ketnet.be due to channel_id_filter.conf
[EPGImport] INFO : skipping play4.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping ketnet.be due to channel_id_filter.conf
[EPGImport] INFO : skipping play4.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping ketnet.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping ketnet.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping ketnet.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping ketnet.be due to channel_id_filter.conf
Serviceref is in ignore list: 1:0:19:1B1D:802:2:11A0000:0:0:0:
[XMLTVConverter] Enumerating event information
Unknown channel:  play4.be
Unknown channel:  ketnet.be
Unknown channel:  eurosport1.be
Unknown channel:  mtv.be
Unknown channel:  fox.be
Unknown channel:  tlc.be
Unknown channel:  elevenproleague2nl.be
Unknown channel:  elevenproleague3nl.be
Unknown channel:  actua.be
Unknown channel:  studio100tv.be
Unknown channel:  een.be
Unknown channel:  avsgent.be
Unknown channel:  stories.be
Unknown channel:  njam.be
Unknown channel:  eurosport2.be
Unknown channel:  nicktoons.be
Unknown channel:  q-music.be
Unknown channel:  plattelandstv.be
Unknown channel:  kanaalz.be
Unknown channel:  playmorekicks.be
Unknown channel:  playmorecinema.be
Unknown channel:  playtime.be
Unknown channel:  playmoreblack.be
Unknown channel:  playsportsgolf.be
Unknown channel:  playsportshd1.be
Unknown channel:  playsportshd3.be
Unknown channel:  playsportshd2.be
Unknown channel:  playsportshd4.be
Unknown channel:  playsportshd5.be
Unknown channel:  playsportshd6.be
Unknown channel:  playsportshd7.be
Unknown channel:  playsportshd8.be
Unknown channel:  discoveryvlaanderen.be
Unknown channel:  disneyvl.be
Unknown channel:  nickelodeonspike.be
Unknown channel:  vlaamsparlement.tv
[EPGImport] ### thread is ready ### Events: 3440
[EPGImport] imported 3440 events
[EPGImport] Save last import date and count event
[EPGImport] Run check deep standby after import
[EPGImport] #### Finished ####

I defined in channel_id_filter.conf:

een\.be
Play4\.be
Ketnet\.be
een\.be

1. The first line is not treated.

No big problem.  Just ignore the first line.

When I just add something on the first line alone.  EPG is till imported.

 

2.  It filters also the custom.channels.xml file which it is not  meant to do:

You can see this in the log:

[EPGImport] Parsing channels from '/etc/epgimport/custom.channels.xml'
[EPGImport] Parsing channels from 'rytec.channels.xml.xz'
[EPGImport] INFO : final regex een\.be|play4\.be|ketnet\.be|een\.be compiled successfully.

It parses the the custom file first and then filters it.

 

3.  I re-added both Play4.be and Ketnet.be in the customs file.  But no EPG is imported.

<!-- 23.5E --><channel id="Play4.be">1:0:19:1B91:C88:3:EB0000:0:0:0:</channel><!-- Vier HD -->
<!-- 23.5E --><channel id="Ketnet.be">1:0:16:5278:C96:3:EB0000:0:0:0:</channel><!-- Ketnet -->

Also this can be circumvented by using another channel-id.  But I described above that local files should not be filtered.  And not touched.

 

So the filtering should occur before adding the custom.channels.xml or any local channels file.

 

Maybe it is too much to ask.

 

An idea:

In the log you see this:

[EPGImport] Parsing channels from '/etc/epgimport/custom.channels.xml'
[EPGImport] Parsing channels from 'rytec.channels.xml.xz'

Maybe you should reverse this order of parsing.

And filter just after the parsing of the downloaded file.

And only then parse the custom.channels.xml file.  Parsing the custom.channels.xml file last and only after the filtering.

 

The idea behind this is that you can add things from the custom.chanels.xml file.

But these are totally under your own control.  After all this file is something the user construct themselves, and has full control over it.

Edited by doglover, 5 April 2021 - 07:18.

[Pr2]

Hello Pr2,


Thanks for this change.


Tried it,

- for information EPGConfig.pyo is not created after reboot but after executing epgimport plugin and apply changes (green save button).

- applied some filters and everything seems to be fine


An advantage of your method is that we can filter channel ids available in rytec epg sources files and not yet added to rytec.channels.xml (hope I'm clear).


Is this change will be committed to oe-a branch ?


Thanks

 

Thanks for the feedback and I am happy that it fulfill your expectations.

I am not handling the XMLTVimport which is the OE-A version of EPGImport but some people keep them in sync.

This patch is still experimental and not yet included into EPGImport neither.

[Pr2]

Hi @doglover,

 

1. I am really surprises that you mention that the 1st line is not filtered your log doesn't show evidence of this problem.

We clearly saw that the full regex is:  een\.be|play4\.be|ketnet\.be|een\.be   so een\.be is included twice but I don't think that this is necessary.

 

Can you please try again and show me evidence that the 1st line is ignored?

 

2. As already explained by WanWizard it is not possible to make a distinction without changing all the EPGImport logic, the goal is rather to fix multi-language channels that have the same service reference. So I don't understand what you are trying to do with Play4.be and Ketnet.be  so filtering out channel to add them later on with the custom.channels.xml I don't see the point here.

 

We are not processing text file line by line but we are processing .xml file so we retrieve the different elements and it is when the channel id element is read that I decide to filter it out or not. So this process is the same if it is a downloaded file or a local file. There is one unique function to parse the xml data (which is nice).

[Pr2]

Be sure to download the very latest version because I commit the file several times yesterday.

[doglover]

I donloaded the .py file this morning.

I assume it is the correct one.

 

2. As already explained by WanWizard it is not possible to make a distinction without changing all the EPGImport logic, the goal is rather to fix multi-language channels that have the same service reference. So I don't understand what you are trying to do with Play4.be and Ketnet.be  so filtering out channel to add them later on with the custom.channels.xml I don't see the point here.

 

 

Will this was just to test if the custom.channels.xml file was also filtered.  Of course the example I used is stupid.

But it has uses.  f.i. removing it from the normal import, but allow it in the custom file to assign the EPG to an IPTV channel for instance.

Of course, the main objective is still full-filled.  i.e. filtering out multi-language assignments.

 

Will try with only one line in the filter file.

Edited by doglover, 5 April 2021 - 08:18.

[doglover]

Tried with only this:  een\.be

 

and no EPG on een.be.

Must have been a fluke before.

 

PS:  Restarted enigma2 before I tried anything.

 

Log:

[EPGImport] autostart (0) occured at 1617607366.01
[EPGImport] autostart (0) occured at 1617607367.37
[EPGImport] WakeUpTime now set to -1 (now=1617607367)
[EPGImport] Selected source:  ['Vlaanderen - Basis (xz)']
sourcesDone():  False None
[EPGImport] nextImport, source= Vlaanderen - Basis (xz)
[EPGImport] Downloading: http://epg.vuplus-community.net/rytecBE_VL_Basic.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://epg.vuplus-community.net/rytecBE_VL_Basic.xz
[EPGImport] afterDownload /tmp/epgimport.xz
[EPGImport] unlink /tmp/epgimport.xz
[EPGImport] Downloading: http://epg.vuplus-community.net/rytec.channels.xml.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://epg.vuplus-community.net/rytec.channels.xml.xz
[EPGImport] afterChannelDownload /tmp/epgimport.xz
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/custom.channels.xml'
[EPGImport] Parsing channels from 'rytec.channels.xml.xz'
[EPGImport] INFO : final regex een\.be compiled successfully.
[EPGImport] Parsing channels from 'rytec.channels.xml.xz'
[EPGImport] INFO : final regex een\.be compiled successfully.
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
[EPGImport] INFO : skipping een.be due to channel_id_filter.conf
Serviceref is in ignore list: 1:0:19:1B1D:802:2:11A0000:0:0:0:
[XMLTVConverter] Enumerating event information
Unknown channel:  eurosport1.be
Unknown channel:  mtv.be
Unknown channel:  fox.be
Unknown channel:  tlc.be
Unknown channel:  elevenproleague2nl.be
Unknown channel:  elevenproleague3nl.be
Unknown channel:  actua.be
Unknown channel:  studio100tv.be
Unknown channel:  een.be
Unknown channel:  avsgent.be
Unknown channel:  stories.be
Unknown channel:  njam.be
Unknown channel:  eurosport2.be
Unknown channel:  nicktoons.be
Unknown channel:  q-music.be
Unknown channel:  plattelandstv.be
Unknown channel:  kanaalz.be
Unknown channel:  playmorekicks.be
Unknown channel:  playmorecinema.be
Unknown channel:  playtime.be
Unknown channel:  playmoreblack.be
Unknown channel:  playsportsgolf.be
Unknown channel:  playsportshd1.be
Unknown channel:  playsportshd3.be
Unknown channel:  playsportshd2.be
Unknown channel:  playsportshd4.be
Unknown channel:  playsportshd5.be
Unknown channel:  playsportshd6.be
Unknown channel:  playsportshd7.be
Unknown channel:  playsportshd8.be
Unknown channel:  discoveryvlaanderen.be
Unknown channel:  disneyvl.be
Unknown channel:  nickelodeonspike.be
Unknown channel:  vlaamsparlement.tv
[EPGImport] ### thread is ready ### Events: 4016
[EPGImport] imported 4016 events
[EPGImport] Save last import date and count event
[EPGImport] Run check deep standby after import
[EPGImport] #### Finished ####

 

[Pr2]

Implementation is finished a PR is waiting approuval:

 

https://github.com/O...gimport/pull/49

[doglover]

Nice.  Thanks

[WanWizard]

As I mentioned on the PR:

 

Why so complex? Enigma already knows what languages you want in your EPG, so the filtering can be done automatically?

[jeepcook]

As I mentioned on the PR:

 

Why so complex? Enigma already knows what languages you want in your EPG, so the filtering can be done automatically?

These parameters have no effect on EPGImport.

[WanWizard]

I know, the remarks here are about the changes suggested by Pr2.

 

My question was why a separate filter mechanism with files that require specific OS knowledge to create if a user can already configure in the GUI which EPG languages the user wishes to see?

[jeepcook]

 

I know, the remarks here are about the changes suggested by Pr2.

 

My question was why a separate filter mechanism with files that require specific OS knowledge to create if a user can already configure in the GUI which EPG languages the user wishes to see?

I think it's a good idea but need an EPGImport dev with EPG gui language conf treatment for each source. EPGImport has a lot of sources with multiple language in each one. Different languages can be found for a channel in the same source file or in multiple source files.

[WanWizard]

I think you missed a part of the discussion.

 

This is about https://github.com/O...port/pull/49...

[jeepcook]

 

I think you missed a part of the discussion.

 

This is about https://github.com/O...port/pull/49...

I know ! I participated to the test with pr2!

[WanWizard]

Ok. So you should know what I mean.

 

95% of OpenPLi users have no clue how to create a file in the filesystem of the box, and add a regex (a what?) to filter on a specific language.

 

So my question was: why address this feature by using this for most users complicated method if the user can select which language they want in the GUI?

[jeepcook]

you're right, I know the difficulties for users to customize the os because they have no knowledge in unix/linux. A lot of users don't know for example they can customise EPGImport with custom channels or sources xml files.

 

I'm ok with you, that's why I said that the dev has to be done into the EPGImport code and gui adapted for users to customize the plugin configuration. But the evolution proposed by DogLover and Pr2 are very interesting.

[doglover]

The problem of channels which are shared by different providers (or from the same provider in different countries) for which the users want the EPG in their own language.

I tried to solve it by splitting up the sources.  The General files in each county source should not contain only channels for that country only.

Shared channels are in the Misc and Sport/Movie files (or named so in the local language).  Not importing these from another country should solve most of the problems. But some people insist of also importing these.

(Although some channels can reside in the wrong file - I do not know everything)

 

 

On a regular interval, I get questions that a channel have the wrong language EPG.  I each time try to explain they can omit importing the EPG for the offending language.

But this comes back time after time.

 

So I tried to think of a method to exclude some EPG on an individual base.  And this is what Pr2 has programmed to do.

 

Of course it is true that setting the EPG language you want should filter out and give preference to language you want.  And the XMLTV file contains that info.  But for one reason or another the language setting does not take effect on imported EPG.  I can not figure out why.  Nor do I have clue how this can be enforced.

So, I though of the next best thing, exclude some channel_id's (EPG references) from importing

 

And yes the system that Pr2 programmed uses regex.  But it is not as bad as it looks.

f.i. Animal Planet

<!-- DK --><!-- 0.8W --><channel id="AnimalPlanet.dk">1:0:1:4274:4:46:E080000:0:0:0:</channel><!-- Animal Planet -->
<!-- FI --><!-- 0.8W --><channel id="AnimalPlanet.fi">1:0:1:4274:4:46:E080000:0:0:0:</channel><!-- Animal Planet -->
<!-- NO --><!-- 0.8W --><channel id="AnimalPlanet.no">1:0:1:4274:4:46:E080000:0:0:0:</channel><!-- Animal Planet -->
<!-- SE --><!-- 0.8W --><channel id="AnimalPlanet.se">1:0:1:4274:4:46:E080000:0:0:0:</channel><!-- Animal Planet -->

All have the same service ref. 

 

If you want the Finnish EPG.  Then it depends on the order which you import the EPG.  The last one wins.  But requires that the rytec.sources.xml file is edited

(Of course you can choose not to import Danish, Swedish or Norwegian EPG)

 

With this filter you add to the "channel_id_filter.conf" file the following:

AnimalPlanet\.dk
AnimalPlanet\.no
AnimalPlanet\.se

This is the correct regex expression to achieve the desired effect.

However, the following does exactly the same.

AnimalPlanet.dk
AnimalPlanet.no
AnimalPlanet.se

In regex a . (period) just means any character.  So it also matches also the desired "channel-id"'s for exclusion.
And since no other combinations exist, it does simply what you want.

 

The knowledge of regex you have to know is very limited.

 

Of course if you want more complicated matters like excluding all French channels (.fr) is a bit more complicated.

But is a rather stupid example.  In this case just do not select anything to import from the French XMLTV files.

But hé, it can be done.