88 replies to this topic

[adriankoooo]

Running the latest OpenPli & EPGImport, but for some reason automatic EPG import at certain time is not working (manual update works correctly):

 

 IMG_1413.JPG   115.51KB   7 downloads

 

In /etc/enigma2/settings I have only 3 lines for epgimport:

 

config.plugins.epgimport.import_onlybouquet=true
config.plugins.epgimport.longDescDays=5
config.plugins.epgimport.wakeup=19:8

 

I tried to modify settings, but it is not saved - I think this is the reason why it is not working.

 

Any idea what cause the problem or if I can report the bug somewhere?

Edited by adriankoooo, 26 December 2021 - 19:27.

[doglover]

These lines for epgimport in settings are normal.

I have these:

config.plugins.epgimport.clear_oldepg=true
config.plugins.epgimport.import_onlybouquet=true
config.plugins.epgimport.longDescDays=7
config.plugins.epgimport.wakeup=9:10
config.plugins.extra_epgimport.last_import=Sun Dec 26 09:15:53 2021, 167379

But remember you can only edit the settings file with enigma2 stopped.  Changes you make here with enigma2 running, will be overwritten by what enigma2 has in memory.

So this is not the recommended method.

 

You better change the settings in the epgimport plug-in itself.

 

Why is it not importing automatically?

 

It has been a long time since I heard this happening.  The reason was that time that the internal clock was not set properly when the EPGimport plug-in first started up.

Our enigma receivers do not have an RTC clock.  The correct time has to come from the transponder the receiver was tuned to when starting up.

Now if you start receiver on an IP source as channel, the clock is never set properly, as no timing signal is present.

Also the time from the transponder could take some time to load in your receiver.  (5-10 seconds) and after EPGimport has already started.

This can cause EPGimport never to be activated, because the new start time is being calculated from 1 jan 1970 00:00 onwards.

 

The solution in that time was to use the plug-in system-time and sync your receiver from the internet with an NTP server.

Now this functionality is build into OpenPLi.  It is located on the menu under setting, System, Divers settings.  (see screen shot)

 

The standard values should work.  However I have a receiver where these do not work properly (on my other receiver they work just right).

So I changed the location of the NTP server to my router.  Most routers can act as NTP server.  Just try it.

 

You can test this in EPGimport.

Open EPGimport plug-in.  Go to log (Press menu on the RC)

The first line should read this:

[EPGImport] WakeUpTime now set to 1640673797 (now=1640592039)

The number are in Epoch time.  which can be converted to human readable time with

https://www.epochconverter.com/

You see the time when it is set (now) and the time of the next import.

Attached Files

•  4097_0_2_2_1_3200_0_0_0_0_20211227085528.jpg   78.54KB   4 downloads

Edited by doglover, 27 December 2021 - 09:06.

[adriankoooo]

Thanks for your reply. You gave me the hint to find the issue. I wanted to insert your config lines manually in settings, so I stopped enigma with init 4 command and opened settings file. And my settings were finally here. What, how?! I tested more and it looks like these settings are saved only when you restart enigma. I verified this by changing some settings. This should be indicated somewhere in the plugin. 

[Pr2]

The plugin is storing the setting into Enigma2 "memory", Enigma2 is flushing it to the flash when you shutdown or restart your box (init 4 is flushing the settings to flash).

If you power off the box without going first to standby your settings changes are not saved, this is the "normal" behavior of Enigma2 and is valid for your Enigma2 settings too, this is not specific to EPGImport.

You are not supposed to power off your box when while it is running.

Edited by Pr2, 27 December 2021 - 16:43.

[adriankoooo]

I am trying to use some advanced filtering in EPGImport. I found the corresponding code for importing from channel_id_filter.conf file:

 

 Screenshot 2022-01-03 at 17.40.41@2x.png   362.96KB   5 downloads

 

From the code you can see that it should log this process, but I can't find anything related in the log file (Blue button -> Log). Is it logging elsewhere?

 

I am trying to use this line, I checked on regex101 and it seem to be a valid regex.

 

 Screenshot 2022-01-03 at 17.52.46.png   67.66KB   5 downloads

Edited by adriankoooo, 3 January 2022 - 17:53.

[Pr2]

Don't care about regex just type in the name NationalGeographic.hu and it will do the job.

[adriankoooo]

Don't care about regex just type in the name NationalGeographic.hu and it will do the job.

 

I linked my other post, this is not always possible, example here: https://forums.openp...dpost&p=1416458

 

I am so curious why it is not logging as it should from the code. 

[doglover]

The logging only happens if you have a channel_id_filter.conf file in the /etyx/epgimport folder.

 

If this file is not present, this option does not do anything.  And nothing is logged.

[adriankoooo]

I have it and I think it is working, because I don't see wrong EPG in the results:

# ls /etc/epgimport/
EPGfilter.sh            FILTERpattern.txt       channel_id_filter.conf  ignore.conf             rytec.sources.xml

...but can't get logging to work. I am trying to understand the code of EPGImpot, but from what I see the logging should work, but actually isn't.

 

Log looks like this after manual importing:

[EPGImport] Selected source:  ['Magyarorszag - Basic (xz)']
sourcesDone():  False None
[EPGImport] nextImport, source= Magyarorszag - Basic (xz)
[EPGImport] Downloading: http://www.xmltvepg.nl/rytecHU_Basic.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://www.xmltvepg.nl/rytecHU_Basic.xz
[EPGImport] afterDownload /tmp/epgimport.xz
[EPGImport] unlink /tmp/epgimport.xz
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[XMLTVConverter] Enumerating event information
Unknown channel:  euronews.hu
Unknown channel:  animalplanet.hu
Unknown channel:  discoveryhdshowcase.hu
Unknown channel:  idxtra.hu
Unknown channel:  fix.hu
Unknown channel:  musicmix.hu
Unknown channel:  foodnetwork.hu
Unknown channel:  babytv.hu
Unknown channel:  hetitv.hu
Unknown channel:  fit.hu
Unknown channel:  sorozat+.hu
Unknown channel:  film+.hu
Unknown channel:  cartoonnetworktcm.hu
Unknown channel:  discoveryscience.hu
Unknown channel:  dtx.hu
Unknown channel:  id.hu
Unknown channel:  kiwitv.hu
[EPGImport] ### thread is ready ### Events: 22973
[EPGImport] imported 22973 events
[EPGImport] Save last import date and count event
[EPGImport] Run check deep standby after import
[EPGImport] #### Finished ####

channels.xml.xz is downloaded and deleted while import? It is downloaded at every import?

 

From my understanding channel_id_filter.conf is parsed (and should be logged) at every import.

Edited by adriankoooo, 3 January 2022 - 19:56.

[adriankoooo]

Looking at the code in EPGConfig.py:

 

 Screenshot 2022-01-03 at 19.59.01@2x.png   363.77KB   7 downloads

 

Function parse

def parse(self, filterCallback, downloadedFile, FilterChannelEnabled):
  print>>log, "[EPGImport] Parsing channels from '%s'" % self.name
  channel_id_filter = set_channel_id_filter()

should call the set_channel_id_filter() function to process channel_id_filter.conf, but I can't even see the line above in the log (Parsing channels from...). Anyone knows when this parse() function is called?

[Pr2]

What is the exact content of your file:

 

channel_id_filter.conf

 

I don't provide support by PM this is written in my signature.

[adriankoooo]

Oh no, I am really sorry. I haven't read it before.

 

channel_id_filter.conf:

# cat /etc/epgimport/channel_id_filter.conf
NationalGeographicWild.cz
\"NationalGeographic.hu\">1:0:19:77F0:2C3:600:E080000:0:0:0:
BBCEarth.hr
BBCEarth.nordic
BBCEarth.ro
BBCEarth.svn
EpicDrama.ro
EpicDrama.cz
NatGeoWildHD.dk
NatGeoWild.fi
NatGeoWildHD.no
NatGeoWild.ro
NatGeoWildHD.se
ViasatExplorer.ro
ViasatExplorer.cz
ViasatHistory.eu
ViasatHistory.ro
ViasatHistory.cz
ZoneReality.ro
CBSReality.eu
DuckTV.ro
JimJam.hr
Jimjam.ro
Boomerang.ro
HistoryEurope.eu
DaVinciLearning.ro
ViasatNature.ro
Eurosport1.ro
Eurosport2.ro
Nickelodeon.ro
TVR2.ro
AXNSciFi.ro
AXNCrime.ro
AnimalPlanet.ro
AXNBlack.hu
root@vuuno4k:/etc/epgimport#

For test I even tried to leave only last line in the file, but the result is the same.

 

 

Is the log file size limited? I have a feel that in Blue -> Show LOG I see only half of the log, because it start like this:

channel:  rik.cz
Unknown channel:  primaplus.sk
Unknown channel:  213.cz
Unknown channel:  ta3.sk
Unknown channel:  tv2.sk
Unknown channel:  jojplus1.sk
Unknown channel:  jojcinemaplus1.cz
Unknown channel:  markizaplus1.sk
[EPGImport] ### thread is ready ### Events: 8320
[EPGImport] nextImport, source= Ceska - Bazicky (xz)
[EPGImport] Downloading: http://www.xmltvepg.nl/rytecCZ_Basic.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://www.xmltvepg.nl/rytecCZ_Basic.xz... etc

Edited by adriankoooo, 3 January 2022 - 21:56.

[Pr2]

this entry is not valid:

 

\"NationalGeographic.hu\">1:0:19:77F0:2C3:600:E080000:0:0:0:

 

Simply use:

 

NationalGeographic.hu

 

The filtering is only based on the channel_id and not on the service reference.

Edited by Pr2, 3 January 2022 - 21:55.

[Pr2]

Please tell us what is your hardware: STB Brand and model and what is the version of OpenPLi that you are using?
 
It seems to me that you are not using OpenPLi 8.1 which is the latest one because your log miss some entries:

[EPGImport] ### thread is ready ### Events: 152459
[EPGImport] nextImport, source= UK/Ireland - Sky 1 (Live on sat) (xz)
[EPGImport] Downloading: http://....../rytecUK_SkyLive.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://..../rytecUK_SkyLive.xz
[EPGImport] afterDownload /tmp/epgimport.xz
[EPGImport] unlink /tmp/epgimport.xz
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/custom.channels.xml'
[EPGImport] Parsing channels from 'rytec.channels.xml.xz'
[EPGImport] INFO: no channel_id_filter.conf file found.
[XMLTVConverter] Enumerating event information

Why didn't I saw this in your log?

[EPGImport] Parsing channels from '/etc/epgimport/custom.channels.xml'
[EPGImport] Parsing channels from 'rytec.channels.xml.xz'

And you can see real EPGImport clearly display in my setup that:
INFO: no channel_id_filter.conf file found.

 

So if you remove the file you should also see this, if you don't you are not using the latest EPGImport.

Edited by Pr2, 3 January 2022 - 22:31.

[adriankoooo]

Please tell us what is your hardware: STB Brand and model and what is the version of OpenPLi that you are using?
 
It seems to me that you are not using OpenPLi 8.1 which is the latest one because your log miss some entries:

[EPGImport] ### thread is ready ### Events: 152459
[EPGImport] nextImport, source= UK/Ireland - Sky 1 (Live on sat) (xz)
[EPGImport] Downloading: http://....../rytecUK_SkyLive.xz to local path: /tmp/epgimport.xz
[EPGImport] checkValidServer serverurl http://..../rytecUK_SkyLive.xz
[EPGImport] afterDownload /tmp/epgimport.xz
[EPGImport] unlink /tmp/epgimport.xz
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/custom.channels.xml'
[EPGImport] Parsing channels from 'rytec.channels.xml.xz'
[EPGImport] INFO: no channel_id_filter.conf file found.
[XMLTVConverter] Enumerating event information

Why didn't I saw this in your log?

[EPGImport] Parsing channels from '/etc/epgimport/custom.channels.xml'
[EPGImport] Parsing channels from 'rytec.channels.xml.xz'

And you can see real EPGImport clearly display in my setup that:
INFO: no channel_id_filter.conf file found.

 

So if you remove the file you should also see this, if you don't you are not using the latest EPGImport.

 

VU Uno 4K:

root@vuuno4k:/etc/epgimport# cat /etc/issue
openpli 8.1-release \n \l
root@vuuno4k:/etc/epgimport# opkg list-installed | grep epgimport
enigma2-plugin-extensions-epgimport - 1.0+git207+f8be073-r0.5
enigma2-plugin-extensions-epgimport-rytec - 20210910-r0.0
root@vuuno4k:/etc/epgimport#

I think this is the latest version. Tomorrow I will continue investigating.

 

Why didn't I saw this in your log? That's what I'm trying to figure out. 

 

custom.channels.xml -> is not needed for channel id filtering, right?

[adriankoooo]

Ok, I think I figured it out. The file "custom.channels.xml" must be present, without it the channel_id_filter.conf is not readed.

 

After creating an empty custom.channels.xml:

 

 Screenshot 2022-01-04 at 00.08.46.png   362.53KB   4 downloads

 

Why?

def update(self, filterCallback, downloadedFile=None):
  customFile = '/etc/epgimport/custom.channels.xml'
  # Always read custom file since we don't know when it was last updated
  # and we don't have multiple download from server problem since it is always a local file.
  if os.path.exists(customFile):
   print>>log, "[EPGImport] Parsing channels from '%s'" % customFile
   self.parse(filterCallback, customFile, filterCustomChannel)
  if downloadedFile is not None:
   self.mtime = time.time()
   return self.parse(filterCallback, downloadedFile, True)
  elif (len(self.urls) == 1) and isLocalFile(self.urls[0]):
   mtime = os.path.getmtime(self.urls[0])
   if (not self.mtime) or (self.mtime < mtime):
    self.parse(filterCallback, self.urls[0], True)
    self.mtime = mtime

def parse(self, filterCallback, downloadedFile, FilterChannelEnabled):
  print>>log, "[EPGImport] Parsing channels from '%s'" % self.name
  channel_id_filter = set_channel_id_filter()

It is interesting why set_channel_id_filter() is running only when custom.channels.xml is exists, it should even when we are downloading the channels.xml (if downloadedFile && elif - in the code). I am only guessing, I am not expert in python.

[adriankoooo]

Oh no. After testing, I see final regex compiled successfully, but it is still not use the file. I don't see any "skipped xy line": https://pastebin.com/0qj5fdMZ (removed URL's for files)

 

I still have no idea why I get only half of the log. I have 9 source enabled.

Edited by adriankoooo, 4 January 2022 - 00:46.

[doglover]

The log is limited in size.

So, if you want to debug, import only one source.  And look then at the log.

At least I have to do it this way, to see the relevant info. 

\"NationalGeographic.hu\">1:0:19:77F0:2C3:600:E080000:0:0:0:

Is an entry which is possible in FILTERpattern.txt but not in channel_id_filter.conf

[Pr2]

If you start mixing up several filtering mechanism we will never figure out what is happening.

 

Flash an image do not restore anything just install EPGImport with a valid channel_id_filter.conf so just with channel_id and test. I am sure that filtering will do the job.

I can't understand why you have so many problem with your image, you probably install plugins that interfered with each other.

Keep it simple and everything will runs smoothly.

[adriankoooo]

Verdict:

• channel_id_filter.conf is working, filtering by channel id
• I think there would be a room for improvement in the future by adding option to filter by service id
• log size is limited (this would be awesome to increase or write to file immediately while importing)

For now, I keep using EPGFilter.sh script, it gives me more freedom and I can fine tune my settings.