164 replies to this topic

[littlesat]

[Huevos]

@Littlesat,

[stripReference(x) for x in BouquetChannelListList]

IMO, that list should be created once and maybe stored as a global. You don't want to run that code every time you test an individual service ref. I've got more than 3000 services in my bouquets and a typical XML download might have 5 to 10 thousand. So in theory your "stripReference" function is going to do some 30, million cycles when it really just needs to do a few thousand.

[WanWizard]

That was the point I was trying to make a few posts back.

[Huevos]

Also,surely, the proper way to do this is compare the numbers as integers, not try to do a string match. Then we wouldn't care less about upper/lower case or leading zeros, etc

Edited by Huevos, 20 June 2019 - 11:42.

[Huevos]

I'll post some code later.

[littlesat]

No need to do that.. in post #39 I already fixed that - as I also discovered it...

Better test it and when issues are discovered fix it..

 

I think string match here is more sufficient. No need to make it extra complex by converting the stuff to e.g. a list of integers and then compare lists- this is more complicated... The service references come as string with hexadecimal characters and : as seperator. We can consider to remove the ":".join(of course and then compare list of strings).

 

This is just an example how we can work together!

Edited by littlesat, 20 June 2019 - 12:14.

[littlesat]

This will also work indeed... no need to compile a string with join... and no need to convert stuff to integers...

 def channelFilter(ref):
+       def stripReference(ref):
+               try:
+                       return ref.upper().split(':')[3:7]
+               except:
+                       return ref

Edited by littlesat, 20 June 2019 - 12:16.

[littlesat]

Question is... is this faster than comparing strings??? I'm afraid it isn't so I think better go for #39

diff --git a/src/EPGImport/plugin.py b/src/EPGImport/plugin.py
index ef3cd15..d458ec4 100644
--- a/src/EPGImport/plugin.py
+++ b/src/EPGImport/plugin.py
@@ -173,20 +173,27 @@ def getBouquetChannelList():
 
 # Filter servicerefs that this box can display by starting a fake recording.
 def channelFilter(ref):
+
+       def stripReference(ref):
+               try:
+                       return ref.upper().split(':')[3:7]
+               except:
+                       return ref
+
        if not ref:
                return False
        sref = enigma.eServiceReference(ref)
-       refstr = ':'.join(sref.toString().split(':')[:11])
+       refstr = stripReference(':'.join(sref.toString().split(':')[:11]))
        if config.plugins.epgimport.import_onlybouquet.value:
                global BouquetChannelListList
                if BouquetChannelListList is None:
-                       BouquetChannelListList = getBouquetChannelList()
+                       BouquetChannelListList = [stripReference(x) for x in getBouquetChannelList()]
                if refstr not in BouquetChannelListList:
                        print>>log, "Serviceref not in bouquets:", refstr
                        return False
        global serviceIgnoreList
        if serviceIgnoreList is None:
-               serviceIgnoreList = filtersServices.filtersServicesList.servicesList()
+               serviceIgnoreList = [stripReference(x) for x in filtersServices.filtersServicesList.servicesList()]
        if refstr in serviceIgnoreList:
                print>>log, "Serviceref is in ignore list:", refstr
                return False

Attached Files

•  plugin.py   45.19KB   1 downloads

Edited by littlesat, 20 June 2019 - 12:21.

[doglover]

Test performed on OpenPLi7.0 installed on a Zgemma StarS - a very slow machine

 

Test 1:  Import normally (no use of the bouquet filtering)

 

[EPGImport] ### thread is ready ### Events: 177460

Import was normal. 

(So nothing broken at first sight)

 

Test 2:  Deleted a bunch of bouquets, and activated the option " Only import EPG for channels in Bouquets"

 

First observation:  At the beginning of the import the spinner started running for half a minute approx.

Secondly:  Once the import started,  The import went through it fairly fast.  (A lot less to import)

Although sometimes it seems that the import hangs, because the counter is not running.  This is normal as it processes data for channels which are not in bouquets.

 

 

In the log appeared a bunch of these:

Serviceref not in bouquets: FF5:3070:13E:820000
Serviceref not in bouquets: FF1:3070:13E:820000
Serviceref not in bouquets: FF2:3070:13E:820000
Serviceref not in bouquets: FF3:3070:13E:820000
Serviceref not in bouquets: FF4:3070:13E:820000

 

[EPGImport] ### thread is ready ### Events: 26138

 

A lot less is imported

But the channels in the bouquets have EPG.  And none of the channels f.i. on 28.2E  ( I removed all bouquets which had channels from 28.2 E) have EPG imported which is the desired effect.

 

Test 3:  Same as Test 2 but with changed service ref - replaced 19 with 1F, and activated the option " Only import EPG for channels in Bouquets"

 

Same result:  [EPGImport] ### thread is ready ### Events: 26138

And all changed channels have EPG

 

 

Conclusions:

 

1. The desired effect is obtained.

2. Although you lose a bit of time at the start of the import,  You gain this multiple times back by not importing unwanted EPG.

3. This are only 3 simple tests.  Hopefully other problems do not crop up when using this.

4. I am a happy bunny.  It will save me a lot of time explaining to users why channels are missing when importing EPG.

 

Willy

[littlesat]

Did you test with changes from post #48? So this means I can commit?

Edited by littlesat, 20 June 2019 - 14:13.

[littlesat]

At the beginning of the import the spinner started running for half a minute approx.

 

->

 

Maybe we need to 'discover' in which phase you see this spinner...

Edited by littlesat, 20 June 2019 - 14:25.

[doglover]

Did you test with changes from post #48? So this means I can commit?

 

With the plugin.py from post 39

 

Will rerun the second test with the plug-in from #48.  (Did not see a file was attached)

 

Willy

[littlesat]

What also can be considered is strip the reference when they are collected... this reduces an additional looping

[WanWizard]

Which is what I have been saying all along. Now you still strip and strip again every time the code loops...

[littlesat]

Better....? This saves a loop....  Indeed this might be a long long long loop with spinners... Maybe it can also streamlined a bit more....

diff --git a/src/EPGImport/plugin.py b/src/EPGImport/plugin.py
index ef3cd15..5a44cf4 100644
--- a/src/EPGImport/plugin.py
+++ b/src/EPGImport/plugin.py
@@ -140,11 +140,11 @@ def getBouquetChannelList():
                                                                                altrernative_list = getAlternatives(service)
                                                                                if altrernative_list:
                                                                                        for channel in altrernative_list:
-                                                                                               refstr = ':'.join(channel.split(':')[:11])
+                                                                                               refstr = service.toString().upper().split(':')[3:7]
                                                                                                if refstr not in channels:
                                                                                                        channels.append(refstr)
                                                                        else:
-                                                                               refstr = ':'.join(service.toString().split(':')[:11])
+                                                                               refstr = service.toString().upper().split(':')[3:7]
                                                                                if refstr not in channels:
                                                                                        channels.append(refstr)
        else:
@@ -161,11 +161,11 @@ def getBouquetChannelList():
                                                altrernative_list = getAlternatives(service)
                                                if altrernative_list:
                                                        for channel in altrernative_list:
-                                                               refstr = ':'.join(channel.split(':')[:11])
+                                                               refstr = channel.upper().split(':')[3:7])
                                                                if refstr not in channels:
                                                                        channels.append(refstr)
                                        else:
-                                               refstr = ':'.join(service.toString().split(':')[:11])
+                                               refstr = service.toString().upper().split(':')[3:7])
                                                if refstr not in channels:
                                                        channels.append(refstr)
        isFilterRunning = 0
@@ -176,7 +176,7 @@ def channelFilter(ref):
        if not ref:
                return False
        sref = enigma.eServiceReference(ref)
-       refstr = ':'.join(sref.toString().split(':')[:11])
+       refstr = sref.toString().upper().split(':')[3:7]
        if config.plugins.epgimport.import_onlybouquet.value:
                global BouquetChannelListList
                if BouquetChannelListList is None:
@@ -186,7 +186,7 @@ def channelFilter(ref):
                        return False
        global serviceIgnoreList
        if serviceIgnoreList is None:
-               serviceIgnoreList = filtersServices.filtersServicesList.servicesList()
+               serviceIgnoreList = [x.upper().split(':')[3:7] for x in filtersServices.filtersServicesList.servicesList()]
        if refstr in serviceIgnoreList:
                print>>log, "Serviceref is in ignore list:", refstr
                return False

Attached Files

•  plugin.py   45.05KB   4 downloads

Edited by littlesat, 20 June 2019 - 14:47.

[doglover]

plugin.py from #48

 

No spinner now.

Box remains responsive.

Same amount of shows imported.

 

Which one of the plugin.py should I test from #55

(I cannot keep up with testing.  Each time the box needs a restart.)

 

 

Willy

[doglover]

plugin.py from #55 will not compile after a restart.

Also EPGimport is gone from the blue button.

 

Willy

[WanWizard]

It had two copy/paste errors. This one better?

Attached Files

•  plugin.py   45.05KB   5 downloads

[littlesat]

I already see that #55 has syntax errors...

This one is a try with gutemine's all services retreiver...

diff --git a/src/EPGImport/plugin.py b/src/EPGImport/plugin.py
index ef3cd15..6fa7904 100644
--- a/src/EPGImport/plugin.py
+++ b/src/EPGImport/plugin.py
@@ -111,72 +111,23 @@ def getAlternatives(service):
              return alternativeServices and alternativeServices.getContent("S", True)
 
 def getBouquetChannelList():
-             channels = [ ]
+            allservices = []
              global isFilterRunning, filterCounter
              isFilterRunning = 1
-             serviceHandler = enigma.eServiceCenter.getInstance()
-             mask = (enigma.eServiceReference.isMarker | enigma.eServiceReference.isDirectory)
-             altrernative = enigma.eServiceReference.isGroup
-             if config.usage.multibouquet.value:
-                           bouquet_rootstr = '1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "bouquets.tv" ORDER BY bouquet'
-                           bouquet_root = enigma.eServiceReference(bouquet_rootstr)
-                           list = serviceHandler.list(bouquet_root)
-                           if list:
-                                         while True:
-                                                       s = list.getNext()
-                                                       if not s.valid():
-                                                                     break
-                                                       if s.flags & enigma.eServiceReference.isDirectory:
-                                                                     info = serviceHandler.info(s)
-                                                                     if info:
-                                                                                    clist = serviceHandler.list(s)
-                                                                                    if clist:
-                                                                                                  while True:
-                                                                                                                service = clist.getNext()
-                                                                                                                filterCounter += 1
-                                                                                                                if not service.valid(): break
-                                                                                                                if not (service.flags & mask):
-                                                                                                                              if service.flags & altrernative:
-                                                                                                                                            altrernative_list = getAlternatives(service)
-                                                                                                                                            if altrernative_list:
-                                                                                                                                                          for channel in altrernative_list:
-                                                                                                                                                                        refstr = ':'.join(channel.split(':')[:11])
-                                                                                                                                                                        if refstr not in channels:
-                                                                                                                                                                                      channels.append(refstr)
-                                                                                                                              else:
-                                                                                                                                            refstr = ':'.join(service.toString().split(':')[:11])
-                                                                                                                                            if refstr not in channels:
-                                                                                                                                                          channels.append(refstr)
-             else:
-                           bouquet_rootstr = '1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.favourites.tv" ORDER BY bouquet'
-                           bouquet_root = enigma.eServiceReference(bouquet_rootstr)
-                           services = serviceHandler.list(bouquet_root)
-                           if not services is None:
-                                         while True:
-                                                       service = services.getNext()
-                                                       filterCounter += 1
-                                                       if not service.valid(): break
-                                                       if not (service.flags & mask):
-                                                                     if service.flags & altrernative:
-                                                                                    altrernative_list = getAlternatives(service)
-                                                                                    if altrernative_list:
-                                                                                                  for channel in altrernative_list:
-                                                                                                                refstr = ':'.join(channel.split(':')[:11])
-                                                                                                                if refstr not in channels:
-                                                                                                                              channels.append(refstr)
-                                                                     else:
-                                                                                    refstr = ':'.join(service.toString().split(':')[:11])
-                                                                                    if refstr not in channels:
-                                                                                                  channels.append(refstr)
+            services = enigma.eServiceCenter.getInstance().list(eServiceReference('1:7:1:0:0:0:0:0:0:0:(type == 1) || (type == 17) || (type == 22) || (type == 25) || (type == 31) || (type == 134) || (type == 195) ORDER BY name'))
+            channels = services and services.getContent("SN", True)
+            for channel in channels:
+                          filterCounter += 1
+                          allservices.append(channel[0].upper().split(':')[3:7])
              isFilterRunning = 0
-             return channels
+            return allservices
 
 # Filter servicerefs that this box can display by starting a fake recording.
 def channelFilter(ref):
              if not ref:
                            return False
              sref = enigma.eServiceReference(ref)
-             refstr = ':'.join(sref.toString().split(':')[:11])
+            refstr = sref.toString().upper().split(':')[3:7]
              if config.plugins.epgimport.import_onlybouquet.value:
                            global BouquetChannelListList
                            if BouquetChannelListList is None:
@@ -186,7 +137,7 @@ def channelFilter(ref):
                                          return False
              global serviceIgnoreList
              if serviceIgnoreList is None:
-                           serviceIgnoreList = filtersServices.filtersServicesList.servicesList()
+                          serviceIgnoreList = [x.upper().split(':')[3:7] for x in filtersServices.filtersServicesList.servicesList()]
              if refstr in serviceIgnoreList:
                            print>>log, "Serviceref is in ignore list:", refstr
                            return False

Attached Files

•  plugin.py   43.37KB   3 downloads

Edited by littlesat, 20 June 2019 - 15:13.

[littlesat]

And from post 55 the syntax error fixed.... (but with our all channel list grabber)

P.S. when Gutemine's all channel grabber works I prefer his...

 

diff --git a/src/EPGImport/plugin.py b/src/EPGImport/plugin.py
index ef3cd15..068959c 100644
--- a/src/EPGImport/plugin.py
+++ b/src/EPGImport/plugin.py
@@ -140,11 +140,11 @@ def getBouquetChannelList():
                                                                                altrernative_list = getAlternatives(service)
                                                                                if altrernative_list:
                                                                                        for channel in altrernative_list:
-                                                                                               refstr = ':'.join(channel.split(':')[:11])
+                                                                                               refstr = service.toString().upper().split(':')[3:7]
                                                                                                if refstr not in channels:
                                                                                                        channels.append(refstr)
                                                                        else:
-                                                                               refstr = ':'.join(service.toString().split(':')[:11])
+                                                                               refstr = service.toString().upper().split(':')[3:7]
                                                                                if refstr not in channels:
                                                                                        channels.append(refstr)
        else:
@@ -161,11 +161,11 @@ def getBouquetChannelList():
                                                altrernative_list = getAlternatives(service)
                                                if altrernative_list:
                                                        for channel in altrernative_list:
-                                                               refstr = ':'.join(channel.split(':')[:11])
+                                                               refstr = channel.upper().split(':')[3:7]
                                                                if refstr not in channels:
                                                                        channels.append(refstr)
                                        else:
-                                               refstr = ':'.join(service.toString().split(':')[:11])
+                                               refstr = service.toString().upper().split(':')[3:7]
                                                if refstr not in channels:
                                                        channels.append(refstr)
        isFilterRunning = 0
@@ -176,7 +176,7 @@ def channelFilter(ref):
        if not ref:
                return False
        sref = enigma.eServiceReference(ref)
-       refstr = ':'.join(sref.toString().split(':')[:11])
+       refstr = sref.toString().upper().split(':')[3:7]
        if config.plugins.epgimport.import_onlybouquet.value:
                global BouquetChannelListList
                if BouquetChannelListList is None:
@@ -186,7 +186,7 @@ def channelFilter(ref):
                        return False
        global serviceIgnoreList
        if serviceIgnoreList is None:
-               serviceIgnoreList = filtersServices.filtersServicesList.servicesList()
+               serviceIgnoreList = [x.upper().split(':')[3:7] for x in filtersServices.filtersServicesList.servicesList()]
        if refstr in serviceIgnoreList:
                print>>log, "Serviceref is in ignore list:", refstr
                return False

Attached Files

•  plugin.py   45.05KB   4 downloads