Jump to content


Photo

Only import EPG for channels in Bouquets - change requested


  • Please log in to reply
164 replies to this topic

Re: Only import EPG for channels in Bouquets - change requested #41 littlesat

  • PLi® Core member
  • 56,123 posts

+685
Excellent

Posted 20 June 2019 - 10:59

:)


WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: Only import EPG for channels in Bouquets - change requested #42 Huevos

  • PLi® Contributor
  • 4,229 posts

+158
Excellent

Posted 20 June 2019 - 11:35

@Littlesat,

[stripReference(x) for x in BouquetChannelListList]

IMO, that list should be created once and maybe stored as a global. You don't want to run that code every time you test an individual service ref. I've got more than 3000 services in my bouquets and a typical XML download might have 5 to 10 thousand. So in theory your "stripReference" function is going to do some 30, million cycles when it really just needs to do a few thousand.



Re: Only import EPG for channels in Bouquets - change requested #43 WanWizard

  • PLi® Core member
  • 68,303 posts

+1,718
Excellent

Posted 20 June 2019 - 11:40

That was the point I was trying to make a few posts back. ;)


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: Only import EPG for channels in Bouquets - change requested #44 Huevos

  • PLi® Contributor
  • 4,229 posts

+158
Excellent

Posted 20 June 2019 - 11:41

Also,surely, the proper way to do this is compare the numbers as integers, not try to do a string match. Then we wouldn't care less about upper/lower case or leading zeros, etc


Edited by Huevos, 20 June 2019 - 11:42.


Re: Only import EPG for channels in Bouquets - change requested #45 Huevos

  • PLi® Contributor
  • 4,229 posts

+158
Excellent

Posted 20 June 2019 - 12:01

I'll post some code later.



Re: Only import EPG for channels in Bouquets - change requested #46 littlesat

  • PLi® Core member
  • 56,123 posts

+685
Excellent

Posted 20 June 2019 - 12:08

No need to do that.. in post #39 I already fixed that - as I also discovered it...

Better test it and when issues are discovered fix it..

 

I think string match here is more sufficient. No need to make it extra complex by converting the stuff to e.g. a list of integers and then compare lists- this is more complicated... The service references come as string with hexadecimal characters and : as seperator. We can consider to remove the ":".join(of course and then compare list of strings).

 

This is just an example how we can work together!


Edited by littlesat, 20 June 2019 - 12:14.

WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: Only import EPG for channels in Bouquets - change requested #47 littlesat

  • PLi® Core member
  • 56,123 posts

+685
Excellent

Posted 20 June 2019 - 12:14

This will also work indeed... no need to compile a string with join... and no need to convert stuff to integers...

 def channelFilter(ref):
+       def stripReference(ref):
+               try:
+                       return ref.upper().split(':')[3:7]
+               except:
+                       return ref

Edited by littlesat, 20 June 2019 - 12:16.

WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: Only import EPG for channels in Bouquets - change requested #48 littlesat

  • PLi® Core member
  • 56,123 posts

+685
Excellent

Posted 20 June 2019 - 12:18

Question is... is this faster than comparing strings??? I'm afraid it isn't so I think better go for #39

diff --git a/src/EPGImport/plugin.py b/src/EPGImport/plugin.py
index ef3cd15..d458ec4 100644
--- a/src/EPGImport/plugin.py
+++ b/src/EPGImport/plugin.py
@@ -173,20 +173,27 @@ def getBouquetChannelList():
 
 # Filter servicerefs that this box can display by starting a fake recording.
 def channelFilter(ref):
+
+       def stripReference(ref):
+               try:
+                       return ref.upper().split(':')[3:7]
+               except:
+                       return ref
+
        if not ref:
                return False
        sref = enigma.eServiceReference(ref)
-       refstr = ':'.join(sref.toString().split(':')[:11])
+       refstr = stripReference(':'.join(sref.toString().split(':')[:11]))
        if config.plugins.epgimport.import_onlybouquet.value:
                global BouquetChannelListList
                if BouquetChannelListList is None:
-                       BouquetChannelListList = getBouquetChannelList()
+                       BouquetChannelListList = [stripReference(x) for x in getBouquetChannelList()]
                if refstr not in BouquetChannelListList:
                        print>>log, "Serviceref not in bouquets:", refstr
                        return False
        global serviceIgnoreList
        if serviceIgnoreList is None:
-               serviceIgnoreList = filtersServices.filtersServicesList.servicesList()
+               serviceIgnoreList = [stripReference(x) for x in filtersServices.filtersServicesList.servicesList()]
        if refstr in serviceIgnoreList:
                print>>log, "Serviceref is in ignore list:", refstr
                return False

Attached Files


Edited by littlesat, 20 June 2019 - 12:21.

WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: Only import EPG for channels in Bouquets - change requested #49 doglover

  • Rytec EPG Team
  • 16,974 posts

+635
Excellent

Posted 20 June 2019 - 13:28

Test performed on OpenPLi7.0 installed on a Zgemma StarS - a very slow machine

 

Test 1:  Import normally (no use of the bouquet filtering)

 

[EPGImport] ### thread is ready ### Events: 177460

Import was normal. 

(So nothing broken at first sight)

 

Test 2:  Deleted a bunch of bouquets, and activated the option " Only import EPG for channels in Bouquets"

 

First observation:  At the beginning of the import the spinner started running for half a minute approx.

Secondly:  Once the import started,  The import went through it fairly fast.  (A lot less to import)

Although sometimes it seems that the import hangs, because the counter is not running.  This is normal as it processes data for channels which are not in bouquets.

 

 

In the log appeared a bunch of these:

Serviceref not in bouquets: FF5:3070:13E:820000
Serviceref not in bouquets: FF1:3070:13E:820000
Serviceref not in bouquets: FF2:3070:13E:820000
Serviceref not in bouquets: FF3:3070:13E:820000
Serviceref not in bouquets: FF4:3070:13E:820000

 

[EPGImport] ### thread is ready ### Events: 26138

 

A lot less is imported

But the channels in the bouquets have EPG.  And none of the channels f.i. on 28.2E  ( I removed all bouquets which had channels from 28.2 E) have EPG imported which is the desired effect.

 

Test 3:  Same as Test 2 but with changed service ref - replaced 19 with 1F, and activated the option " Only import EPG for channels in Bouquets"

 

Same result:  [EPGImport] ### thread is ready ### Events: 26138

And all changed channels have EPG

 

 

Conclusions:

 

1. The desired effect is obtained.

2. Although you lose a bit of time at the start of the import,  You gain this multiple times back by not importing unwanted EPG.

3. This are only 3 simple tests.  Hopefully other problems do not crop up when using this.

4. I am a happy bunny.  It will save me a lot of time explaining to users why channels are missing when importing EPG.

 

Willy


~~Rytec Team~~
Maxytec Multibox SE OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: Only import EPG for channels in Bouquets - change requested #50 littlesat

  • PLi® Core member
  • 56,123 posts

+685
Excellent

Posted 20 June 2019 - 14:12

Did you test with changes from post #48? So this means I can commit?


Edited by littlesat, 20 June 2019 - 14:13.

WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: Only import EPG for channels in Bouquets - change requested #51 littlesat

  • PLi® Core member
  • 56,123 posts

+685
Excellent

Posted 20 June 2019 - 14:14

At the beginning of the import the spinner started running for half a minute approx.

 

->

 

Maybe we need to 'discover' in which phase you see this spinner...


Edited by littlesat, 20 June 2019 - 14:25.

WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: Only import EPG for channels in Bouquets - change requested #52 doglover

  • Rytec EPG Team
  • 16,974 posts

+635
Excellent

Posted 20 June 2019 - 14:20

Did you test with changes from post #48? So this means I can commit?

 

With the plugin.py from post 39

 

Will rerun the second test with the plug-in from #48.  (Did not see a file was attached)

 

Willy


~~Rytec Team~~
Maxytec Multibox SE OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: Only import EPG for channels in Bouquets - change requested #53 littlesat

  • PLi® Core member
  • 56,123 posts

+685
Excellent

Posted 20 June 2019 - 14:30

What also can be considered is strip the reference when they are collected... this reduces an additional looping


WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: Only import EPG for channels in Bouquets - change requested #54 WanWizard

  • PLi® Core member
  • 68,303 posts

+1,718
Excellent

Posted 20 June 2019 - 14:36

Which is what I have been saying all along. Now you still strip and strip again every time the code loops...


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: Only import EPG for channels in Bouquets - change requested #55 littlesat

  • PLi® Core member
  • 56,123 posts

+685
Excellent

Posted 20 June 2019 - 14:38

Better....? This saves a loop.... ;) Indeed this might be a long long long loop with spinners... Maybe it can also streamlined a bit more....

diff --git a/src/EPGImport/plugin.py b/src/EPGImport/plugin.py
index ef3cd15..5a44cf4 100644
--- a/src/EPGImport/plugin.py
+++ b/src/EPGImport/plugin.py
@@ -140,11 +140,11 @@ def getBouquetChannelList():
                                                                                altrernative_list = getAlternatives(service)
                                                                                if altrernative_list:
                                                                                        for channel in altrernative_list:
-                                                                                               refstr = ':'.join(channel.split(':')[:11])
+                                                                                               refstr = service.toString().upper().split(':')[3:7]
                                                                                                if refstr not in channels:
                                                                                                        channels.append(refstr)
                                                                        else:
-                                                                               refstr = ':'.join(service.toString().split(':')[:11])
+                                                                               refstr = service.toString().upper().split(':')[3:7]
                                                                                if refstr not in channels:
                                                                                        channels.append(refstr)
        else:
@@ -161,11 +161,11 @@ def getBouquetChannelList():
                                                altrernative_list = getAlternatives(service)
                                                if altrernative_list:
                                                        for channel in altrernative_list:
-                                                               refstr = ':'.join(channel.split(':')[:11])
+                                                               refstr = channel.upper().split(':')[3:7])
                                                                if refstr not in channels:
                                                                        channels.append(refstr)
                                        else:
-                                               refstr = ':'.join(service.toString().split(':')[:11])
+                                               refstr = service.toString().upper().split(':')[3:7])
                                                if refstr not in channels:
                                                        channels.append(refstr)
        isFilterRunning = 0
@@ -176,7 +176,7 @@ def channelFilter(ref):
        if not ref:
                return False
        sref = enigma.eServiceReference(ref)
-       refstr = ':'.join(sref.toString().split(':')[:11])
+       refstr = sref.toString().upper().split(':')[3:7]
        if config.plugins.epgimport.import_onlybouquet.value:
                global BouquetChannelListList
                if BouquetChannelListList is None:
@@ -186,7 +186,7 @@ def channelFilter(ref):
                        return False
        global serviceIgnoreList
        if serviceIgnoreList is None:
-               serviceIgnoreList = filtersServices.filtersServicesList.servicesList()
+               serviceIgnoreList = [x.upper().split(':')[3:7] for x in filtersServices.filtersServicesList.servicesList()]
        if refstr in serviceIgnoreList:
                print>>log, "Serviceref is in ignore list:", refstr
                return False

Attached Files


Edited by littlesat, 20 June 2019 - 14:47.

WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: Only import EPG for channels in Bouquets - change requested #56 doglover

  • Rytec EPG Team
  • 16,974 posts

+635
Excellent

Posted 20 June 2019 - 14:48

plugin.py from #48

 

No spinner now.

Box remains responsive.

Same amount of shows imported.

 

Which one of the plugin.py should I test from #55

(I cannot keep up with testing.  Each time the box needs a restart.)

 

 

Willy


~~Rytec Team~~
Maxytec Multibox SE OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: Only import EPG for channels in Bouquets - change requested #57 doglover

  • Rytec EPG Team
  • 16,974 posts

+635
Excellent

Posted 20 June 2019 - 14:53

plugin.py from #55 will not compile after a restart.

Also EPGimport is gone from the blue button.

 

Willy


~~Rytec Team~~
Maxytec Multibox SE OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: Only import EPG for channels in Bouquets - change requested #58 WanWizard

  • PLi® Core member
  • 68,303 posts

+1,718
Excellent

Posted 20 June 2019 - 15:07

It had two copy/paste errors. This one better?

Attached Files


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: Only import EPG for channels in Bouquets - change requested #59 littlesat

  • PLi® Core member
  • 56,123 posts

+685
Excellent

Posted 20 June 2019 - 15:11

I already see that #55 has syntax errors... ;)

This one is a try with gutemine's all services retreiver...

diff --git a/src/EPGImport/plugin.py b/src/EPGImport/plugin.py
index ef3cd15..6fa7904 100644
--- a/src/EPGImport/plugin.py
+++ b/src/EPGImport/plugin.py
@@ -111,72 +111,23 @@ def getAlternatives(service):
              return alternativeServices and alternativeServices.getContent("S", True)
 
 def getBouquetChannelList():
-             channels = [ ]
+            allservices = []
              global isFilterRunning, filterCounter
              isFilterRunning = 1
-             serviceHandler = enigma.eServiceCenter.getInstance()
-             mask = (enigma.eServiceReference.isMarker | enigma.eServiceReference.isDirectory)
-             altrernative = enigma.eServiceReference.isGroup
-             if config.usage.multibouquet.value:
-                           bouquet_rootstr = '1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "bouquets.tv" ORDER BY bouquet'
-                           bouquet_root = enigma.eServiceReference(bouquet_rootstr)
-                           list = serviceHandler.list(bouquet_root)
-                           if list:
-                                         while True:
-                                                       s = list.getNext()
-                                                       if not s.valid():
-                                                                     break
-                                                       if s.flags & enigma.eServiceReference.isDirectory:
-                                                                     info = serviceHandler.info(s)
-                                                                     if info:
-                                                                                    clist = serviceHandler.list(s)
-                                                                                    if clist:
-                                                                                                  while True:
-                                                                                                                service = clist.getNext()
-                                                                                                                filterCounter += 1
-                                                                                                                if not service.valid(): break
-                                                                                                                if not (service.flags & mask):
-                                                                                                                              if service.flags & altrernative:
-                                                                                                                                            altrernative_list = getAlternatives(service)
-                                                                                                                                            if altrernative_list:
-                                                                                                                                                          for channel in altrernative_list:
-                                                                                                                                                                        refstr = ':'.join(channel.split(':')[:11])
-                                                                                                                                                                        if refstr not in channels:
-                                                                                                                                                                                      channels.append(refstr)
-                                                                                                                              else:
-                                                                                                                                            refstr = ':'.join(service.toString().split(':')[:11])
-                                                                                                                                            if refstr not in channels:
-                                                                                                                                                          channels.append(refstr)
-             else:
-                           bouquet_rootstr = '1:7:1:0:0:0:0:0:0:0:FROM BOUQUET "userbouquet.favourites.tv" ORDER BY bouquet'
-                           bouquet_root = enigma.eServiceReference(bouquet_rootstr)
-                           services = serviceHandler.list(bouquet_root)
-                           if not services is None:
-                                         while True:
-                                                       service = services.getNext()
-                                                       filterCounter += 1
-                                                       if not service.valid(): break
-                                                       if not (service.flags & mask):
-                                                                     if service.flags & altrernative:
-                                                                                    altrernative_list = getAlternatives(service)
-                                                                                    if altrernative_list:
-                                                                                                  for channel in altrernative_list:
-                                                                                                                refstr = ':'.join(channel.split(':')[:11])
-                                                                                                                if refstr not in channels:
-                                                                                                                              channels.append(refstr)
-                                                                     else:
-                                                                                    refstr = ':'.join(service.toString().split(':')[:11])
-                                                                                    if refstr not in channels:
-                                                                                                  channels.append(refstr)
+            services = enigma.eServiceCenter.getInstance().list(eServiceReference('1:7:1:0:0:0:0:0:0:0:(type == 1) || (type == 17) || (type == 22) || (type == 25) || (type == 31) || (type == 134) || (type == 195) ORDER BY name'))
+            channels = services and services.getContent("SN", True)
+            for channel in channels:
+                          filterCounter += 1
+                          allservices.append(channel[0].upper().split(':')[3:7])
              isFilterRunning = 0
-             return channels
+            return allservices
 
 # Filter servicerefs that this box can display by starting a fake recording.
 def channelFilter(ref):
              if not ref:
                            return False
              sref = enigma.eServiceReference(ref)
-             refstr = ':'.join(sref.toString().split(':')[:11])
+            refstr = sref.toString().upper().split(':')[3:7]
              if config.plugins.epgimport.import_onlybouquet.value:
                            global BouquetChannelListList
                            if BouquetChannelListList is None:
@@ -186,7 +137,7 @@ def channelFilter(ref):
                                          return False
              global serviceIgnoreList
              if serviceIgnoreList is None:
-                           serviceIgnoreList = filtersServices.filtersServicesList.servicesList()
+                          serviceIgnoreList = [x.upper().split(':')[3:7] for x in filtersServices.filtersServicesList.servicesList()]
              if refstr in serviceIgnoreList:
                            print>>log, "Serviceref is in ignore list:", refstr
                            return False

Attached Files


Edited by littlesat, 20 June 2019 - 15:13.

WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: Only import EPG for channels in Bouquets - change requested #60 littlesat

  • PLi® Core member
  • 56,123 posts

+685
Excellent

Posted 20 June 2019 - 15:18

And from post 55 the syntax error fixed.... (but with our all channel list grabber)

P.S. when Gutemine's all channel grabber works I prefer his...

 

diff --git a/src/EPGImport/plugin.py b/src/EPGImport/plugin.py
index ef3cd15..068959c 100644
--- a/src/EPGImport/plugin.py
+++ b/src/EPGImport/plugin.py
@@ -140,11 +140,11 @@ def getBouquetChannelList():
                                                                                altrernative_list = getAlternatives(service)
                                                                                if altrernative_list:
                                                                                        for channel in altrernative_list:
-                                                                                               refstr = ':'.join(channel.split(':')[:11])
+                                                                                               refstr = service.toString().upper().split(':')[3:7]
                                                                                                if refstr not in channels:
                                                                                                        channels.append(refstr)
                                                                        else:
-                                                                               refstr = ':'.join(service.toString().split(':')[:11])
+                                                                               refstr = service.toString().upper().split(':')[3:7]
                                                                                if refstr not in channels:
                                                                                        channels.append(refstr)
        else:
@@ -161,11 +161,11 @@ def getBouquetChannelList():
                                                altrernative_list = getAlternatives(service)
                                                if altrernative_list:
                                                        for channel in altrernative_list:
-                                                               refstr = ':'.join(channel.split(':')[:11])
+                                                               refstr = channel.upper().split(':')[3:7]
                                                                if refstr not in channels:
                                                                        channels.append(refstr)
                                        else:
-                                               refstr = ':'.join(service.toString().split(':')[:11])
+                                               refstr = service.toString().upper().split(':')[3:7]
                                                if refstr not in channels:
                                                        channels.append(refstr)
        isFilterRunning = 0
@@ -176,7 +176,7 @@ def channelFilter(ref):
        if not ref:
                return False
        sref = enigma.eServiceReference(ref)
-       refstr = ':'.join(sref.toString().split(':')[:11])
+       refstr = sref.toString().upper().split(':')[3:7]
        if config.plugins.epgimport.import_onlybouquet.value:
                global BouquetChannelListList
                if BouquetChannelListList is None:
@@ -186,7 +186,7 @@ def channelFilter(ref):
                        return False
        global serviceIgnoreList
        if serviceIgnoreList is None:
-               serviceIgnoreList = filtersServices.filtersServicesList.servicesList()
+               serviceIgnoreList = [x.upper().split(':')[3:7] for x in filtersServices.filtersServicesList.servicesList()]
        if refstr in serviceIgnoreList:
                print>>log, "Serviceref is in ignore list:", refstr
                return False

Attached Files


WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W



0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users