23 replies to this topic

[digitalija]

hi!

 

I wold like to export , convert enigma2 epg xml to rytec xmltv format

If someone would like to help I will appriciate it

thanks in advance

cheers!

 

[doglover]

If you are using an image which has OpenWebif installed, I have have whipped up a grabber for this.

 

Use Webgrab++: http://www.webgrabplus.com/

The Siteini for this is attached.  (in the zipped file)

 

A couple of adaptions to be made fro your situation:

Open the the attached SiteIni in an editor like Notepad++ or PsPad.

 

Change this to your timezone: timezone=UTC+01:00

Change this to the IP address of your box: http://192.168.1.19

 

Save the file in the webgrab data dir.

 

Now construct a Webgrab++.config.xml file with the data on which channels to grab.

 

f.i.

 

    <channel update="i" site="openwebif" site_id="1:0:19:320C:45F:35:C00000:0:0:0:" xmltv_id="2Be">2Be</channel>

 

1:0:19:320C:45F:35:C00000:0:0:0:  is the service ref of the channel.  The others are self explanatory.

 

Willy

Attached Files

•  Webgrab++.config.xml.zip   1.38KB   201 downloads

[bugmenot]

Hi doglover, some time ago but anyway. I just started to implement this one for my OpenATV image with OpenWebIf. So far so good.

 

It seems to be stuck while parsing the output of the EPG request for channel. In your sitiini I can see that obviously the channel block should start with <table> tag. For not it seems to be that the whole list ist a table and every epg entry is within a tbody. Maybe you can help me to correc the parsing.

 

Thanks a lot!

[doglover]

It seems that the OpenWebif changed the format some time ago.

Use the attached.

 

In the config file list your channels as:

 

 

<channel update="i" site="openwebif-org.v2" site_id="1:0:19:1B91:C88:3:EB0000:0:0:0:" xmltv_id="2Be">2Be</channel>

 

I will see later today if I can whip up something to grab the service refs and channel names from the bouquet files automatically

 

Willy

 

Attached Files

•  openwebif-org.v2.zip   713bytes   46 downloads

[doglover]

Now includes a creator for the channels list.

 

Willy

Attached Files

•  openwebif-org.v2.zip   832bytes   81 downloads

[bugmenot]

Hi doglover, thank you so much for your really fast help. It works like a charme!

[jenseneverest]

Great work as always @doglover thank you, this works perfectly and is surprisingly fast. I no longer need to use the online websites (that are now been hammered by iptv servers )

Do you have a script or any easy way of converting webgrab.channels to myxxx.channels for use with epgimporter ??

Like in this example:

From this:

   <channel id="1:0:1:D6:2:F020:FFFF0000:0:0:0:"</channel>"Racing UK">Racing UK</channel>

To this:

   <channel id="Racing UK">1:0:1:D6:2:F020:FFFF0000:0:0:0:</channel>

TIA

[doglover]

No, I do not have.

You can use some editting helps in Notepad++ or PSPad, but that is it.

Another trick I used is importing it in Excell an play around with it.

 

 

Willy

[Abu Baniaz]

You can use a custom mix file with ABM to create a csv file for you

[meitevau]

Hi,

 

I wanted to test this with Webgrab++ and Linux Mint 18.3 with Mono 4.2.1 installed. I get the following error when running "./run.sh":

 

 

**------------------------------------------------------------------------------------------------
* @header_start
* WebGrab+Plus ini for grabbing EPG data from TvGuide websites
* @Site: OpenWebIf - Enigma box
* @MinSWversion:
* @Revision 1 - [08/12/2016] doglover
*    -  New OpenWebif
* @Revision 0 - [27/10/2014] doglover
*   - creation
* @Remarks:
* @header_end
**------------------------------------------------------------------------------------------------
 
site {url=openwebif|timezone=Europe/Berlin|maxdays=8.1|cultureinfo=de-DE|charset=UTF-8|titlematchfactor=90|retry=<retry time-out="30">2</retry>}
 
urldate.format {daycounter|0}
url_index{url ()|http://192.168.2.36/...sref=|channel|}
 
index_showsplit.scrub {multi ()|<tbody data-id=||</tbody}
index_start.scrub {single ()|<div class="esl2 panel ui-state-hover">|<br/>|-|</}
*index_duration.scrub {single ()|<div class="esl2 panel ui-state-hover">|(| min.)|</}
index_title.scrub {single (includeblock=1)|<div class="esl2 panel ui-state-hover">||<br/>|</}
index_subtitle.scrub {single ()|<td class="eff"|<span>|</span>|}
index_description.scrub {single ()|<td class="eff"|</div>|</td>|}
 
index_title.modify            {cleanup}
index_subtitle.modify         {cleanup}
 
**  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _
**      #####  CHANNEL FILE CREATION (only to create the xxx-channel.xml file)
**
**@auto_xml_channel_start
*url_index{url|http://192.168.2.36/ajax/channels}
*index_site_id.scrub {multi|open_epg_dialog|('|'|)}
*index_site_channel.scrub {multi|open_epg_dialog|,'|'|)}
**@auto_xml_channel_end

 

The IP of my VU+ Solo 4K is 192.168.2.36, so the openwebif-org.v2.ini looks like:

 

 

**------------------------------------------------------------------------------------------------
* @header_start
* WebGrab+Plus ini for grabbing EPG data from TvGuide websites
* @Site: OpenWebIf - Enigma box
* @MinSWversion:
* @Revision 1 - [08/12/2016] doglover
*    -  New OpenWebif
* @Revision 0 - [27/10/2014] doglover
*   - creation
* @Remarks:
* @header_end
**------------------------------------------------------------------------------------------------
 
site {url=openwebif|timezone=Europe/Berlin|maxdays=8.1|cultureinfo=de-DE|charset=UTF-8|titlematchfactor=90|retry=<retry time-out="30">2</retry>}
 
urldate.format {daycounter|0}
url_index{url ()|http://192.168.2.36/...sref=|channel|}
 
index_showsplit.scrub {multi ()|<tbody data-id=||</tbody}
index_start.scrub {single ()|<div class="esl2 panel ui-state-hover">|<br/>|-|</}
*index_duration.scrub {single ()|<div class="esl2 panel ui-state-hover">|(| min.)|</}
index_title.scrub {single (includeblock=1)|<div class="esl2 panel ui-state-hover">||<br/>|</}
index_subtitle.scrub {single ()|<td class="eff"|<span>|</span>|}
index_description.scrub {single ()|<td class="eff"|</div>|</td>|}
 
index_title.modify            {cleanup}
index_subtitle.modify         {cleanup}
 
**  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _
**      #####  CHANNEL FILE CREATION (only to create the xxx-channel.xml file)
**
**@auto_xml_channel_start
*url_index{url|http://192.168.2.36/ajax/channels}
*index_site_id.scrub {multi|open_epg_dialog|('|'|)}
*index_site_channel.scrub {multi|open_epg_dialog|,'|'|)}
**@auto_xml_channel_end

 

My WebGrab++.config.xml looks like:

 

 

<?xml version="1.0"?>
<!--
Configuration file for WebGrab+Plus, the incremental Electronic-Program-Guide web grabber
by Jan van Straaten, Nov 2010  
Version V1.0.2 Beta
added mode w, proxy, channel attribute site_channel
 
-->
<settings>
     
  <filename>testwebgrab.xml</filename>
<!--   
  <mode>m, n, f</mode>
 -->  
  <mode>m, n, #</mode>  
  <proxy>automatic</proxy>  
  <logging>on</logging>
  <retry>3</retry>
  <skip>13, 1</skip>
  <user-agent>random</user-agent>
  <timespan>5</timespan>
   
   <!-- update mode
  i or incremental     only updates of changes , gabs, repairs and new shows
  l or light           forces update of today and new shows, rest as incremental
  s or smart           forces udate of today and tomorrow and new shows, rest as light
  f or full or force   forces full update  
  If one of these values is entered here it will apply to all channels selected for update  
  (see channel).  This value overrules the value of 'update' for in the individual channels  
  If no value is entered here the individual 'update' values from the channellist are taken -->
  <update>f</update>
     
  <!--Polen-->
    <channel update="i" site="openwebif-org.v2" site_id="1:0:19:283D:3FB:1:C00000:0:0:0:" xmltv_id="Das Erste HD">Das Erste HD</channel>
    <channel update="i" site="openwebif-org.v2" site_id="1:0:19:2B66:3F3:1:C00000:0:0:0:" xmltv_id="ZDF HD">ZDF HD</channel>
 
</settings>

 

So what's going wrong? Where is the error?

 

meitevau

Edited by meitevau, 15 March 2018 - 21:59.

[doglover]

Can you post the WebGrab++.log.txt file pls. so I can see what the error exactly is.

The above does not tell me anything.

 

Willy

[meitevau]

Here is the Log file:

 

[        ]
[        ]              WebGrab+Plus/w MDB & REX Postprocess -- version  V2.1              
[        ]
[        ]                                 Jan van Straaten                                
[        ]                              Francis De Paemeleere                              
[        ]
[        ]             thanks to Paul Weterings and all the contributing users             
[        ] --------------------------------------------------------------------------------
[        ]
[        ] Job started at 15/03/2018 21:51:26
[  Debug ]
[  Debug ] Running  on: Unix 4.13.0.37
[  Debug ] Environment: 4.0.30319.17020
[  Debug ] Mono version: 4.2.1 (Debian 4.2.1.102+dfsg2-7ubuntu4)
[  Debug ]
[  Debug ] Loading timezone data
[  Debug ] Embedded timezones source: WGconsole.WG.Common.timezonesdata.txt
[  Debug ] Reading config file: /home/user/Programme/Webgrab/./WebGrab++.config.xml
[  Info  ] found: /home/user/Programme/Webgrab/./siteini.user/openwebif-org.v2.ini -- Revision 01
[  Debug ] xmltv input file - /home/user/Programme/Webgrab/testwebgrab.xml - found
[  Debug ] 0 superfluous shows removed
[  Info  ] Found existing channel (xmltv_id=Das Erste HD) in the config file
[  Info  ] Found existing channel (xmltv_id=ZDF HD) in the config file
[  Info  ]
[  Info  ]
[  Info  ]       i=index  .=same  c=change  g=gab  r=replace  n=new
[  Info  ]
[  Info  ]
[  Info  ] Group (0) :
[  Info  ] update requested for - 2 - out of - 2 - channels for 6 day(s)
[  Debug ]
[  Info  ] (   1/2   ) OPENWEBIF-ORG.V2 -- chan. (xmltv_id=Das Erste HD) -- mode Force
[Error   ] Unable to update channel Das Erste HD
[Critical] Generic syntax exception:
[Critical]    message:
[Error   ] no index page data received from Das Erste HD
[Error   ] unable to update channel, try again later
[  Info  ] Existing guide data restored!
[  Debug ]
[  Info  ] (   1/2   ) OPENWEBIF-ORG.V2 -- chan. (xmltv_id=ZDF HD) -- mode Force
[Error   ] Unable to update channel ZDF HD
[Critical] Generic syntax exception:
[Critical]    message:
[Error   ] no index page data received from ZDF HD
[Error   ] unable to update channel, try again later
[  Info  ] Existing guide data restored!
[  Debug ]
[  Debug ] 0 shows in 2 channels
[  Debug ] 0 updated shows
[  Debug ] 0 new shows added
[  Info  ]
[  Info  ]
[        ] Job finished at 15/03/2018 21:51:27 done in 0s

 

meitevau

[doglover]

So, something is wrong.  I still cannot tell what is going on.  But lets try to find out what could be wrong.

 

1. Service ref wrong?

 

I checked them on my own receiver and are correct.  This cannot be the problem.

 

2. Do you have a user and password defined for openwebif.

 

This can be a problem;

Try to read the EPG in a browser by using the URL:

http://192.168.2.36/ajax/epgpop?sref=1:0:19:283D:3FB:1:C00000:0:0:0:

Do you see the EPG? Yes, then we have to look further.

No, then we have your problem.  Either there is no EPG in you receiver for this channel, or you cannot access it because you use a password on your receiver.

 

3. You are using an old version/different version of Openwebif.

The SiteIni is written for use with the latest OpenWebif.

I am using:

Open Source web interface voor Linux set-top boxen
Versie: OWIF 1.2.7
Site en broncode: Github

4. You can try change the showsplit by adding a debug statement.

index_showsplit.scrub {multi (debug)|<tbody data-id=||</tbody}

Run WG again and look into the log file of WG.

There will be also a file html.source.htm created.  Can you find out the content of this file.  Is there the info present?

This action will not resolve the problem, but will possible tell us where the problem is.

 

 

If I can think of something else, I will post it here.

 

Willy

[meitevau]

Hi,

 

To 1.:

 

My M3U-List looks like:

 

#EXTM3U
#EXTVLCOPT--http-reconnect=true
#EXTINF:-1 tvg-id="Das Erste HD" tvg-name="Das Erste Hd", Das Erste HD
#EXTVLCOPT:program=10301
http://root:XXXXXXXX...1:C00000:0:0:0:
#EXTINF:-1 tvg-id="ZDF HD" tvg-name="Zdf Hd",ZDF HD
#EXTVLCOPT:program=11110
http://root:XXXXXXXX...1:C00000:0:0:0:
#EXTINF:-1 tvg-id="SWR Fernsehen BW HD" tvg-name="Swr Fernsehen Bw Hd",SWR HD
#EXTVLCOPT:program=10303
http://root:XXXXXXXX...1:C00000:0:0:0:
#EXTINF:-1 tvg-id="BR Nord HD" tvg-name="Br Nord Hd",BR Fernsehen Nord HD
#EXTVLCOPT:program=10326
http://root:XXXXXXXX...1:C00000:0:0:0:
#EXTINF:-1 tvg-id="hr-fernsehen HD" tvg-name="Hr-fernsehen Hd",hr-fernsehen HD
#EXTVLCOPT:program=10355
http://root:XXXXXXXX...1:C00000:0:0:0:
 

 

This M3U-List works with Kodi without any problems.

 

The only thing I miss is a complete EPG for my list.

 

To 2.:

When I open the URL, I get an error 404 Page not found!

 

So I think, perhaps the URL is wrong.

 

To 3.:

I'm using the newest Version of OpenWebif:

 

 

OpenWebif Quelloffenes Webinterface für Linux-Receiver Version: OWIF 1.2.8

 

To 4.:

Will test it!

 

meitevau

Edited by meitevau, 17 March 2018 - 14:27.

[meitevau]

OK, testet it with and without authentification for OpenWebig, same result!

 

Changed the line as statet from you at point 4. Here is the log file:

[        ]
[        ]              WebGrab+Plus/w MDB & REX Postprocess -- version  V2.1              
[        ]
[        ]                                 Jan van Straaten                                
[        ]                              Francis De Paemeleere                              
[        ]
[        ]             thanks to Paul Weterings and all the contributing users             
[        ] --------------------------------------------------------------------------------
[        ]
[        ] Job started at 17/03/2018 14:36:24
[  Debug ]
[  Debug ] Running  on: Unix 4.13.0.37
[  Debug ] Environment: 4.0.30319.17020
[  Debug ] Mono version: 4.2.1 (Debian 4.2.1.102+dfsg2-7ubuntu4)
[  Debug ]
[  Debug ] Loading timezone data
[  Debug ] Embedded timezones source: WGconsole.WG.Common.timezonesdata.txt
[  Debug ] Reading config file: /home/marco/Programme/Webgrab/./WebGrab++.config.xml
[  Info  ] found: /home/marco/Programme/Webgrab/./siteini.user/openwebif-org.v2.ini -- Revision 01
[  Debug ] xmltv input file - /home/marco/Programme/Webgrab/testwebgrab.xml - found
[  Debug ] 0 superfluous shows removed
[  Info  ] Found existing channel (xmltv_id=Das Erste HD) in the config file
[  Info  ] Found existing channel (xmltv_id=ZDF HD) in the config file
[  Info  ]
[  Info  ]
[  Info  ]       i=index  .=same  c=change  g=gab  r=replace  n=new
[  Info  ]
[  Info  ]
[  Info  ] Group (0) :
[  Info  ] update requested for - 2 - out of - 2 - channels for 6 day(s)
[  Debug ]
[  Info  ] (   1/2   ) OPENWEBIF-ORG.V2 -- chan. (xmltv_id=Das Erste HD) -- mode Force
[Error   ] Unable to update channel Das Erste HD
[Critical] Generic syntax exception:
[Critical]    message:
[Error   ] no index page data received from Das Erste HD
[Error   ] unable to update channel, try again later
[  Info  ] Existing guide data restored!
[  Debug ]
[  Info  ] (   1/2   ) OPENWEBIF-ORG.V2 -- chan. (xmltv_id=ZDF HD) -- mode Force
[Error   ] Unable to update channel ZDF HD
[Critical] Generic syntax exception:
[Critical]    message:
[Error   ] no index page data received from ZDF HD
[Error   ] unable to update channel, try again later
[  Info  ] Existing guide data restored!
[  Debug ]
[  Debug ] 0 shows in 2 channels
[  Debug ] 0 updated shows
[  Debug ] 0 new shows added
[  Info  ]
[  Info  ]
[        ] Job finished at 17/03/2018 14:36:25 done in 0s

 

I can't find a file named "html.source.htm".

 

meitevau

[meitevau]

OK, tested it with and without authentification for OpenWebif, same result!

 

Changed the line as statet from you at point 4. Here is the log file:

[        ]
[        ]              WebGrab+Plus/w MDB & REX Postprocess -- version  V2.1              
[        ]
[        ]                                 Jan van Straaten                                
[        ]                              Francis De Paemeleere                              
[        ]
[        ]             thanks to Paul Weterings and all the contributing users             
[        ] --------------------------------------------------------------------------------
[        ]
[        ] Job started at 17/03/2018 14:36:24
[  Debug ]
[  Debug ] Running  on: Unix 4.13.0.37
[  Debug ] Environment: 4.0.30319.17020
[  Debug ] Mono version: 4.2.1 (Debian 4.2.1.102+dfsg2-7ubuntu4)
[  Debug ]
[  Debug ] Loading timezone data
[  Debug ] Embedded timezones source: WGconsole.WG.Common.timezonesdata.txt
[  Debug ] Reading config file: /home/marco/Programme/Webgrab/./WebGrab++.config.xml
[  Info  ] found: /home/marco/Programme/Webgrab/./siteini.user/openwebif-org.v2.ini -- Revision 01
[  Debug ] xmltv input file - /home/marco/Programme/Webgrab/testwebgrab.xml - found
[  Debug ] 0 superfluous shows removed
[  Info  ] Found existing channel (xmltv_id=Das Erste HD) in the config file
[  Info  ] Found existing channel (xmltv_id=ZDF HD) in the config file
[  Info  ]
[  Info  ]
[  Info  ]       i=index  .=same  c=change  g=gab  r=replace  n=new
[  Info  ]
[  Info  ]
[  Info  ] Group (0) :
[  Info  ] update requested for - 2 - out of - 2 - channels for 6 day(s)
[  Debug ]
[  Info  ] (   1/2   ) OPENWEBIF-ORG.V2 -- chan. (xmltv_id=Das Erste HD) -- mode Force
[Error   ] Unable to update channel Das Erste HD
[Critical] Generic syntax exception:
[Critical]    message:
[Error   ] no index page data received from Das Erste HD
[Error   ] unable to update channel, try again later
[  Info  ] Existing guide data restored!
[  Debug ]
[  Info  ] (   1/2   ) OPENWEBIF-ORG.V2 -- chan. (xmltv_id=ZDF HD) -- mode Force
[Error   ] Unable to update channel ZDF HD
[Critical] Generic syntax exception:
[Critical]    message:
[Error   ] no index page data received from ZDF HD
[Error   ] unable to update channel, try again later
[  Info  ] Existing guide data restored!
[  Debug ]
[  Debug ] 0 shows in 2 channels
[  Debug ] 0 updated shows
[  Debug ] 0 new shows added
[  Info  ]
[  Info  ]
[        ] Job finished at 17/03/2018 14:36:25 done in 0s

 

I can't find a file named "html.source.htm".

 

meitevau

[doglover]

To 2.:
When I open the URL, I get an error 404 Page not found!
So I think, perhaps the URL is wrong.

This is the reason you do not have data.  On the enigma2 receiver is there EPG present?

Maybe you need to obtain the EPG first on the receiver with EPGrefresh or even EPGimport?

 

Your remark, about the URL not correct.  It could be.

Lets try to find out.  Open the Webif in a browser.  Request the EPG for a channel (ZDF)

Now open this page on a separate page. (by clicking on the button at the bottom op the pop-up)

Look at the URL.  This will contain %3A's which is safeurl equivalent of :

 

But in your first remark you start talking about an m3u file.  So this is for IPTV channels.

Now I am wondering what your are trying todo.

 

Watching the channels from your enigma2 receiver (sat receiver) on a Kodi device?

The contents of the m3u file indicate this sort of set-up.

 

Can you enlighten me about your set-up pls.

 

But the first thing you need is EPG on the enigma2 receiver.  If there is none, you cannot obtain it.

 

Willy

 

PS: you have different version of OWIF.  Will see if I can get hold of this newer version, and see if anything has changed.

Edited by doglover, 17 March 2018 - 15:46.

[doglover]

On my second receiver I have:

Open Source web interface voor Linux set-top boxen
Versie: OWIF 1.2.8
Site en broncode: Github

Tried it, and no problem getting EPG from it.

 

Willy

[doglover]

I can't find a file named "html.source.htm".

It should be in the same folder as you placed the SiteIni (openwebif-org.v2.ini)

 

Willy

[meitevau]

Hi,

 

it works now!!!

 

This was the tip that helped me:

 

Request the EPG for a channel (ZDF)

Now open this page on a separate page. (by clicking on the button at the bottom op the pop-up)

 

Because I couldn't find that button, I switched from new responsive design to the old "classic" design from Openwebif.

 

Now I could open the EPG-Data in a new window, and could find the url.

 

After that I testet Webgrab, and it worked!

 

meitevau

Edited by meitevau, 18 March 2018 - 18:03.