Jump to content


Photo

The end of Rytec EPG?


  • Please log in to reply
38 replies to this topic

#1 doglover

  • Rytec EPG Team
  • 16,974 posts

+635
Excellent

Posted 31 March 2019 - 13:26

It is no secret that the EPG is grabbed from all kinds of websites using Webgrab++

 

While we received help for writing SiteIni's for difficult webpages from the users on the Webgrab++ forum in the past (see http://www.webgrabplus.com/forum), has this now ground to a halt.
Reason: We are a so called commercial provider of EPG.  And they will not engage in help for commercial purposes.

 

If we ask for help, some of postings will be:  I can do it, but I wont tell you.
They do not actually say it this way, but the result is the same.

(They post a sample grab from that website, but do not want to share the actual SiteIni.)

An example of this can seen in:  http://www.webgrabpl...brs-and-mojtvhr

 

We will continue to provide EPG as long as we can.  As long as I can write the SiteIni's myself.

Actually parsing a website is not that difficult.  The problem is in obtaining the correct webpages.  Just because you can see it in a browser does not guarantee you you can download that webpage.  Finding the solutions for this escapes me more and more, since I am not an internet specialist.

 

I am posting this, because I feel very low now and disappointed.

 

Willy

 

 


~~Rytec Team~~
Maxytec Multibox SE OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: The end of Rytec EPG? #2 Alex_NL

  • Senior Member
  • 1,953 posts

+10
Neutral

Posted 31 March 2019 - 14:09

Why do they think Rytec EPG is commercial???


Dreambox DM920:2x Triple tuner (2x DVB-S2X MIS and DVB-C/T2) Xtrend ET10000: 2x DVB-S2, 2x DVB-C, IPTV Xtrend ET9500 (defect)
Channel Master 120cm offset + Black Ultra LNB (Ku-band): 42°E-37,5°W - 100cm SAB draaibaar: 45.1°E-60°E - 80cm vast: 28.2°E/23.5°E/19.2°E/13°E
DVB-T2 en DAB via een combi VHF/UHF-antenne op zolder gericht op Schoten.


Re: The end of Rytec EPG? #3 Lost in Space

  • Senior Member
  • 876 posts

+69
Good

Posted 31 March 2019 - 14:51

well, feeling bad and dissapointed is not unusual in the Open Source 'community'

Sometimes this is like communism ... even China ... only kept the name for nostalgic purpose ....

 

Maybe your approach is simply too old-fashioned and altruistic :rolleyes:

 

If you would be on a college in the US you would have found a startup with your room mate, get 1 Million from a venture capitalist and after your XMLTV format became a de-facto standard for EPG sharing and exchange you would try to sell yourself to Google or Facebook ;)


Edited by gutemine, 31 March 2019 - 14:52.


Re: The end of Rytec EPG? #4 WanWizard

  • PLi® Core member
  • 68,309 posts

+1,719
Excellent

Posted 31 March 2019 - 14:57

In which way is this response in any way helpful?


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: The end of Rytec EPG? #5 doglover

  • Rytec EPG Team
  • 16,974 posts

+635
Excellent

Posted 31 March 2019 - 15:10

Why do they think Rytec EPG is commercial???

 

Because one of the culprits over there said so...


~~Rytec Team~~
Maxytec Multibox SE OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: The end of Rytec EPG? #6 Rob van der Does

  • Senior Member
  • 7,766 posts

+184
Excellent

Posted 31 March 2019 - 15:16

In which way is this response in any way helpful?

LOL, a rhetorical question....
Did you ever notice any of Gutemine's remarks to be useful?

Re: The end of Rytec EPG? #7 Lost in Space

  • Senior Member
  • 876 posts

+69
Good

Posted 31 March 2019 - 15:24

I just tried to hint the OP what he achieved by NOT going this route, which you could consider a polite way of cheering him up :unsure:

 

If you insist. I could also start writing what you consider usefull remarks on some subjects here ... but you might not enjoy that either :rolleyes:


Edited by gutemine, 31 March 2019 - 15:25.


Re: The end of Rytec EPG? #8 Taykun345

  • Senior Member
  • 1,296 posts

+41
Good

Posted 31 March 2019 - 15:42

It seems more that some guys just do not want to share the grabber if its a bit more advanced. I dont bite into commercial thing because Rytec is free lol. Everyone can see that unless you are blind asf.

 

I for example made great student website now more than 10 years old for my business school. Well, students are less and less interested in sharing last few years. They rather sell their student notes. I scanned them for free on the other hand...

 

Dont be dissapointed, lets enjoy this hobby.


Army MoodBlue HD skin modification by me: https://github.com/T...-MoodBlueHD-mod
Matrix10 MH-HD2 skin modification by me: https://github.com/B...-MX-HD2-OpenPli
MetrixHD skin modification by me: https://github.com/T...xHD-WPstyle-mod
Slovenian translation for OpenPLi E2: https://github.com/T...ion-for-OpenPLi

Re: The end of Rytec EPG? #9 kiddac

  • Member
  • 4 posts

+2
Neutral

Posted 31 March 2019 - 17:56

The principles behind webgrab plus is simple. Although the program is over complicated because it tries to be a catch all. And you need a degree in geekness to be able to use it.

 

All it is doing is reading various json files from a website or seperate url and traversing through the keys.

 

Everything you need can actually be done on your enigma2 box with python. Downloading the json files, building the xml files, cross referencing the lame files etc.

 

My next months project is going to be exactly this. I have already doing something extremely similar for my latest project, which is a picon download called "e2piconizer". (soon to be released)

 

For example sky uk tv guide. All this initial data can be found in the tv guide html

<script id="initialData" type="application/json"> ........

 

or sourced from the epg APIs.

 

http://awk.epgsky.co...linear/services

http://awk.epgsky.co...linear/schedule

 

Virgin UK TV, is exactly the same except it uses Horizon TV API for the data.

Horizon TV has many worldwide channels that can be scraped.

https://web-api.hori...n/web/channels/ # romania
https://web-api.hori...l/web/channels/ # poland
https://web-api.hori...d/web/channels/  # Netherlands
https://web-api.hori...n/web/channels/  # Hungary
https://web-api.hori...s/web/channels/  # Czech
https://web-api.hori...u/web/channels/  # Switzerland/German
https://web-api.hori...k/web/channels/  # Slovakia
https://web-api.hori...u/web/channels/  # Austria/German

 

here is an old link of someone that made a horizon tv grabber. Lots of useful information in his code.

 

https://github.com/t...horizon.tv.json

 

There is always a way ;)


Edited by kiddac, 31 March 2019 - 17:57.


Re: The end of Rytec EPG? #10 doglover

  • Rytec EPG Team
  • 16,974 posts

+635
Excellent

Posted 31 March 2019 - 18:38

Everything you need can actually be done on your enigma2 box with python. Downloading the json files, building the xml files, cross referencing the lame files etc.

 

 

 

Firstly.  it are not always json files.  But that is besides the point.

 

The problem is finding the exact URL to download the json files.

Can you find them for telemach.ba?  or tv2go.t-2.net?

There are special problems here, which I cannot figure out.  Solutions exits but they do not want to share it.

Once I know the correct URL (and request headers) it is easy.

 

Willy


~~Rytec Team~~
Maxytec Multibox SE OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: The end of Rytec EPG? #11 Abu Baniaz

  • PLi® Contributor
  • 2,414 posts

+61
Good

Posted 31 March 2019 - 18:55

The problem Willy is that you do all the hard work on your own and in the background. Not many people truly appreciate the amount of hard work you put into EPG for E2 receivers for free.

If you remember, I suggested that the channels file should be on GitHub, people can submit changes that way.

There are many people with IT backgrounds who frequent the forum. I am sure if you open up and allow others to assist, they will gladly do so.

You do so much for the community. Don't let any misplaced labelling of your efforts keep you down. You have been doing a fantastic job.

Re: The end of Rytec EPG? #12 kiddac

  • Member
  • 4 posts

+2
Neutral

Posted 31 March 2019 - 19:42

I didn't realise you where after complicated files that require a php setup and tokens just to get the data.

 

Just reading through some of them posts on webgrabplus forum what a strange place that forum has become.

The attitude of the major player on there of  I can do it, but I am not showing anyone else is ridiculous. Apart from resentment what is he trying to achieve.

 

...and doglover just reading through some of your later posts on there, I feel your pain.



Re: The end of Rytec EPG? #13 jeanclaude

  • Senior Member
  • 866 posts

+28
Good

Posted 1 April 2019 - 09:33

The problem Willy is that you do all the hard work on your own and in the background. Not many people truly appreciate the amount of hard work you put into EPG for E2 receivers for free.

If you remember, I suggested that the channels file should be on GitHub, people can submit changes that way.

There are many people with IT backgrounds who frequent the forum. I am sure if you open up and allow others to assist, they will gladly do so.

You do so much for the community. Don't let any misplaced labelling of your efforts keep you down. You have been doing a fantastic job.

the problem with this is that if doglover makes his scripts publically available to anyone, that many more people will start scraping those websites....... which in return may cause counter-measures from these websited to stop these site scrapings.

so I understand his reluctance to make this public - best keep a low profile on this to remain 'unnoticed' by the websites.


DreamBox 7000S+8000HD (eindelijk), openPLi, CCcam, 85 cm schotel, draaibare opstelling en VEEL te weinig slaap.

Re: The end of Rytec EPG? #14 jenseneverest

  • Senior Member
  • 65 posts

+5
Neutral

Posted 1 April 2019 - 16:45

 

The problem Willy is that you do all the hard work on your own and in the background. Not many people truly appreciate the amount of hard work you put into EPG for E2 receivers for free.

If you remember, I suggested that the channels file should be on GitHub, people can submit changes that way.

There are many people with IT backgrounds who frequent the forum. I am sure if you open up and allow others to assist, they will gladly do so.

You do so much for the community. Don't let any misplaced labelling of your efforts keep you down. You have been doing a fantastic job.

the problem with this is that if doglover makes his scripts publically available to anyone, that many more people will start scraping those websites....... which in return may cause counter-measures from these websited to stop these site scrapings.

so I understand his reluctance to make this public - best keep a low profile on this to remain 'unnoticed' by the websites.

 

 

The channel list is used to ref the epg data by the epgimporter plugin, anyone can download it, the plugin dose so everytime you update epg. Its not the actual grab ini for data.

 

The problem is with the commercial epg.....IMO mostly pay server IPTV providers.  Now everyone is at it, where as it was not an issue previously.

Further more the TV guide websites dont just give the data out, a lot have protection to stop exactly what is being scraped from them.....

 

How many EPG data stream channels are out there like the one on 28.2 ?? Do other sats have them too ??

That would be the best option for e2 users, provided the code could be added to the image to collect it.... i seem to remember a post about 28.2 a while back....

 

@doglover just to say thanks for all the work over the years, much appreciated.



Re: The end of Rytec EPG? #15 WanWizard

  • PLi® Core member
  • 68,309 posts

+1,719
Excellent

Posted 1 April 2019 - 18:32

That would be the best option for e2 users, provided the code could be added to the image to collect it.... i seem to remember a post about 28.2 a while back....

 

The code exists, all it needs is someone with time and enough C knowledge to implement an EPG reader for OpenTV.


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: The end of Rytec EPG? #16 doglover

  • Rytec EPG Team
  • 16,974 posts

+635
Excellent

Posted 2 April 2019 - 07:01

OpenTV helps indeed for 28.2E, but not for all channels.  Also 13 E - Sky will be covered.

It is no solution for other sats, cable and DVB-T

 

Willy


~~Rytec Team~~
Maxytec Multibox SE OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: The end of Rytec EPG? #17 WanWizard

  • PLi® Core member
  • 68,309 posts

+1,719
Excellent

Posted 2 April 2019 - 11:54

I think the big plus for the current method is that, although very complex, it doesn't need programming skills. Any other solution requires writing a custom HTML parser per web page. Not very complex, but it requires the right skills all the same...


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: The end of Rytec EPG? #18 doglover

  • Rytec EPG Team
  • 16,974 posts

+635
Excellent

Posted 2 April 2019 - 12:32

Writing SiteIni's for WG++ kan be very easy or very complex.

As an example I wrote and tested the attached In 1 hours time.  Admit-tingly it is not a complex website.

 

Willy

Attached Files


Edited by doglover, 2 April 2019 - 12:34.

~~Rytec Team~~
Maxytec Multibox SE OpenPli (used as mediaplayer)
Mutant HD2400 OpenPli
Vu+ Duo OpenPli (backup)

Synology NAS

Sat: 13E, 19.2E, 23.5E and 28.2E
*Pli/Rytec EPG POWERED*


Re: The end of Rytec EPG? #19 WanWizard

  • PLi® Core member
  • 68,309 posts

+1,719
Excellent

Posted 2 April 2019 - 13:27

Rather you than me. ;)


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: The end of Rytec EPG? #20 Pr2

  • PLi® Contributor
  • 6,046 posts

+256
Excellent

Posted 6 April 2019 - 08:47

OpenTV will not cover all the satellite channels, so this can be helpful for a bunch of channels but not all.
I don't think that there is still solution available to download the EPG directly from the satellite from the french provider (CSat on 19.2E) for example.

NO SUPPORT by PM, it is a forum make your question public so everybody can benefit from the question/answer.
If you think that my answer helps you, you can press the up arrow in bottom right of the answer.

Wanna help with OpenPLi Translation? Please read our Wiki Information for translators

Sat: Hotbird 13.0E, Astra 19.2E, Eutelsat5A 5.0W
VU+ Solo 4K: 2*DVB-S2 + 2*DVB-C/T/T2 (used in DVB-C) & Duo 4K: 2*DVB-S2X + DVB-C (FBC)

AB-Com: PULSe 4K 1*DVB-S2X (+ DVB-C/T/T2)
Edision OS Mio 4K: 1*DVB-S2X + 1*DVB-C/T/T2
 



1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users