38 replies to this topic

[doglover]

It is no secret that the EPG is grabbed from all kinds of websites using Webgrab++

 

While we received help for writing SiteIni's for difficult webpages from the users on the Webgrab++ forum in the past (see http://www.webgrabplus.com/forum), has this now ground to a halt.
Reason: We are a so called commercial provider of EPG.  And they will not engage in help for commercial purposes.

 

If we ask for help, some of postings will be:  I can do it, but I wont tell you.
They do not actually say it this way, but the result is the same.

(They post a sample grab from that website, but do not want to share the actual SiteIni.)

An example of this can seen in:  http://www.webgrabpl...brs-and-mojtvhr

 

We will continue to provide EPG as long as we can.  As long as I can write the SiteIni's myself.

Actually parsing a website is not that difficult.  The problem is in obtaining the correct webpages.  Just because you can see it in a browser does not guarantee you you can download that webpage.  Finding the solutions for this escapes me more and more, since I am not an internet specialist.

 

I am posting this, because I feel very low now and disappointed.

 

Willy

 

 

[Alex_NL]

Why do they think Rytec EPG is commercial???

[Lost in Space]

well, feeling bad and dissapointed is not unusual in the Open Source 'community'

Sometimes this is like communism ... even China ... only kept the name for nostalgic purpose ....

 

Maybe your approach is simply too old-fashioned and altruistic

 

If you would be on a college in the US you would have found a startup with your room mate, get 1 Million from a venture capitalist and after your XMLTV format became a de-facto standard for EPG sharing and exchange you would try to sell yourself to Google or Facebook

Edited by gutemine, 31 March 2019 - 14:52.

[WanWizard]

In which way is this response in any way helpful?

[doglover]

Why do they think Rytec EPG is commercial???

 

Because one of the culprits over there said so...

[Rob van der Does]

In which way is this response in any way helpful?

LOL, a rhetorical question....
Did you ever notice any of Gutemine's remarks to be useful?

[Lost in Space]

I just tried to hint the OP what he achieved by NOT going this route, which you could consider a polite way of cheering him up

 

If you insist. I could also start writing what you consider usefull remarks on some subjects here ... but you might not enjoy that either

Edited by gutemine, 31 March 2019 - 15:25.

[Taykun345]

It seems more that some guys just do not want to share the grabber if its a bit more advanced. I dont bite into commercial thing because Rytec is free lol. Everyone can see that unless you are blind asf.

 

I for example made great student website now more than 10 years old for my business school. Well, students are less and less interested in sharing last few years. They rather sell their student notes. I scanned them for free on the other hand...

 

Dont be dissapointed, lets enjoy this hobby.

[kiddac]

The principles behind webgrab plus is simple. Although the program is over complicated because it tries to be a catch all. And you need a degree in geekness to be able to use it.

 

All it is doing is reading various json files from a website or seperate url and traversing through the keys.

 

Everything you need can actually be done on your enigma2 box with python. Downloading the json files, building the xml files, cross referencing the lame files etc.

 

My next months project is going to be exactly this. I have already doing something extremely similar for my latest project, which is a picon download called "e2piconizer". (soon to be released)

 

For example sky uk tv guide. All this initial data can be found in the tv guide html

<script id="initialData" type="application/json"> ........

 

or sourced from the epg APIs.

 

http://awk.epgsky.co...linear/services

http://awk.epgsky.co...linear/schedule

 

Virgin UK TV, is exactly the same except it uses Horizon TV API for the data.

Horizon TV has many worldwide channels that can be scraped.

https://web-api.hori...n/web/channels/ # romania
https://web-api.hori...l/web/channels/ # poland
https://web-api.hori...d/web/channels/  # Netherlands
https://web-api.hori...n/web/channels/  # Hungary
https://web-api.hori...s/web/channels/  # Czech
https://web-api.hori...u/web/channels/  # Switzerland/German
https://web-api.hori...k/web/channels/  # Slovakia
https://web-api.hori...u/web/channels/  # Austria/German

 

here is an old link of someone that made a horizon tv grabber. Lots of useful information in his code.

 

https://github.com/t...horizon.tv.json

 

There is always a way

Edited by kiddac, 31 March 2019 - 17:57.

[doglover]

Everything you need can actually be done on your enigma2 box with python. Downloading the json files, building the xml files, cross referencing the lame files etc.

 

 

 

Firstly.  it are not always json files.  But that is besides the point.

 

The problem is finding the exact URL to download the json files.

Can you find them for telemach.ba?  or tv2go.t-2.net?

There are special problems here, which I cannot figure out.  Solutions exits but they do not want to share it.

Once I know the correct URL (and request headers) it is easy.

 

Willy

[Abu Baniaz]

The problem Willy is that you do all the hard work on your own and in the background. Not many people truly appreciate the amount of hard work you put into EPG for E2 receivers for free.

If you remember, I suggested that the channels file should be on GitHub, people can submit changes that way.

There are many people with IT backgrounds who frequent the forum. I am sure if you open up and allow others to assist, they will gladly do so.

You do so much for the community. Don't let any misplaced labelling of your efforts keep you down. You have been doing a fantastic job.

[kiddac]

I didn't realise you where after complicated files that require a php setup and tokens just to get the data.

 

Just reading through some of them posts on webgrabplus forum what a strange place that forum has become.

The attitude of the major player on there of  I can do it, but I am not showing anyone else is ridiculous. Apart from resentment what is he trying to achieve.

 

...and doglover just reading through some of your later posts on there, I feel your pain.

[jeanclaude]

The problem Willy is that you do all the hard work on your own and in the background. Not many people truly appreciate the amount of hard work you put into EPG for E2 receivers for free.

If you remember, I suggested that the channels file should be on GitHub, people can submit changes that way.

There are many people with IT backgrounds who frequent the forum. I am sure if you open up and allow others to assist, they will gladly do so.

You do so much for the community. Don't let any misplaced labelling of your efforts keep you down. You have been doing a fantastic job.

the problem with this is that if doglover makes his scripts publically available to anyone, that many more people will start scraping those websites....... which in return may cause counter-measures from these websited to stop these site scrapings.

so I understand his reluctance to make this public - best keep a low profile on this to remain 'unnoticed' by the websites.

[jenseneverest]

 

The problem Willy is that you do all the hard work on your own and in the background. Not many people truly appreciate the amount of hard work you put into EPG for E2 receivers for free.

If you remember, I suggested that the channels file should be on GitHub, people can submit changes that way.

There are many people with IT backgrounds who frequent the forum. I am sure if you open up and allow others to assist, they will gladly do so.

You do so much for the community. Don't let any misplaced labelling of your efforts keep you down. You have been doing a fantastic job.

the problem with this is that if doglover makes his scripts publically available to anyone, that many more people will start scraping those websites....... which in return may cause counter-measures from these websited to stop these site scrapings.

so I understand his reluctance to make this public - best keep a low profile on this to remain 'unnoticed' by the websites.

 

 

The channel list is used to ref the epg data by the epgimporter plugin, anyone can download it, the plugin dose so everytime you update epg. Its not the actual grab ini for data.

 

The problem is with the commercial epg.....IMO mostly pay server IPTV providers.  Now everyone is at it, where as it was not an issue previously.

Further more the TV guide websites dont just give the data out, a lot have protection to stop exactly what is being scraped from them.....

 

How many EPG data stream channels are out there like the one on 28.2 ?? Do other sats have them too ??

That would be the best option for e2 users, provided the code could be added to the image to collect it.... i seem to remember a post about 28.2 a while back....

 

@doglover just to say thanks for all the work over the years, much appreciated.

[WanWizard]

That would be the best option for e2 users, provided the code could be added to the image to collect it.... i seem to remember a post about 28.2 a while back....

 

The code exists, all it needs is someone with time and enough C knowledge to implement an EPG reader for OpenTV.

[doglover]

OpenTV helps indeed for 28.2E, but not for all channels.  Also 13 E - Sky will be covered.

It is no solution for other sats, cable and DVB-T

 

Willy

[WanWizard]

I think the big plus for the current method is that, although very complex, it doesn't need programming skills. Any other solution requires writing a custom HTML parser per web page. Not very complex, but it requires the right skills all the same...

[doglover]

Writing SiteIni's for WG++ kan be very easy or very complex.

As an example I wrote and tested the attached In 1 hours time.  Admit-tingly it is not a complex website.

 

Willy

Attached Files

•  tv.wp.pl.zip   9.36KB   21 downloads

Edited by doglover, 2 April 2019 - 12:34.

[WanWizard]

Rather you than me.

[Pr2]

OpenTV will not cover all the satellite channels, so this can be helpful for a bunch of channels but not all.
I don't think that there is still solution available to download the EPG directly from the satellite from the french provider (CSat on 19.2E) for example.