62 replies to this topic

[WanWizard]

Great, quite possible that the morons get it from sites like this...

[MiLo]

Perhaps we should think to change the way of working of XMLTV import, why not use torrent to download the files that we need for our EPG?

Actually, while technically interesting, using torrents won't help at all.

The trouble is not the bandwidth, but the number of requests. Even when using torrents, you still need a server that provides the torrent file at a fixed location. That server would still get hit by the same silly number of requests.

[doglover]

At the moment I have put a dummy (700 bytes large) files on the servers with a the filenames which are most requested with these silly requests.

Deleting the file helped in some,  but the servers were very busy sending not found messages.

 

Hopefully this will lessen the problem a bit.  There are very little bytes transmitted.  No exception are generated and the receivers have their request fulfilled, so repeat request are hopefully not send.

 

Willy.

[MiLo]

Maybe you should replace the files with a gzipped 1GB file filled with zeroes. That might kill the offenders. The gz file would be about 1MB, and unpack into 1GB which on a settop would be disastrous when unpacked into /tmp/...

[doglover]

The culprit:

 

http://linux.mjnet.e...ion-2012-05-09/

 

OpenElec for Kodi.

 

Willy

 

PS: Milo:  I will think about this.  Is rather cruel locking up somebody's machine.

[pluizebol]

PS: Milo:  I will think about this.  Is rather cruel locking up somebody's machine.

 

Are they not doing the same thing ?

[WanWizard]

I tried adding a http redirect (302) to a 10G download file at a large ISP, but the client in use seem to ignore it, so that was useless.

 

Having such a file on the server itself won't really help; instead of lots of short connection requests, you have a lot of long running requests. Also, if your bandwidth it capped, you might run into a lot of additional costs...

[MiLo]

Maybe we should add XZ support to the XMLTV plugin, which compresses better. A 10GB zeroes file is only 1k or so with LZMA compressors like XZ
It also has the benefit of using a bit less bandwith on the legitimate files.

[doglover]

Posted yesterday on Kodinerds.

 

an alle,

die die rytecxmltvskyde als source noch in den einstellungen drin stehen haben, bitte ich,
diese zu entfernen, da bei den jeweiligen hostern nur unnötig serverressourcen verbraucht werden.

die xml dateien werden auch nicht mehr unter dieser adresse zurück kommen.

danke

 

 

Willy

[doglover]

Response from Kodi (german userbase)

 

ok here is the result of some testing.

how does the iptv simple pvr addon work?
there is a setting for epg where you can insert an url to an xml file, e.g. the rytec xml url.
and there is a setting to cache this file. default is off i think, which is pretty bad in this case.
if cache is off, unfortunatly, for every single channel it is requesting the file .
if cache is on and the file is not an xml or corrupted, it ignores that and is also trying to download the url for every channel.

so it actually doesnt help if you create a dummy(corrupted) 1gb zipped file, because it is still downloading the small file for every channel, even if cache is on. and it doesn't look like its extracting the file.

pls get in contact with afedchin(author of iptv simple) at forum.kodi.tv and/or make a new thread here: forum.kodi.tv/forumdisplay.php?fid=215

you probably have to block kodi user somehow(user agent) if that helps - creating dummy files dont

 

 

So this is a disaster.  If somebody start zapping.  Request for every channel........

 

Willy

[Sjaaky]

Put a legit xml file with a clear statement about this situation on the server, so the user gets to see this statement instead of the scheduled program.

[WanWizard]

It is possible to create a conditional rewrite on user-agent string, to always serve such a file on Kodi requests.

 

But that still takes more resources than simply sending a 404 back, which I do on the firewall, the webserver doesn't even see the requests...

[doglover]

Put a legit xml file with a clear statement about this situation on the server, so the user gets to see this statement instead of the scheduled program.

 

This is one of the options I was considering, but have not put it in action. 

The problem with this is, I have to include a fair number of channels, otherwise the user (abuser) would not see it.

And this require a bit of work from me.

 

Willy

[doglover]

Jin wrote:

if you have an m3u playlist of 50 channels, it will/should request the epg file max 50 times.
the more channels people have in their playlist the more requests you get. the playlist i was providing had 60-70 channels and now 120+ (including radio

50 requests from each user. If you have 1000 users (not unlikely)
50 000 requests/day
German file is 5.12 MB. Or 256 GB /day
7.78 TB/month

Do I need to say more.

Willy

[doglover]

Answer from Kodi:

 

This does not appear to be due to our official version of IPTV Simple Client. The version of that add-on that we use in Kodi has EPG cacheing turned on by default and will abort an EPG download attempt after 3 failed tries. I believe the traffic you are seeing is due to a modified version of the add-on that is distributed from another site, which has these options turned off and is pre-configured to use your EPG URLs, which is why so many 404 requests come in.

I will attempt to find the people who are distributing the modified version and ask them to make the changes you have requested. Until then, blocking the Kodi user agent is probably the best way to deal with this.

-- Ned Scott,
Team Kodi

 

 

So a rogue, pimped version produced by a moron!!!!!

And everbody suffers.

 

As long as this version exist,  I cannot publish the sources.  Because everything will start over.

 

Willy

Edited by doglover, 7 April 2015 - 10:54.

[Rob van der Does]

At least the culprit has been found. At that's the first thing required for a solution.

[WanWizard]

In the meantime: let's hire "number 47".

[doglover]

In the meantime: let's hire "number 47".

 

I am sorry. Number 47?

 

Willy

[WanWizard]

I should have said "Agent 47": http://en.wikipedia.org/wiki/Agent_47

[doglover]

Answer from the author of the Kodi plug-in:

 

 

Hi,
 

Hi,

    Quote:
    there is a setting for epg where you can insert an url to an xml file, e.g. the rytec xml url.
    and there is a setting to cache this file. default is off i think, which is pretty bad in this case.

It's not true. Caching is enabled by default. GitHub

    Quote:
    if cache is off, unfortunatly, for every single channel it is requesting the file .

It's not true. The addon loads an EPG data (from cache if it is enabled and cache is not expired) only once for first channel GitHub. As you can see if EPG loaded and requested period is same the addon doesn't try to load EPG again. And it does not matter whether caching is enabled or not.

    Quote:
    if cache is on and the file is not an xml or corrupted, it ignores that and is also trying to download the url for every channel.

It's not true. If the addon cannot load file it try to load it again two times and if the file still cannot be loaded the addon marks EPG as loaded GitHub. And it does not matter whether caching is enabled or not.

The caching will work correctly if a server supports and sends Last-Modified header. If the server doesn't support this header the addon cannot compare time of modification of the cached file and the remote file and will try to load file again.

    Quote:
    The XMLTV files are only updated once a day, so continuous checks on updates are not needed. They only cause an heavy load on the servers.

Your servers is not alone. Another servers can update EPG a more often. The addon checks for updates only by HEAD request and if server replied correctly the addon will not load file again and again. Also users can configure update interval in Kodi settings TV -> Guide -> EPG update interval.

    Quote:
    plug-in seems to be badly programmed:

Wellcome to the GitHub with fixes. The addon is open sourced.

BR,
Anton

 

 

 

 

It will not be fixed very soon.

 

Willy