62 replies to this topic

[doglover]

I received this e-mail from one of the host of the XMLTV files:

 

My provider response:

Thank you for your email and request. In answer to your question:

All additional traffic above the 5TB Commitment which you have will be
charged against Euro 0,10 / GB. Where these costs not added to the
confirmation email?

At this moment the calculation is that you will be doing 30TB, this
will mean a surcharge of Euro 2500,00.

THE END!

 

All the caused by this: http://forums.openpl...y/?hl=stupidity

 

Will those responsible for this stop with their actions, pls.

Or we will be forced to close down.

 

Will

Edited by doglover, 17 March 2015 - 09:15.

[diglam]

last time you solved problem with fail2ban, isn't it working now?

[WanWizard]

This is about another server.

 

The message doesn't say if all traffic can be contributed to XMLTV data, or if other stuff is running on that server too.

[doglover]

On one website we see for March ( 18 days)

 

  /rytec.channels-greece-en.xml.gz 233 589,549 1.39 GB 2.47 KB

 

589549  (over 500 hundred thousand) downloads of a channels file.  1.39 GB which is more than the other combined normal downloads.

Stupidity? or sabotage?

 

Will those responsible for this, stop doing this.

I guess those downloading the greek english EPG will know they are doing this.  Will they check the set-up pls.

 

The problem started somewhere in the middle of February 2015.  Before this file has been downloaded 0 times.

 

Willy

Edited by doglover, 18 March 2015 - 10:37.

[jeanclaude]

if there is no response, you could try emptying that channels file and see what happens.

if the downloads start again, but with another channels file then you know it's deliberate sabotage.

[doglover]

BTW:  whois  93.49.233.12

An Italian provider

 

Stop with requesting enormous amount of files!!!!!!

 

 

Willy

[WanWizard]

Tell them to run fail2ban too.

 

Since it's only useful to retreive the EPG once a day, I ban every IP with too many requests in a certain time. If you have multiple boxes behind NAT, just have them collect the EPG an hour apart to avoid the ban.

 

To give an impression of scale:

 

I have 47696 IP's currently in my ban list, Out of 3337993 requests. Some details:

 pkts bytes target     prot opt in     out     source               destination
 105K 6299K REJECT     all  --  *      *       87.196.132.248       0.0.0.0/0           reject-with icmp-port-unreachable
 122K 7313K REJECT     all  --  *      *       89.95.46.201         0.0.0.0/0           reject-with icmp-port-unreachable
 120K 7208K REJECT     all  --  *      *       178.166.7.127        0.0.0.0/0           reject-with icmp-port-unreachable
 109K 6546K REJECT     all  --  *      *       84.90.169.116        0.0.0.0/0           reject-with icmp-port-unreachable
 118K 7052K REJECT     all  --  *      *       213.22.122.16        0.0.0.0/0           reject-with icmp-port-unreachable
 117K 6996K REJECT     all  --  *      *       37.189.229.97        0.0.0.0/0           reject-with icmp-port-unreachable
 113K 6777K REJECT     all  --  *      *       84.91.98.237         0.0.0.0/0           reject-with icmp-port-unreachable
 119K 7123K REJECT     all  --  *      *       84.73.83.29          0.0.0.0/0           reject-with icmp-port-unreachable
 123K 7369K REJECT     all  --  *      *       37.189.254.210       0.0.0.0/0           reject-with icmp-port-unreachable
 120K 7186K REJECT     all  --  *      *       84.90.40.41          0.0.0.0/0           reject-with icmp-port-unreachable
 120K 7218K REJECT     all  --  *      *       89.114.33.186        0.0.0.0/0           reject-with icmp-port-unreachable
 102K 6114K REJECT     all  --  *      *       82.224.43.140        0.0.0.0/0           reject-with icmp-port-unreachable
 121K 7278K REJECT     all  --  *      *       89.155.49.214        0.0.0.0/0           reject-with icmp-port-unreachable
 120K 7212K REJECT     all  --  *      *       81.13.137.239        0.0.0.0/0           reject-with icmp-port-unreachable
 123K 7351K REJECT     all  --  *      *       88.219.157.156       0.0.0.0/0           reject-with icmp-port-unreachable
 202K   12M REJECT     all  --  *      *       89.154.176.103       0.0.0.0/0           reject-with icmp-port-unreachable
 211K   13M REJECT     all  --  *      *       188.251.13.3         0.0.0.0/0           reject-with icmp-port-unreachable
 110K 6574K REJECT     all  --  *      *       84.91.21.167         0.0.0.0/0           reject-with icmp-port-unreachable
 113K 6788K REJECT     all  --  *      *       176.79.134.169       0.0.0.0/0           reject-with icmp-port-unreachable
 111K 6618K REJECT     all  --  *      *       84.91.119.199        0.0.0.0/0           reject-with icmp-port-unreachable
 246K   15M REJECT     all  --  *      *       84.91.208.86         0.0.0.0/0           reject-with icmp-port-unreachable
 171K   10M REJECT     all  --  *      *       193.91.35.252        0.0.0.0/0           reject-with icmp-port-unreachable
 169K   10M REJECT     all  --  *      *       79.169.39.206        0.0.0.0/0           reject-with icmp-port-unreachable
 210K   13M REJECT     all  --  *      *       178.166.64.136       0.0.0.0/0           reject-with icmp-port-unreachable
 243K   15M REJECT     all  --  *      *       89.154.142.236       0.0.0.0/0           reject-with icmp-port-unreachable
 241K   14M REJECT     all  --  *      *       95.95.20.72          0.0.0.0/0           reject-with icmp-port-unreachable
 177K   11M REJECT     all  --  *      *       78.130.68.229        0.0.0.0/0           reject-with icmp-port-unreachable
 332K   20M REJECT     all  --  *      *       84.91.121.158        0.0.0.0/0           reject-with icmp-port-unreachable
 241K   14M REJECT     all  --  *      *       188.93.193.64        0.0.0.0/0           reject-with icmp-port-unreachable
 263K   16M REJECT     all  --  *      *       89.180.4.149         0.0.0.0/0           reject-with icmp-port-unreachable
 199K   12M REJECT     all  --  *      *       88.214.171.251       0.0.0.0/0           reject-with icmp-port-unreachable
 183K   11M REJECT     all  --  *      *       82.236.5.85          0.0.0.0/0           reject-with icmp-port-unreachable
 186K   11M REJECT     all  --  *      *       84.101.241.80        0.0.0.0/0           reject-with icmp-port-unreachable
 313K   19M REJECT     all  --  *      *       46.127.223.233       0.0.0.0/0           reject-with icmp-port-unreachable
 709K   35M REJECT     all  --  *      *       188.37.99.166        0.0.0.0/0           reject-with icmp-port-unreachable 

So you see there are IP's with a rediculous amount of requests!

 

332K requests from a single IP? You got to be kidding, there is nothing normal about this anymore...

[doglover]

As a consequence of all this bullying SIFteam has quit as an distributor of the XMLTV files.  Since they also generate and distributed the XEPGdb files for Crossepg.  These file have also disappeared from the net.

 

Willy

[pluizebol]

As a consequence of all this bullying SIFteam has quit as an distributor of the XMLTV files.  Since they also generate and distributed the XEPGdb files for Crossepg.  These file have also disappeared from the net.

 

Willy

 

Willy

Does it mean that we should block SIFTeam in our Rytec-sources ?

[doglover]

Not really.  The website is unavailable now.  The plug-in takes care of this itself.

In the next update this website will be removed.

 

Willy

[Pr2]

Hi,

 

Perhaps we should think to change the way of working of XMLTV import, why not use torrent to download the files that we need for our EPG?

This will spread the workload/bandwidth on everybody.

 

Pr2

[MiLo]

Interestingly, the torrent approach would make it best if all boxes start downloading at the same time.

We already have CTorrent and Transmission in the feeds, they work well, so it should be possible to build a working prototype by simply replacing the "gz" download in the plugin by a call to transmission and waiting for the data to arrive. Any takers?

Guess that for the download, clients just need to get today's "hash" from a server, and with just a few seeders it'll run pretty smooth regardless of how many downloaders there are.

Edited by MiLo, 20 March 2015 - 11:47.

[WanWizard]

Can that same mechanism be used for the feeds? Because those suffer from the same issue at the moment.

 

It would also mean you can take a server down (for maintenance or other reasons) without client being impacted too much.

[MiLo]

Technically, sure, you can transfer anything by torrent. But that'd mean some heavy coding in opkg I guess...

[WanWizard]

I wonder.

 

Currently opkg downloads happen through a wget script, externally to the opkg binary. You could convert that to a torrent download instead of a wget?

[doglover]

For XMLTV plugin the rewrite should not be that dramatic.  The plug-in can import local files. 

If the existing torrent client could be configured to download the needed XMLTV files each day via a torrent, and upon completion of the download start the XMLTV import plug-in, the problem is solved.

 

Of course we will need to construct a set of torrents and provide them one way or another.

 

Willy

[WanWizard]

Seems we're not the first with this idea: http://www.ebaytechb...e/#.VQwYSsv1neQ

 

I think it's worth a look at.

[topol9]

Hallo. say mee please whot happened ned kanals. befor i have epg  not problem this   http://xmltvepg.wanw...tecxmltvbenl.gz    naw  not work epg    maby yuo have anase link work?

[doglover]

Hallo. say mee please whot happened ned kanals. befor i have epg  not problem this   http://xmltvepg.wanw...tecxmltvbenl.gz    naw  not work epg    maby yuo have anase link work?

 

For what purpose do you use this file?

 

If it is on a enigma box.  Pls. update the importer.

 

Willy

[topol9]

iptv media stalker