22 replies to this topic

[doglover]

There seems to be an image (or programme) around which downloads each minute the rytec.channels.xml.gz file from our servers.  This download occurs a complete day from several hundreds of IP addresses.  (Each IP address downloads the file repeatedly).

 

This sort of action causes DoS (Denial of Service) and crashes the server.

 

Is this an attempt to sabotage our efforts?  Or is there a stupid programmer, who thinks this is a valid action?

 

Willy

 

 

 

[needz]

I would rather think it's a programmer.. DoS would happen much more often than each minute..

 

http://xmltvepg.wanwizard.eu doesn't work now by the way, could you check it? Thanks.

[doglover]

There are several hundreds of IP address doing the same each minute.  Resulting in dozens of requests each second for the same file.

 

And yes wanwizard does not work today due to this reason.  The XMLTV EPG is piggybacked on a server which has also another purpose, and the whole server grinded to an halt.

 

Willy

Edited by doglover, 3 September 2014 - 07:56.

[MiLo]

There seems to be an image (or programme) around which downloads each minute the rytec.channels.xml.gz file from our servers.  This download occurs a complete day from several hundreds of IP addresses.  (Each IP address downloads the file repeatedly).

 

This sort of action causes DoS (Denial of Service) and crashes the server.

 

Is this an attempt to sabotage our efforts?  Or is there a stupid programmer, who thinks this is a valid action?

 

Willy

 

 

I suspect it's just sheer stupidity. A cron job which got the "day" and "minute" fields wrongs and now runs every minute instead of every day. Or a programmer who wanted to test something and later forgot to change it back before releasing his plugin.

 

This is probably combined with users who think that that plugin is the best thing since frozen pizza and don't have the foggiest idea that their boxes are actually performing a DDOS.

[WanWizard]

From log file analysis it seems to be located in Spain / Portugal, the same IP's also download the tvportugal and tvdplus epg files...

[doglover]

In Spain/Portugal the AZ-Box and Amiko Alien is very popular.

They now run also a sort of enigma2.  And they have also included EPG.  I assume the XMLTV files (or Crossepg) are adapted to download the Rytec files.

A bug in there?

 

Willy

[WanWizard]

Possible.

 

It looks like it downloads the "index" every minute, to see if there is a new version of the EPG, and when there is, it downloads it. Which seems to result in hunderds of hits per minute on an 80Kb file.

[doglover]

Possible.

 

It looks like it downloads the "index" every minute, to see if there is a new version of the EPG, and when there is, it downloads it. Which seems to result in hunderds of hits per minute on an 80Kb file.

 

 

Stupid action.  The files are only updated once a day. 

 

Willy

Edited by doglover, 3 September 2014 - 15:09.

[doglover]

What can we do?

Because this is a disaster.

 

Willy

[daniel2005]

Isn't it possible to record the ipadresses and ban them

[WanWizard]

The download system is distributed, not centrally managed, and we're talking hunderds of IP's. Banning them manually is unmanagable.

 

I'm looking into setting up fail2ban, and auto ban IP's that do more than x downloads in a given time period.

 

My server als has the problem of being the first one in the list, so code that doesn't select a mirror at random (like the xmlltvepg plugin does) will always hit my server...

[WanWizard]

And this solution is running now.

 

An IP will receive a ban of 12 hours if it tries to download the channel index more than 5 times in 10 minutes.

 

I've started xmltvepg.wanwizard.eu again, and it was immediately flooded with request again. After a bit of startup lag, I now see IP's being banned in rapid succession, and the number of requests only go down slowly. So this might be a more widespread problem than we think.

 

I really like to meet the person that thought this was a good idea, and give him my 2 cents...   

[WanWizard]

Until now, 5750 unique IP's have gotten a ban. Whatever it is, it's a sizable operation.
 

[Robinson]

For me, this forum is working slowly today and sometimes does not fully load.

Is this related to what you are talking about?

Or is my internet provider not performing as they should?

Edited by Robinson, 3 September 2014 - 21:50.

[WanWizard]

Yes, working on getting fail2ban operational, and to test that, I need to enable the rytec server again every now and then, which in turn enables the "ddos attack" again.

[Ampersand]

Until now, 5750 unique IP's have gotten a ban. Whatever it is, it's a sizable operation.
 

 

All of them are from Spain/Portuguese? If so maybe it is worth to ask some friends (if you have any) from these coutries to write such info on couple of most visited sat forums. If not users, then admins of this forum will be able to check/contact appropriate people to stop this.

[WanWizard]

I haven't checked all individual IP's, but they all seem to be downloading the Rytec files for Portugal and Dplus.

[doglover]

Dplus is Spain.

 

Willy

 

PS: I could produce a fake file for Portugal and Spain with a schedule each hour, saying: EPG disabled - check openpli.org for more info.

And publish here an explanation.  This could bring the culprit above water.

However, this will punish all users of the Spain and Portugal files.

 

Willy

[WanWizard]

Willy,

 

Can you check if other mirrors have the same problem, or it this a custom plugin (or something) that just happens to pick the first mirror in the list (which unfortunately is me)?

[doglover]

We will ask them.  But I have no idea when they will answer.

 

However your action, may provoke a stirr somewhere -  The WanWizard server is not up-to-date.  If this happens that would be very happy, which means there is a means of identifying the culprit.

 

Willy