25 replies to this topic

[Sicilian]

I'm told you may be able to help with this issue.

 

We've been hosting EPG Importer files on a standalone dedicated server for years.

 

Anyway we can no longer keep paying for this server so I've moved the hosting back to the server that hosts OpenViX website, feeds etc... Problem we have is when these EPG files are hosted the server gets hammered like a DDOS attack.

 

Do you have a way of blocking requests from devices other than E2 boxes?

[WanWizard]

I'm not sure I can be of any help, because our situation is rather different, we use CDN servers from our hosting provider, which are basically just caching proxies on OVH VPS in front of of the actual server.

 

They use this config (NGINX, replaced the upstrream hostname by xxxxxxxxxxxxx), which only accepts Python Twisted clients. It doesn't help with the hammering, but it does greatly reduce the bandwidth used and the load on the server.

 

Still, at peak time, the two servers still pump out around 200Mbps...

proxy_cache_path /data/rytecepg levels=1:2 keys_zone=cache_rytecepg:32m max_size=10g inactive=7d use_temp_path=off;

server {
    listen 80;
#    listen 443 ssl;
    listen [::]:80;
#    listen [::]:443 ssl;

    set $cache_uri $uri;
    server_name rytecepg.wanwizard.eu;

#    ssl_certificate /etc/nginx/certs/openpli-chained.crt;
#    ssl_certificate_key /etc/nginx/certs/openpli.key;
#    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
#    ssl_prefer_server_ciphers on;
#    ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";

    access_log  /var/log/nginx/rytecepg-access.log main;
    error_log /var/log/nginx/rytecepg-error.log error;

    # last update timestamp, do not cache

    location ~ "^/LastUpdate.txt$" {
        proxy_set_header Host xxxxxxxxxxxxx;
        proxy_hide_header access-control-allow-origin;
        add_header Access-Control-Allow-Origin "*";

        proxy_set_header X-Forwarded-For $remote_addr;

        proxy_pass http://xxxxxxxxxxxxx;
    }

    # sources xml's

    location ~ "^/rytec\.sources\.xml\.\d{8}\.gz$" {

        # block concurrent connections
        limit_conn addr 1;
        limit_conn_status 429;

        # data should not be zipped, already compressed
        gzip off;

        # and doesn't expire
        expires off;

        proxy_set_header Host xxxxxxxxxxxxx;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Accept-Encoding "";
        proxy_hide_header access-control-allow-origin;

        add_header Access-Control-Allow-Origin "*";
        add_header X-Cache $upstream_cache_status;

        proxy_hide_header X-Cache;
        proxy_ignore_headers Vary;
        proxy_ignore_headers Expires;
        proxy_ignore_headers Set-Cookie;
        proxy_ignore_headers Cache-Control;

        proxy_pass http://xxxxxxxxxxxxx;

        proxy_cache cache_rytecepg;
        proxy_cache_lock on;
        proxy_cache_lock_timeout 5s;
        proxy_cache_valid 200 1h;
        proxy_cache_valid 404 1h;
        proxy_cache_revalidate on;
        proxy_cache_use_stale error timeout http_500 http_502 http_503;
        proxy_cache_key rytecepg$uri$is_args$args;
    }

    # default for all other files

    location / {

        # Zabbix monitoring
        if ($http_user_agent = "Zabbix") {
            return 200;
        }

        set $ret 0;

        # Python Twisted
        if ($http_user_agent = "Twisted Client") {
            set $ret 200;
        }
        if ($http_user_agent = "Twisted client") {
            set $ret 200;
        }

        if ($ret != 200) {
            return 403;
        }

        # limit concurrent connections
        limit_conn addr 4;
        limit_conn_status 429;

       # data should not be zipped
        gzip off;

        # and doesn't expire
        expires off;

        proxy_set_header Host xxxxxxxxxxxxx;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Accept-Encoding "";
        proxy_hide_header access-control-allow-origin;

        add_header Access-Control-Allow-Origin "*";
        add_header X-Cache $upstream_cache_status;

        proxy_hide_header X-Cache;
        proxy_ignore_headers Vary;
        proxy_ignore_headers Expires;
        proxy_ignore_headers Set-Cookie;
        proxy_ignore_headers Cache-Control;

        proxy_pass http://xxxxxxxxxxxxx;

        proxy_cache cache_rytecepg;
        proxy_cache_lock on;
        proxy_cache_lock_timeout 5s;
        proxy_cache_valid 200 10m;
        proxy_cache_valid 404 1h;
        proxy_cache_revalidate on;
        proxy_cache_use_stale error timeout http_500 http_502 http_503;
        proxy_cache_key rytecepg$uri$is_args$args;
    }

}

 

[Sicilian]

Thanks for that, but I don't think that will be a solution with our server.

 

I'm wondering if there's anyway to block Android devices, something like this maybe using a htaccess file: .htaccess - Android/iOS htaccess detection and redirecting - Stack Overflow

[WanWizard]

This

        # Python Twisted
        if ($http_user_agent = "Twisted Client") {
            set $ret 200;
        }
        if ($http_user_agent = "Twisted client") {
            set $ret 200;
        }

        if ($ret != 200) {
            return 403;
        }

effectly blocks all clients not using Python twisted, this is basically a "block all except", instead of an "allow all except".

 

You can do the same in Apache, something like

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ! "Twisted Client" [NC]
RewriteCond %{HTTP_USER_AGENT} ! "Twisted client" [NC]
RewriteRule ^ - [F,L]

which will return a 403 for all requests with other user agents. (the C and c checks are because of different twisted versions).
 

[WanWizard]

According to our provider, it blocks quite a lot.

 

This is from today on one of the CDN nodes:

[admin@cdn2 nginx]# grep -c " 200 " rytecepg-access.log
57540

[admin@cdn2 nginx]# grep -c " 403 " rytecepg-access.log
187788

[admin@cdn2 nginx]# grep -c " 404 " rytecepg-access.log
11307

which is about 75% blocked...

[WanWizard]

BTW, it might be something we all should look into.

 

Instead of Willy having to update lots of different servers (with public FTP access, brrrrr), why not have one master server (like we use) which is updated by Willy but not publicly accessable, and have all other EPG nodes simply run an nginx caching proxy?

[doglover]

A lot of the problems start with IPTV players which want EPG.

 

Some of these can be configured, to retrieve EPG each time you switch a channel.  Since each of our files contain a number of channels (some over 300 channels)  so each time you request one of these channels, the complete file is being downloaded.  Up to 300 times, if you start zapping.

 

Secondly, Kodi EPG plug-ins can all be setup to download EPG at regular intervals, like once each hour.  Which is overkill as the EPG is refreshed only once a day in the morning.

Teaching all Kodi adepts to only download the EPG once a day is likely to be impossible.

 

Rytec EPG has 69 XMLTV files.

Most user can do with selecting less than 20 files.  But there are always guys who will select the lot.

 

Why not set a limit how much files each IP-address can download in a day. Once the limit is reached, the IP-address will be blocked for the rest of the day.
f.i. a limit of 40 files per IP-address would be enough for each user.  There are 5 mirrors, so if one blocks them, they can download 40 files from each of the other mirrors.  (if all the mirrors have the same limitation)

 

I am no network specialist.  So I do not have any idea how to implement this.

[Sicilian]

This

        # Python Twisted
        if ($http_user_agent = "Twisted Client") {
            set $ret 200;
        }
        if ($http_user_agent = "Twisted client") {
            set $ret 200;
        }

        if ($ret != 200) {
            return 403;
        }

effectly blocks all clients not using Python twisted, this is basically a "block all except", instead of an "allow all except".

 

You can do the same in Apache, something like

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ! "Twisted Client" [NC]
RewriteCond %{HTTP_USER_AGENT} ! "Twisted client" [NC]
RewriteRule ^ - [F,L]

which will return a 403 for all requests with other user agents. (the C and c checks are because of different twisted versions).
 

 

I'm testing out that htaccess now, thx. I'll report back.

[Sicilian]

Actually looks like that htaccess blocks e2 boxes too:

 

login as: root

root@192.168.11.49's password:

root@vuultimo4k:~# wget http://openvix.co.uk.../LastUpdate.txt

--2024-10-10 16:03:56--  http://openvix.co.uk.../LastUpdate.txt

Resolving openvix.co.uk... 176.31.181.161

Connecting to openvix.co.uk|176.31.181.161|:80... connected.

HTTP request sent, awaiting response... 500 Internal Server Error

2024-10-10 16:03:56 ERROR 500: Internal Server Error.

 

root@vuultimo4k:~# wget http://openvix.co.uk...channels.xml.xz

--2024-10-10 16:05:03--  http://openvix.co.uk...channels.xml.xz

Resolving openvix.co.uk... 176.31.181.161

Connecting to openvix.co.uk|176.31.181.161|:80... connected.

HTTP request sent, awaiting response... 500 Internal Server Error

2024-10-10 16:05:03 ERROR 500: Internal Server Error.

[OpenSourcerer]

Because you haven't allowed wget. It will work if you use epgimport which uses python Twisted Client.

[Sicilian]

Ok, thanks, I'll give it a try.

[WanWizard]

Because you haven't allowed wget. It will work if you use epgimport which uses python Twisted Client.

 

Exactly, that was the point of this excersise!

[WanWizard]

Why not set a limit how much files each IP-address can download in a day. Once the limit is reached, the IP-address will be blocked for the rest of the day.
f.i. a limit of 40 files per IP-address would be enough for each user.  There are 5 mirrors, so if one blocks them, they can download 40 files from each of the other mirrors.  (if all the mirrors have the same limitation)

 

I am no network specialist.  So I do not have any idea how to implement this.

 

That is very complex, requires a lot of IP tracking, and it still means you serve data (which uses bandwidth) to clients you don't want to send a single byte to.

 

Also, if I have one Enigma box and three Android stream boxes in my home network, all 4 will connect with the same public IP, so the android boxes will quickly use up my allowance, and Enigma can't download anymore because my IP is blocked for the rest of the day.

[Sicilian]

Anyway to manually run Epgimporter via telnet and force it to my server to test if it will still download?

[WanWizard]

Don't think so.

 

You can force it to go to your server by editing /etc/enigma2/epgimport/rytec.sources.xml and remove all the other servers, or add those hostnames to /etc/hosts with a 127.0.0.1 IP, so the download would fail for them.

[Sicilian]

This seems to also, thx to Ev0:

 

Un-hash the wget lines to test and allow wget.

#SetEnvIfNoCase User-Agent .*wget.* systems 
#SetEnvIfNoCase User-Agent .*Wget.* systems
SetEnvIfNoCase User-Agent .*Twisted Client.* systems 
SetEnvIfNoCase User-Agent .*Twisted client.* systems  
Order Deny,Allow 
Deny from All 
Allow from env=systems

[Sicilian]

Don't think so.

 

You can force it to go to your server by editing /etc/enigma2/epgimport/rytec.sources.xml and remove all the other servers, or add those hostnames to /etc/hosts with a 127.0.0.1 IP, so the download would fail for them.

Thanks, I'll try that tomorrow. I run it on a box and log it in Telnet.

[WanWizard]

This seems to also, thx to Ev0

 

Yup, that works too. Note that the "Order / Deny / Allow" syntax depends on your Apache version.

[Sicilian]

Well sadly this didn't help, completely killed the server. So I've got no choice but to take it offline for now until a solution can be found.

[littlesat]

With the option '--user-agent=' you can still work-a-round it when using wget....