Forums

Every night, here, there is a sync/backup over the network between 2 USB drives (both connected to a different AP which are connected by UTP (cat5)) with:

 

rsync -av --delete /tmp/mnt/USB1/ -e "ssh -i /opt/usr/authorized_keys/id_rsa" 192.168.5.9:/tmp/mnt/USB2/

 

This is running at about 1,1MB/s. That's not much, but the data changes aren't huge, say between 0-2gb. I decided to lower the security (as the backup is done within the LAN, security is not a real issue) and I changed the above line to:

 

rsync -av --delete /tmp/mnt/USB1/ --rsh="ssh -i /opt/usr/authorized_keys/id_rsa" 192.168.5.9:/tmp/mnt/USB2/

This gave an increase of about 36% to 1,5 MB/s.

 

 With a simple "cp"  I reached speeds around 5 MB/s. Maybe I should check where the bottle neck is with a tool like "iperf" ? If someone has any idea thanks in advance
 

Edited by 40H3X, 17 March 2016 - 09:05.

Perhaps you find something here: The fastest remote directory rsync over ssh archival I can muster (40MB/s over 1gb NICs)

Most likely the limiting factor is your usb port (2.0)

rsync is bidirectional checking/comparing (technical not correct, my user point of view) while copy is mostly one way.

Edited by Jan Gruuthuse, 17 March 2016 - 13:12.

Also:

 

- AP's are mentioned, which suggests there is wireless in the path, which can slow things down considerably.

- cp are large packets, one way (a bit like ftp), rsync are a lot of small packets.

- depending on the hardware at either end, ssh encryption calculations can slow things down.

I doubt this:

 

Also:

 

- AP's are mentioned, which suggests there is wireless in the path, which can slow things down considerably.

 

probably not as:

>8 >8 different AP which are connected by UTP (cat5)) with: >8 >8

as the backup is done within the LAN

If wifi is/was used, I'm put on the wrong foot, seeing the references to wired connection

Edited by Jan Gruuthuse, 17 March 2016 - 13:47.

I read that as "the two AP's are connected to each other via UTP". If there was an end-to-end utp connection, why mention access points?

Could be right. On the other hand AP could mean plenty : cat5 wall socket, combined router, ... (Dutch: toegangspunt)

If Wifi is in play, start laying out a cat6 network, probably the biggest speed gain there is.

If distance would be an issue cat6 -> fibre -> cat6

Edited by Jan Gruuthuse, 17 March 2016 - 13:55.

Don't argee about context with an autistic person. There isn't any...

@ Jan and @WW thanks for the reply's.

 

Sorry, I can understand the confusion, apart from the questionable sanity of the setup I only mentioned the AP's as to explain how the USB drives are connected to the network. Rsync is installed on both AP's

 

USB1drive----AP1=======UTP=======AP2----USB2drive

Edited by 40H3X, 17 March 2016 - 15:22.

Ah, I didn't even get that. So they are like fritzboxes or so, routers with a USB port to be used as a poor-mans fileserver?

 

In that case you can count on it, it's a combination of a poor performing USB port, and a very weak CPU for the required encryption calculations. And that is probably true for both sides.

 

I tried it once, connect a USB disk to my router, which is quite a high-end Draytek, and performance was a nightmare. I now connect them to the USB port of my Synology, with better results.

Yes, as you so rightly pointed out this a poor man's setup ....to say something for the WiFi router ....at least they aren't fritzboxes and a course a server in this setup would be preferable, but for now it has to do. Is it possbile to check if the bottle neck is the CPU?

Edited by 40H3X, 17 March 2016 - 15:48.

If the router can export the disk using nfs, you can mount the disk directly on the other router, and use a local path rsync without the ssh overhead. This is how I do it (from my LAN devices to my syno USB disk).

RSync by default calculates checksums and tries to "partially" update files.

If the files aren't being "changed" but just added/removed, there's a "whole file only" flag that you can pass to rsync, which will then skip the useless checksumming, resulting in much better speeds.

Update: Thanks everyone for the time and good suggestions. In this case (after some debate ) I convinced the person in question to move over to a more sane and professional setup.

Edited by 40H3X, 18 March 2016 - 10:28.

Enable the rsync inplace option and when using ssh disable rsync compression. Also look at the fuzzy option which can speed up things.