Jump to content


Photo

rsync v3.1.2

Increasing speeds

  • Please log in to reply
13 replies to this topic

#1 40H3X

  • Forum Moderator
    PLi® Contributor
  • 5,782 posts

+186
Excellent

Posted 17 March 2016 - 09:03

Every night, here, there is a sync/backup over the network between 2 USB drives (both connected to a different AP which are connected by UTP (cat5)) with:

 

rsync -av --delete /tmp/mnt/USB1/ -e "ssh -i /opt/usr/authorized_keys/id_rsa" 192.168.5.9:/tmp/mnt/USB2/

 

This is running at about 1,1MB/s. That's not much, but the data changes aren't huge, say between 0-2gb. I decided to lower the security (as the backup is done within the LAN, security is not a real issue) and I changed the above line to:

 

rsync -av --delete /tmp/mnt/USB1/ --rsh="ssh -i /opt/usr/authorized_keys/id_rsa" 192.168.5.9:/tmp/mnt/USB2/

This gave an increase of about 36% to 1,5 MB/s.

 

 With a simple "cp"  I reached speeds around 5 MB/s. Maybe I should check where the bottle neck is with a tool like "iperf" ? If someone has any idea thanks in advance ;)
 


Edited by 40H3X, 17 March 2016 - 09:05.

Hardware: Vu+ Uno 4K SE - Vu+ Duo 4K  - Fuba 78 cm - Tripleblock LNB Quad 19.2/23.5/28.2 - DS918+
Software : OpenPLi - OSCam - Settings van Hans - Autotimer - EPGImport

---------------------------------------------------------------------------------------------------------------------------------------

Remember: Upvote with the rep_up.png button for any user/post you find to be helpful, informative, or deserving of recognition!

---------------------------------------------------------------------------------------------------------------------------------------

Many answers to your question can be found in our new and improved wiki

Note: I do not provide support via PM !.


Re: rsync v3.1.2 #2 Jan Gruuthuse

  • Senior Member
  • 985 posts

+20
Neutral

Posted 17 March 2016 - 13:12

Perhaps you find something here: The fastest remote directory rsync over ssh archival I can muster (40MB/s over 1gb NICs)

Most likely the limiting factor is your usb port (2.0)

rsync is bidirectional checking/comparing (technical not correct, my user point of view) while copy is mostly one way.


Edited by Jan Gruuthuse, 17 March 2016 - 13:12.

Sf8HD, sf8008, OpenPli 9.0, fixed lnb 28.2E, 19.2E

Re: rsync v3.1.2 #3 WanWizard

  • PLi® Core member
  • 68,309 posts

+1,719
Excellent

Posted 17 March 2016 - 13:32

Also:

 

- AP's are mentioned, which suggests there is wireless in the path, which can slow things down considerably.

- cp are large packets, one way (a bit like ftp), rsync are a lot of small packets.

- depending on the hardware at either end, ssh encryption calculations can slow things down.


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: rsync v3.1.2 #4 Jan Gruuthuse

  • Senior Member
  • 985 posts

+20
Neutral

Posted 17 March 2016 - 13:46

I doubt this:

 

Also:

 

- AP's are mentioned, which suggests there is wireless in the path, which can slow things down considerably.

 

probably not as:

>8 >8 different AP which are connected by UTP (cat5)) with: >8 >8

as the backup is done within the LAN

If wifi is/was used, I'm put on the wrong foot, seeing the references to wired connection :P


Edited by Jan Gruuthuse, 17 March 2016 - 13:47.

Sf8HD, sf8008, OpenPli 9.0, fixed lnb 28.2E, 19.2E

Re: rsync v3.1.2 #5 WanWizard

  • PLi® Core member
  • 68,309 posts

+1,719
Excellent

Posted 17 March 2016 - 13:48

I read that as "the two AP's are connected to each other via UTP". If there was an end-to-end utp connection, why mention access points?


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: rsync v3.1.2 #6 Jan Gruuthuse

  • Senior Member
  • 985 posts

+20
Neutral

Posted 17 March 2016 - 13:53

Could be right. On the other hand AP could mean plenty : cat5 wall socket, combined router, ... (Dutch: toegangspunt)

If Wifi is in play, start laying out a cat6 network, probably the biggest speed gain there is.

If distance would be an issue cat6 -> fibre -> cat6


Edited by Jan Gruuthuse, 17 March 2016 - 13:55.

Sf8HD, sf8008, OpenPli 9.0, fixed lnb 28.2E, 19.2E

Re: rsync v3.1.2 #7 WanWizard

  • PLi® Core member
  • 68,309 posts

+1,719
Excellent

Posted 17 March 2016 - 14:22

Don't argee about context with an autistic person. There isn't any... :wacko: :P


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: rsync v3.1.2 #8 40H3X

  • Forum Moderator
    PLi® Contributor
  • 5,782 posts

+186
Excellent

Posted 17 March 2016 - 15:20

@ Jan and @WW thanks for the reply's.

 

Sorry, I can understand the confusion, apart from the questionable sanity of the setup I only mentioned the AP's as to explain how the USB drives are connected to the network. Rsync is installed on both AP's

Although I known this setup is far from ideal, it was choosen with the philosophy .....better some of a pudding than none of a pie, you could say ;)

 

USB1drive----AP1=======UTP=======AP2----USB2drive


Edited by 40H3X, 17 March 2016 - 15:22.

Hardware: Vu+ Uno 4K SE - Vu+ Duo 4K  - Fuba 78 cm - Tripleblock LNB Quad 19.2/23.5/28.2 - DS918+
Software : OpenPLi - OSCam - Settings van Hans - Autotimer - EPGImport

---------------------------------------------------------------------------------------------------------------------------------------

Remember: Upvote with the rep_up.png button for any user/post you find to be helpful, informative, or deserving of recognition!

---------------------------------------------------------------------------------------------------------------------------------------

Many answers to your question can be found in our new and improved wiki

Note: I do not provide support via PM !.


Re: rsync v3.1.2 #9 WanWizard

  • PLi® Core member
  • 68,309 posts

+1,719
Excellent

Posted 17 March 2016 - 15:29

Ah, I didn't even get that. So they are like fritzboxes or so, routers with a USB port to be used as a poor-mans fileserver?

 

In that case you can count on it, it's a combination of a poor performing USB port, and a very weak CPU for the required encryption calculations. And that is probably true for both sides.

 

I tried it once, connect a USB disk to my router, which is quite a high-end Draytek, and performance was a nightmare. I now connect them to the USB port of my Synology, with better results.


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: rsync v3.1.2 #10 40H3X

  • Forum Moderator
    PLi® Contributor
  • 5,782 posts

+186
Excellent

Posted 17 March 2016 - 15:48

Yes, as you so rightly pointed out this a poor man's setup ....to say something for the WiFi router ....at least they aren't fritzboxes ;) and a course a server in this setup would be preferable, but for now it has to do. Is it possbile to check if the bottle neck is the CPU?


Edited by 40H3X, 17 March 2016 - 15:48.

Hardware: Vu+ Uno 4K SE - Vu+ Duo 4K  - Fuba 78 cm - Tripleblock LNB Quad 19.2/23.5/28.2 - DS918+
Software : OpenPLi - OSCam - Settings van Hans - Autotimer - EPGImport

---------------------------------------------------------------------------------------------------------------------------------------

Remember: Upvote with the rep_up.png button for any user/post you find to be helpful, informative, or deserving of recognition!

---------------------------------------------------------------------------------------------------------------------------------------

Many answers to your question can be found in our new and improved wiki

Note: I do not provide support via PM !.


Re: rsync v3.1.2 #11 WanWizard

  • PLi® Core member
  • 68,309 posts

+1,719
Excellent

Posted 17 March 2016 - 16:02

If the router can export the disk using nfs, you can mount the disk directly on the other router, and use a local path rsync without the ssh overhead. This is how I do it (from my LAN devices to my syno USB disk).


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: rsync v3.1.2 #12 MiLo

  • PLi® Core member
  • 14,042 posts

+298
Excellent

Posted 17 March 2016 - 18:25

RSync by default calculates checksums and tries to "partially" update files.

If the files aren't being "changed" but just added/removed, there's a "whole file only" flag that you can pass to rsync, which will then skip the useless checksumming, resulting in much better speeds.
Real musicians never die - they just decompose

Re: rsync v3.1.2 #13 40H3X

  • Forum Moderator
    PLi® Contributor
  • 5,782 posts

+186
Excellent

Posted 18 March 2016 - 10:28

Update: Thanks everyone for the time and good suggestions. In this case (after some debate ;) ) I convinced the person in question to move over to a more sane and professional setup.


Edited by 40H3X, 18 March 2016 - 10:28.

Hardware: Vu+ Uno 4K SE - Vu+ Duo 4K  - Fuba 78 cm - Tripleblock LNB Quad 19.2/23.5/28.2 - DS918+
Software : OpenPLi - OSCam - Settings van Hans - Autotimer - EPGImport

---------------------------------------------------------------------------------------------------------------------------------------

Remember: Upvote with the rep_up.png button for any user/post you find to be helpful, informative, or deserving of recognition!

---------------------------------------------------------------------------------------------------------------------------------------

Many answers to your question can be found in our new and improved wiki

Note: I do not provide support via PM !.


Re: rsync v3.1.2 #14 openpli4me

  • Senior Member
  • 74 posts

0
Neutral

Posted 18 March 2016 - 10:29

Enable the rsync inplace option and when using ssh disable rsync compression. Also look at the fuzzy option which can speed up things.


1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users