Questions tagged [rsync]
Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon
1,614
questions
385
votes
20
answers
376k
views
Showing total progress in rsync: is it possible?
I have searched for this option already, but have only found solutions that involve custom patching. The fact that it does not show in --help and no more info can be found probably indicates the ...
- 8,589
340
votes
18
answers
447k
views
Copying a large directory tree locally? cp or rsync?
I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp.
I'm worried about permissions ...
- 32.5k
278
votes
3
answers
211k
views
What is archive mode in rsync?
I know you can use -a or --archive to activate archive mode when using rsync. Unfortunately, I have no idea what archive mode is supposed to do, and the man page is not at all explicit about what this ...
- 2,927
215
votes
18
answers
390k
views
How to use rsync over FTP
Any unix:
I have the following cmd line which works fine.
rsync -avr -e ssh /home/dir [email protected]:/home/
But I need to set it up now to rsync to a remote server that only has an FTP server on ...
- 2,353
140
votes
5
answers
193k
views
Rsync difference between --checksum and --ignore-times options
Can anyone clarify the differences between the --checksum and --ignore-times options of rsync?
My understanding is as follows:
--checksum
If the file size and time match, it will do a checksum at ...
- 1,667
103
votes
4
answers
92k
views
Perform rsync while following sym links
I have a directory that contains symbolic links to other directories located on different media on my system:
/opt/lun1/2011
/opt/lun1/2010
/opt/lun2/2009
/opt/lun2/2008
/opt/lun3/2007
But the ...
- 2,242
96
votes
18
answers
155k
views
Why hasn't rsync caught on in the Windows world? [closed]
It seems that rsync is the de-facto standard for efficient file backup and sync in Unix/Linux.
Does anyone have any thoughts on why it wouldn't have caught on in the Windows world?
Why hasn't it ...
93
votes
2
answers
58k
views
Rsync creates a directory with the same name inside of destination directory
When I run this command
rsync -avzp --del -e "ssh -p myport" user@hostname:/var/www/tests /var/www/tests
files get synchronized but instead of saving files in /var/www/tests, Rsync creates one more ...
- 933
89
votes
12
answers
112k
views
protocol version mismatch -- is your shell clean?
When following the instructions to do rsync backups given here: http://troy.jdmz.net/rsync/index.html
I get the error "protocol version mismatch -- is your shell clean?"
I read somewhere ...
- 1,255
88
votes
4
answers
96k
views
How to keep rsync from chown'ing transferred files?
I have an sshfs connection setup with a remote filesystem on a Linux server. I'm doing an rsync from my local server to the ftpfs-filesystem. Because of the nature of this setup, I can't chown ...
- 8,964
85
votes
2
answers
127k
views
Why is rsync skipping the main directory?
I'm trying to use rync locally (on a windows machine) to a remote server (my osx box) in order to test a remote deploy build script. I've done rsync before just fine between 2 linux servers, but I'm ...
- 1,007
84
votes
8
answers
158k
views
Is it possible to use rsync over sftp (without an ssh shell)?
Rsync over ssh, works great every time.
However, trying to rsync to a host which allows only sftp logins, but not ssh logins, provides the following error:
rsync -av /source ssh user@remotehost:/...
- 18.2k
80
votes
4
answers
54k
views
How to keep the full path with rsync?
When backing up with rsync, How do I keep the full directory structure?
For example, the remote server is saturn, and I want to backup saturn's /home/udi/files/pictures to a local directory named ...
- 13.7k
67
votes
3
answers
8k
views
Rsync triggered Linux OOM killer on a single 50 GB file
I have a single 50 GB file on server_A, and I'm copying it to server_B. I run
server_A$ rsync --partial --progress --inplace --append-verify 50GB_file root@server_B:50GB_file
Server_B has 32 GB ...
- 833
64
votes
8
answers
47k
views
Rsync : copying over timestamps only
Currently I have two directories A/ and B/ which are identical in every respect, with the exception of the timestamps. Therefore if I run the command :
rsync --dry-run -crvv A/ B/
then all files ...
- 1,049
63
votes
2
answers
75k
views
Can rsync display current average speed?
When I'm transferring large quantities of data using rsync, it would be helpful if I could have the average speed up until now at a glance, rather than a bunch of different speeds for each file.
- 1,283
60
votes
18
answers
30k
views
Favorite rsync tips and tricks
The more I use rsync the more I realise that it's a swiss army knife of file transfer. There are so many options. I recently found out that you can go --remove-source-files and it'll delete a file ...
- 32.5k
60
votes
1
answer
44k
views
rsync - Exclude files that are over a certain size?
I am doing a backup of my desktop to a remote machine. I'm basically doing rsync -a ~ example.com:backup/ However there are loads of large files, e.g. wikipedia dumps etc. Most of the files I care a ...
- 32.5k
58
votes
5
answers
57k
views
Can I make rsync output only the summary?
I use rsync to backup a directory which is very big, containing many sub-directories and files, so I don't want to see the "incremental file list". I just want to know the summary in the end. If I use ...
- 583
57
votes
4
answers
164k
views
Why is my rsync so slow?
My Laptop and my workstation are both connected to a Gigabit Switch. Both are running Linux. But when I copy files with rsync, it performs badly.
I get about 22 MB/s. Shouldn't I theoretically get ...
- 844
54
votes
3
answers
86k
views
Rsync to AWS S3 bucket
For a server I am hosting a website on I want to backup the data and settings to an S3 bucket. I found out that you can't directly use rsync to backup to an S3 bucket. Is there another way to achieve ...
- 663
54
votes
4
answers
133k
views
run rsync as root but keep user ownership
I would like to backup user files from one server to another with rsync.
but I noticed that the user folders change to root.
how can I keep the user permissions with rsync (running by root)?
- 1,968
54
votes
3
answers
27k
views
Why is it not possible to use two remotes for rsync? [closed]
Note to serverfault users:
This closed question is answered here by four comments by hynekcer. It is then well accepted in other comments as a regularly answerable question. It's now being voted for ...
- 1,405
49
votes
6
answers
63k
views
Rsync daemon: is it really useful?
Are there any practical benefits in using rsyncd compared to rsync over ssh? Does it really increase speed, stability, anything?
- 11.3k
48
votes
7
answers
66k
views
How can one efficiently use S3 to back up files incrementally?
I understand how rsync works on a high-level, but there are 2 sides. With S3 there is no daemon to speak of — well there is, but it's basically just HTTP.
There look to be a few approaches.
s3rsync ...
- 581
47
votes
7
answers
56k
views
Why rsync is faster than NFS?
Few days ago I noticed something rather odd (at least for me). I ran rsync copying the same data and deleting it afterwards to NFS mount, called /nfs_mount/TEST. This /nfs_mount/TEST is hosted/...
- 2,245
45
votes
5
answers
121k
views
Rsync command issues, owner and group permissions doesn´t change
I´m trying to set owner and group via rsync and it doesn't seem to be working.
This is the command:
sudo rsync -rlptDvz --owner=cmsseren --group=cmsseren /home/serena/public_html/ -e ssh root@ip:/...
- 549
44
votes
4
answers
72k
views
How to output a list of changed files from rsync?
I am using rsync in a bash script to keep files in sync between a few servers and a NAS. One issue I have run into is trying to generate a list of the files that have changed from the during the rsync....
- 565
44
votes
1
answer
56k
views
What exactly will --delete-excluded do for rsync?
I use rsync with great pleasure for backing up my servers, although I do have a question about the --delete-excluded parameter.
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded ...
- 4,043
43
votes
7
answers
35k
views
Handling renamed files or directories in rsync
I am working on an rsync script for directory replication. I have it syncing only new and modified files or directories but I don't like the fact that it's copying renamed files or directories as a ...
- 543
42
votes
7
answers
66k
views
Using RSYNC with Amazon S3 [closed]
I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools)
Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and ...
- 1,730
41
votes
5
answers
41k
views
rsync 'cannot delete non-empty directory' errors, even with --force option
When running this command:
$ sudo rsync -r --delete --force --checksum --exclude=uploads /data/prep/* /data/app/
I'm getting the following output:
cannot delete non-empty directory: html/js/...
- 513
39
votes
3
answers
48k
views
SCP transfer only modified files
I'm using below command to transfer files cross server
scp -rc blowfish /source/directory/* [email protected]:/destination/directory
Is there a way to transfer only files modified files just like ...
39
votes
2
answers
84k
views
How do I set destination permissions with rsync (chown chmod)
I am trying to do a local rsync, from a mount point to a local folder. I need to set the owner, group, and permissions to specific settings. Here is what I am using:
rsync -rtlv --chown=process:...
- 961
38
votes
4
answers
54k
views
How to repeatedly call rsync until files are sucessfully transferred
I'm trying to syncronize files from a remote server that is not reliable, meaning the connection tends to fail "randomly" with
rsync: connection unexpectedly closed
Rsync is called with --...
- 655
36
votes
2
answers
57k
views
Using rsync's archive flag without copying symbolic links
As stated in rsync's man page, the -a (archive) switch is equivalent to -rlptgoD. However, I have a situation where I don't want symbolic links retained. Is there any way to keep using the -a switch ...
- 483
36
votes
7
answers
34k
views
How can I use rsync with a FAT file system?
I would like to write a simple backup script that saves some data to a FAT drive. Should I reformat the drive and use a better file system or is it possible to use rsync with FAT? If so, what problems ...
- 819
36
votes
3
answers
40k
views
compare 2 directories and copy differences in a 3rd directory
Running ubuntu 12.04, I want to compare 2 directories, say folder1/ and folder2/ and copy any files that are different to folder3/. There are also nested files, so matching subdirectories should be ...
- 421
35
votes
2
answers
22k
views
Copy directory tree without empty directories?
I have the following tree
# upper letters = directory
# lower letters = files
A
|-- B
|-- C
|-- D
|-- e <= file
|-- F
|-- G
I need to copy this tree to another destination, ...
- 1,329
34
votes
3
answers
52k
views
rsync won't delete files on destination
I am trying to mirror a directory that changes over time to another directory. My problem is that rsync is not deleting files on destination if they aren't existing in source directory anymore. Here ...
- 2,368
33
votes
7
answers
35k
views
rsync files to a kubernetes pod
I need to rsync a file tree to a specific pod in a kubernetes cluster. It seems it should be possible if only one can convince rsync that kubectl acts sort of like rsh. Something like:
rsync --rsh='...
- 3,130
33
votes
9
answers
51k
views
Best way to compare (diff) a full directory structure?
What's the best way to compare directory structures?
I have a backup utility which uses rsync. I want to tell the exact differences (in terms of file sizes and last-changed dates) between the source ...
- 13.7k
32
votes
8
answers
49k
views
Rsync seems incompatible with .bashrc (causes "is your shell clean?")
It turns out rsync can't work with a remote server which has a .bashrc file?
At local client i got when run rsync:
protocol version mismatch -- is your shell clean?
(see the rsync man page for an ...
- 473
32
votes
6
answers
31k
views
Using rsync from msysgit for binary files
I have three windows boxes that are connected to a linux box on my network in various ways (one's local to the network, one is a laptop and may or may not be local, and the other is my work desktop). ...
- 331
32
votes
1
answer
19k
views
Why does rsync spawn multiple processes for me?
I am using the following cron statement to backup from one folder to another folder in the same machine:
19 21 * * * root rsync -ac --delete /source/folder /dest/folder
When I use pstree, I see the ...
- 6,101
31
votes
5
answers
20k
views
Converting UTF-8 NFD filenames to UTF-8 NFC, in either rsync or afpd
I have a home file server running FreeNAS 8. A few days ago I used rsync to upload my entire iTunes library from Mac so that I could load my library over the network instead of off a slow USB drive. ...
- 653
30
votes
8
answers
27k
views
How can I use rsync to duplicate a directory tree, creating hardlinks to files?
From time to time, I have to perform several large migration changes on data files on my server, and I'm looking for a good way to do this. I was thinking about using rsync to duplicate my directory ...
30
votes
5
answers
30k
views
Do you need -e ssh for rsync?
I always used to use the following command when copying from a server:
rsync --progress -avze ssh user@host:/path/to/files ./here
However, a friend of mine showed me that I can simply do:
rsync --...
- 403
30
votes
5
answers
18k
views
Nice rsync on remote machine
When using rsync+ssh to access a remote machine, is there a way to "nice" the rsync process on the remote machine (to lower its priority)?
Editing the question to clarify:
USER PID %CPU %MEM ...
- 532
29
votes
2
answers
53k
views
rsync : transferring files in one direction only
Is it possible to use rsync to copy files in one direction only?
For example, suppose we have:
left/a.txt
right/a.txt
where the files are initially identical.
If one then modifies right/a.txt, ...
- 1,049