One of the most versatile utilities developed is rsync, however; learning to effectively use the application can be a daunting task. Rsync is useful for conducting backups to remote file servers or even mirroring a local drive to a removable one. It supports transferring files over ssh as well as it’s own protocol. Unfortunately, to use the built in rsync protocol you need to set up an rsync server, which on a Mac can be quite tricky.
On the one hand, you can simply type rsync —daemon and it will start a rsync daemon running on TCP port 873. But without the appropriate rsyncd.conf things can get a little messy. In addition, if you reboot the ‘server’ the process will not restart automatically. The worst thing is to have a system that has been operational for several months suddenly stop because someone rebooted the hardware and no one remembered that the process needed to be relaunched. Personally, I think it is much better to have the system offer some more resiliency by automating this process.
On the Mac, unfortunately inetd is no longer a viable option, thus you need to use launchd and launchdctl to load your XML described process file. So, I created the following plist (property list) file that I installed as root into /Library/LaunchDaemon.
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Disabled</key> <false/> <key>Label</key> <string>org.samba.rsync</string> <key>Program</key> <string>/opt/local/bin/rsync</string> <key>ProgramArguments</key> <array> <string>/opt/local/bin/rsync</string> <string>--daemon</string> <string>--config=/usr/local/etc/rsyncd/rsyncd.conf</string> </array> <key>inetdCompatibility</key> <dict> <key>Wait</key> <false/> </dict> <key>Sockets</key> <dict> <key>Listeners</key> <dict> <key>SockServiceName</key> <string>rsync</string> <key>SockType</key> <string>stream</string> </dict> </dict> </dict> </plist>
You should also note that although I could have used the default 2.6.9 version of rsync that ships with most Macs, I have actually upgraded mine to 3.0.8 using the MacPorts.org system. In addition, I have created this plist to look for the rsyncd.conf in /usr/local/etc/rsyncd, because it is a more unified best practice way of doing things. Besides, like Mac OS X I am a fan of FreeBSD and it’s just the way I roll. The following is an example of a rsyncd.conf file that I have used in the past:
# rsyncd.conf - Example file, see rsyncd.conf(5) # # # Set this if you want to stop rsync daemon with rc.d scripts pid file = /var/run/rsyncd.pid # Remember that rsync will supposedly reread this file before each new client connection # so you should not need to HUP the daemon ever. motd=/usr/local/etc/rsyncd/rsyncd.motd uid = nobody gid = nobody use chroot = no max connections = 4 syslog facility = local5 [mk] path = /Volumes/Data/home/mikel/stuff comment = Mikel King Repository uid = www gid = www list = yes read only = no auth users = mking secrets file = /usr/local/etc/rsyncd/mking.secrets
Once I have completed the basic setup it’s time to launch the daemon. To do this we need to use launchdctl to load the plist into the lauchd registry. I find it is easiest to use pushed to temporarily move to /Library/LaunchDaemons and run the command locally as follows;
sudo launchctl load org.samba.rsync.plist
At this point we have told the Mac (in my case a Snow Leopard Server) to make this service available. If you were to perform a ps ax | grep rsync you would likely not see anything. Once you make a connection attempt on the appropriate TCP port 873 launchd will setup the daemon. On my laptop at the command prompt I enter the appropriate command that will make the connection to the rsync service.
rsync --stats mking@olivent.com::mk
This above command will connect to the rsync daemon, which is a geeky way of saying service causing launchd on the remote server to instantiate a copy of rsyncd to launch and run answering the request. It does this on the fly in order to save system resources. Honestly there isn’t much reason to keep rsyncd around running just in case someone makes the call and supplies the correct credentials. You don’t keep your car running just in case you might decide to hop in and run up to 7 Eleven for a burrito and cup of Brazilian Bold do you? No, because that would be a gross waste of resources! On the server side when we make the call it answers with the following;
isis:~ $ ps ax |grep rsync 85366 ?? Ss 0:00.00 /usr/libexec/launchproxy /opt/local/bin/rsync --daemon --config=/usr/local/etc/rsyncd/rsyncd.conf
As soon as the connection to rsync has completed it’s transaction the daemon will end it’s run allowing those cycles and ram to return to the pool of resources that the server needs to use for doing other things like serving Minecraft or WordPress web sites. The following is an example of what it looks like from the client perspective,which in geek speak is basically a way of saying what happened on my laptop;
djehuty: mking$ rsync --stats mking@olivent.com::mk Password: drwxrwxrwt 374 2011/11/19 11:39:11 . -rw-r--r-- 382258 2011/11/10 22:16:56 ThumbtackMap.png -rwxr-xr-x 71 2011/07/30 00:48:29 addRoute -rw-r--r-- 255809 2011/10/24 09:03:27 mk-mib.jpg -rw-r--r-- 78922 2011/11/03 14:47:54 rei-press-mug.png -rw-r--r-- 1362 2011/07/29 23:56:50 rsyncd.conf -rw-r--r-- 681399 2011/11/18 15:03:15 stargate.png -rw-r--r-- 66468 2011/11/01 15:04:52 terminal.app.png -rw-r--r-- 715 2011/11/18 18:19:07 tftp.plist -rw-r--r-- 10274 2011/11/18 17:42:13 admin-ssh-bundle.tbz Number of files: 10 Number of files transferred: 0 Total file size: 1477278 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 225 File list generation time: 0.007 seconds File list transfer time: 0.000 seconds Total bytes sent: 61 Total bytes received: 300 sent 61 bytes received 300 bytes 144.40 bytes/sec total size is 1477278 speedup is 4092.18
As you can see I am running rsync on my laptop with the –stats option which yields this handy output of what transpired during the session. After issuing the rsync command it prompts me for my password on the rsync server for that resource, which rsync calls a module. Assuming that I am listed in the module definition in rsyncd.conf as an auth user and enter the correct password noted in the appropriate “secrets” file then rsyncd will send the appropriate data to rsync on my laptop.
I understand all of this client server protocol negotiation may sound like “Blah blah blah blah” or one of the adults from a Peanuts comic because it’s definitely geek speak. Just keep the basics in mind; If you run rsync on your side of the connection to call rsyncd on the other end. This means that you are the client and the destination is the server. Of course this gets very muddy when you start talking about the X Windowing System but we shall save that for another day.
In summary rsync is an extremely useful service to have in your utility belt. I have used rsync to copy huge amounts of data to sites all over the world. When I was working on a project that required deliverables in Malaysia, China and Turkey from the US I used rsync to transport the data. The main reason I chose rsync is it’s ability to be automated and of course if you are using the rsync protocol you can not forget the ability resume a transfer if something breaks.
I hope this article helps you understand the power of rsync and sheds some insight into it’s uses. Please leave a comment on how you use rsync.
Related articles
- Advanced Mac OS X Shell Scripting (jafdip.com)
- How To rsync Server Setup for Centos (vijaynayani.wordpress.com)
- Performing MacPorts Magick (jafdip.com)
I have mac os 10.8 rsync –deamon trow this: rsync: –deamon:
unknown option rsync error: syntax or usage error (code 1) at
/SourceCache/rsync/rsync-42/rsync/main.c(1333) [client=2.6.9] There
is anything that I need to install, before to use rsync command to
load the deamon? Thanks.
John,
Errors noting main.c usually indicate a permission issue. Have you tried to just connect via ssh to determine if rsync is running properly? Try something like rsync –stats -avzrp john@someplac.com:documents and see if it successfully builds a list of files from that example. Obviously you will need to adjust things for something relevant to your systems.
Good luck,
Mikel
This was an excellent article and was invaluable for me to get rsyncd up and running. A few small suggestions if I may:
1. Just before your plist codeblock there’s a typo, it should be /Library/LaunchDaemons
2. Also around here, it would be nice to mention that the plist file you created was called ‘org.samba.rsync.plist’. It’s obvious later on but for those working through this sequentially.
3. The only point where I got stuck was auth errors. This was solved by making sure the .secrets file is owned by root with 600 permissions.
Thanks again for taking the time to write this.
Happy to provide this and I will update the documentation as you’ve noted. Thank you for taking the time to leave a carefully written and honest comment.
Regards,
Mikel