RSYNC For Code Deployments

In the past we have looked at several different deployment scenarios from sneaker net file wrangling to SFTP and even git cloning/checkouts. Today we need to look at the next level of deploying code. The next level is rsync and if you are not familiar with or never really delved into rsync then today’s the day we crack this nut open.

While you can effectively use SCP or even SFTP to move files around between hosts there are a number of limitations. For one while scripting can be done it is a bit tedious. Furthermore as with SCP and SFTP you will need to properly setup Passwordless SSH Authentication in order to use rsync for automagick code deployments. One of the big advantages taht Rsync offers is the ability to ship only the blocks of data that have actually changed. In addition it has the ability to keep the target in sync with the changes made in the source which makes it particularly well suited for code deploys.

Because the rsync man page has a huge list of options lets take a look at what a typical command might look like. We shall start by deconstructing the following filesystems backup:

rsync —partial —append —status —avzrp SRC DEST

Let’s start with the partial option. This command line switch allows you to resume failed transfers. Normally rsync will discard partially transferred files however this will flag the system to keep them which can be handy with large binaries like image, video or audio files.

The append option is NOT one I recommend for code deploys, but is fantastic for file backups. Essentially this option will append the changes to the destination file if it already exists. This can have unexpected results for code deploys.

The status or stats option simply displays a section of transfer stats that I personally find very helpful when trouble shooting deployment problems. Feel free to omit.

a   archive mode
v   verbose
z   gzip compress during transfer
r   recurse into subdirectories
p   preserve permissions

These remaining options are relatively self explanatory so there’s little need to dig in deeper. I do think it is important that we take a moment to remember that rsync offers a –dry-run option so you can test the commands before doing any irreparable damage to your system.

The append option is NOT recommended for code deploys

The following are both powerful and very dangerous. They are also essential for us to efficiently use rsync for code deployments therefore we will look at them in greater detail.

--del                   an alias for --delete-during
--delete                delete extraneous files from dest dirs
--delete-before         receiver deletes before xfer, not during
--delete-during         receiver deletes during the transfer
--delete-delay          find deletions during, delete after
--delete-after          receiver deletes after transfer, not during
--delete-excluded       also delete excluded files from dest dirs
--force-delete          force deletion of dirs even if not empty

As previously mentioned you should use the –dry-run option until you feel extremely confident that you will not break things. In addition maintaining good backups is a must.

So starting with the –delete option while it may seem obvious and even logical it is one that get misused more then not. If you delete a file from the source path then it will be deleted from the destination. This applies to directories and files equally. This makes the option a good candidate for code deployments but a bad one for filesystem backups.

Well that was the easy one as each of the remaining delete options are complex. For instance the –delete-delay will delete the files in question during the transfer but after the files are done being shipped. This is probably one of the more confusing aspects of working with rsync. In essence it stores a stack of files marked for deletion that it discovers during the transfer process and once it’s done transferring it deletes them.

Reading that I am certain you are confused as to why or how that is different from the –delete-after option. Well the –delete-after option does not begin the search for the files to delete until after the transfer is complete. This also happens on the receiver side of the equation.

Similarly the –delete-before instructs the receiver to scan for files deletions prior and then remove them prior to transferring the changes. In addition the –delete-during performs this during that actual data transfer essentially it performs a just in time operation.

The –delete-excluded option is potentially problematic for for code deploys as most files systems have a bevy of files that you want excluded from rsync process. This options instructs the receiver to analyze the –exclude option for additional items to remove from the destination. I recommend that you use this one extreme caution. For instance assume you have files like minified JavaScript and CSS in your git exclusions which is the same driver for your code deploy. Using this option means that you would deploy those minified files to the destination and then delete them.

The final option –force-delete is another that I recommend you use with extreme caution. This option has an alias –force so once again use with care. Let’s say for the sake of argument you included a file named cache in your code base under wp-content then deploy your code changes to a live WordPress installation. This option will replace the cache directory with your file and while it may not break your site completely it would render the local caching system useless thus degrading server performance.

Now that you have a basic understanding of how rsync we will in part two we will go into more detail by testing actual scripts. As with everything that is scripto-magick you need to test, test again and then test some more. There is no magickal silver bullet for efficient code deployments and your needs can change over time.

Related Content:

How to setup rsyncd on Mac OS X