Recently I was asked about maintaining a data center full of servers. More specifically about maintaining a repository of the configuration files for all servers in the data center.
I am sure it’s no surprise that as our data centers and systems in general become more sophisticated managing the complex array of all the configuration data in and of itself is nearly as important as the user data stored in. Anyone who’s ever lost a server as a result of some catastrophic failure, be it failed equipment or some other nefarious means, knows it is not easy to rebuild a system to its pre-failure state.
Honestly it even the best backups can yield less than accurate results. Archive media can become faulty and let’s not rule out human error. Therefore as a layer of redundancy I like the idea of a configuration management solution. Of course disaster preparedness is not the only reason one might consider implementing a some sort of configuration management solution.
If you have a large installation of equipment it becomes increasingly difficult to keep track of the numerous system updates and configuration files. Especially if you are in an environment with inconsistent technical staff as result high employee turn over for instance. Having a configuration management procedure in place is a good way to offset this sort of issue.
Several years ago during a large coding project I was introduced to subversion (svn for short) and although I had been familiar with other versioning solutions for whatever reason svn stuck. Initially we started with just the code base, however the more we used it the more we put into the repository. It started with storing documentation about the application and ended up dropping all of the apache, php and mysql configuration files into it.
Shortly after completing the beta testing we had to replicate the entire server installation into numerous front end production web servers. It was then that it hit me that if we had the svn client on each server all we would have to do is run a checkout to have 90% of the configuration completed.
This certainly helped expedite server rollouts. Of course it did add an additional step in the pre-deployment built out. In addition to installing php, apache, and mysql we would now have to install svn. Although this is not a huge task it does add a layer of complexity to the overall scheme. One could use a svn repository to manage the build options for your system to ensuring that you are creating nearly identical deployments.
As you can see this can snowball rather quickly. It is a delicate balancing act determining where to draw the line. I my data center I have opted for maintaining a repository of /etc and /usr/local/etc of each system for each server.
To keep things simple I shall assume that you have a working subversion server with a repository ready for use. While there are several different ways to organize this repository, I have found that the best is to start with a group of like servers based on function. For instance let’s start with the named servers. Of course I am assuming that each server only performs a single function. If you are a jails jockey then this is likely to be the case, however if you are constrained by space, power and hardware it is more likely that each machine fills at least two billets.
Still for the sake of simplicity let’s roll with the assumption that you only have one service per server. In addition we shall limit out discussion to a single division, as some organizations will have multiple divisions as well as being dispersed across multiple geographic locations. Again for the sake so simplicity let’s assume that you only have the one.
Very well with the basic assumptions in place we need to construct our server repository. After confirming that I am able to access the svn server I begin with adding the server to the ‘Servers’ repository. From the prompt on the server named thoth, I would run the following command;
thoth:svn mkdir svn://svn.olivent.com/Servers/dns thoth:svn mkdir svn://svn.olivent.com/Servers/dns/THOTH
Notice that I placed the server name in all capital letters. This is a habit I picked up from customizing kernels in FreeBSD where one would copy GENERIC to the host name in all caps. You are certainly free to setup your system as best suits your personal style.
The next step is to start importing etc and /usr/local/etc into the system. The easiest way to accomplish this is to execute the following in root;
thoth:svn mkdir svn://svn.olivent.com/Servers/dns/THOTH/etc thoth:cd /etc thoth:sudo svn import svn://svn.olivent.com/Servers/dns/THOTH/etc
Although the import command should recursively create the target for you at the destination I have found it is better to explicitly create it yourself. The import command assumes that your current working directory is the one you wish to import. If the command is successful then you will see numerous files listed ending with ‘Committed revision XX.’ Where XX is the actual number of the revision.
Using the same methodology let’s add /usr/local/etc into the repository.
thoth:svn mkdir svn://svn.olivent.com/Servers/dns/THOTH/usr thoth:svn mkdir svn://svn.olivent.com/Servers/dns/THOTH/usr/local thoth:svn mkdir svn://svn.olivent.com/Servers/dns/THOTH/usr/local/etc thoth:cd /usr/local/etc thoth:svn import svn://svn.olivent.com/Servers/dns/THOTH/usr/local/etc
Observer that once again I explicitly created and specified the destination. Because import will assume the you wish to import everything in the present working directory I change the path to /usr/local/etc to ensure that I do not collect and collateral files. you can imagine what would happen if you were to accidentally import all of /usr.
Ok so now we have all of our current configuration files imported into the repository, but that really only helps us half way. One of the main advantages of using a versioning system like subversion is to improve the ability to capture changes to system configuration files as well as document why those changes are being made. Therefore in order to make use of this we need to checkout and place into service our versioned copies of these files. This actually can get a bit tricky
thoth:cd /usr/local/ thoth:mv etc old-etc thoth:svn checkout svn://svn.olivent.com/Servers/dns/THOTH/usr/local/etc
At this point I have accomplished storing both /etc and /usr/local/etc in the repository for the machine known as THOTH. In addition I have successfully checked out the current repository version of /usr/local/etc. Depending on your system and it’s activity you should perform the checkout to a temp folder and drop down to single user mode. Also keep in mind that on some systems namely Mac OS X /etc is a symbolic link to /private/etc which can make things rather touchy if you do not proceed with caution. Be certain to take the time to make note of your systems’ peculiarities.
Continuing with the original assumption that we are experimenting on a FreeBSD execute the two command blocks outline below. I’m using tce which of course is just etc backwards. Next reboot to single user mode, remember to ‘mount -w /’ so that you’re not spinning your wheels for nothing and execute the later command block.
thoth:mkdir /tce thoth:cd /tce thoth:svn checkout svn://svn.olivent.com/Servers/dns/THOTH/etc
*****Reboot to single user mode*****
thoth:cd / thoth:mv etc old-etc thoth:mv tce/etc etc
If all went as planned then you are now running on your versioned system all that remains to do is boot back up to multi user mode. Once safely back into multi user mode let’s try a senario. Suppose that you assign one of your BSDAs to install a new port that requires modifications to your rc.conf as well as installing its new configuration directory in /usr/local/etc and a new startup script in /usr/local/etc/rc.d.
Your Jr sysadmin successfully builds the port and installs the new application and even performs the the appropriate check-ins to the repository complete with commentary documentation as follows;
thoth:cd /etc
thoth:svn commit
The above should only transmit rc.conf if you added the new_app_enable=”YES” statement as required. Next you will want to add the new configuration to you /usr/local/etc section of the repository.
thoth:cd /use/local/etc
thoth:svn add new_app
thoth:svn add rc.d/new_app.sh
thoth:svn commit
Alright I know that this seems like a lot more work but consider what happens a few weeks later when your Jr sysadmin reboots the sever after some other maintenance and it hangs. Of course it does not take a versioning system to locate the missing quote on the entry named_enable=”YES” but it’s nice to be able to review the logs and determine who was the last person to modify the rc.conf and why.
Obviously I have demonstrated a rather time consuming manual process for all of this and it is quite possible to script much of the check-in and update process once you are familiar with the manual procedure. Additionally after reading this brief introduction to versioning you may be wondering. Why oh why would I even submit myself to all that effort and action tracking. I do have a good answer for you, concise documentation.
Consider that the server you just added to versioning is not really touched by you for several years. Your Jr sysadmin faithfully maintains the system checking in all of his changes over the years and one day, he leaves the company. Now what do you do? How do you know all of the systems that this person maintained? You could start logging in and cataloging this manually, but perhaps is you have a reliable versioning solution in place that you could simply run a report on his activity over the last few months.
Another fine example is you have to perform a site audit of all you systems. Perhaps you’ve wanted to build a network topology diagram for years but of course you just haven’t had the resources necessary to catalog hundreds of servers. Suddenly the university you work for has received a small grant to introduce some ‘GREEN’ initiatives. You could sell them on the idea that server consolidation could reduce their power consumption by a potentially sizable amount if only you had the resources namely personnel to complete the task in a timely fashion.
Utilizing your new team of student helpers you task them with the job of cataloging all of the servers. However do you really want to grant them direct access to everything? Perhaps if one were to use a subversion configuration file management system they could be granted temporary read only access to the repository. Ultimately allowing this temporary support staff to complete the task in a safe environment.
I hope you have enjoyed out brief look into the world of configuration file management, using subversion. Obviously there other possibilities out there, like CVS, OpenCVS, and git to name a few. Perhaps next issue we can look into some ways of automating this process.
Related articles
- Name Based Vhosting in Mac OS X Snow Leopard Server (jafdip.com)
- Advanced Mac OS X Shell Scripting (jafdip.com)
- Performing MacPorts Magick (jafdip.com)
- Trolling For A Quality Operating System (jafdip.com)
