Archives for May 2009

Recently I had the distinct pleasure of working with a client during a network merger. As anyone who’s ever experienced a corporate merger knows; the first rule of mergers is that things never mesh as well as planned. Honestly, this experience has been no exception. However one of the issues that developed was the inability to simply renumber both IP scopes as that would have created numerous inconsistencies in the two existing LANs. In fact we theorized that the flux created by simply renumbering the two LANs onto one scope would have cause an event horizon effect that would most likely shift the Earth off of it’s axis and truly ruining everyone’s day.

Truth be told it would have create far more problems than it would have solved by just renumbering the scopes. It was determined that the best course of action would be to transition one of the scope into the other over time, minimizing the fluctuations and affording us the opportunity to carefully adjust for any in consistencies that developed. Not to mention avoiding the whole Earth axis shifting space time continuum ripping thing which I am fairly certain everyone could agree would have been bad. Very bad.

Initially the thought was to bridge the two IP scopes. Of course this is not as easy as one would have expected since both scope had distinctly different internet connections as well as default routes. The first scope existed on a 172.16.1.x/24 and the second scope 192.168.1.x/24 luckily on the 172 LAN side we had an Adtran 1224R routing firewall switch that made the initial process a virtual walk in the park. All that I had to do was to create a new VLAN on this switch and assign it an address from the 192 scope as well as assign several ports to the new VLAN. Like magick the the networks were now connected and from this device I was able to send and receive traffic in both directions. This would save countless hours by simply avoiding adding anything to the hundreds of workstations on the 172 LAN.

Unfortunately the same was not so easy on the 192 LAN’s configuration, as a result of, and without naming a specific firewall manufacturer and identifying their soon to be EOL’d, equipment utterly lacking the ability to add a secondary route. The only option left was to add a secondary route on the Servers and workstations the reside on this 192.168.1.x network that instructed each how to reach the 172.16.1.x network. Therefore, I manually executed the following command on the Xserve in question;

route add -net 172.16.7.0/24 192.168.1.254

Figure 1 Adding a route via cli under MAC OS X

This worked wonderfully, as I could now ping in both directions without issue. The Web services were now available on the 172 side, as well as the services of any other server that I added the above command to. The down side to this approach is that in order to bring a server online some one with admin privileges would have to manually enter this command after each reboot. Obviously that solution would not hold water for very long.

In a search for a more permanate solution, I read /etc/rc and after finding the excerpt in Fig 1 I thought to myself eureka I’ve got it. I happily opened my favorite text editor and rolled up my BSD sleeves creating my very own rc.local consisting of the route command in Fig 1. However for some reason this option failed to work correctly as expected. In fact it really did not work well at all. For some reason the Mac OS X Servers seemed to work and the workstations did not.

if [ -f /etc/rc.local ]; then
        sh /etc/rc.local
fi

Figure 2 rc.local excerpt from /etc/rc

So at this point I was pretty much beside myself. I seriously puzzled over this for a while, and nearly considered the whole black hole flux issue as an acceptable risk. Yet after reading numerous entries on yahoo and google that stated this should have worked I found a page on afp548 and Apple’s own developer connection site that referenced launchd. All indications pointed to Apple deprecating the rc construct over time anyway and I figured now is as good a time as any to hop on the bandwagon. So I began diddling with launchd, and besides from reading the specs it seemed to be really cool. I mean who wouldn’t want to learn how to manipulate the uber startup system, that has replaced the familiar BSD rc systems as well as init and inetd (xinetd to be exact).

Therefore after many attempts I ended up with the addBridgeRoute script in Figure 4. Believe me when I tell you this was not my first attempt, nor the way I had expected things to unfold when I began this little endeavor. In fact if you examine Figure #3 you will see what I had initially attempted. I tried over and over again frustrating as it was to get launchctl load to accept my little plst as gospel and just do what I told it to. You know it’s not easy bending the world to meet your will, when you don’t receive useful feedback.

While technically this did satisfy all of the syntactical requirements, for one reason or another it did not succeed. Most likely through my own lack of knowledge about all of the launchd options. Personally I feel lucky to have discovered all that I had about this amazing facility apple has included into Mac OS X. In any event dead lines drew nearer and as much as I do enjoy the sounds that they make when the go whooshing passed, I was in no mood to miss this one. Finally the client became a little more than antsy and I had to come up with something positive, as I am not one to accept defeat so easily.

	Label
	com.olivent.route
	ProgramArguments

		/sbin/route
		add
		-net
		172.16.7.0/24
		192.168.2.254

	StandardErrorPath
		/dev/null
	RunAtLoad

Figure 3 /Library/LaunchDaemons/com.olivent.route.plist

Therefore, as I am certain many other administrators have faced similar hurdles, through trial and error, in this case mostly error, one perseveres achieving fantastic results. I shifted gears and began working with the script you see in Figure 4 so that I could push my own entries into the syslog. Ok once again I’ll admit that this wasn’t really all that fantastic but at the end of the day the client is happy and I am documenting the entire process for your reading pleasure. In my book that would be known as a win win.

Feeling a wee bit of pressure at this point I quickly shifted gears, as the goal was to move the artists workstations from the current 172 based LAN to the 192 scoped LAN. Thus placing them on the same net work fabric as the imaging and production servers while still being able to access the administrative servers on the 172 LAN.

In figure four you will see the wrapper I drafted to facilitate this operation. I basically built this script in stages adding simple methods to aid in troubleshooting the problem with this particular launch daemon. One thing that became evident after the first reboot with the new plist and associated wrapper script was the launchdctl list showed that my process was loaded but unfortunately at this point I was still unable to pass traffic from the 192 to the 172. So at this point I started adding logging method to determine what was inhibiting the route processes. Ultimately I performed a couple of Hail Mary’s and inserted the sleep statement which viola, eureka, bam that was the magick ticket.

#!/bin/sh

#       /usr/local/sbin/addBridgeRoute
#
#       by: Mikel King for Olivent Technologies, llc
#       contact: http://twitter.com/mikelking
#
#       Purpose: to launch alternative routing for net bridge.
#
#	Date stamp and initiating the log entry
#	I added this basically for troubleshooting purposes.
STAMP=`date +"%b %e %H:%M:%S %Y"`
echo "${STAMP} addBridgeRoute: started" >> /var/log/system.log

#	Pause long enough to allow the other network services to properly start
sleep 30

#	Actually adding the route
route -q add -net 172.16.1.0/24 192.168.1.254

#	Just another log entry
echo "${STAMP} addBridgeRoute: Added the bridging route 172.16.1.0/24 192.168.1.254" >> /var/log/system.log

#	YALE (yet another log entry)
echo "${STAMP} addBridgeRoute: completed" >>/var/log/system.log
exit 0

Figure 4 /usr/local/sbin/addBridgeRoute

	Label
	com.kbeny.addBridgeRoute
	Program
		/usr/local/sbin/addBridgeRoute
	StandardErrorPath
	/dev/null
	RunAtLoad

Figure 5 /Library/LaunchDaemons/com.olivent.addBridgeRoute.plist

I hope that you have enjoyed this little stroll through the underbelly of the Mac OS X operating system. Expect Apple to continue the changes to the launch system in future versions of the the OS.

ABOUT THE AUTHOR: Mikel King (http://twitter.com/mikelking) has been a leader in the Information Technology Services field for over 20 years. He is currently the CEO of Olivent Technologies, a professional creative services partnership in NY. Additionally he is currently serving as the Secretary of the BSD Certification group as well as a Senior Editor for the BSD News Network.