Using radmind to install Tiger – Part 1

radmind is a wonderful tool for managing OS X machines. We use it to install and uninstall third-party software, apply software updates and security patches, remove unauthorized software, and generally keep machines in a known, consistent state.

radmind works really well for minor OS updates – going from 10.4.3 to 10.4.4, for example. And it easily does the reverse, going from 10.4.4 to 10.4.3.

But I (and others) have run into challenges when using radmind to roll out a major OS update – from 10.2 to 10.3, or 10.3 to 10.4. For each of these transitions, I could usually get the correct files installed, but the process would hang at the end, requiring a manual power-off, power-on to reset the machine.

This happens because unlike Apple’s OS X Installers, where you are generally booted from CD or DVD or a NetBoot volume, with radmind, you are booted from the volume you are updating. This is like doing brain surgery on yourself! Many Mac OS X executables rely on shared libraries. During a major OS X upgrade, these shared libraries are replaced with different versions. When the radmind update is complete and your script tries to restart the machine, the “reboot” executable tries to link to a shared library that no longer exists on the machine, and so it dies, usually with a segmentation fault.

Andrew Mortenson at the University of Michigan, one of the radmind developers, did some excellent work puzzling all this out and coming up with a workaround. In broad strokes, you need to figure out which executables, shared libraries and frameworks are needed by your radmind scripts. You then copy them somewhere on the filesystem not managed by radmind. You also copy the dynamic linker, and set some environment variables so the system will use your copies of the dynamic linker, shared libraries, frameworks, and executables. When radmind swaps these out for different versions, it won’t matter, because you are using other copies of these tools.

I’ll quote Andrew from the radmind mailing list:

The cause of hangs and crashes following a major system upgrade with radmind, such as Mac OS X 10.3 to 10.4, can be attributed to modified libraries that don’t agree with the running kernel, which happens to be that of the old system. To get around this, you need to make copies of crucial libraries and tools before you replace them with the versions included with the new OS, and to know what libraries are crucial, you need the help of otool.

otool is part of Apple’s Xcode tools. Among other uses, it can show you what shared libraries a binary links with to function properly. Following a major OS revision radmind update, we want at a minimum to be able to reboot the system, so we know we want a copy of the old “reboot” tool, in this case the one that ships with Mac OS X 10.3. It obviously requires some libraries, which we use otool to see:

% otool -L /sbin/reboot
/sbin/reboot:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.0.0)

So we need libSystem for reboot to work. But libSystem itself requires another shared library:

% otool -L /usr/lib/libSystem.B.dylib
/usr/lib/libSystem.B.dylib:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.2.1)
/usr/lib/system/libmathCommon.A.dylib (compatibility version 1.0.0, current version 93.0.0)

When you’ve exhausted the otool -L output for each library, you have a list of libraries you need for reboot to execute successfully. Using a script, copy these libraries–as well as all required tools and the dynamic linker, /usr/lib/dyld–to a temporary location *prior* to starting a radmind update, i.e., before letting lapply do its work.

Now you have copies of the old OS’s basic tools and libraries to use of the new versions that lapply will download, but to make use of them you have to set a few environment variables. We want the dynamic linker to use the old versions of the libraries, so we tell it to look in our temporary directory for them instead of using the default paths (/usr/lib, etc.):

sh/bash: DYLD_LIBRARY_PATH=/path/to/dir/containing/old/libs; export DYLD_LIBRARY_PATH
csh/tcsh: setenv DYLD_LIBRARY_PATH /path/to/dir/containing/old/libs

(Note that the same thing can be done with frameworks, using DYLD_FRAMEWORK_PATH. If a tool you want to use links with a framework, according to otool, you’ll need to make a copy of the framework, as well.)

With that variable set, dyld will look first in $DYLD_LIBRARY_PATH for the libraries a tool links with. (You can verify that this works by telling dyld to print out the libraries as it links the tool with them, using another environment variable:

sh/bash: DYLD_PRINT_LIBRARIES=1; export DYLD_PRINT_LIBRARIES
csh/tcsh: setenv DYLD_PRINT_LIBRARIES 1

Now when you run a command, dyld will display the paths to the libraries it’s using:

% ls
dyld: loaded: /bin/ls
dyld: loaded: /tmp/test/libncurses.5.4.dylib
dyld: loaded: /tmp/test/libSystem.B.dylib, cpu-sub-type: 0
dyld: loaded: /tmp/test/libmathCommon.A.dylib, cpu-sub-type: 0
[…]

Of course, this is unnecessary when things are configured properly, but it’s a helpful debugging tool.)

Now you need to make sure that the search path is set to look in your temporary directory for tools first:

sh/bash: PATH=/tmp/test:${PATH}; export PATH
csh/tcsh: set path=( /tmp/test $path )

% ls
dyld: loaded: /tmp/test/ls
dyld: loaded: /tmp/test/libncurses.5.4.dylib
dyld: loaded: /tmp/test/libSystem.B.dylib, cpu-sub-type: 0
dyld: loaded: /tmp/test/libmathCommon.A.dylib, cpu-sub-type: 0

When the radmind update script executes the reboot command, it will use the old copy and the old libraries, and restart the machine.

Following is a Perl subroutine from my run_radmind script that copies the needed tools and libraries and sets the correct environment variables.


sub copyVitalTools {
   # this subroutine copies tools and libraries that may be
   # needed after lapply starts changing the filesystem. This
   # ensures that we can call them even if needed tools and
   # libraries are swapped out from under us during an lapply
   # session. This is often the case during a major OS change
   # (10.3.x->10.4.x)
   #
   # Thanks to Andrew Mortenson of UMich for the main concepts
   # here. If you add to the list of tools, you'll need to use
   # 'otool' to generate a list of library and framework
   # dependancies.
 
   my $toolDir = "/tmp/tool";
   my $OSversion = `/usr/bin/uname -r | cut -f 1 -d '.'`;
 
   # current OS is 10.3.x
   if ($OSversion == 7 ) {
 
      `mkdir -p $toolDir`;
      `cp /usr/lib/dyld $toolDir`;
      `cp /usr/bin/touch $toolDir`;
      `cp /usr/bin/grep $toolDir`;
      `cp /usr/bin/curl $toolDir`;
      `cp /bin/cat $toolDir`;
      `cp /bin/sh $toolDir`;
      `cp /bin/sync $toolDir`;
      `cp /sbin/reboot $toolDir`;
      `cp /usr/lib/libSystem.B.dylib $toolDir`;
      `cp /usr/lib/libbsm.dylib $toolDir`;
      `cp /usr/lib/libcurl.2.dylib $toolDir`;
      `cp /usr/lib/libz.1.dylib $toolDir`;
      `cp /usr/lib/libssl.0.9.7.dylib $toolDir`;
      `cp /usr/lib/libcrypto.0.9.7.dylib $toolDir`;
      `cp /usr/lib/libncurses.5.dylib $toolDir`;
      `cp /usr/lib/system/libmathCommon.A.dylib $toolDir`;
 
      $ENV{'PATH'} = "$toolDir:$ENV{'PATH'}";
      $ENV{'DYLD_LIBRARY_PATH'} = "$toolDir";
 
      # uncomment the next line to turn on dylib debugging
      #$ENV{'DYLD_PRINT_LIBRARIES'} = 1;
 
   # current OS is 10.4.x
   } elsif ($OSversion == 8 ) {
 
      `mkdir -p $toolDir`;
      `cp /usr/lib/dyld $toolDir`;
      `cp /usr/bin/touch $toolDir`;
      `cp /usr/bin/grep $toolDir`;
      `cp /usr/bin/curl $toolDir`;
      `cp /bin/cat $toolDir`;
      `cp /bin/sh $toolDir`;
      `cp /bin/sync $toolDir`;
      `cp /sbin/reboot $toolDir`;
      `cp /usr/lib/libSystem.B.dylib $toolDir`;
      `cp /usr/lib/libbsm.dylib $toolDir`;
      `cp /usr/lib/libcurl.3.dylib $toolDir`;
      `cp /usr/lib/libz.1.dylib $toolDir`;
      `cp /usr/lib/libssl.0.9.7.dylib $toolDir`;
      `cp /usr/lib/libcrypto.0.9.7.dylib $toolDir`;
      `cp /usr/lib/libncurses.5.4.dylib $toolDir`;
      `cp /usr/lib/system/libmathCommon.A.dylib $toolDir`;
 
      $ENV{'PATH'} = "$toolDir:$ENV{'PATH'}";
      $ENV{'DYLD_LIBRARY_PATH'} = "$toolDir";
 
      # uncomment the next line to turn on dylib debugging
      #$ENV{'DYLD_PRINT_LIBRARIES'} = 1;
 
   } else {
      # not 10.3.x or 10.4.x, so
      # no idea what to do yet
   }
}


I’ve successfully used radmind to take a machine from 10.3.9 to 10.4.3 and back, several times. There’s more to the solution than copying the libraries, though. I’ll continue in a subsequent post.

Advertisement