OpenSolaris upgrade -- BFU

Building and Installing OpenSolaris (BFU)

Introduction

This is the second of two articles in which we describe how to build and install OpenSolaris. The first article provides all the necessary background information (terminology, where to get the tools, and so on) and describes a basic compilation and installation. There are circumstances under which the steps we performed in the first article will be insufficient for our needs; this article describes what we must do to address this shortcoming.

This article assumes that readers are familiar with the contents of the first article--or at the very least know how to acquire the OpenSolaris source code, build it, and Install it. For the sake of consistency, this article will deal with the same version of code the first one did, even though newer versions have been made available between the two publication dates.

Note that while these articles were accurate at the time they were written, OpenSolaris is continually evolving. For the most recent details, please consult the ON Developers Reference.


A Brief Recap

In the first article we defined much of the terminology we encounter when dealing with OpenSolaris. We also described how to obtain the source code and other bits necessary to compile it, and finally, we described the steps necessary to perform a simple compilation and installation of OpenSolaris. These steps are:

1. Obtain the source code, closed binaries, compilers and build tools from www.opensolaris.org or one of its mirrors.
2. Install the ON build tools and Sun Studio compilers into /opt and add their directories to our PATH.
3. Unpack the source and closed binaries tar balls.
4. Copy and edit the opensolaris.sh script.
5. Build the source code using the nightly script.
6. Install the resulting new kernel using the Install script.
7. Reboot using the new kernel.

These steps are fine if all we want to do is build and test a new kernel, but there are times when they are insufficient: we either want to install some new userland components, or our newly built kernel is incompatible with the userland libraries and applications that are currently installed. The latter scenario is referred to as a Flag Day. Another example of a Flag Day is when we need to update a library and a command that relies on that library at the same time.


Flag Days

A Flag Day is when newer versions of one or more system packages must be installed before we can use our new version of the ON components. When they occur, Flag Day notices will be posted on the OpenSolaris web site, at www.opensolaris.org/os/community/onnv. These notices will contain instructions for building or installing the newer required software, which may not be compatible with the usual Solaris packaging tools. In particular, the use of the BFU utility, which is the focus of this article, to update components renders those components unupgradable with the standard Solaris packages. If this is cause for concern, then using the latest Solaris Express release is probably the best approach. Indeed, using BFU is recommended only for active developers of the ON code (or those who must live on the very bleeding edge).


A BFU Primer

The process by which we install all the new bits of ON is referred to as BFU, which is an abbreviation of Blindingly Fast Upgrade (or Bonwick-Faulkner Upgrade, after the original developers, Jeff Bonwick and Roger Faulkner). BFU uses a set of cpio archives to directly overwrite the existing components of the system (which is why the usual Solaris packaging tools can no longer be used once we've performed a BFU); these archives are created as part of the nightly build process, which we described in the previous article.


Before we can use BFU to install our newly compiled components, we must define three environment variables. These are:

FASTFS This would usually be /opt/onbld/bin/`uname -p`/fastfs.
BFULD This would usually be /opt/onbld/bin/`uname -p`/bfuld.
GZIPBIN This would usually be /bin/gzip.

Setting these in our shell's start-up dot file (e.g., .profile) is probably the most convenient way to set these environment variables, especially if we plan to BFU frequently. The opensolaris.sh environment file we described in the previous article also sets these variables. Using bldenv opensolaris.sh (having first copied this file from usr/src/tools/env/opensolaris.sh and editing as necessary) before we run bfu is probably best. It is a good idea to get into the habit of updating our copy of opensolaris.sh with each new source delivery, because this will enable us to pick up any new environment variables automatically.


Using BFU to Install ON

Using BFU to install the ON parts of OpenSolaris is a relatively simple task: invoke the bfu command with a single argument, which is the path of the archive to install.

In the previous article we used $HOME/open_solaris/build-17/testws as our workspace directory. Assuming we're still in the same directory, we can run bfu like this:

# bfu `pwd`/archives/`uname -p`/nightly

Note that because system files are overwritten, bfu must be run as root. Also, ensure that /opt/onbld/bin is in root's PATH.

Here's some of the output from bfu:

# bfu `pwd`/archives/`uname -p`/nightly
Copying /opt/onbld/bin/bfu to /tmp/bfu.1100
Executing /tmp/bfu.1100 /home/rich/open_solaris/build-17/testws/archives/i386/nightly

Loading /home/rich/open_solaris/build-17/testws/archives/i386/nightly on /

Creating bfu execution environment ...
/tmp/bfu.1100[2261]: /net/onnv.eng/export/gate/public/bin/acr: cannot open
chmod: WARNING: can't access /tmp/bfubin/acr
Verifying archives ...
Performing basic sanity checks ...
/etc/svc/repository.db: passed integrity check
Disabling kernel module unloading ... moddebug: 0 = 0x20000
Quiescing init ...
Unmounting /lib/libc.so.1 ...
Disabling sendmail temporarily ...
Disabling remote logins ...
Disabling syslog temporarily ...
Killing httpd ...
Disabling fmd temporarily ...
Killing nscd ...
Turning on delayed i/o ...
Filesystem Mode
/ safe
/usr safe
2423 blocks

Saving configuration files in /bfu.child ... 4464 blocks
Removing init.d links ... done.
Removing obsolete rc.d scripts ... done.
Preserving user-installed perl modules...
cp: cannot access /usr/perl5/site_perl/5.8.3/*
Removing perl 5.8.3 packages
Removing perl 5.8.3 from /var/sadm/install/contents
Removing perl 5.8.3 from /usr/perl5
Extracting ufs modules for boot block ... 410 blocks
Extracting generic.usr ... 269143 blocks
Extracting i86pc.usr ... 410 blocks
Extracting generic.lib ... 34359 blocks
Extracting generic.sbin ... 1420 blocks
Extracting generic.kernel ... 108072 blocks
Extracting generic.root ... 3962 blocks
Extracting i86pc.root ... 6790 blocks
Extracting i86pc.boot ... 2430 blocks

Removing duplicate kernel binaries ...

Simulating SUNWcry* installation...

Cleaning up old Kerberos GSS-API mechanisms...

[... Conflict info (see later) omitted for brevity ...]

Create /platform/i86pc/boot_archive
updating /platform/i86pc/boot_archive...this may take a minute

For each file in conflict, your version has been restored.
The new versions are under /bfu.conflicts.

MAKE SURE YOU RESOLVE ALL CONFLICTS BEFORE REBOOTING.

To install resolved changes required for reboot in the boot
archive, invoke 'bootadm update-archive'

Removing obsolete smf services ...
Disabling unneeded inetd.conf entries ...
Connecting platform and name service profiles ...
Marking converted services as enabled ...
cp: cannot access /net/greenline.eng/meta0/smf/post-5090532/sysidtool.xml
bfu: could not copy /net/greenline.eng/meta0/smf/post-5090532/sysidtool.xml
cp: cannot access /net/greenline.eng/meta0/smf/post-5090532/kdmconfig.xml
bfu: could not copy /net/greenline.eng/meta0/smf/post-5090532/kdmconfig.xml
Upgrade of orac took 2:09.
Turning off delayed i/o and syncing filesystems ...
Filesystem Mode
/ safe
/usr safe

Entering post-bfu protected environment (shell: ksh).
Edit configuration files as necessary, then reboot.

When the BFU procedure (which takes about two minutes on the author's Acer Ferrari 3400 laptop) completes, we are left in a newly-created subshell. This is because it's possible that the new commands and libraries will not be compatible with the old, currently running, kernel. The new subshell has its PATH set to /tmp/bfubin and LD_LIBRARY_PATH set to /tmp/bfulib. These directories contain old versions of commands and libraries that are commonly needed to resolve problems and reboot the system. (The errors about accessing /net/greenline.eng are due to bug 4865419 in bfu.sh, and can be ignored.) Note that neither init nor shutdown should be used to reboot the system after BFUing; use the reboot command instead.

Notice that the bfu output talks about resolving conflicts. These can be identified in the bfu output by looking for lines similar to the following:

NEW conflict: boot/grub/menu.lst
NEW conflict: boot/solaris/bootenv.rc
NEW conflict: boot/solaris/devicedb/master
update: etc/.login
update: etc/aggregation.conf
NEW conflict: etc/auto_home
update: etc/auto_master
update: etc/bootrc

These conflicts must be resolved before the system is rebooted if brickification is to be avoided.


Resolving Conflicts

Every machine running Solaris has numerous configuration files, many of which get modified from a default installation. Examples of these configuration files include /etc/auto_home, /etc/nsswitch.conf, and /etc/path_to_inst. The bfu command keeps a list of these files, which is called the conflicts database.

When BFU overwrites a configuration file, a copy of the current file is first saved under /bfu.child. A copy of the new file (from the cpio archive) is also placed under /bfu.parent, having moved the previous bfu's /bfu.parent (if it exists) to /bfu.ancestor (the latter is, essentially, a grand parent). The files under these directories are referred to as the child, parent, and ancestor respectively.

When bfu has finished extracting the cpio archives, it diffs the various versions of these files and follows these rules:

• If the file is unchanged (i.e., the parent and child are the same), nothing is reported as there is nothing to do.
• If the file was accepted from the parent during the previous run of BFU (i.e., the child is the same as the ancestor), then the parent is automatically accepted at this time and the file is marked as "update: ". This is also what happens the first time we run BFU and one of the other rules doesn't apply.
• If the file is the same as the beginning of the previous file, BFU assumes that a user added lines to the end (in other words, the child consists of the parent plus concatenated lines). The child version is restored and the file is marked "restore: ".
• If the parent and child differ, but the parent and ancestor are the same, then the conflict is deemed to be old, and the file is marked "old: ".
• Finally, if the parent and child differ, and the parent and ancestor differ too, then conflict is deemed to be new and the file is marked "NEW: ".

So, a conflict that is NEW refers to a file that was different from the default, and the default has been updated. To resolve this, whatever changes were introduced in the new build must be ported to the existing file.

Although we could possibly accept the parent without further investigation, doing so will lose all our customizations from the child. A lot of these customizations are performed by class action scripts from non-ON packages, so blindly accepting the parent can lead to many lost hours. Conflicts should therefore be resolved, either manually or automatically. We'll describe how to do this later, but we should say that if not all conflicts can be resolved automatically (or we decide to resolve them manually), it is a very good idea to have a proper baseline conflicts database. And this can be accomplished by installing a BFU archive corresponding to the release currently on our system.

When we BFU to the same build that we have already installed by other means, we can safely ignore all conflicts because we know that our current configuration files work. A reboot must be performed prior to BFUing to a newer build.


Automatic Conflict Resolution

Most, if not all, conflicts can be resolved automatically using a script called acr. This script is usually invoked in the post-BFU protected environment; when called in this manner, no command line options are necessary. (For more specialized applications acr accepts one or two command line parameters. The first specifies the name of an alternate root directory, and the second is the name of the directory containing the BFU archives.)

acr uses a file called conflict_resolution.gz, which is created whenever nightly is run with the -a option, and is stored in the directory containing the BFU archive. The acr script lists the files it is processing on standard out, and places detailed results in a file called allresults, which is stored in a subdirectory of /tmp. The following is an example acr invocation:

bfu# acr
Getting conflict resolution information from /home/rich/open_solaris/build-17/testws/archives/i386/nightly/conflict_resolution.gz: 590 blocks
Building command list for the class action scripts:

Begin processing files
PROCESSING etc/devlink.tab
PROCESSING etc/driver_classes
PROCESSING etc/security/device_policy
PROCESSING etc/path_to_inst
PROCESSING etc/shadow
PROCESSING etc/vold.conf
PROCESSING etc/rmmount.conf
PROCESSING etc/inet/hosts
PROCESSING etc/inet/ipnodes
PROCESSING var/spool/cron/crontabs/root
PROCESSING etc/passwd
PROCESSING etc/inet/inetd.conf
PROCESSING etc/default/init
PROCESSING etc/remote
PROCESSING etc/nsswitch.conf
PROCESSING etc/inet/services
PROCESSING etc/security/auth_attr
PROCESSING etc/security/exec_attr
PROCESSING etc/security/prof_attr
PROCESSING etc/user_attr
PROCESSING etc/vfstab
PROCESSING etc/auto_home
PROCESSING etc/datalink.conf
PROCESSING boot/grub/menu.lst
PROCESSING etc/krb5/krb5.conf
PROCESSING etc/openwin/server/etc/OWconfig
PROCESSING kernel/drv/sd.conf
PROCESSING boot/solaris/bootenv.rc
PROCESSING boot/solaris/devicedb/master
See /tmp/acr.Yda49i/allresults for more information.

And this is the corresponding allresults file:

bfu# cat /tmp/acr.Yda49i/allresults
PROCESSING etc/devlink.tab
RETURN CODE : 0
PROCESSING etc/driver_classes
RETURN CODE : 0
PROCESSING etc/security/device_policy
RETURN CODE : 0
PROCESSING etc/path_to_inst
RETURN CODE : 0
PROCESSING etc/shadow
RETURN CODE : 0
PROCESSING etc/vold.conf
RETURN CODE : 0
PROCESSING etc/rmmount.conf
RETURN CODE : 0
PROCESSING etc/inet/hosts
RETURN CODE : 0
PROCESSING etc/inet/ipnodes
RETURN CODE : 0
PROCESSING var/spool/cron/crontabs/root
RETURN CODE : 0
PROCESSING etc/passwd
RETURN CODE : 0
PROCESSING etc/inet/inetd.conf
RETURN CODE : 0
PROCESSING etc/default/init
RETURN CODE : 0
PROCESSING etc/remote
RETURN CODE : 0
PROCESSING etc/nsswitch.conf
RETURN CODE : 0
PROCESSING etc/inet/services
RETURN CODE : 0
PROCESSING etc/security/auth_attr
RETURN CODE : 0
PROCESSING etc/security/exec_attr
RETURN CODE : 0
PROCESSING etc/security/prof_attr
RETURN CODE : 0
PROCESSING etc/user_attr
RETURN CODE : 0
PROCESSING etc/vfstab
RETURN CODE : 0
PROCESSING etc/auto_home
RETURN CODE : 0
PROCESSING etc/datalink.conf
RETURN CODE : 0
PROCESSING boot/grub/menu.lst
RETURN CODE : 0
PROCESSING etc/krb5/krb5.conf
RETURN CODE : 0
PROCESSING etc/openwin/server/etc/OWconfig
Converting old OWconfig file to new format.
RETURN CODE : 0
PROCESSING kernel/drv/sd.conf
RETURN CODE : 0
PROCESSING boot/solaris/bootenv.rc
RETURN CODE : 0
PROCESSING boot/solaris/devicedb/master
RETURN CODE : 0

The allresults file should be examined to ensure that to errors occurred during the acr process. If any errors do occur, the conflicts must be resolved manually. An acr failure might also be indicative of a bug in one or more upgrade scripts, so a posting to the opensolaris-bugs mailing list might be a good idea.


Manual Conflict Resolution

We manually resolve conflicts by diffing the ancestor and parent versions of the file in question. More generally, we run the following command

bfu# diff /bfu.ancestor/$file /bfu.parent/$file

and apply the diffs manually (note that if this is the first time we run BFU, /bfu.ancestor will not exist, hence our earlier recommendation about creating a baseline conflicts database). For many files a shortcut can be used. Given that most changes are additions or modifications, and that the BFU process leaves us in a directory called /bfu.conflicts, simply running

bfu# pw
/bfu.conflicts
bfu# diff $file /$file

and ensuring that all the diffs point to the right (i.e., ">" rather than "<") will do the job. Here's an example:

bfu# diff boot/solaris/bootenv.rc /boot/solaris/bootenv.rc
5d4
< # CDDL HEADER START
7,10c6
< # The contents of this file are subject to the terms of the
< # Common Development and Distribution License, Version 1.0 only
< # (the "License"). You may not use this file except in compliance
< # with the License.
---
> #pragma ident "@(#)bootenv.rc 1.34 05/04/15 SMI"
12,26d7
< # You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
< # or http://www.opensolaris.org/os/licensing.
< # See the License for the specific language governing permissions
< # and limitations under the License.
< #
< # When distributing Covered Code, include this CDDL HEADER in each
< # file and include the License file at usr/src/OPENSOLARIS.LICENSE.
< # If applicable, add the following below this CDDL HEADER, with the
< # fields enclosed by brackets "[]" replaced with your own identifying
< # information: Portions Copyright [yyyy] [name of copyright owner]
< #
< # CDDL HEADER END
< #
< #pragma ident "@(#)bootenv.rc 1.35 05/06/08 SMI"
< #
38a20,22
> setprop prealloc-chunk-size 0x2000
> setprop bootpath /pci@0,0/pci-ide@11,1/ide@0/cmdk@0,0:a
> setprop console 'text'

In this example, the important diff lines point to the right, so we're OK.

Although this process works most of the time, there are a few exceptions which we detail below. These exceptions are etc/name_to_major, etc/security/*_attr, and /etc/path_to_inst.

The file etc/name_to_major maps device names to their major numbers; each device must have a unique major number. Diffing the ancestor with the parent is a good starting point; the device name is important but the major number may vary. Ideally, devices that are no longer supported should be removed, but leaving them in is usually harmless. When in doubt, leave them in.

New devices should be added with an unused major number—preferably the one from the diffs if it is available.

When we're finished, we can check our changes by running two sort commands:

bfu# sort -k1,1 /etc/name_to_major | sort -uc -k1,1
bfu# sort -k2n /etc/name_to_major | sort -uc -k2n

The first command reports the first duplicated device name, and the second command reports the lowest duplicated major number. Both commands should result in no output before rebooting. (The kernel will warn about such conflicts when rebooting, but by then it is likely to be too late...)

The etc/security/*_attr files are modified quite a lot by class-action scripts, so there can be a big difference between the parent and child versions. This means that the diff shortcut we described previously won't be very effective, although the more general diff of the parent and ancestor can still be useful.

The final exception is /etc/path_to_inst. Unless we know exactly what we're doing, this file shoudl not be touched.

Generally, only conflicts that are marked as NEW need to be examined.


BFU and Zones

If the machine we're BFUing hosts zones in addition to the global one, we have some more work to do. Because the contents of the global zone are copied each time we create a zone (unless that zone is "sparse"), we must ensure that the contents of the all the non-global zones are consistent with the global zone. Fortunately, BFU handles this for us, by updating each zone after the global zone has been updated. Because each zone may have configuration files that differ from the global zone's, each zone has its own set of conflict management directories. Note that it is a good idea to shut down all zones before BFUing, because this ensures that any dependancies that might be violated by the new files won't break them (the zones). It is also a good idea to rerun bfu if we add a new zone after BFUing, to ensure that the new zone gets updated.

It is not possible to BFU a single zone. If we want to test userland changes in a zone, we must manually copy the files from our build workspace into that zone.

Conflicts must be resolved in the global zone and then in each non-global zone prior to rebooting. Resolving conflicts in a zone works in the same way as in the global zone, but because many files will be identical to their global-zone counterparts many files won't need to me manually merged: copying the file form the global zone will be sufficient.

When all conflicts in the global and all non-global zones have been resolved, we can reboot the system. Note that the system must be rebooted before any non-global zone is rebooted.


BFUing Without Building

So far this article has assumed that we will be BFUing using archives we built ourselves (as a side effect of building OpenSolaris). What happens if we want to use the latest and greatest bits, without compiling them? Fortunately, pre-built BFU archives are available. To BFU without building from source, we need to download the BFU archive and ON build tools corresponding to the release we want to BFU to. We should also be aware that if there is an appreciable difference between our current build and the one we wish to BFU to, we might need to BFU several times.

Let's look at an example, BFUing our recently BFUed Build 17 system to Build 18. As in Part 1 of this series, we'll assume that we've downloaded the two required files into a directory called $HOME/open_solaris/build-18, and that our current directory is that directory. For convenience, we'll refer to this directory as $OPEN_SOL.

First we'll unpack the BFU archive:

$ bzip2 -dc opensolaris-bfu-20050720.i386.tar.bz2 | tar xf -

This will create a directory called archives-20050720, which contains the BFU archive (the name of this file and similar ones will change with each release). We must now become root and install the ON build tools.

# cd /tmp
# bzip2 -dc $OPEN_SOL/SUNWonbld-20050720.i386.tar.bz2 | tar xf -
# cd onbld
# pkgadd -d . SUNWonbld


For brevity, we've omitted the output from pkgadd. Next we BFU, using a command similar to our previous example (having first ensured that FASTFS, BFULD, and GZIPBIN have been defined appropriately, and that /opt/onbld/bin is in our PATH):

# bfu $OPEN_SOL/archives-20050720/`uname -p`

Next we resolve any conflicts (in this example, there are none), and reboot:

bfu# acr
No conflicts to resolve.
bfu# reboot
updating /platform/i86pc/boot_archive...this may take a minute


The message about updating /platform/i86pc/boot_archive is x86 specific. Assuming all goes well, our machine will boot the new build (in this case, Build 18). We can check (as always) by logging in and running uname:

$ uname -a
SunOS orac 5.11 opensol-20050720 i86pc i386 i86pc


Summary

This article has described how we can use BFU to update our version of OpenSolaris, either from source that we built ourselves or from BFU archives we download from the opensolaris.org web site or elsewhere. To illustrate the former we BFUed to the Build 17 we built in the previous article, and we BFUed from Build 17 to Build 18 as an example of the latter.

The steps we used to BFU from source we built ourselves are:

1. Ensure the FASTFS, BFULD, and GZIPBIN environment variables are set appropriately.
2. Run the bfu command as root from our workspace directory, passing the name of the archive as the command line option. More often than not, this will be `pwd`/archives/`uname -p`/nightly.
3. Resolve any conflicts using the acr script.
4. Resolve any remaining conflicts manually.
5. Reboot using the reboot command (rather than init or shutdown).

The steps required to install pre-built BFU archives are similar:

1. Download the BFU archive and ON build tools for the build we want to BFU to from the opensolaris.org web site or one of its mirrors.
2. Unpack the BFU archive.
3. Install the ON build tools.
4. Ensure the FASTFS, BFULD, and GZIPBIN environment variables are set appropriately.
5. Run the bfu command as root, passing it the name of the BFU archive we unpacked in step 2.
6. Resolve any conflicts using the acr script
7. Resolve any remaining conflicts manually
8. Reboot using the reboot command.

We also warned that once it has been BFUed, the normal methods for updating the system (e.g., patches or Upgrade installs) cannot subsequently be used: a fresh install or another BFU is required. Finally, we stated that BFU is intended for active OpenSolaris developers only; regular mortals are probably best served using Solaris Express.


Recommended Reading

The opensolaris.org web site contains several documents, including the OpenSolaris Developer's Reference. Also, be sure to check out details regarding the OpenSolaris Tools Community site and specifically the Sun Studio download site at opensolaris.org/os/community/tools/sun_studio_tools/. The author's book, Solaris Systems Programming [Teer 2005], is essential for readers wishing to work with the userland OpenSolaris code, and Solaris Internals [Mauro and McDougall 2001] by Jim Mauro and Richard McDougall should be on every OpenSolaris kernel hacker's bookshelf. The community blogs at www.opensolaris.org/os/blogs/ are another great source of information.

[Mauro and McDougall 2001]: Solaris Internals, ISBN 0-13-022496-0. See www.solarisinternals.com/ for more details.
[Teer 2005]: Solaris Systems Programming, ISBN 0-201-75039-2. See www.rite-group.com/rich/ssp/ for more details.