Installing and Testing ON

This chapter describes several flexible methods for installing your ON bits. Please note that because ON does not include all the programs needed for a working system, you must have an existing full install (typically Solaris or Solaris Express) before you can perform these procedures successfully.

Additionally, some of the common testing procedures for the kernel and core userland components are covered. Although these tests are intended to cover as much of the system as possible, and to be flexible enough that additional tests can be written and added to the infrastructure, most testing is still done by project-specific test suites. When fixing bugs or adding new features, you are well-advised to contact the owner(s) of the code you are changing to obtain any existing tests. You should also contribute new tests that detect the bug you are fixing or verify the functionality of your new features.

5.1 Installation Overview

Other than manually copying specific files from your proto area into your live system, there are three main ways to install your bits on your system. Which one you use will depend on what you have changed: if you have changed only the kernel, see section 5.1.1 to learn about Install. If you have changed the kernel and userland components of ON and your changes must be applied together, you must either hand-copy your userland changes before using Install, or use BFU; see section 5.3 for information on BFU.

To accommodate fully functional builds even though some sources are missing, a set of closed binaries is available, and the build system has been modified to make use of them. You will need to use the closed-bins.<platform>.tar.gz components along with your build products to build either Install or BFU archives that work. Please see 4 Building OpenSolaris and the latest Release Notes for more information on building and the use of the closed binaries.

Each of these installation methods is progressively more complex and time-consuming, but upgrades a larger part of your system. Because most ON developers only modify the ON sources and are not responsible for integration testing, BFU is by far the most popular method for performing system upgrades. Developers working heavily on the kernel will often make use of Install during development and use BFU to keep current between testing phases.

Each method is described briefly in this section. Detailed instructions are provided in sections 5.2 and 5.3.

5.1.1 Cap-Eye-Install

Install (pronounced cap-eye-install) is used to update only the kernel and its associated modules on a specific system. It will place the new kernel in a nonstandard location and install only the platform-specific modules for your particular host. This allows you to test your changes without removing the normal kernel; if your new kernel does not boot or crashes, this makes recovery much easier.

5.1.2 BFU

BFU is used to update all ON bits, both kernel and userland. It is capable of updating some configuration files and is aware of the impact of the changes that have been made to ON. BFU is more thorough than Install, and takes longer. Also, unlike Install, the new kernel will be installed over the existing one, so if it does not work properly you may have to boot from alternate media to recover.

5.1.3 Flag Days and Other Hazards

In some cases, you will need to install newer versions of one or more system packages before you will be able to use a new version of the ON bits. When this happens, it is known as a Flag Day. Flag Day notices will be posted at http://opensolaris.org/os/community/onnv/ and will include instructions for building and/or installing the newer software you will need. It should be noted that this installation procedure is not guaranteed to interoperate cleanly with the standard packaging tools such as pkgadd(1M). In particular, use of BFU (see section 4.1.4) or ad-hoc replacement of Solaris components means that those components can no longer be updated using Solaris packages. If this is of concern to you, we recommend that you utilize exclusively Solaris Express for managing your system package installation, and build and test only against the source tree current at the time of the latest Express build. You can

5.2 Using Cap-Eye-Install to Install Kernels

First, you must have a workspace containing a built kernel. If you need more information on building kernels, see chapter 4 and especially section 4.3. Once you have built a kernel, you need to make an Install tarball using the Install command.

The Install(1) utility creates a tar file that can be extracted into an appropriate machine's root directory. It utilizes an existing built kernel tree and the kernel makefiles to determine the correct contents of the tar file. See Install(1) or the output of 'Install -h' for a complete list of options; only the most commonly-used options are described here.

The tar file constructed by Install is specific to an architecture, such as sun4u or i86pc. There are two ways to specify the architecture for which you want Install to create an archive. The first is to be in the architecture's subdirectory (usr/src/uts/<arch>) when running Install; the second and preferred method is to use the -k <arch> option. Note that current releases of ON support only one architecture in each ISA: sun4u on SPARC and i86pc on x86/amd64.

Another setting which is usually specified is the ``glomname'', using the -G option. This is the name of the subdirectory in /platform/<arch>/ that the binaries will go into, and is generally of the form ``kernel.something''. If you don't use a glomname, you will overwrite the current kernel and modules on the target machine, and are likely to make BFUing the machine later a more difficult task, as well as running the risk of having to boot from alternate media to fix your machine if the new kernel does not boot.

A simple invocation of Install might look like:

        $ Install -G kernel.foo -k sun4u
        ... lots of spew ...
        Creating tarfile /tmp/Install.username/Install.sun4u.tar
        Install complete

You can now copy /tmp/Install.username/Install.sun4u.tar to your test machine and extract it in the root directory.

On x86 with build 14 or newer, you will need to add your kernel to the boot archive before you will be able to boot it. To do this, add the line:

        platform/i86pc/kernel.foo

to /boot/solaris/filelist.ramdisk, where kernel.foo is the glomname (the argument to -G). This requirement is eliminated in build 18 and newer, and does not apply to SPARC.

After installing your kernel, reboot the test machine and have it use the new kernel by running:

        (SPARC)
        # reboot -- 'kernel.foo/sparcv9/unix'

        (AMD64)
        # reboot -- 'kernel.foo/amd64/unix'

        (x86)
        # reboot -- 'kernel.foo/unix'

Note that you will need to use either this reboot syntax each time you wish to boot the test kernel, or use similar arguments to the bootloader or OBP. Otherwise, the normal kernel installed by BFU or the regular installation will be booted.

5.2.1 Caveats

Although Install is useful for developers who have changed only kernel code, it is of limited value for others. In particular, if a recent flag day notice indicates that newer kernels are incompatible with existing userland libraries or commands, Install cannot be used to test the updated kernel until you have upgraded your userland via BFU or some other mechanism such as the regular installation or upgrade procedure.

Like bfu (see section 5.3), Install is rather closely attached to its particular release, so you should use the current version from the gate matching the release you are building. Normally this is in the public/bin subdirectory of the gate; however, for installations outside Sun it is located in /opt/onbld/bin.

It is critical that Install users install the correct set of platform-specific modules, especially on SPARC systems. Failure to do so can result in an unbootable system. See section 5.2.2 below for more information on how platform-specific modules relate to Install.

One major advantage of Install over BFU is the ability to keep your existing kernel in place so that you can still boot if the test kernel proves toxic. We strongly recommend that if you use Install to test kernels, you take advantage of this feature and use distinct locations (see the -G option described in section 5.2.1 above) for each new kernel you test. Otherwise, you will likely have to boot from alternate media to repair your system following the installation of a bad kernel.

5.2.2 Platform-Specific Information

Ordinarily, Install does not generate archives with implementation-specific modules. If these archives are installed onto a system which requires the missing modules, the system may fail to boot or work properly. If you do this, you will need to boot from a known-working kernel and correct the problem.

An example symptom of the problem (on an Enterprise 3500):

        SunOS Release on81 Version jrhyason_[ws-vmstat]_05/15/01 64-bit
        Copyright 1983-2001 Sun Microsystems, Inc.  All rights reserved.
        DEBUG enabled
        obpsym: symbolic debugging is available.
        Read 297063 bytes from misc/forthdebug
  ====>  WARNING: consconfig: consconfig_util_openvp failed: err 6 vnode
 0x2803c80
  ====>  WARNING: consconfig: consconfig_util_link I_PLINK failed: error
 22
        configuring IPv4 interfaces: hme0.
        ...

The console is gone!

To include the needed modules in your Install tarball, make sure to use

        $ Install -i <implementation>

to include implementation-specific modules. But how do you know what sun4u implementation you need? First, obtain your machine's ``official'' implementation name from the output of 'uname -i'. Then, in usr/src/uts/sun4u, run ``grep IMPLEMENTED_PLATFORM */Make*'' to see a list of implementations and the corresponding platform name reported by uname(1).

In the example above, the E3500 reports:

        $ uname -i
        SUNW,Ultra-Enterprise

And we see from the grep output:

        $ grep IMPLEMENTED_PLATFORM */Make*
        ...
        sunfire/Makefile:IMPLEMENTED_PLATFORM   = SUNW,Ultra-Enterprise
        ...

In this case, the ``-i sunfire'' argument must be added to get the correct behavior.

Additionally, one of the easiest ways to get tripped up with Install wads comes from the fact that not all drivers are delivered by ON. This has been particularly noticeable with x86, but it also happens with SPARC, especially framebuffer drivers. One way to work around this is to do:

 # cd /platform/{sun4u,i86pc}
 # mkdir glomname
 # (cd kernel; tar cf - .) | (cd glomname; tar xf -)

and then install the Install glom image. This will copy your existing drivers to the new kernel installation, ensuring that the drivers which are not part of ON (or OpenSolaris as a whole) are available when you reboot.

5.3 Using BFU to Install ON

The Blindingly Fast Upgrade (or Bonwick-Faulkner Upgrade) is a process used to update ON bits on a system. The ordinary Solaris upgrade procedure requires the complete WOS and takes at least 30 minutes (usually much longer). To save time, BFU uses a set of cpio(1) archives to directly overwrite the existing contents of the system. BFU often takes less than 10 minutes to run to completion, and if you are upgrading from a recent build, conflict resolution will usually take only a few minutes. Over the course of a year, using BFU can save dozens of hours of development time.

In order to use BFU, you will need to set three additional environment variables first. You can set these in your login dot-files, or on the command line. If you prefer, you could create a local wrapper for bfu(1) that sets them first. The environment variables are:

FASTFS

This should normally be set to /opt/onbld/bin/`uname -p`/fastfs.

BFULD

This should normally be set to /opt/onbld/bin/`uname -p`/bfuld.

GZIPBIN

This should normally be set to /usr/bin/gzip.

BFU is simple to use and normally takes only a single argument: a path to the set of archives you wish to install. For example, if your workspace is located in /home/jdoe/workspace, and you have completed a nightly(1) build, you would invoke bfu(1) as follows:

        # bfu /home/jdoe/workspace/archives/`uname -p`/nightly

Note that, since it modifies the system software installation, bfu(1) must always be run as root.

When bfu completes there's no guarantee that the new commands and libraries are compatible with the currently running (old) kernel. Therefore, instead of exiting, bfu puts you into a subshell in which PATH=/tmp/bfubin and LD_LIBRARY_PATH=/tmp/bfulib. These directories contain the old versions of the commands and libraries commonly needed to resolve conflicts and reboot the system. They have also been modified to work with a saved copy of the old dynamic linker.

Note that you may receive warnings from BFU about being unable to copy files from ``greenline.eng'' or other systems or locations. In general, these warnings should be reported as bugs. However, at the time of this writing, they are harmless provided that your system is running at least Solaris 10 build 74 prior to your BFU attempt. See the latest release notes for any additional requirements and restrictions.

5.3.1 Caveats

Although it saves time, BFU is not a panacea. This section contains information about BFU's drawbacks. You should carefully evaluate these drawbacks against the benefits and decide whether BFU is appropriate to your needs. In general, unless you are an active developer, we recommend that you do not use BFU.

BFU does not update package information. Therefore you will most likely be unable to install official Solaris patches or run a full system upgrade procedure on a system which has been BFU'd. The importance of this cannot be overemphasized: IF YOU BFU YOUR SYSTEM, DO NOT ATTEMPT TO USE ``NORMAL'' SYSTEM MANAGEMENT PROCEDURES TO UPDATE IT IN THE FUTURE. USE BFU OR REINSTALL.

BFU does not update non-ON packages, even if newer versions of those packages are required in order to successfully install or run the version of ON you are installing. You may need to update those packages before and/or after running BFU. To understand what package updates may be needed, consult http://opensolaris.org/os/community/onnv/ for the full list of flag days between the build you are currently running and the build you wish to BFU to. Each flag day notice will instruct you as to what package updates, if any, are needed, and whether they must be completed before or after BFUing. It is critical that you read, understand, and follow these instructions exactly before running BFU. If you fail to do so you will almost certain brickify your system. See section 5.1.3 for more information about flag days.

Although the core functionality of BFU consists of the simple activity of unpacking cpio(1) archives into the running system, it also performs numerous other tasks related to the update of your system. Many of these tasks are specific to particular changes that have been made in the sources over a period of years. If your system has unusual characteristics, these additional updates can fail, which may result in a nonfunctional system. Because these updates vary greatly, it is impossible to know in advance which updates could fail, or what failure modes are possible. Although such failures are rare, they can occur. If you use BFU, you should follow the development of ON to understand changes being made that could have an adverse effect on your system. If you have doubts as to how well BFU will update a particular aspect of your system following a major change to ON, read bfu(1) and consult with the engineers who made the change.

When bfu finishes, it invokes ksh with a limited PATH. The PATH contains programs which have been specially modified to work regardless of what changes the BFU archives may contain. You can use these programs to resolve conflicts (see section 5.3.2). In particular, ``reboot'' works, but ``init 6'' does not. You can exit from this protected environment if you want, but it is not advisable unless you are sure that there have not been any ``flag days'' (synchronized kernel/userland changes) since your last bfu.

Never BFU in a window; always use the system console. If BFU or the system crashes in midstream, or you use the window system and it crashes or hangs, your system will be in an inconsistent state. You may be unable to boot. Therefore, you should ensure that as much of the system as possible is quiescent before starting a BFU, and be sure you have a copy of suitable system software media handy to reinstall if necessary.

Never BFU a production system. Production systems should always be updated to approved releases using the supported upgrade mechanism.

5.3.2 Resolving Conflicts

Every machine has several configuration files which get modified from the default installation; bfu keeps a list of these. This list is known as the conflicts database.

BFU saves a copy of each configuration file it would overwrite under /bfu.child; likewise, it stores a copy of each such file from the cpio archives it is extracting under /bfu.parent, having moved the previous BFU's /bfu.parent (if any) to /bfu.ancestor. We refer to the files saved under those directories as the child, parent and ancestor respectively.

When the extraction is complete, BFU diffs the various versions of these configuration files and acts according to the following rules:

* If the file is unchanged (i.e., there is no difference between the child and the parent), there is nothing to do or report.

* If the file was accepted from the parent the previous run (i.e., the child is identical to the ancestor) then the parent is accepted automatically this time; such files are marked as ``update:''.

* If the file is the same as the beginning of the previous file, it is assumed that the user has added lines to the end (i.e., the child consists of the parent plus some added lines); the child version is restored; such files are marked as ``restore:''.

* If the file differs between the parent and the child, but the parent and ancestor are identical, then it is deemed an old conflict; such files are marked as ``old:''.

* Lastly, if the file differs between the parent and the child, and the parent and ancestor differ as well, then it is deemed a NEW conflict; such files are marked as ``NEW:''.

So now we know that a NEW conflict means a file which was already different from the default, where the default has been updated. To resolve this difference, whatever changes were introduced in the new build must be ported to the existing file on the system.

Although it is possible to blindly accept the parent, this will cause any customizations in the child to be lost. As it is very common for these files to have such customizations made automatically by class action scripts from non-ON packages, this is usually a mistake, which can lead to hours of lost productivity. Therefore, there are two forms of conflict resolution which are discussed in the following two sections. They are automatic and manual conflict resolution.

If you elect to resolve conflicts manually, or if the automatic tools are unable to resolve all conflicts, you will benefit from having a proper BFU baseline installation. To establish such a baseline, you should install BFU archives corresponding to your distribution immediately after installing it. It is strongly recommended that you begin this process only if your installation is sufficiently recent that such BFU archives are available. In particular, BFUing a system older than Solaris 11 build 16 will fail and may render your system unbootable.

When BFUing to the same build as your distribution include, you can ignore all conflicts, since your existing installed configuration files are known to work correctly for this build. You must then reboot before BFUing to a later build.

5.3.2.1 Automatic Conflict Resolution

Automatic conflict resolution is performed by a script called acr. This script is available in /opt/onbld/bin and in the usr/src/tools/scripts directory of the ON workspace. In the usr/src/tools/scripts directory, the acr executable gets made from acr.sh if you do a nightly(1) build with the -t option. There is man page, acr.1, in /opt/onbld/man/man1 and in the usr/src/tools/scripts directory of the ON workspace.

The standard way to use acr is to invoke it while still in the protected environment after a BFU is performed. In this mode there is no need to specify any command line parameters.

For more specialized applications, acr accepts one or two parameters. The first parameter is the name of an alternate root directory. The second parameter, if specified, is the name of the directory containing the archives.

acr uses a file called conflict_resolution.gz in the directory containing the BFU cpio archives. This conflict resolution file is constructed whenever nightly(1) creates archives. That is to say whenever nightly(1) is run with the -a option.

When acr runs, it lists the files that it is processing to standard out. Detailed results are written to a file called allresults in a subdirectory of /tmp. When acr exits, it prints the full path name of the allresults file. The allresults should be examined to ensure that no errors occurred during acr processing. If errors occurred, you will need to resort to manual conflict resolution discussed in the next section.

5.3.2.2 Manual Conflict Resolution

Manual conflict resolution requires you to resolve each of the conflicts by hand. The general way to resolve conflicts is:

        % diff /bfu.ancestor/$file /bfu.parent/$file

then manually apply the diffs to /$file. Note that you will not have /bfu.ancestor if you have not previously BFU'd; therefore you will find manual conflict resolution easier if you have established a baseline as described above. For many files, a short-cut can be employed. BFU drops you in /bfu.conflicts when it completes, and most changes are additions or modifications, so doing:

        % diff $file /$file

and making sure the diffs all point to the right (i.e., ``>'' rather than ``<'') will do the trick. This will not work in all cases, however; some notable exceptions are detailed below.

etc/name_to_major

This file matches device names with major numbers; it is critical that each device have a unique major number. Diffing the ancestor with the parent is a good start -- just remember that the name is important, but the major number may vary. Old devices which have been removed in the parent should be deleted, but if you're not sure, just leave them alone as they are generally harmless. New devices should be added with an unused major number (if it's available, use the one from the diffs). Never change the number for an existing entry unless you are sure you know what you are doing. If you are in doubt as to what changed, you can consult the history of the files directly from your workspace (e.g. usr/src/uts/{intel|sparc}/os/name_to_major).

Once finished, you can sanity-check your changes by running:

        % sort -k1,1 /etc/name_to_major | sort -uc -k1,1
        % sort -k2n /etc/name_to_major | sort -uc -k2n

These report the first duplicated device name and the lowest duplicated major number, respectively. Either of these indicates a problem that you must correct before rebooting. If there is no output, you're all set. Note that the kernel will also warn you of such conflicts after you reboot, but by then it may be too late.

etc/security/*_attr

These files tend to get shuffled around quite a bit by class-action scripts, so the parent and child versions can differ wildly. For these, the shortcut described above is ineffective, but the more general diff of the ancestor with the parent works well.

In general, only NEW conflicts need to be examined.

5.3.3 BFU and Zones

The contents of each zone are copied from the global zone when the zone is created. To ensure that all zones remain operational after BFUing, it is necessary to keep the zone contents consistent with those in the global zone. To do this, BFU will update each zone in turn once the global zone has been updated; however, because each zone may have configuration files that differ from the global zone's, each zone has its own BFU conflict management directories. Unfortunately, it is not possible to BFU only a single zone. BFU affects the entire system including all zones. If you want to test userspace changes in a zone, you will need to copy files from your proto area into that zone manually.

It is safest to shut down all zones before BFUing your system. This will ensure that dependencies which may be violated by the files being installed will not break a zone.

After BFUing a system with zones other than the global zone, you will need to first resolve conflicts in the global zone, then resolve conflicts in each remaining zone before rebooting the system. Resolving BFU conflicts in a zone works exactly the same way as for the global zone; however, in many cases it will not be necessary to manually merge the file contents. Instead you are likely to find that the file should be identical to that in the global zone, and you can simply copy it over. After resolving conflicts in all zones, you must reboot your system before rebooting any zone.

5.4 Test Suites

Sun has a number of test suites used with ON. Unfortuantely these aren't yet available; check out the roadmap at http://opensolaris.org/os/about/roadmap/.