Device Tree Cleanup and Reset (Controller Renumbering)

This procedure assumes that both nodes of the cluster are configured more or less identical.  This will completely reset all logical disk pathing including those entries for the system boot disks.  Systems having different hardware configurations may still not match after this procedure.

1. Create a checkpoint of the current system to fall back to if necessary.
2. Determine the current checkpoint associated syspool dataset that is used for the root filesystem.  (ie. syspool/rootfs-nmu-000)  This will be listed as the "Current" and "Default" entry for the "show appliance checkpoint" command.
3. Shutdown and reboot into GRUB or boot from CDROM
4. Select "Safe Mode" from regular GRUB menu or "Recovery Console" if booted from CDROM.
5. If asked for a login, login as "root" with no password.
6. Force import syspool.  "zpool import -f syspool"
7. Create a mountpoint in /tmp and mount the current checkpoint to it.  For example,  "mkdir /tmp/a ; mount -F zfs syspool/rootfs-nmu-000 /tmp/a"
8. Remove all entries from /tmp/a/dev/[r]dsk "rm /tmp/a/dev/dsk/* ; rm /tmp/a/dev/rdsk/*"
9. Remove all entries from /tmp/a/dev/cfg "rm -rf /tmp/a/dev/cfg/*"
10. Remove /tmp/a/etc/devices/devid_cache
11. Remove /tmp/a/etc/devices/mdi_scsi_vhci_cache
12. Put down a new device tree via devfsadm directed to the current checkpoint.  "devfsadm -r /tmp/a"
13. Update the boot archive of the current checkpoint "bootadm update-archive -R /tmp/a"
14. Unmount the current checkpoint "umount /tmp/a"
15. Reboot
16. Repeat same steps for the other node.
17. If the system fails to boot, fall back to the checkpoint created in step 1.