Xen Live Migration with iSCSI

Xen Live Migration with iSCSI

Overview

This is a document about how to set up XEN and iSCSI on 3 Debian Stable (3.1 / Sarge) machines so that you can have a virtual host that can be migrated live between 2 of the machines. The third machine will be the common disk for the virtual machine that is seen via the 2 machines running XEN. I have assumed you have some knowledge of Linux specifically Debian, but not limited to. You may need to solve some dependencies that I have left out by yourself. I can't think of everything, and I didn't start with completely empty machines.

In this example we will use dangermouse character names for the machine names. Buggles and Dangermouse will be the XEN hosts, and Penfold will house the disk that is common for both. Virtual01 will be the name we give to the virtual machine, that can be live migrated between dangermouse and buggles, using the disk on penfold over iSCSI. That was a mouthful, but don't worry we will break it down into what is hopefully simple steps.

---

XEN

Installation

Assuming you have already got running debian stable machines using the 2.6.8 kernel that was shipped with Debian, we are ready to move on to installing XEN from backports.org. This process needs to be done on all three machines.

Prepare the apt sources by adding the backports repositories to the file /etc/apt/sources.list

dangermouse:~# echo "deb http://www.backports.org/debian/ sarge-backports main" >> /etc/apt/sources

Now tell apt that we don't normally want to install packages from backports.org unless we specifically ask.

dangermouse:~# cat <<! >> /etc/apt/preferencesPackage: *Pin: release a=sarge-backportsPin-Priority: 200!

Update the apt cache:

dangermouse:~# apt-get update

Now we want to install the xen kernel and hypervisor on every machine. This is so we get a consistent 2.6.16 kernel on every machine including the machine we will use for the disk so we can use the latest version of iSCSI Target software from IET.

dangermouse:~# aptitude -t sarge-backports install linux-image-2.6.16-1-xen-686 linux-modules-2.6.16-1-xen-686 grub xen-hypervisor-3.0-i386 xen-utils-3.0 udev

There is probably a few other packages that I have left out that will get automatically updated as well. They should all be fine.

We need to disable tls as it is not supported by XEN at this time. There may be a libc686-xen package available, and I would suggest installing that if possible instead of disabling /lib/tls.

dangermouse:~# mv /lib/tls /lib/tls.disabled

The Debian package of XEN kernel or hypervisor doesn't yet make an initrd image, so we need to make one manually.

dangermouse:~# mkinitrd -o /boot/initrd.img-2.6.16-1-xen-686 2.6.16-1-xen-686

Now prepare the boot loader. Firstly run update-grub to automatically populate /boot/grub/menu.1st, then fixup the created XEN entries, by adding two module lines and removing the initrd line.

dangermouse:~# update-grub

turn this:

kernel /boot/vmlinuz-2.6.16-1-xen-686 root=/dev/hda1 roinitrd /boot/initrd.img-2.6.16-1-xen-686

into this in the /boot/grub/menu.1st:

kernel /boot/xen-3.0-i386.gz console=vga dom0_mem=256000module /boot/vmlinuz-2.6.16-1-xen-686 root=/dev/hda1 ro xencons=offmodule /boot/initrd.img-2.6.16-1-xen-686

Replace /dev/hda1 with whatever is your root partition. dom0_mem is the amount of memory available to the host virtual machine or domain0. xencons turns off the use of the serial port. I need the serial ports for other purposes, so this one is up to you.

Now you need to reboot and confirm that the XEN hypervisor boots and domain0 starts as well. If this fails go over the steps and make sure you didn't leave a step out. Because you still have your Debian standard kerenl installed you can always recover to that with no ill affect

XEN Configuration

Now that you have XEN installed and you are successfully booting off the XEN Hypervisor we can configure the xen daemon. The configuration file is /etc/xen/xend-config.sxp.

(xend-relocation-server yes)(xend-relocation-hosts-allow '')(network-script 'network-bridge netdev=eth0')(vif-script vif-bridge)(dom0-min-mem 196)(dom0-cpus 0)

Now restart the xend process.

dangermouse:~# /etc/init.d/xend restart

iSCSI

uDev Setup

We setup udev to make the iSCSI devices a consistent name across all the machines, and you don't end up with /dev/sda on one machine and /dev/sdb on another.

On each of the clients do the following:

dangermouse:~# echo "BUS==\"scsi\", SYSFS{vendor}==\"IET\", SYSFS{model}==\"VIRTUAL-DISK\", \KERNEL==\"sd*\", NAME=\"%k\", PROGRAM=\"/lib/udev/getlun.sh $id\", SYMLINK+=\"iscsi_%c%n\"" \>>/etc/udev/rules.d/10_persistant_scsi.rulesdangermouse:~# cat <<! >> /lib/udev/getlun.sh#!/bin/bashecho $1 | awk -F":" '{print $NF}'!

What this will do is create a link in /dev called iscsi_01 for instance, where 0 is the LUN number, and 1 is the first partition, which might be a link to say /dev/sdb1.

iSCSI Target

Now we need to set up the target on the machine which won't run the virtual machine. This is penfold.

I used a block device for this on a SATA raid controller, but you could just use an ordinary file or a normal disk, or logical volume. The iSCSI Target documentation will give you more information on setting up something other than a block device.

penfold:~# apt-get install linux-headers-2.6.16-1-xen-686 openssl libssl-dev

Get the latest IET source code from your nearest sourceforge mirror, then uncompress and extract the files from the archive ready to build.

penfold:~# cd /usr/srcpenfold:~# wget http://sf.net...../iscsitarget-0.4.13.tar.gzpenfold:~# tar -zxf iscsitarget-0.4.13.tar.gzpenfold:~# cd iscsitarget-0.4.13penfold:~# makepenfold:~# make install

edit /etc/ietd.conf and make it look like the following.

IncomingUser joe secretOutgoingUser jack secret2    Target iqn.2006-05.com.performancemagic:penfold.disk1        IncomingUser jim othersecret        OutgoingUser james yetanothersecret        Lun 0 Path=/dev/sdb,Type=fileio        Alias Test        HeaderDigest None        DataDigest None        MaxConnections 1        InitialR2T Yes        ImmediateData No        MaxRecvDataSegmentLength 8192        MaxXmitDataSegmentLength 8192        MaxBurstLength 262144        FirstBurstLength 65536        DefaultTime2Wait 2        DefaultTime2Retain 20        MaxOutstandingR2T 8        DataPDUInOrder Yes        DataSequenceInOrder Yes        ErrorRecoveryLevel 0

Now start the iSCSI target.

penfold:~# /etc/init.d/iscsi-target startStarting iSCSI enterprise target service: succeeded.

(dmesg output abreviated)

penfold:~# dmesgiSCSI Enterprise Target Software - version 0.4.13iotype_init(90) register fileioiotype_init(90) register nulliosess_param(173) 1 1 1 8192 8192 262144 65536 2 20 1 1 1 0 1 1sess_param(173) 1 1 1 8192 8192 262144 65536 2 20 1 1 1 0 1 1

iSCSI Initiator

Now lets build and setup the iSCSI initiator.

dangermouse:~# cd /usr/srcdangermouse:~# svn co svn://svn.berlios.org/open-iscsi/

The revision I got was 581

dangermouse:~# cd open-iscsidangermouse:~# makedangermouse:~# make installdangermouse:~# echo "iqn.2006-05.com.performancemagic:dangermouse.01" > /etc/initiatorname.iscsi

Change dangermouse in the above example to buggles for the other XEN host.

Edit /etc/iscsid.conf and make it look like the following.

node.active_cnx = 1node.startup = manualnode.session.auth.username = jimnode.session.auth.password = othersecretnode.session.timeo.replacement_timeout = 120node.session.err_timeo.abort_timeout = 10node.session.err_timeo.reset_timeout = 30node.session.iscsi.InitialR2T = Nonode.session.iscsi.ImmediateData = Yesnode.session.iscsi.FirstBurstLength = 262144node.session.iscsi.MaxBurstLength = 16776192node.session.iscsi.DefaultTime2Wait = 0node.session.iscsi.DefaultTime2Retain = 0node.session.iscsi.MaxConnections = 1node.conn[0].iscsi.HeaderDigest = Nonenode.conn[0].iscsi.DataDigest = Nonenode.conn[0].iscsi.MaxRecvDataSegmentLength = 65536discovery.sendtargets.auth.username = joediscovery.sendtargets.auth.password = secret

Start the initiator.

dangermouse:~# /etc/init.d/open-iscsi startStarting iSCSI initiator service: succeeded.

(dmesg output abreviated)

dangermouse:~# dmesgSCSI subsystem initializediscsi: registered transport (tcp)

iSCSI Setup

Now we have the target setup and the initiator started, we need to find the targets on the initiator.

dangermouse:~# iscsiadm -m discovery -t sendtargets -p 127.0.0.1:3260[0f6cce] 127.0.0.1:3260,1 iqn.2006-05.com.performancemagic:penfold.disk1

now take the number in the [ ] brackets and login.

dangermouse:~# iscsiadm -m node -r 0f0cce -l

(dmesg output abreviated)

dangermouse:~# dmesgscsi0 : iSCSI Initiator over TCP/IP, v1.0-574sess_param(173) 1 0 1 8192 8192 262144 65536 2 0 1 1 1 0 1 1Vendor: IET Model: VIRTUAL-DISK Rev: 0Type: Direct-Access ANSI SCSI revision: 04SCSI device sda: 10000 512-byte hdwr sectors (5 MB)sda: Write Protect is offsda: Mode Sense: 77 00 00 08SCSI device sda: drive cache: write backSCSI device sda: 10000 512-byte hdwr sectors (5 MB)sda: Write Protect is offsda: Mode Sense: 77 00 00 08SCSI device sda: drive cache: write backsda: sda1sd 0:0:0:1: Attached scsi disk sda

XEN Virtual Machine

Create partition and File-system

Now that you can see the iSCSI device we can create partitions for the virtual machine to use. We just need to create the simplest of patitions, one for the root file-system of the virtual machine and one for swap. You can use fdisk or cfdisk for this, or any other partitioning programme. Your partitions in fdisk might look something like this:

dangermouse:~# fdisk /dev/iscsi_0Command (m for help): pDisk /dev/iscs_0: 60.0 GB, 60011642880 bytes255 heads, 63 sectors/track, 7296 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes   Device Boot      Start         End      Blocks   Id  System/dev/iscsi_01               1         249     2000061   82  Linux swap / Solaris/dev/iscsi_02             250        7296    56605027+  83  Linux

Now that we have created the partitions we need to create file-systems on them.

dangermouse:~# mkswap /dev/iscsi_01Setting up swapspace version 1, size = 5115 kBno label, UUID=055d2ab4-d2ab-4779-818c-71d4950c08dfdangermouse:~# mke2fs -j /dev/iscsi_02Filesystem label=OS type: LinuxBlock size=1024 (log=0)Fragment size=1024 (log=0)1256 inodes, 5000 blocks250 blocks (5.00%) reserved for the super userFirst data block=1Maximum filesystem blocks=52428801 block group8192 blocks per group, 8192 fragments per group1256 inodes per groupWriting inode tables: doneCreating journal (1024 blocks): doneWriting superblocks and filesystem accounting information: doneThis filesystem will be automatically checked every 28 mounts or180 days, whichever comes first.  Use tune2fs -c or -i to override.

Create Virtual Machine

I am going to create a Debian virtual machine using debootstrap.

dangermouse:~# mount /dev/iscsi_02 /mntdangermouse:~# debootstrap sarge /mntI: Retrieving ReleaseI: Retrieving Packages......

Once debootstrap is completed we need to setup the machine a little bit. We need to disable tls again, setup the /etc/fstab and the network interfaces and create tty and console devices. We also need to copy the kernel modules to the virtual machine.

dangermouse:~# mv /mnt/lib/tls /mnt/lib/tls.disableddangermouse:~# cat <<! > /mnt/etc/fstab# /etc/fstab: static file system information.## <file system> <mount point>   <type>  <options>       <dump>  <pass>proc            /proc           proc    defaults        0       0/dev/hda1       none            swap    sw              0       0/dev/hda2       /               ext3    defaults,errors=remount-ro 0       1!dangermouse:~# echo "virtaul01" > /mnt/etc/hostnamedangermouse:~# cat <<! >/mnt/etc/network/interfacesauto lo eth0iface lo inet loopbackiface eth0 inet static        address 192.168.1.10        netmask 255.255.255.0        broadcast 192.168.1.255        network 192.168.1.0        gateway 192.168.1.254!dangermouse:~# cd /mnt/devdangermouse:/mnt/dev# ./MAKEDEV console tty1 tty2 tty3 tty4 tty5 tty6dangermouse:~# cp -a /lib/modules/`uname -r` /mnt/lib/modulesdangermouse:~# umount /mnt

Make sure you unmount the disk ready for XEN to run the machine. If you don't you will destroy the filesystem.

Setup and Start Virtual Machine

Now that we have a basic Debian machine ready, we need to setup the XEN virtual machine config file. This file needs to be on both clients, buggles and dangermouse.

dangermouse:~# cat <<! >/etc/xen/virtual01kernel = "/boot/vmlinuz-2.6.16-1-xen-686"memory = 128name = "virtual01"vif = [ 'ip=192.168.1.10' ]disk = [ 'phy:/dev/iscsi_02,hda2,w','phy:/dev/iscsi_01,hda1,w' ]root = "/dev/hda2 ro"extra = "2"ramdisk = '/boot/initrd.img-2.6.16-1-xen-686'!

Now that we have a config file lets try starting the virtual machine on both of the clients one after the other, shuting it down each time.

dangermouse:~# xm create -c virtual01

This command should produce what looks to be a machine booting, and hopefully it will get all the way to the console where you can log in as root. Once logged in, shut the machine down in the normal way, eg shutdown -h now.

Celebrate! Well done we have a working XEN virtual machine, also known as a domain U. Now repeat the xm create -c virtual01 command on buggles (the other client) to ensure everything is setup and working on there. In each instance test you can ping the ethernet interface from a remote machine.

XEN Live Migration

Now for what is probably the easiest part of the whole process. Now that we know that you have a working iSCSI configuration and a working XEN virtual machine on both clients we are ready to test the live migration process.

With the virtual machine running on one of the clients, (dangermouse in this example) run this command as root on that same machine:

dangermouse:~# xm migrate --live virtual01 buggles

Hopefully after a few seconds the command should return without printing anything. If that is that case then that is probably a very good thing. Check on both clients using xm list to see that the virtual machine has indeed migrated to the second node. If it has, congratulations!

转自：www.performancemagic.com/iscsi-xen-howto/index.html