Consolidation Demo Using Solaris Containers

来源：互联网发布：lua调用java方法编辑：程序博客网时间：2024/05/21 09:17

Consolidation Demo Using Solaris Containers

Angel Camacho, September, 2004

Equipment (Minimum)

System based on UltraSPARC or x86 processor, with one CPU (two or more preferred)
256 Mbyte of RAM (1 GB preferred)
20-Gbyte hard disks (two or more hard disks preferred)
Solaris 10 Operating System build 63 (Solaris Express 8/04)
MySQL 3.23 (other RDMBS may be substituted)
Apache Web Server 1.3 (or Sun Java System Web Server)

Objective

The goal is to demonstrate the capabilities of the Solaris 10 Operating System, using the Solaris Zones feature and Solaris Containers technology in an everyday situation, to facilitate and encourage customer adoption.

Prerequisites

Knowledge of system administration using the Solaris OS or another UNIX-like OS is required to set up the environment (the steps will be provided). Also, an understanding of RDBMS and web server technologies is recommended.

Description

Two zones are configured to be used as containers for an RDBMS and a web server. Some parameters are modified for each zone, to control the CPU and other resources in use, according to each application.

The demo is carried out simulating the load from the web server that is accessing information contained in the database. These two applications were selected because they solve a real business problem, but their different natures make them unsuited to share resources inside a traditional server or partition.

Steps

Preparations: Before installing the OS, prepare three IP addresses and CDs with Solaris 10 build 63, MySQL, and the Apache web server.

Note: You can get the Solaris 10 OS from the web site of the Sun Software Express program, and the other software from SunFreeware. If you need to simulate the load on the applications to check the boundaries of the containers, use the "CPU spinner" as described in Step 8 of the following Cookbook section.

Install the Solaris OS.
Configure the system after installation.
Create the structure to hold the zones.
Create and install two new zones.
Update the resources for each zone including the global zone.
Install and configure the RDBMS.
Install and configure the web server.
Create workloads.
Monitor performance and manage resources.

Cookbook

Preparations: You will need at least three different IP addresses, not DHCP. For this demo you will not need network equipment. Everything will be contained in the same system, unless you need network access to get the software from an NFS server or a server running Solaris JumpStart software.

In this case, my machine is a Sun Blade 100 workstation, with 2 Gbyte RAM and two 20-Gbyte hard disks.

IP addresses must be from the same class and netmask, and within the same subnetwork:

IP for server: xx.xx.xx.10
IP for zone #1: xx.xx.xx.11
IP for zone #2: xx.xx.xx.12

1. Install the Solaris 10 OS. If you have one hard disk, make three partitions: the first for swap, the second for Solaris, and the third to be used by the zones. Allocate between 8 and 10 Gbyte for each zone as a separate partition, as partitions will hold the local files for each new zone, and some extra software. In this example, because we are not loading extra applications, the partition size might be as small as 3 Gbyte for each zone, or 9 Gbyte of disk space.

If you have two hard disks, use one to hold the Solaris OS and swap, and leave the other one for the zones. For this example, one disk will hold all the zone file systems. Just be aware that in doing so, if we lose one partition, we lose all the zones. A better approach would be to create a partition for each zone to be used. This partition should be large enough to hold the root file system, plus the applications to run. You can find more information in the documentation for Solaris Containers.

Server name: servidor
Primary IP address: xx.xx.xx.10
Naming service: None (files under /etc to be used)
Root file system: /dev/dsk/c0t0d0s0

2. Customize the server as needed for your setup, but be sure you can reach the IP addresses for the zones. For this example, files under /etc would be used.

/etc/hosts

127.0.0.1 localhost

xx.xx.xx.10 servidor

xx.xx.xx.11 myweb

xx.xx.xx.12 mydb

3. Assign the third partition to /zones as the mount point, update the vfstab file and create two subdirectories called 1 (for myweb) and 2 (for mydb). This will keep things under a simple directory structure, but any mountpoint or name will work; just avoid using long directory structures.

Add the following line (as needed) to /etc/vfstab:

/dev/dsk/c0t2d0s2 /dev/rdsk/c0t2d0s2 /zones ufs 2 yes -

Mount the /zones file system and create the two subdirectories, and for security reasons change access permissions to 700:

servidor# mkdir /zones

servidor# mount /zones

servidor# mkdir /zones/1 /zones/2

servidor# chmod 700 /zones/1 /zones/2

4. Create and install the zones called myweb and mydb, and assign them the root file system /zones/1 or /zones/2 accordingly, using the zonecfg, zoneadm, and zlogin commands.

servidor# zonecfg -z myweb

myweb: No such zone configured

Use 'create' to begin configuring a new zone.

zonecfg:myweb> create

zonecfg:myweb> set zonepath=/zones/1

zonecfg:myweb> add net

zonecfg:myweb:net> set address=xx.xx.xx.11

zonecfg:myweb:net> set physical=eri0

zonecfg:myweb:net> end

zonecfg:myweb> set autoboot=true

zonecfg:myweb> info

zonepath: /zones/1

autoboot: true

pool:

inherit-pkg-dir:

dir: /lib

inherit-pkg-dir:

dir: /platform

inherit-pkg-dir:

dir: /sbin

inherit-pkg-dir:

dir: /usr

net:

address: xx.xx.xx.11

physical: eri0

zonecfg:myweb> commit

zonecfg:myweb> exit

servidor# zoneadm -z myweb install

Preparing to install zone <myweb>.

Creating list of files to copy from the global zone.

Copying <2369> files to the zone.

Initializing zone product registry.

Determining zone package initialization order.

Preparing to initialize <782> packages on the zone.

Initialized <782> packages on zone.

Successfully initialized zone <myweb>.

servidor# zoneadm list -cv

ID NAME STATUS PATH

0 global running /

- myweb installed /zones/1

servidor#

Do the same for the zone called mydb, but use xx.xx.xx.12 instead, and verify.

servidor# zoneadm list -cv

ID NAME STATUS PATH

0 global running /

- myweb installed /zones/1

- mydb installed /zones/2

servidor#

To finalize the server configuration, boot the zone and log in using the console.

servidor# zoneadm -z myweb boot

servidor# zoneadm -z mydb boot

servidor# zoneadm list -cv

ID NAME STATUS PATH

0 global running /

1 myweb running /zones/1

2 mydb running /zones/2

servidor# zlogin -C myweb

(Note: similar messages to those presented here should appear at the terminal.)

[Connected to zone 'myweb' console] (hit enter once)

...

... answer configuration questions to fit your environment

...

>System identification is completed.

rebooting system due to change(s) in /etc/default/init

[NOTICE: zone rebooting]

SunOS Release 5.10 Version s10_54 64-bit

Use is subject to license terms.

Hostname: myweb

The system is coming up. Please wait.

starting rpc services: rpcbind done.

syslog service starting.

...

The system is ready.

myweb console login: root

Password: ********

Apr 21 10:25:08 myweb login: ROOT LOGIN /dev/console

Sun Microsystems Inc. SunOS 5.10 s10_54 May 2004

# hostname

myweb

# exit

myweb console login: ~. (~. to exit console session)

[Connection to zone 'myweb' console closed]

servidor#

Same for mydb.

servidor# zlogin -C mydb

...

servidor# zoneadm list -cv

ID NAME STATUS PATH

0 global running /

3 myweb running /zones/1

4 mydb running /zones/2

servidor#

Done!

The system now has three zones: one global zone, and two extra zones, myweb and mydb. The ID number of myweb and mydb changed because of the reboot of the zones.

Log in to the zones (using zlogin), to customize variables and defaults for each zone so that you can log in as root from any terminal, and update the prompt, and so on.

Note that each zone has its own resources; look at the end of the output.

servidor# prctl -i process $$

1659: -sh

...

. more output

...

zone.cpu-shares [ no-basic no-local-action ]

1 privileged none

65535 system none [ max ]

servidor#

This means that the process $$, which happens to be in the global zone (0), has one share (the minimum for a zone). Use zlogin to log in to the other zones (myweb or mydb), and run the same command, and they bring the same result. Up to this moment, we have three containers that share all the same resources at the same time, with no limits on each other, in particular on CPU time allocation.

5. Use prctl to allocate cpu-shares to each of the three containers (zones). In this case, for simplicity, use a number that adds up to 100: Assign 45 to mydb, 30 to myweb, and 25 to the global zone. Then in the same way, a new "subdirectory" structure will be added as /usr/local, where all the local installations will be made.

First create the directory structure that will hold the /usr/local directory and update the number of shares for myweb:

servidor# mkdir /zones/1/local

servidor# chmod 700 /zones/1/local

servidor# zonecfg -z myweb

zonecfg:myweb> info

zonepath: /zones/1

autoboot: true

pool:

inherit-pkg-dir:

dir: /lib

inherit-pkg-dir:

dir: /platform

inherit-pkg-dir:

dir: /sbin

inherit-pkg-dir:

dir: /usr

net:

address: xx.xx.xx.11

physical: eri0

zonecfg:myweb> add rctal

zonecfg:myweb:rctl> set name=zone.cpu-shares

zonecfg:myweb:rctl> set value=(priv=privileged,limit=30,action=none)

zonecfg:myweb:rctl> end

zonecfg:myweb> add fs

zonecfg:myweb:fs> set dir=/usr/local

zonecfg:myweb:fs> set special=/zones/1/local

zonecfg:myweb:fs> set type=lofs

zonecfg:myweb:fs> add options [rw,nodevices]

zonecfg:myweb:fs> end

zonecfg:myweb> info

zonepath: /zones/1

autoboot: true

pool:

inherit-pkg-dir:

dir: /lib

inherit-pkg-dir:

dir: /platform

inherit-pkg-dir:

dir: /sbin

inherit-pkg-dir:

dir: /usr

fs:

dir: /usr/local

special: /zones/1/local

type: lofs

options: [rw,nodevices]

net:

address: xx.xx.xx.11

physical: eri0

rctl:

name: zone.cpu-shares

value: (priv=privileged,limit=30,action=none)

zonecfg:myweb> verify

zonecfg:myweb> commit

zonecfg:myweb> exit

servidor# zoneadm -z myweb reboot

servidor# prctl -i zone (myweb new zone ID)

3064: /usr/lib....

...

zone.cpu-shares [ no-basic no-local-action ]

30 privileged deny

65535 system deny [ max ]

servidor#

Here is an alternative method to obtain information regarding the number of shares the zone can use:

servidor# zlogin myweb

[Connected to zone 'myweb' pts/7]

Last login: Tue Apr 27 16:40:31 on pts/7

Sun Microsystems Inc. SunOS 5.10 s10_54 May 2004

myweb# prctl -i process $$

3080: -sh

...

zone.cpu-shares [ no-basic no-local-action ]

30 privileged deny

65535 system deny [ max ]

myweb# df -k

Filesystem kbytes used avail capacity Mounted on

/ 19239673 227300 18819977 2% /

/dev 19239673 227300 18819977 2% /dev

/lib 12641267 4660252 7854603 38% /lib

/platform 12641267 4660252 7854603 38% /platform

/sbin 12641267 4660252 7854603 38% /sbin

/usr 12641267 4660252 7854603 38% /usr

/usr/local 19239673 227300 18819977 2% /usr/local

proc 0 0 0 0% /proc

mnttab 0 0 0 0% /etc/mnttab

fd 0 0 0 0% /dev/fd

swap 3655608 40 3655568 1% /var/run

swap 3655568 0 3655568 0% /tmp

myweb#

Note that 30 zone.cpu-shares are allocated for this zone, which only privileged users can change. In case the zone requires more shares, they will be denied. On the other output, verify /usr/local and play around creating some files and deleting them. Whatever is created inside the zone's /usr/local is not available to other zones nor the global zone, but if you check /zones/1/local, the files are there. Can you delete files from /zones/1/local while in the global zone? Yes, so be careful.

Do the same for the zone called mydb, but allocate 45 zone.cpu-shares instead, and provide /usr/local for it as well.

servidor# mkdir /zones/2/local

servidor# chmod 700 /zones/2/local

servidor# zonecfg -z mydb

zonecfg:mydb> info

zonepath: /zones/2

...

mydb# prctl -i process $$

3275: -sh

...

zone.cpu-shares [ no-basic no-local-action ]

45 privileged deny

65535 system deny [ max ]

mydb#

After these changes, the global zone still has only one share, myweb has 30 shares, and mydb has 45. Each one has its own /usr/local with total isolation for local installs. Remember that at this moment all the shares account for 76 shares.

Changing resources in the global zone involves a different process than the one used for the non-global zones, but it is simpler.

servidor# prctl -i process $$

3363: -sh

...

zone.cpu-shares [ no-basic no-local-action ]

1 privileged none

65535 system deny [ max ]

servidor# prctl -r -n zone.cpu-shares -v 25 -i process $$

servidor# prctl -i process $$

3367: -sh

...

zone.cpu-shares [ no-basic no-local-action ]

25 privileged none

65535 system deny [ max ]

servidor#

The only problem with this modification is that every time that the system boots, we lose the global zone configuration. We need to add this change into the boot scripts, in order to get the shares we need in the global zone, even after reboots.

servidor# cd /etc/init.d

servidor# vi zonesshares

#!/sbin/sh

# Global zone share update

# Angel Camacho

/usr/bin/prctl -r -n zone.cpu-shares -v 25 -i process $$

exit 0

servidor# ln /etc/rc3.d/S99zonesshares zonesshares

servidor# ls -i zonesshares

176404 zonesshares

servidor# ls -i /etc/rc3.d/S99zonesshares

176404 /etc/rc3.d/S99zonesshares

Now the system has three containers, well defined in terms of CPU usage, and a zone-owned /usr/local file system where you can make local installs. I encourage you to start using other parameters to manipulate resources like memory, buffers, and lwps, to adjust your containers to be as airtight as you wish.

Here is a list of values that might be interesting to use, but be aware that the wrong configuration may bring down the server performance.

process.max-sem-ops

process.max-address-space

process.max-file-descriptor

process.max-file-size

process.max-cpu-time

task.max-lwps

project.max-shm-memory

6. Install and configure MySQL: You can carry out this installation in two different ways. One is to log in as root into mydb and perform an exclusive installation, into the local /usr/local subdirectory, to achieve total isolation of the application. The other one is to perform a "global" installation, so we can access the application from other zones. The decision is yours.

While in the global zone, download MySQL from Sunfreeware.com. Get all the software MySQL depends on, which may not be present in the Solaris 10 build you are using. MySQL 3.23.53 depends on zlib 1.2, ncurses 5.3, and libgcc 3.3.

Other versions should work, but these are part of the tested configuration. The software might be in the global zone, under a directory called /sw, already uncompressed and ready to add as a package with pkgadd. The process will include mounting /sw into the local zone and extracting the software, then proceeding with the configuration.

servidor# vi /etc/dfs/dfstab

...

share -F nfs -o rw,root=myweb,root=mydb /sw

servidor# chmod 777 /sw

servidor# chmod 666 /sw/*

servidor# /etc/init.d/nfs.server start

servidor# dfshares

RESOURCE SERVER ACCESS TRANSPORT

servidor:/sw servidor - -

servidor# zlogin -C mydb

[Connected to zone 'mydb' console]

mydb console login: root

Password:

May 6 17:46:54 mydb login: ROOT LOGIN /dev/console

Last login: Thu May 6 16:13:19 on console

Sun Microsystems Inc. SunOS 5.10 s10_54 May 2004

mydb# mkdir /sw

mydb# mount -F nfs -o rw servidor:/sw /sw

mydb# ls /sw

libgcc-3.3-sol9-sparc-local

mysql-3.23.53-sol9-sparc-local

ncurses-5.3-sol9-sparc-local

zlib-1.2.1-sol9-sparc-local

mydb# pkgadd -d zlib-1.2.1-sol9-sparc-local

mydb# pkgadd -d ncurses-5.4-sol9-sparc-local

mydb# pkgadd -d libgcc-3.3-sol9-sparc-local

mydb# pkgadd -d mysql-3.23.53-sol9-sparc-local

mydb# cd /usr/local/mysql/bin

mydb# ./safe_mysqld &

mydb# ./mysqladmin -u root password 'new password'

7. Install and configure Apache HTTP Server: The same issues are present with Apache, but in this case, Apache web server is already present inside the Solaris software, so you only need to configure it.

servidor# zlogin -C myweb

servidor# cd /etc/apache

servidor# cp httpd.conf-example httpd.conf

servidor# vi httpd.conf

8. At the global zone, use prctl and prstat to show workloads, open a web browser, and connect to myweb and use the web server.

Note: If you want to stress the CPU/memory of your system, you can get spinners from BigAdmin. The applications shown so far wouldn't be enough to stress the system, as the load that can be generated is minimum even for a single CPU machine with limited resources.

To get the spinners, download the tar file into the global zone, in a working directory, /sw. Uncompress the file and copy and rename the file to its corresponding zone, into /usr/local/bin.

The tar file includes the spinners for both the UltraSPARC and x86 platforms. For both platforms, you will also need MySQL, zlib, ncurses, and libgcc.

The following example is for the x86 architecture. (The equivalent for a system based on the UltraSPARC CPU uses nspins.gz.)

servidor# cd /sw

servidor# gunzip nspinx.gz

servidor# chmod +x nspinx

servidor# cp nspinx /usr/local/bin/clientspinx

servidor# cp nspinx /zones/1/local/webspinx

servidor# cp nspinx /zones/2/local/dbspinx

9. Play around with the zones, change cpu-shares, and shut down zones, to demonstrate the different capabilities of the system, but first change the scheduler from Time Share (TS) to Fair Share Scheduler (FSS).

servidor# dispadmin -d FSS

servidor# reboot

After reboot you can check that the FSS is in use.

servidor# ps -cafe|more

UID PID PPID CLS PRI STIME TTY TIME CMD

root 0 0 SYS 96 09:20:54 ? 0:03 sched

root 1 0 FSS 59 09:20:55 ? 0:00 /etc/init -r

root 2 0 SYS 98 09:20:55 ? 0:00 pageout

root 3 0 SYS 60 09:20:55 ? 1:00 fsflush

root 42 1 FSS 59 09:21:40 ? 0:00 /usr/lib/saf/sac -t

root 25 1 FSS 59 09:21:29 ? 0:00 /usr/lib/utmpd

daemon 15 1 FSS 10 09:21:26 ? 0:00 /usr/sbin/rpcbind

root 60 1 FSS 29 09:21:13 ? 0:00 /usr/lib/syseventd

daemon 68 1 FSS 29 09:21:14 ? 0:00 /usr/lib/crypto/kcfd

root 71 1 FSS 29 09:21:14 ? 0:02 /usr/lib/picl/picld

root 129 12 FSS 1 11:17:33 pts/2 0:00 /

-More-

For the following demonstration, it's preferable to log in from a different system. The reason is that the Xsun server might interfere with the overall results as it is running directly from the global zone. However, if the demonstration does not require high accuracy on the performance metrics, you can do it all from the global zone. For simplicity the assumption is that the demo will be carried out logging in from a remote system.

Open four terminals, and telnet into the three zones: global, myweb, and mydb. On the fourth one, telnet into the global zone as well and monitor the processes with prstat.

mydb# dbspinx &

myweb# webspinx &

servidor# clientspinx &

servidor# prstat -J

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP

1720 root 1096K 672K run 2 0 0:00:23 43% dbspinx/1

1721 root 1096K 672K run 1 0 0:00:11 26% webspinx/1

1722 root 1096K 672K run 4 0 0:00:05 14% clientspinx/1

1536 root 71M 19M sleep 47 0 0:00:50 2.8% gnome-term/1

1156 root 45M 41M sleep 25 0 0:04:34 2.0% Xsun/1

1294 root 64M 13M sleep 28 0 0:02:59 0.5% gnome-panel/1

1290 root 58M 8352K sleep 38 0 0:00:36 0.5% metacity/1

1593 root 6184K 5112K cpu0 27 0 0:00:01 0.2% prstat/1

1698 root 4160K 2464K sleep 44 0 0:00:00 0.0% zlogin/1

1274 root 4928K 3072K sleep 59 0 0:00:01 0.0% xscreensaver/1

1690 root 1240K 968K sleep 59 0 0:00:00 0.0% sh/1

1689 root 4160K 2464K sleep 59 0 0:00:00 0.0% zlogin/1

1699 root 1240K 968K sleep 59 0 0:00:00 0.0% sh/1

527 daemon 2400K 984K sleep 5 0 0:00:00 0.0% rpcbind/1

213 root 2560K 752K sleep 25 0 0:00:00 0.0% cron/1

PROJID NPROC SIZE RSS MEMORY TIME CPU PROJECT

1 39 610M 199M 9.9% 0:10:03 89% user.root

0 86 258M 130M 6.4% 0:00:05 0.0% system

Total: 125 processes, 295 lwps, load averages: 2.28, 0.75, 0.34

And finally, use prctl, dispadmin, and zonecfg to change the variables and modify resource usage. Try to find out what would happen if we start a second dbspinx, or try using some other kind of application.