work with solaris contract subsystem

A process contract is the formal definition of the relationship thatexists between a Process A and its monitoring rocess. In case process Aterminates abnormally, the monitoring process will be able to restartit.
This is a small exercise that is using the contract subsystem and its associated commmands.

0. Get some information

# man contract

1. SMF and contracts

Check the state of the “syslog” service :

# svcs system-log
STATE STIME FMRI
online Dec_10 svc:/system/system-log:default
#
# pgrep -fl syslog
658 /usr/sbin/syslogd

# ps -o pid,comm,ctid | grep syslog
658 /usr/sbin/syslogd 35

# svcs -pv system-log
STATE NSTATE STIME CTID FMRI
online - Dec_10 35 svc:/system/system-log:default
Dec_10 658 syslogd

Allthese commands show that the service is enabled ( syslogd is startedwith pid 658 ) and is monitored by the contract subsystem usingContractID 35.

Another command shows more information about the contract :

#ctstat -i 35
CTID ZONEID TYPE STATE HOLDER EVENTS QTIME NTIME
35 0 process owned 7 0 - -

Or in verbose mode :

#ctstat -vi 35
CTID ZONEID TYPE STATE HOLDER EVENTS QTIME NTIME
35 0 process owned 7 0 - -
cookie: 0×20
informative event set: none
critical event set: core signal hwerr empty
fatal event set: none
parameter set: inherit regent
member processes: 658
inherited contracts: none

The HOLDER field indicates that the process holding the contract is having pid 7 :

# ps -fp 7
UID PID PPID C STIME TTY TIME CMD
root 7 1 0 Dec 10 ? 0:41 /lib/svc/bin/svc.startd

which is the main SMF daemon. Let’s find out more about the contract held by this process. The /proc file system
can help there :

# cd /proc/7/contracts/
# ls
17 18 19 22 27 30 32 34 35 36 37 38 39 40 42 43 46 50 83

where we find out why ’svc.startd’, also knows as svc://system/svc/restarter:default is called the master
restarter in the SMF framework : it is holding contracts for many many services.

Comingback to the verbose output of the ctstat command, we see that someevents received by the holder of contract 35 are considered critical.Among them, “signal” is an event that indicates the reception of afatal signal from another process. The restarter has the job ofrestarting syslogd if one ‘critical’ event is received.

Let’s check that…

2. The master restarter

Before helping syslogd to die, let’s open another terminal and type

#ctwatch -rv 35
CTID EVID CRIT ACK CTTYPE SUMMARY

which can be used to see all the events related to contract 35.
We can now use svcadm :

#svcadm refresh system-log

Nothing visible, a quick look at the Pid of syslogd tells us that it only reread its config file. We did the
famous “pkill -HUP syslogd” in the SMF way. Let’s try again :

#svcadm restart system-log

#svcs -pv system-log
STATE NSTATE STIME CTID FMRI
online - 10:24:37 89 svc:/system/system-log:default
10:24:37 2173 syslogd

while the other “ctwatch” terminal shows :

#ctwatch -rv 35
CTID EVID CRIT ACK CTTYPE SUMMARY
35 33 crit no process contract empty

Whathappend ? The syslogd process was terminated by svcadm. The contractbeing linked to the process is then also terminated and is ‘empty’.While starting another syslogd, another contract, number 89, wascreated.

Now in the “ctwatch” terminal, let’s type

#ctwatch -rv 89
CTID EVID CRIT ACK CTTYPE SUMMARY

while in the other window, we kill the syslog daemon (only in the global zone) :

#pkill -9 -z 0 syslogd

We observe in the “ctwatch terminal” :

#ctwatch -rv 89
CTID EVID CRIT ACK CTTYPE SUMMARY
89 34 crit no process process 2173 received a fatal signal
signal: 9 (SIGKILL)
sender pid: 2187
sender ctid: 86
89 35 crit no process contract empty

Whichshows that the contract subsystem was notified that the syslogd processreceived a signal. We even may know who sent it. In our case, it is the“kill” command, which terminated already. The result of the signal isthat
contract is now empty, ended.

But :

# svcs -pv system-log
STATE NSTATE STIME CTID FMRI
online - 10:27:04 90 svc:/system/system-log:default
10:27:04 2193 syslogd

shows that the master restarter has done its job. The system-log service is still online because “svc.startd” has
instructed “init” to fork and exec a new version of syslogd to keep the service running.

3. The contract file system

Everyoneknows the /proc filesystem, used to provide information to systemadministrators about the running processes in a nice well-knownfile-based manner. Commands like “ps”, “pfile”, “pgrep”,… get theirinformation by opening and reading files in /proc which are actually aninterface to the process structures maintained by the kernel.

Thesame is true for ctfs, the contract file system. All the contractcommands get their input from the kernel through another pseudofilesystem mounted on /system/contract.

Example :

# truss -t open ctstat -i 35
open(”/var/ld/ld.config”, O_RDONLY) Err#2 ENOENT
open(”/lib/libcontract.so.1″, O_RDONLY) = 3
open(”/lib/libuutil.so.1″, O_RDONLY) = 3
open(”/lib/libc.so.1″, O_RDONLY) = 3
open(”/lib/libnvpair.so.1″, O_RDONLY) = 3
open(”/lib/libnsl.so.1″, O_RDONLY) = 3
open(”/platform/SUNW,Sun-Blade-100/lib/libc_psr.so.1″, O_RDONLY) = 3
open(”/usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.3″, O_RDONLY) = 3
CTID ZONEID TYPE STATE HOLDER EVENTS QTIME NTIME
open64(”/system/contract/all/35/status”, O_RDONLY) = 3
35 0 process owned 7 0 - -

4. Monitor and restart ANY application

Thecommand ‘ctrun’ can be used to create a contract for any application.It will then monitor the application process for all the events thatyou specify and restart the application if some fatal event occurs.Check it out!

#ctrun -r 0 -o noorphan -f signal /usr/openwin/bin/xclock &

Myxclock will be restarted any number of times ( -r 0 ), ctrun will makesure that all processes get killed before restarting ( -o noorphan )and we monitor the ’signal’ type of fatal event.
You get a contractid and so on.

Then just kill the clock and see it nicely reappear…

 

 

1   What is the contract file system?

From contract(4): " The /system/contract file system acts as the primary interface to the contract subsystem."

From learning solaris: "A process contract is the formal definition of the relationship that exists between a Process A and its monitoring process."

In my own words : "The contract file system sets up a framework in the kernel that enables real time monitoring of processes and enables us to act on changes in process states immediately."

2   How to use the contract file system>

The contract file system is primarily used by the SMF(5) framework when services are started. The service restarter uses contracts to keep track of the processes in a contract.

It is also possible for applications and users to create contracts on the fly. The following commands are used by users to work with and manage contracts:

• ctstat
• ctrun
• ctwatch
• /system/contract

3   Examine your system

The contract framework is implemented as a pseudo file system.At system boot the contract file system is mounted at/system/boot.

www2.petervg.nl # df -n /system/contract/system/contract : ctfs www2.petervg.nl # grep ctfs /etc/vfstabctfs - /system/contract ctfs - no -www2.petervg.nl # ls /system/contract//all22260 22362 6063 6064 6065 6066 6073 6078 6079 6083 6090 6091 6093 6094 6095 6096 6097 6098

4   Create our own contract

With the ctrun command we can run a command and place it in a new contract.

4.1   New process without a new contract

www2.petervg.nl # ptree -c $$14536 zsched [process contract 22362] 3168 /usr/lib/ssh/sshd 3171 /usr/lib/ssh/sshd 3173 -bash 3265 sleep 555 3279 ptree -c 3173www2.petervg.nl # sleep 555 &[2] 3280www2.petervg.nl # ptree -c $$14536 zsched [process contract 22362] 3168 /usr/lib/ssh/sshd 3171 /usr/lib/ssh/sshd 3173 -bash 3280 sleep 555 3281 ptree -c 3173

As we can see, the sleep 555 commandis running in the same contract (22362) as the parent bash shell.

4.2   New command with a new contract

www2.petervg.nl # ptree -c $$14536 zsched [process contract 22362] 3168 /usr/lib/ssh/sshd 3171 /usr/lib/ssh/sshd 3173 -bash 3300 ptree -c 3173www2.petervg.nl # ctrun sleep 666 &[1] 3301www2.petervg.nl # ptree -c $$14536 zsched [process contract 22362] 3168 /usr/lib/ssh/sshd 3171 /usr/lib/ssh/sshd 3173 -bash 3301 ctrun sleep 666 [process contract 22366] 3302 sleep 666 3303 ptree -c 3173

And in this example we see that the sleep 666command is actually run under a new contract (22366).

But does this contract automagically restart the sleep process whenit is killed? The answer is no. By a contract will not restart theprocesses in a contract.

5   Checking the status of a contract

When a contract is started we can use the ctstat command to get informationabout the contract. The ctwatch command can then be usedto monitor the use of the contract over time.

www2.petervg.nl # ps -o ctid,pid,ppid,args | grep sleep22362 3577 3173 ctrun sleep 55522385 3578 3577 sleep 55522362 3599 3591 grep sleep

With the ps -o ctid command it ispossible to identify the contract a particular command is runningin.

www2.petervg.nl # ptree -c $$14536 zsched [process contract 22362] 3168 /usr/lib/ssh/sshd 3171 /usr/lib/ssh/sshd 3173 -bash 3591 ksh 4323 ctrun sleep 666 [process contract 22449] 4324 sleep 666 4336 ptree -c 3591www2.petervg.nl # ctstat -i 22449CTID ZONEID TYPE STATE HOLDER EVENTS QTIME NTIME 22449 12 process owned 4323 0 - -

In the previous output we see that the contract under investigationis contract 22449 which is running in zone 12. The current contractis a process contract which is held by process 4323. This meansthat process 4323 created the contract and started the process inthis contract.

With the -v option we can obtain more verbose information abouta specific contract.

www2.petervg.nl # ctstat -v -i 22452CTID ZONEID TYPE STATE HOLDER EVENTS QTIME NTIME 22452 12 process owned 4408 0 - - cookie: 0 informative event set: core critical event set: hwerr empty fatal event set: hwerr parameter set: none member processes: 4409 inherited contracts: none