pmon 的实例健康检查处理逻辑分析

来源：互联网发布：log4j2 日志数据库编辑：程序博客网时间：2024/06/05 17:56

PMON进程的功能

用于恢复失败的数据库用户的强制性进程，它先获取失败用户的标识，释放该用户占有的所有数据库资源。PMON有规律地被呼醒，检查是否需要，或者其它进程发现需要时可以被调用。

PMON进程负责在反常中断的连接之后的清理工作。例如，如果因某些原因专用服务“故障”或被kill掉，PMON就是负责处理（恢复或回滚工作）和释放你的资源。

PMON将发出未提交工作的回滚，释放锁，和释放分配给故障进程的SGA资源。除了在异常中断之后的清理外，PMON监控其他oracle后台进程，如果有必要（和有可能）重新启动他们。如果共享服务或一个分配器故障（崩溃），PMON将插手并且重启另一个（在清理故障进程之后）。

PMON将观察所有Oracle进程，只要合适或重启他们或中止进程。例如，在数据库日志写进程事件中，LGWR故障，实例故障。这是一个严重的错误，最安全的处理方法就是去立即终止实例，让正常的恢复处理数据。

PMON为实例做的另一件事是去使用Oracle TNS监听器登记。当一个实例开启的时候，PMON进程投出众所周知的端口地址，除非指向其他，来看是否监听器正在开和运行着。众所周知/默认端口是使用1521。

现在，如果监听器在一些不同端口开启会发生什么？这种情况，机制是相同的，除了监听器地址需要被LOCAL_LISTENER参数明确指定。如果监听器运行在库实例开启的时候，PMON和监听器通讯，传到它相关参数，譬如服务器名和实例的负载度量。如果监听器没被开启，PMON将周期性的试着和它联系来登记自己。

The background process PMON cleans up afterfailed processes by:

1. Rolling back the user’s current transaction

2. Releasing all currently held table or row locks 其实还有latch 资源

3. Freeing other resources currently reserved by the user

4. Restarts dead dispatchers

FROM：

http://blog.csdn.net/tianlesoftware/article/details/5587788

pmon 是如何来实现PMON将观察所有Oracle进程，只要合适或重启他们或中止进程。例如，在数据库日志写进程事件中，LGWR故障，实例故障。这是一个严重的错误，最安全的处理方法就是去立即终止实例，让正常的恢复处理数据。这个功能的呢？

在开始前我们需要先了解一些知识

关于信号量，共享内存：

Linux进程间通信——使用信号量

http://blog.csdn.net/ljianhui/article/details/10243617

Linux进程间通信——使用共享内存

http://blog.csdn.net/ljianhui/article/details/10253345

0)“signal 0″

0)“signal 0″ is kind oflike a moral equivalent of “ping”. 进程状态检查

Using “kill -0 NNN” in a shell script is agood way to tell if PID “NNN” is alive or not:

signal 0 is just used to check process isexists or not.

检测进程是否存在的原理：

$ echo $$ # show our process id

12833

$ /bin/bash # create new process

$ echo $$ # show new process id

12902

$ kill -0 12902

$ echo $? # exists, exit code is 0

$ exit # return to previous shell

$ kill -0 12902

bash: kill: (12902) - No such process

$ echo $? # doesn't exist, exit code is 1

18) SIGCONT

让一个停止(stopped)的进程继续执行. 本信号不能被阻塞. 可以用一个handler来让程序在由stopped状态变为继续执行时完成特定的工作.例如,重新显示提示符

/proc目录

Linux 内核提供了一种通过 /proc 文件系统，在运行时访问内核内部数据结构、改变内核设置的机制。proc文件系统是一个伪文件系统，它只存在内存当中，而不占用外存空间。它以文件系统的方式为访问系统内核数据的操作提供接口。

用户和应用程序可以通过proc得到系统的信息，并可以改变内核的某些参数。由于系统的信息，如进程，是动态改变的，所以用户或应用程序读取proc文件时，proc文件系统是动态从系统内核读出所需信息并提交的。下面列出的这些文件或子文件夹，并不是都是在你的系统中存在，这取决于你的内核配置和装载的模块。另外，在/proc下还有三个很重要的目录：net，scsi和sys。 Sys目录是可写的，可以通过它来访问或修改内核的参数，而net和scsi则依赖于内核配置。例如，如果系统不支持scsi，则scsi 目录不存在。

除了以上介绍的这些，还有的是一些以数字命名的目录，它们是进程目录。系统中当前运行的每一个进程都有对应的一个目录在/proc下，以进程的 PID号为目录名，它们是读取进程信息的接口。而self目录则是读取进程本身的信息接口，是一个link。

From：

http://blog.csdn.net/zdwzzu2006/article/details/7747977

read() 内核函数

相关函数：readdir, write, fcntl, close, lseek, readlink, fread

头文件：#include <unistd.h>

定义函数：ssize_t read(int fd, void * buf, size_t count);

函数说明：read()会把参数fd 所指的文件传送count 个字节到buf 指针所指的内存中. 若参数count 为0, 则read()不会有作用并返回0.返回值为实际读取到的字节数, 如果返回0, 表示已到达文件尾或是无可读取的数据,此外文件读写位置会随读取到的字节移动.

附加说明：

如果顺利 read()会返回实际读到的字节数, 最好能将返回值与参数count 作比较, 若返回的字节数比要求读取的字节数少, 则有可能读到了文件尾、从管道(pipe)或终端机读? ?蛘呤莚ead()被信号中断了读取动作.

当有错误发生时则返回-1, 错误代码存入errno 中, 而文件读写位置则无法预期.

错误代码：

EINTR 此调用被信号所中断.

EAGAIN 当使用不可阻断I/O 时(O_NONBLOCK),若无数据可读取则返回此值.

EBADF 参数fd 非有效的文件描述词, 或该文件已关闭.

time内核函数

Stat内核函数

[root@LIXORA 7100]# stat cmdline

File: `cmdline'

Size: 0 Blocks:0 IO Block: 4096 regular empty file

Device: 3h/3d Inode: 465305612 Links: 1

Access: (0444/-r--r--r--) Uid: ( 500/ oracle) Gid: ( 501/oinstall)

Access: 2015-10-12 10:16:58.800293919 +0800

Modify: 2015-10-12 10:16:58.800293919 +0800

Change: 2015-10-12 10:16:58.800293919 +0800

相关函数：fstat, lstat, chmod, chown, readlink, utime

头文件：#include <sys/stat.h> #include <unistd.h>

定义函数：int stat(const char * file_name, struct stat *buf);

函数说明：stat()用来将参数file_name 所指的文件状态, 复制到参数buf 所指的结构中。

下面是struct stat 内各参数的说明：

struct stat

{

dev_t st_dev; //device 文件的设备编号

ino_t st_ino; //inode 文件的i-node

mode_t st_mode; //protection 文件的类型和存取的权限

nlink_t st_nlink; //number of hard links 连到该文件的硬连接数目, 刚建立的文件值为1.

uid_t st_uid; //user ID of owner 文件所有者的用户识别码

gid_t st_gid; //group ID of owner 文件所有者的组识别码

dev_t st_rdev; //device type 若此文件为装置设备文件, 则为其设备编号

off_t st_size; //total size, in bytes 文件大小, 以字节计算

unsigned long st_blksize; //blocksize for filesystem I/O 文件系统的I/O 缓冲区大小.

unsigned long st_blocks; //number of blocks allocated 占用文件区块的个数, 每一区块大小为512 个字节.

time_t st_atime; //time of lastaccess 文件最近一次被存取或被执行的时间, 一般只有在用mknod、utime、read、write 与tructate 时改变.

time_t st_mtime; //time of last modification 文件最后一次被修改的时间,一般只有在用mknod、utime 和write 时才会改变

time_t st_ctime; //time of last change i-node 最近一次被更改的时间, 此参数会在文件所有者、组、权限被更改时更新

};

先前所描述的st_mode 则定义了下列数种情况：

1、S_IFMT 0170000 文件类型的位遮罩

2、S_IFSOCK 0140000scoket

3、S_IFLNK 0120000 符号连接

4、S_IFREG 0100000 一般文件

5、S_IFBLK 0060000 区块装置

6、S_IFDIR 0040000 目录

7、S_IFCHR 0020000 字符装置

8、S_IFIFO 0010000 先进先出

9、S_ISUID 04000 文件的 (setuser-id on execution)位

10、S_ISGID 02000 文件的 (setgroup-id on execution)位

11、S_ISVTX 01000 文件的sticky 位

12、S_IRUSR (S_IREAD)00400 文件所有者具可读取权限

13、S_IWUSR(S_IWRITE)00200 文件所有者具可写入权限

14、S_IXUSR (S_IEXEC)00100 文件所有者具可执行权限

15、S_IRGRP 00040 用户组具可读取权限

16、S_IWGRP 00020 用户组具可写入权限

17、S_IXGRP 00010 用户组具可执行权限

18、S_IROTH 00004 其他用户具可读取权限

19、S_IWOTH 00002 其他用户具可写入权限

20、S_IXOTH 00001 其他用户具可执行权限上述的文件类型在 POSIX 中定义了检查这些类型的宏定义

21、S_ISLNK (st_mode) 判断是否为符号连接

22、S_ISREG (st_mode) 是否为一般文件

23、S_ISDIR (st_mode) 是否为目录

24、S_ISCHR (st_mode) 是否为字符装置文件

25、S_ISBLK (s3e) 是否为先进先出

26、S_ISSOCK (st_mode) 是否为socket 若一目录具有sticky 位 (S_ISVTX), 则表示在此目录下的文件只能被该文件所有者、此目录所有者或root 来删除或改名.

返回值：执行成功则返回0，失败返回-1，错误代码存于errno。

For most files under the /proc directory,stat() does not return the file size in the st_size field; instead the field isreturned with the value 0.

对于/proc 目录下的文件stat（）函数不会返回文件大小。只返回“0“

错误代码：

1、ENOENT 参数file_name 指定的文件不存在

2、ENOTDIR 路径中的目录存在但却非真正的目录

3、ELOOP 欲打开的文件有过多符号连接问题, 上限为16 符号连接

4、EFAULT 参数buf 为无效指针, 指向无法存在的内存空间

5、EACCESS 存取文件时被拒绝

6、ENOMEM 核心内存不足

7、ENAMETOOLONG 参数file_name 的路径名称太长

范例

#include <sys/stat.h>

#include <unistd.h>

main()

{

struct stat buf;

stat("/etc/passwd", &buf);

printf("/etc/passwd file size = %d \n", buf.st_size);

}

执行：

/etc/passwd file size = 705

lstat() 内核函数

相关函数：stat, fstat, chmod, chown, readlink, utime

头文件：#include <sys/stat.h> #include <unistd.h>

定义函数：int lstat (const char * file_name, struct stat * buf);

函数说明：lstat()与stat()作用完全相同, 都是取得参数file_name 所指的文件状态, 其差别在于,当文件为符号连接时, lstat()会返回该link本身的状态.详细内容请参考stat().

返回值：执行成功则返回0, 失败返回-1, 错误代码存于errno.

范例：参考stat(

++++++++++++++++++++++++++++++++++

NAME

semctl - semaphore control operations内核函数

SYNOPSIS

#include <sys/types.h>

#include <sys/ipc.h>

#include <sys/sem.h>

int semctl(int semid, int semnum, int cmd, ...);

DESCRIPTION

semctl() performs the control operation specified by cmdon the semaphore set identified by semid, or on the semnum-th semaphore of thatset. (The semaphores in a set arenumbered starting at 0.)

This function has three or four arguments, depending on cmd. When there are four, the fourth has the typeunion semun. The calling program mustdefine this union as follows:

union semun {

int val; /* Value for SETVAL */

struct semid_ds *buf; /* Buffer for IPC_STAT, IPC_SET */

unsigned short *array; /* Array for GETALL, SETALL */

struct seminfo *__buf; /* Buffer for IPC_INFO

(Linux specific) */

};

The semid_ds data structure is defined in <sys/sem.h> as follows:

struct semid_ds {

struct ipc_perm sem_perm; /* Ownership and permissions

time_t sem_otime; /* Last semop time */

time_t sem_ctime; /* Last change time */

unsigned short sem_nsems; /* No. of semaphores in set */

};

The ipc_perm structure is defined in <sys/ipc.h> as follows (thehighlighted fields are settable using IPC_SET):

struct ipc_perm {

key_t key; /* Key supplied to semget() */

uid_t uid; /* Effective UID of owner */

gid_t gid; /* Effective GID of owner */

uid_t cuid; /* Effective UID of creator */

gid_t cgid; /* Effective GID of creator */

unsigned short mode; /* Permissions */

unsigned short seq; /* Sequence number */

};

Valid values for cmd are:

IPC_STAT Copy information from the kernel data structureassociated with semid into the semid_ds structure pointed to by arg.buf. The argument semnum is ignored. The calling process must have read permissionon the semaphore set.

IPC_SET Write the values ofsome members of the semid_ds structure pointed to by arg.buf to the kerneldata structure associated with this semaphore set, updating also its sem_ctime member. The followingmembers of the structure are updated:

sem_perm.uid,sem_perm.gid, and (the leastsignificant 9 bits of) sem_perm.mode. The effective UID of the calling process must match the owner (sem_perm.uid) or creator (sem_perm.cuid) of the semaphore set, or thecaller must be privileged. The argument semnumis ignored.

IPC_RMID Immediately remove the semaphore set, awakening all processes blocked insemop() calls on the set (with an error return and errno set to EIDRM). The effective user ID of the calling processmust match the creator or owner of the semaphore set, or the caller must beprivileged. The argument semnum isignored.

IPC_INFO (Linux specific)

Returns information about system-widesemaphore limits and parameters in the structure pointed to by arg.__buf.

This structure is of type seminfo, defined in <sys/sem.h> if the_GNU_SOURCE feature test macro is defined:

struct seminfo {

int semmap; /* # of entries in semaphore map;

unused*/

int semmni; /* Max. # of semaphore sets */

int semmns; /* Max. # of semaphores in all

semaphore sets */

int semmnu; /* System-wide max. # of undo

structures; unused */

int semmsl; /* Max. # of semaphores in a set */

int semopm; /* Max. # of operations for semop() */

int semume; /* Max. # of undo entries per

process; unused */

int semusz; /* size of struct sem_undo */

int semvmx; /* Maximum semaphore value */

int semaem; /* Max. value that can be recorded for

semaphore adjustment (SEM_UNDO) */

};

The semmsl, semmns, semopm,and semmni settings can be changed via /proc/sys/kernel/sem; see proc(5) for details.

SEM_INFO (Linux specific)

Returns a seminfo structure containing the sameinformation as for IPC_INFO, except that the following fields are returnedwithinformation about system resources consumed by semaphores: the semusz fieldreturns the number of semaphore sets that currently exist on the system; and the semaem field returns the totalnumber of semaphores in all semaphore sets on the system.

SEM_STAT (Linux specific)

Returns a semid_ds structure as for IPC_STAT. However, the semid argument is not a semaphore identifier, but instead anindexinto the kernel鈥檚 internal array that maintains information about all semaphore setson the system.

GETALL Return semval (i.e.,the current value) for all semaphores of the set into arg.array. The argument semnum is ignored. Thecalling process musthave read permission on the semaphore set.

GETNCNT The system callreturns the value of semncnt (i.e., the number of processes waiting for thevalue of this semaphore to increase) for the semnum-th semaphore of the set(i.e. the number of processes waiting for an increase of semval for the semnum-thsemaphore of theset). The calling process must have readpermission on the semaphore set.

GETPID The system call returns the value of sempidfor the semnum-th semaphore of the set (i.e. the PID of the process thatexecuted the last semop() call for the semnum-th semaphore of the set). The calling process must have read permissionon the semaphore set.

GETVAL The system call returns the value of semval forthe semnum-th semaphore of the set. Thecalling process must have read permis-sion on the semaphore set.

GETZCNT The system callreturns the value of semzcnt (i.e., the number of processes waiting for thevalue of this semaphore to become zeo) for the semnum-th semaphore of the set (i.e. the number of processeswaiting for semval of the semnum-th semaphore of the set to become 0). The calling process must have read permissionon the semaphore set.

SETALL Set semval for allsemaphores of the set using arg.array, updating also the sem_ctime member ofthe semid_ds structure associ-ated with the set. Undo entries (see semop(2)) are cleared for alteredsemaphores in all processes. Ifthechanges tosemaphore values would permit blocked semop() calls in otherprocesses to proceed, then those processes are woken up. The argu- ment semnumis ignored. The calling process musthave alter (write) permission on the semaphore set.

SETVAL Set the value of semval to arg.val forthe semnum-th semaphore of the set, updating also the sem_ctime member of thesemid_dsstructure associated with the set. Undo entries are cleared for altered semaphores in all processes. If the changes tosemaphore values would permit blocked semop() calls in other processesto proceed, then those processes are woken up. The call-ing process must havealter permission on the semaphore set.

RETURN VALUE

On failure semctl() returns -1 with errno indicating the error.

Otherwise the system call returns a nonnegative value depending on cmdas follows:

GETNCNT the value of semncnt.

GETPID the value of sempid.

GETVAL the value of semval.

GETZCNT the value of semzcnt.

IPC_INFO the index of the highest used entry in the kernel鈥檚 internalarray recording information about all semaphore sets. (This infor-mation can be used with repeated SEM_STAT operations to obtaininformation about all semaphore sets on the system.)

SEM_INFO As for IPC_INFO.

SEM_STAT the identifier of thesemaphore set whose index was given in semid.

All other cmd values return 0 on success.

ERRORS

On failure, errno will be set to one of the following:

EACCES The argument cmd has one of the values GETALL,GETPID, GETVAL, GETNCNT, GETZCNT, IPC_STAT, SEM_STAT, SETALL, or SETVAL and thecalling process does not have the required permissions on the semaphore set anddoes not have the CAP_IPC_OWNER capability.

EFAULT The address pointed toby arg.buf or arg.array isn鈥檛 accessible.

EIDRM The semaphore set wasremoved.

EINVAL Invalid value for cmdor semid. Or: for a SEM_STAT operation,the index value specified in semid referred to anarray slot that is currently unused.

EPERM The argument cmd has the value IPC_SET orIPC_RMID but the effective user ID of the calling process is not the creator (asfound in sem_perm.cuid) or the owner (as found in sem_perm.uid) of thesemaphore set, and the process does not have the CAP_SYS_ADMIN capability.

ERANGE The argument cmd has the value SETALL or SETVAL and the value towhich semval is to be set (for some semaphore of the set) is less than 0 or greater than the implementationlimit SEMVMX.

NOTES

The IPC_INFO, SEM_STAT and SEM_INFO operations are used by the ipcs(8)program to provide information on allocated resources. In the future these may modified or moved toa /proc file system interface. Various fields in a struct semid_ds were shorts under Linux 2.2 and havebecome longs under Linux 2.4. To take advantage of this, a recompi-lation underglibc-2.1.91 or later should suffice. (The kernel distinguishes old and new calls by an IPC_64 flag in cmd.)

In some earlier versions of glibc, the semun union was defined in<sys/sem.h>, but POSIX.1-2001 requires that the caller define this union. On versions of glibc where this union is not defined, the macro_SEM_SEMUN_UNDEFINED is defined in <sys/sem.h>.

The following system limit on semaphore sets affects a semctl() call:

SEMVMX Maximum value forsemval: implementation dependent (32767).

For greater portability it is best to always call semctl() with fourarguments.

Under Linux, semctl() is not a system call, but is implemented via thesystem call ipc(2).

CONFORMING TO

SVr4, POSIX.1-2001.

shmctl - shared memory control 内核函数

SYNOPSIS

#include <sys/ipc.h> #include <sys/shm.h>

int shmctl(int shmid, int cmd, struct shmid_ds *buf);

DESCRIPTION

shmctl() performs the control operation specified by cmd on the sharedmemory segment whose identifier is given in shmid.

The buf argument is a pointer to a shmid_ds structure, defined in<sys/shm.h> as follows:

struct shmid_ds {

struct ipc_perm shm_perm; /* Ownership and permissions */

size_t shm_segsz; /* Size of segment (bytes) */

time_t shm_atime; /* Last attach time */

time_t shm_dtime; /* Last detach time */

time_t shm_ctime; /* Last change time */

pid_t shm_cpid; /* PID of creator */

pid_t shm_lpid; /* PID of last shmat()/shmdt() */

shmatt_t shm_nattch; /* No. of current attaches */

...

};

The ipc_perm structure is defined in <sys/ipc.h> as follows (thehighlighted fields are settable using IPC_SET):

struct ipc_perm {

key_t key; /* Key supplied to shmget() */

uid_t uid; /* Effective UID of owner */

gid_t gid; /* Effective GID of owner */

uid_t cuid; /* Effective UID of creator */

gid_t cgid; /* Effective GID of creator */

unsigned short mode; /* Permissions + SHM_DEST and

SHM_LOCKED flags */

unsigned short seq; /* Sequence number */

};

Valid values for cmd are:

IPC_STAT Copy information fromthe kernel data structure associated with shmid into the shmid_ds structurepointed to by buf. The caller must haveread permission on the shared memory segment.

IPC_SET Write the values ofsome members of the shmid_ds structure pointed to by arg.buf to the kerneldata structure associated with this shared memory segment, updating also its shm_ctime member. The following fields can be changed: shm_perm.uid, m_perm.gid, and (the least significant 9bits of) shm_perm.mode. The effectiveUID of the calling process must match the owner (shm_perm.uid) or creator (shm_perm.cuid) of the shared memorysegment, or the caller must be privileged.

IPC_RMID Mark the segment to be destroyed. The segment will only actually be destroyedafter the last process detaches it (i.e., when the shm_nattch member of the associatedstructure shmid_ds is zero). The callermust be the owner or creator, or be privileged. If a segment has been markedfor destruction, then the (non-standard) SHM_DEST flag of the shm_perm.modefield in the associated data structure retrieved by IPC_STAT will be set. The caller must ensure that a segment iseventually destroyed; otherwise its pages that were faulted in will remain inmemory or swap.

IPC_INFO (Linux specific)

Returns information aboutsystem-wide shared memory limits and parameters in the structure pointed to bybuf. This structure is oftype shminfo (thus, a cast is required), defined in <sys/shm.h> if the_GNU_SOURCE feature test macro is defined:

struct shminfo {

unsigned long shmmax; /*Max. segment size */

unsigned long shmmin; /*Min. segment size; always 1 */

unsigned long shmmni; /*Max. # of segments */

unsigned long shmseg; /*Max. # of segments that a

process can attach; unused */

unsigned long shmall; /*Max. # of pages of shared

memory, system-wide */

};

The shmmni, shmmax, and shmallsettings can be changed via /proc files of the same name; see proc(5) fordetails.

SHM_INFO (Linux specific)

Returns a shm_info structure whose fieldscontain information about system resources consumed by shared memory.

This structure is

defined in <sys/shm.h> ifthe _GNU_SOURCE feature test macro is defined:

struct shm_info {

int used_ids; /* # of currently existing

segments */

unsigned long shm_tot; /* Total number of shared

memory pages */

unsigned long shm_rss; /* # of resident shared

memory pages */

unsigned long shm_swp; /* # of swapped shared

memory pages */

unsigned longswap_attempts; /* Unused since Linux 2.4*/

unsigned longswap_successes; /* Unused since Linux 2.4 */

};

SHM_STAT (Linux specific)

Returns a shmid_ds structure asfor IPC_STAT. However, the shmidargument is not a segment identifier, but instead an index into the

kernel鈥檚 internalarray that maintains information about all shared memory segments on thesystem.

The caller can prevent or allow swapping of a shared memory segment withthe following cmd values:

SHM_LOCK (Linux specific)

Prevent swapping of the shared memory segment. Thecaller must fault in any pages that are required to be present afterlocking is enabled. If a segment has been locked, then the(non-standard) SHM_LOCKED flag of the shm_perm.mode field in the associated data structure retrieved by IPC_STAT will be set.

SHM_UNLOCK (Linux specific)

Unlock the segment, allowingit to be swapped out.

In kernels before 2.6.10, only aprivileged process could employ SHM_LOCK and SHM_UNLOCK. Since kernel 2.6.10, an unprivileged processcan employ these operations if itseffective UID matches the owner or creator UID of the segment, and (forSHM_LOCK) the amount of memory to be locked falls within theRLIMIT_MEMLOCK resource limit (see setrlimit(2)).

RETURN VALUE

A successful IPC_INFO or SHM_INFO operation returns the index of thehighest used entry in the kernel鈥檚 internal array recording information about all shared memory segments. (This information can be used with repeatedSHM_STAT operations to obtain information about all shared memory segments on the system.) A successful SHM_STAT operation returns theidentifier of the shared memory segment whose index was given in shmid. Other operations return 0 on success.

On error, -1 is returned, and errno is set appropriately.

ERRORS

EACCES IPC_STAT or SHM_STAT isrequested and shm_perm.mode does not allow read access for shmid, and thecalling process does not have the CAP_IPC_OWNER capability.

EFAULT The argument cmd hasvalue IPC_SET or IPC_STAT but the address pointed to by buf isn鈥檛accessible.

EIDRM shmid points to aremoved identifier.

EINVAL shmid is not a valididentifier, or cmd is not a valid command. Or: for a SHM_STAT operation, the index value

specified in shmid referred to an array slot that is currentlyunused.

ENOMEM (In kernels since2.6.9), SHM_LOCK was specified and the size of the to-be-locked segment wouldmean that the total bytes in locked shared memory segments would exceed the limit forthe real user ID of the calling process. This limit is defined by the RLIMIT_MEMLOCK soft resource limit (see setrlimit(2)).

EOVERFLOW IPC_STAT is attempted,and the GID or UID value is too large to be stored in the structure pointed to bybuf.

EPERM IPC_SET or IPC_RMID isattempted, and the effective user ID of the calling process is not that of thecreator (found in shm_perm.cuid), or the owner (found in shm_perm.uid), and theprocess was not privileged (Linux: did not have the CAP_SYS_ADMIN capability). Or (in kernels before 2.6.9),SHM_LOCK or SHM_UNLOCK was specified, but the process was not privileged(Linux: did not have the CAP_IPC_LOCK capability). (Since Linux 2.6.9, this error can also occurif the RLIMIT_MEMLOCK is 0 and thecaller is not privi-leged.)

NOTES

The IPC_INFO, SHM_STAT and SHM_INFO operations are used by the ipcs(8)program to provide information on allocated

resources. In the future

these may modified or moved to a /proc file system interface.

Linux permits a process to attach(shmat()) a shared memory segment that has already been marked for deletionusing shmctl

(IPC_RMID). This feature is not available onother Unix implementations; portable applications should avoid relying onit. Various fields in a structshmid_ds were shorts under Linux 2.2 and have become longs under Linux 2.4. Totake advantage of this, a recompi-lation under glibc-2.1.91 or later shouldsuffice. (The kernel distinguishes oldand new calls by an IPC_64 flag in cmd.)

CONFORMING TO

SVr4, POSIX.1-2001.

Strace 跟踪跟踪分析

+++++++++++++++++++++++++++++++++++++++

from alert log：

PMON started with pid=2, OS id=7355

PSP0 started with pid=3, OS id=7357

MMAN started with pid=4, OS id=7359

DBW0 started with pid=5, OS id=7361

LGWR started with pid=6, OS id=7363

CKPT started with pid=7, OS id=7365

SMON started with pid=8, OS id=7367

RECO started with pid=9, OS id=7369

CJQ0 started with pid=10, OS id=7371

MMON started with pid=11, OS id=7373

MMNL started with pid=12, OS id=7375

QMNC started with pid=16, OS id=7383

sessio1:

[root@lixora ~]# ps -ef|grep smon

54322 7367 1 0 11:39 ? 00:00:00 ora_smon_ora10g

root 7666 7576 0 12:07 pts/5 00:00:00 grep smon

[root@lixora ~]# kill -0 7367

[root@lixora ~]# kill -1 7367

[root@lixora ~]# kill -9 7367

Case 1 Oracleversion 10201

sessio2: strace –p {pmon pid}

kill(7357, SIG_0) = 0 ----使用 kill -0检查进程状态，这里返回0表明进程存在

open("/proc/7357/stat",O_RDONLY) = 19 ---这里oracle有再次检查了一次/proc/7357/stat 是否有状态信息

read(19, "7357 (oracle) S 1 7357 73570 -1"..., 999) = 244 ------read 函数读取/proc/PID/stat文本信息，并返回实际读取到的字节数。

close(19) = 0

kill(7359, SIG_0) = 0

open("/proc/7359/stat",O_RDONLY) = 19

read(19, "7359 (oracle) S 1 7359 73590 -1"..., 999) = 241

close(19) = 0

kill(7361, SIG_0) = 0

。。。。。

close(19) = 0

kill(7365, SIG_0) = 0

open("/proc/7365/stat",O_RDONLY) = 19

read(19, "7365 (oracle) S 1 7365 73650 -1"..., 999) = 243

close(19) = 0

---------这是在窗口一中手工kill -9 7367进程

kill(7367,SIG_0) = -1 ESRCH(No such process)----pmon 检测不到进程7367

times(NULL) = 437138651

stat("/u02/app/oracle/admin/ora10g/bdump",{st_mode=S_IFDIR|0750, st_size=16384, ...}) = 0------确定bdump目录状态

close(2) = 0

lstat("/u02/app/oracle/admin/ora10g/bdump/ora10g_pmon_7355.trc", 0x7fff227635b0) = -1 ENOENT (No such file or directory ) -------查看是否存在pmon进程对应的日志文件，检测发现没有

stat("/u02/app/oracle/admin/ora10g/bdump/ora10g_pmon_7355.trc", 0x7fff227635b0) = -1 ENOENT (No such file or directory) -------查看是否存在pmon进程对应的软链接日志文件，检测发现没有

open("/u02/app/oracle/admin/ora10g/bdump/ora10g_pmon_7355.trc",O_WRONLY|O_CREAT|O_TRUNC, 0660) = 2

write(2, "", 0) = 0 ----------打开pmon日志文件

write(2,"/u02/app/oracle/admin/ora10g/bdu"..., 55) = 55 ---------以下write函数开始在pmon日志中写日志

write(2, "\n", 1) = 1

write(2, "Oracle Database 10gEnterprise E"..., 128) = 128

write(2, "\n", 1) = 1

write(2, "ORACLE_HOME =/u02/app/oracle/pr"..., 50) = 50 ----这些内容看着都熟悉吧

write(2, "System name:\tLinux\n",19) = 19

write(2, "Node name:\tlixora\n",18) = 18

write(2,"Release:\t2.6.32-300.10.1.el5uek\n", 32) = 32

write(2, "Version:\t#1 SMP Wed Feb 2217:37"..., 45) = 45

write(2, "Machine:\tx86_64\n",16) = 16

write(2, "Instance name:ora10g\n", 22) = 22

write(2, "Redo thread mounted by thisinst"..., 40) = 40

write(2, "Oracle process number:2\n", 25) = 25

write(2, "Unix process pid: 7355,image: o"..., 51) = 51

write(2, "\n", 1) = 1

times(NULL) = 437138651

write(2, "*** 2015-03-0512:09:06.075", 27) = 27

write(2, "\n", 1) = 1

write(2, "*** SERVICENAME:(SYS$BACKGROUND"..., 57) = 57

write(2, "\n", 1) = 1

times(NULL) = 437138651

write(2, "*** SESSION ID:(170.1)2015-03-0"..., 46) = 46

write(2, "\n", 1) = 1

times(NULL) = 437138651

write(2, "Background process SMONfound de"..., 34) = 34

write(2, "\n", 1) = 1

times(NULL) = 437138651

write(2, "Oracle pid = 8",14) = 14

write(2, "\n", 1) = 1

times(NULL) = 437138651

write(2, "OS pid (from detachedprocess) ="..., 38) = 38

write(2, "\n", 1) = 1

times(NULL) = 437138651

write(2, "OS pid (from process state)= 73"..., 34) = 34

write(2, "\n", 1) = 1

。。。。。。。。。。。。。。。。。。。。。。。

write(2, "Dump of memory from0x000000006C"..., 60) = 60

write(2, "error 474 detected inbackground"..., 40) = 40

write(2, "\n", 1) = 1

lseek(18, 1024, SEEK_SET) = 1024

read(18,"\30\0$\0005\0>\0I\0V\0`\0k\0u\0\203\0\241\0\312\0\323\0\332\0\341\0\351\0"...,512) = 512

lseek(18, 22528, SEEK_SET) = 22528

read(18,"\v\0\301\1\0\0J\0\302\1\0\0\207\0\303\1\0\0\254\0\304\1\0\0\320\0\305\1\0\0\10\1"...,512) = 512

times(NULL) = 437138659

write(2, "ORA-00474: SMON processterminat"..., 45) = 45 -------

write(2, "\n", 1) = 1

close(8) = 0 ---------以下write函数开始在pmon日志中写日志结束

open("/u02/app/oracle/admin/ora10g/bdump/alert_ora10g.log", O_WRONLY|O_CREAT|O_APPEND, 0664) = 8

---------在pmon日志写完后，接着打开alert日志，将刚才新生成的pmon日志的位置写到alert日志中

writev(8, [{"Thu Mar 5 12:09:06 2015\n", 25}, {"Errorsin file /u02/app/oracle/a"..., 118}], 2) = 143

times(NULL) = 437138659

close(8) = 0

open("/u02/app/oracle/admin/ora10g/bdump/alert_ora10g.log",O_WRONLY|O_CREAT|O_APPEND, 0664) = 8

writev(8, [{"Thu Mar 512:09:06 2015\n", 25}, {"PMON: terminating instance due t"...,43}, {"\n", 1}], 3) = 69

---------alert 日志中再写下是我pmon干掉了实例，but这里只是先发个声明稿而已，实际还没开始干活呢！！

告诉大家是由于xxx原因被pmon terminating ----（其实这个” terminating “ 是在英语中是进行时，表明正在进行中，这里也从一个侧面反映学好英语是多么重要啊---多么痛的领悟！！！！）

times(NULL) = 437138659

open("/proc/7393/stat",O_RDONLY) = 19 ------照例先检查下检查状态

read(19, "7393 (oracle) S 1 7393 73930 -1"..., 999) = 241

close(19) = 0

kill(7393, SIGKILL) = 0 ----------如果进程存在则干掉： kill -9 PID

kill(7393, SIGCONT) = -1 ESRCH (No suchprocess)-----------再次检查下进程状态

open("/proc/7670/stat",O_RDONLY) = 19

。。。。。。。

open("/proc/7369/stat",O_RDONLY) = 19

read(19, "7369 (oracle) S 1 7369 73690 -1"..., 999) = 241

close(19) = 0

kill(7369, SIGKILL) = 0

kill(7369, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/7367/stat", O_RDONLY) = -1 ENOENT (No such file or directory)

-------7367 是 smon进程，这是我们手工 kill -9 7367杀掉的，所以在 /proc 中相应的信息早就被os清理掉了，所以这里检查进程状态时就报错了，表明该进程已经释放了

open("/proc/7365/stat",O_RDONLY) = 19

read(19, "7365 (oracle) S 1 7365 73650 -1"..., 999) = 243

close(19) = 0

kill(7365, SIGKILL) = 0

kill(7365, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/7363/stat",O_RDONLY) = 19

read(19, "7363 (oracle) S 1 7363 73630 -1"..., 999) = 242

close(19) = 0

kill(7363, SIGKILL) = 0

kill(7363, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/7361/stat",O_RDONLY) = 19

read(19, "7361 (oracle) S 1 7361 73610 -1"..., 999) = 242

close(19) = 0

kill(7361, SIGKILL) = 0

kill(7361, SIGCONT) = -1 ESRCH (No such process)

open("/proc/7359/stat",O_RDONLY) = 19

read(19, "7359 (oracle) S 1 7359 73590 -1"..., 999) = 241

close(19) = 0

kill(7359, SIGKILL) = 0

kill(7359, SIGCONT) = -1 ESRCH (No such process)

open("/proc/7357/stat",O_RDONLY) = 19

read(19, "7357 (oracle) S 1 7357 73570 -1"..., 999) = 244

close(19) = 0

kill(7357, SIGKILL) = 0

kill(7357, SIGCONT) = -1 ESRCH (No such process)

------到这里pmon进程完成了第一次后台进程清理工作

使用 kill -0pid ====ping 再次检查进程是否存在

kill(7393, SIG_0) = -1 ESRCH (No suchprocess)

kill(7670, SIG_0) = -1 ESRCH (No suchprocess)

kill(7391, SIG_0) = -1 ESRCH (No suchprocess)

kill(7383, SIG_0) = -1 ESRCH (No suchprocess)

kill(7381, SIG_0) = -1 ESRCH (No suchprocess)

kill(7379, SIG_0) = -1 ESRCH (No suchprocess)

kill(7377, SIG_0) = -1 ESRCH (No suchprocess)

kill(7375, SIG_0) = -1 ESRCH (No suchprocess)

kill(7373, SIG_0) = -1 ESRCH (No suchprocess)

kill(7371, SIG_0) = -1 ESRCH (No such process)

kill(7369, SIG_0) = -1 ESRCH (No suchprocess)

kill(7367, SIG_0) = -1 ESRCH (No suchprocess)

kill(7365, SIG_0) = -1 ESRCH (No suchprocess)

kill(7363, SIG_0) = -1 ESRCH (No suchprocess)

kill(7361, SIG_0) = -1 ESRCH (No suchprocess)

kill(7359, SIG_0) = -1 ESRCH (No suchprocess)

kill(7357, SIG_0) = -1 ESRCH (No suchprocess)

times(NULL) = 437138663

close(8) = 0

------------当确定所有后台进程（除pmon外）已经全部被清理后，在alert日志中写下实例被pmon ”清理好了“，但是共享内存段和信号量呢？？？

open("/u02/app/oracle/admin/ora10g/bdump/alert_ora10g.log",O_WRONLY|O_CREAT|O_APPEND, 0664) = 8

writev(8, [{"Instance terminated by PMON, pid"..., 39},{"\n", 1}], 2) = 40

semctl(655361, 0, IPC_RMID, 0) = 0 -----------------信号量清理

close(18) = 0

close(13) = 0

shmctl(2424837, IPC_RMID, 0) = 0 -------------------共享内存段清理

shmdt(0x60000000) = 0

open("/proc/7355/stat",O_RDONLY) = 13 --------到这里已经完成99%的后台和信号量，共享内存段的清理工作，但是pmon还活着呢！！

read(13, "7355 (oracle) R 1 7355 73550 -1"..., 999) = 223

close(13) = 0

kill(7355, SIGKILL <unfinished ...> ------------最后的最后 pmon终于 kill -9 自杀了

+++ killed by SIGKILL +++ --------到这里pmon进程的全部工作完成，instance被完全终止

先写 pmon 日志，叙述原因，然后进行内存转储，并且在 alert 中写下新生成的pmon 日志的位置，名称等信息。

然后再检测一下后台进程状态（是否还存活？）；检查2次

确认全被干掉后，再在alert 日志中post 出实例被pmon 干掉了，

然后清理共享内存段，及信号量

最后的最后圆满完成程序的自我异常安全退出机制后。pmon 就自杀了。

oracle 太 tmd的牛逼了！！！

根据上面的分析信息画了一张逻辑处理流程图

这里让我想到额外一个想法：

如果在oracle 实例出现异常在没法正常完成出程序退出逻辑时，可能来不及去写alert 日志，会先去写相应的进程日志，而该进程日志写的信息不一定会写到alert 日志中；也就是说以后再诊断oracle 异常问题时，如果在alert 日志中找不到信息时非常有必要把相关的后台进程日志都去看下，或许还是会记录一些问题信息。

open("/u02/app/oracle/admin/ora10g/bdump/alert_ora10g.log",O_WRONLY|O_CREAT|O_APPEND, 0664) = 8

writev(8, [{"Instance terminated byPMON, pid"..., 39}, {"\n", 1}], 2) = 40

semctl(786433, 0, IPC_RMID, 0) = 0

close(19) = 0

close(14) = 0

shmctl(2457605, IPC_RMID, 0) = 0

shmdt(0x60000000) = 0

open("/proc/7788/stat",O_RDONLY) = 14

read(14, "7788 (oracle) R 1 7788 77880 -1"..., 999) = 222

close(14) = 0

kill(7788, SIGKILL <unfinished ...>

+++ killed by SIGKILL +++

------ Shared Memory Segments --------

key shmid owner perms bytes nattch status

0x00000000 2129921 root 644 80 2

0x00000000 2162690 root 644 16384 2

0x00000000 2195459 root 644 280 2

0xa1728964 2457605 oracle10g 640 211812352 20

------ Semaphore Arrays --------

key semid owner perms nsems

0x17ee0b04 786433 oracle10g 640 154

------ Message Queues --------

key msqid owner perms used-bytes messages

after:

[oracle10g@lixora ~]$ ipcs -a

------ Shared Memory Segments --------

key shmid owner perms bytes nattch status

0x00000000 2129921 root 644 80 2

0x00000000 2162690 root 644 16384 2

0x00000000 2195459 root 644 280 2

------ Semaphore Arrays --------

key semid owner perms nsems

------ Message Queues --------

key msqid owner perms used-bytes messages

Case 2 10203 版本测试：

略有改变：主要是 oracle 不在使用了 kill -0来进行进程状态检查，其他处理机制一样

close(18) = 0

open("/proc/9863/stat",O_RDONLY) = 18

read(18, "9863 (oracle) S 1 9863 98630 -1"..., 999) = 246

close(18) = 0

open("/proc/9865/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)---------日常进程检测机制变化

times(NULL) = 437903209

stat("/oracle/admin/lixora/bdump",{st_mode=S_IFDIR|0750, st_size=32768, ...}) = 0

close(2) = 0

lstat("/oracle/admin/lixora/bdump/lixora_pmon_9853.trc",0x7fff00ea3f90) = -1 ENOENT (No such file or directory)

stat("/oracle/admin/lixora/bdump/lixora_pmon_9853.trc",0x7fff00ea3f90) = -1 ENOENT (No such file or directory)

open("/oracle/admin/lixora/bdump/lixora_pmon_9853.trc",O_WRONLY|O_CREAT|O_TRUNC, 0660) = 2

write(2, "Dump file ", 10) = 10

write(2,"/oracle/admin/lixora/bdump/lixor"..., 47) = 47

write(2, "\n", 1) = 1

write(2, "Oracle Database 10gEnterprise E"..., 128) = 128

write(2, "\n", 1) = 1

write(2, "ORACLE_HOME =/oracle\n", 22) = 22

write(2, "System name:\tLinux\n",19) = 19

write(2, "Node name:\tlixora\n",18) = 18

write(2,"Release:\t2.6.32-300.10.1.el5uek\n", 32) = 32

write(2, "Version:\t#1 SMP Wed Feb 2217:37"..., 45) = 45

write(2, "Machine:\tx86_64\n",16) = 16

write(2, "Instance name:lixora\n", 22) = 22

write(2, "Redo thread mounted by thisinst"..., 40) = 40

write(2, "Oracle process number:2\n", 25) = 25

write(2, "Unix process pid: 9853,image: o"..., 51) = 51

write(2, "\n", 1) = 1

times(NULL) = 437903215

write(2, "*** 2015-10-1306:30:36.052", 27) = 27

write(2, "\n", 1) = 1

write(2, "*** SERVICENAME:(SYS$BACKGROUND"..., 57) = 57

write(2, "\n", 1) = 1

times(NULL) = 437903215

write(2, "*** SESSION ID:(170.1)2015-10-1"..., 46) = 46

write(2, "\n", 1) = 1

times(NULL) = 437903215

write(2, "Background process SMONfound de"..., 34) = 34

write(2, "\n", 1) = 1

times(NULL) = 437903215

write(2, "Oracle pid = 8",14) = 14

write(2, "\n", 1) = 1

times(NULL) = 437903215

write(2, "OS pid (from detachedprocess) ="..., 38) = 38

write(2, "\n", 1) = 1

times(NULL) = 437903215

write(2, "OS pid (from process state)= 98"..., 34) = 34

write(2, "\n", 1) = 1

times(NULL) = 437903215

write(2, "dtp = 0x6001eac8, proc =0x8565a"..., 35) = 35

write(2, "\n", 1) = 1

times(NULL) = 437903215

write(2, "Dump of memory from0x0000000060"..., 60) = 60

write(2, "\n", 1) = 1

times(NULL) = 437903216

write(2, "06001EAC0 0000"..., 65) = 65

write(2, "\n", 1) = 1

。。。。。。。。。。。。。。。。。。。。。。。。

times(NULL) = 437903218

write(2, "error 474 detected inbackground"..., 40) = 40

write(2, "\n", 1) =1

lseek(17, 1024,SEEK_SET) = 1024

read(17,"\30\0$\0004\0=\0H\0T\0^\0j\0t\0\202\0\240\0\310\0\321\0\331\0\340\0\347\0"...,512) = 512

lseek(17, 23040, SEEK_SET) = 23040

read(17,"\f\0\326\1\0\0P\0\327\1\0\0r\0\330\1\0\0\224\0\331\1\0\0\267\0\332\1\0\0\331\0"...,512) = 512

times(NULL) = 437903219

write(2, "ORA-00474: SMON processterminat"..., 45) = 45

write(2, "\n", 1) = 1

close(8) = 0

open("/oracle/admin/lixora/bdump/alert_lixora.log",O_WRONLY|O_CREAT|O_APPEND, 0660) = 8

writev(8, [{"Tue Oct 13 06:30:362015\n", 25}, {"Errors in file /oracle/admin/lix"..., 110}], 2)= 135

times(NULL) = 437903219

close(8) = 0

open("/oracle/admin/lixora/bdump/alert_lixora.log",O_WRONLY|O_CREAT|O_APPEND, 0660) = 8

writev(8, [{"Tue Oct 13 06:30:362015\n", 25}, {"PMON: terminating instance due t"..., 43},{"\n", 1}], 3) = 69

times(NULL) = 437903219

open("/proc/9916/stat",O_RDONLY) = 18

read(18, "9916 (oracle) S 1 9916 99160 -1"..., 999) = 245

close(18) = 0

kill(9916, SIGKILL) = 0

kill(9916, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9909/stat",O_RDONLY) = 18

read(18, "9909 (oracle) S 1 9909 99090 -1"..., 999) = 244

close(18) = 0

kill(9909, SIGKILL) = 0

kill(9909, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9907/stat",O_RDONLY) = 18

read(18, "9907 (oracle) S 1 9907 99070 -1"..., 999) = 244

close(18) = 0

kill(9907, SIGKILL) = 0

kill(9907, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9904/stat",O_RDONLY) = 18

read(18, "9904 (oracle) S 1 9904 99040 -1"..., 999) = 244

close(18) = 0

kill(9904, SIGKILL) = 0

kill(9904, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9900/stat",O_RDONLY) = 18

read(18, "9900 (oracle) S 1 9900 99000 -1"..., 999) = 242

close(18) = 0

kill(9900, SIGKILL) = 0

kill(9900, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9897/stat",O_RDONLY) = 18

read(18, "9897 (oracle) S 1 9897 98970 -1"..., 999) = 243

close(18) = 0

kill(9897, SIGKILL) = 0

kill(9897, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9895/stat",O_RDONLY) = 18

read(18, "9895 (oracle) S 1 9895 98950 -1"..., 999) = 246

close(18) = 0

kill(9895, SIGKILL) = 0

kill(9895, SIGCONT) = 0

open("/proc/9893/stat",O_RDONLY) = 18

read(18, "9893 (oracle) S 1 9893 98930 -1"..., 999) = 243

close(18) = 0

kill(9893, SIGKILL) = 0

kill(9893, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9891/stat",O_RDONLY) = 18

read(18, "9891 (oracle) S 1 9891 98910 -1"..., 999) = 243

close(18) = 0

kill(9891, SIGKILL) = 0

kill(9891, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9889/stat",O_RDONLY) = 18

read(18, "9889 (oracle) S 1 9889 98890 -1"..., 999) = 244

close(18) = 0

kill(9889, SIGKILL) = 0

kill(9889, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9887/stat",O_RDONLY) = 18

read(18, "9887 (oracle) S 1 9887 98870 -1"..., 999) = 242

close(18) = 0

kill(9887, SIGKILL) = 0

kill(9887, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9885/stat",O_RDONLY) = 18

read(18, "9885 (oracle) S 1 9885 98850 -1"..., 999) = 242

close(18) = 0

kill(9885, SIGKILL) = 0

kill(9885, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9883/stat",O_RDONLY) = 18

read(18, "9883 (oracle) S 1 9883 98830 -1"..., 999) = 242

close(18) = 0

kill(9883, SIGKILL) = 0

kill(9883, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9881/stat",O_RDONLY) = 18

read(18, "9881 (oracle) S 1 9881 98810 -1"..., 999) = 242

close(18) = 0

kill(9881, SIGKILL) = 0

kill(9881, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9879/stat",O_RDONLY) = 18

read(18, "9879 (oracle) S 1 9879 98790 -1"..., 999) = 244

close(18) = 0

kill(9879, SIGKILL) = 0

kill(9879, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9877/stat",O_RDONLY) = 18

read(18, "9877 (oracle) S 9850 98779877 0"..., 999) = 257

close(18) = 0

kill(9877, SIGKILL) = 0

kill(9877, SIGCONT) = 0

open("/proc/9875/stat",O_RDONLY) = 18

read(18, "9875 (oracle) S 1 9875 98750 -1"..., 999) = 241

close(18) = 0

kill(9875, SIGKILL) = 0

kill(9875, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9873/stat",O_RDONLY) = 18

read(18, "9873 (oracle) S 1 9873 98730 -1"..., 999) = 243

close(18) = 0

kill(9873, SIGKILL) = 0

kill(9873, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9871/stat",O_RDONLY) = 18

read(18, "9871 (oracle) S 1 9871 98710 -1"..., 999) = 243

close(18) = 0

kill(9871, SIGKILL) = 0

kill(9871, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9869/stat",O_RDONLY) = 18

read(18, "9869 (oracle) S 1 9869 98690 -1"..., 999) = 251

close(18) = 0

kill(9869, SIGKILL) = 0

kill(9869, SIGCONT) = 0

open("/proc/9867/stat",O_RDONLY) = 18

read(18, "9867 (oracle) S 1 9867 98670 -1"..., 999) = 245

close(18) = 0

kill(9867, SIGKILL) = 0

kill(9867, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9865/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9863/stat", O_RDONLY) = 18

read(18, "9863 (oracle) S 1 9863 98630 -1"..., 999) = 246

close(18) = 0

kill(9863, SIGKILL) = 0

kill(9863, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9861/stat", O_RDONLY) = 18

read(18, "9861 (oracle) S 1 9861 98610 -1"..., 999) = 245

close(18) = 0

kill(9861, SIGKILL) = 0

kill(9861, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9859/stat",O_RDONLY) = 18

read(18, "9859 (oracle) S 1 9859 98590 -1"..., 999) = 244

close(18) = 0

kill(9859, SIGKILL) = 0

kill(9859, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9857/stat",O_RDONLY) = 18

read(18, "9857 (oracle) S 1 9857 98570 -1"..., 999) = 244

close(18) = 0

kill(9857, SIGKILL) = 0

kill(9857, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9855/stat",O_RDONLY) = 18

read(18, "9855 (oracle) S 1 9855 98550 -1"..., 999) = 245

close(18) = 0

kill(9855, SIGKILL) = 0

kill(9855, SIGCONT) = -1 ESRCH (No suchprocess)

open("/proc/9916/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9909/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9907/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9904/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9900/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9897/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9895/stat",O_RDONLY) = -1 ENOENT (No such file or directory)

open("/proc/9893/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9891/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9889/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9887/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9885/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9883/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9881/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9879/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9877/stat",O_RDONLY) = 18

read(18, "9877 (oracle) R 9850 98779877 0"..., 999) = 192

close(18) = 0

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) =0

rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [],0}, 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) =0

nanosleep({1, 0}, {1, 0}) = 0

open("/proc/9877/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9875/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9873/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9871/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9869/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9867/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9865/stat",O_RDONLY) = -1 ENOENT (No such file or directory)

open("/proc/9863/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9861/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9859/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9857/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

open("/proc/9855/stat",O_RDONLY) = -1 ENOENT (No such fileor directory)

times(NULL) = 437903326

close(8) = 0

open("/oracle/admin/lixora/bdump/alert_lixora.log",O_WRONLY|O_CREAT|O_APPEND, 0660) = 8

writev(8, [{"Instance terminated byPMON, pid"..., 39}, {"\n", 1}], 2) = 40

semctl(491523, 0, IPC_RMID, 0) = 0

close(17) = 0

close(13) = 0

shmctl(4358148, IPC_RMID, 0) = 0

shmdt(0x60000000) = 0

open("/proc/9853/stat",O_RDONLY) = 13

read(13, "9853 (oracle) R 1 9853 98530 -1"..., 999) = 225

close(13) = 0

kill(9853, SIGKILL <unfinished ...>

+++ killed by SIGKILL +++

0 0