因为用户权限不足引起的进程（QMNC）挂死

早晨检查Alert日志，发现下列错误：

Fri Jun 19 08:24:43 2009
Process q001 died, see its trace file
Fri Jun 19 08:24:43 2009
ksvcreate: Process(q001) creation failed

在日志没有提示日志文件的名称。分析q001是属于QMNC进程的任务，所以找qmnc的日志，在bdump中果然存在data_qmnc_3815000.trc。

 /oracle/admin/data/bdump/data_qmnc_3815000.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
ORACLE_HOME = /oracle/product/10.2.0
System name: AIX
Node name: ibm5703
Release: 3
Version: 5
Machine: 00C4956D4C00
Instance name: data
Redo thread mounted by this instance: 1
Oracle process number: 36
Unix process pid: 3815000, image: oracle@ibm5703 (QMNC)

*** 2009-06-19 08:22:20.577
*** SERVICE NAME:(SYS$BACKGROUND) 2009-06-19 08:22:11.904
*** SESSION ID:(523.5) 2009-06-19 08:22:11.904
Waited for process q001 to initialize for 60 seconds
*** 2009-06-19 08:22:20.577
Dumping diagnostic information for q001:
OS pid = 3150770
loadavg : 3.19 7.56 8.24
swap info: free_mem = 80.98M rsv = 192.00M
           alloc = 21043.50M avail = 49152.00M swap_free = 28108.50M
       F S      UID     PID    PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY  TIME CMD
  240001 A   oracle 3150770       1   0  60 20 1fd079510 85720          08:21:12      -  0:00 ora_q001_data
open: The file access permissions do not allow the specified action.
3150770: ora_q001_data
0x09fffffff0004b48  usl_relocate1(??, ??, ??) + 0x1e0
0x09fffffff00071d0  usl_relocate(??) + 0x50
0x09fffffff000086c  usla_main(??, ??, ??, ??, ??, ??) + 0x160
0x09fffffff0000248  [/usr/ccs/bin/usla64]() + 0x8
*** 2009-06-19 08:23:09.314
*** 2009-06-19 08:23:19.317
Waited for process q001 to initialize for 70 seconds
*** 2009-06-19 08:23:19.317
Dumping diagnostic information for q001:
OS pid = 3150770
loadavg : 3.93 7.08 8.04
swap info: free_mem = 65.36M rsv = 192.00M
           alloc = 21115.99M avail = 49152.00M swap_free = 28036.01M
       F S      UID     PID    PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY  TIME CMD
  240001 A   oracle 3150770       1   0  60 20 1fd079510 90980          08:21:12      -  0:00 ora_q001_data
open: The file access permissions do not allow the specified action.
procstack: open(/proc/3150770/ctl): The requested resource is busy.
*** 2009-06-19 08:24:04.694
*** 2009-06-19 08:24:14.695
Waited for process q001 to initialize for 80 seconds
*** 2009-06-19 08:24:14.695
Dumping diagnostic information for q001:
OS pid = 3150770
loadavg : 5.54 6.96 7.94
swap info: free_mem = 78.14M rsv = 192.00M
           alloc = 21172.87M avail = 49152.00M swap_free = 27979.13M
       F S      UID     PID    PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY  TIME CMD
  242001 T   oracle 3150770       1   0  60 20 1fd079510 91980          08:21:12      -  0:00 ora_q001_data
  240001 A   oracle  255198 6448398   0  60 20 255c93510 29272          08:23:55      -  0:01 /usr/bin/procstack 3150770
open: The file access permissions do not allow the specified action.
procstack: open(/proc/3150770/ctl): The requested resource is busy.
*** 2009-06-19 08:24:38.326
*** 2009-06-19 08:24:43.327
Process q001 is dead (pid=3150770, state=5):

 

文件中有红色字的应该是表示原因，执行/usr/bin/procstack 3150770 的时候权限不足。修改该文件权限后观察。