crsd.log报错: CAAMonitorHandler :: 0:Could not join /oracle/product/cr

一、环境描述:

    AIX 5.3 + ORACLE 10.2.0.4 RAC

二、问题描述:

今天给一套数据库做巡检是发现CRS日志crsd.log频繁报错vip error,以下截取部分日志:

<p> </p><p>2014-08-22 17:03:03.497: [  CRSEVT][11205]32CAAMonitorHandler :: 0:Action Script /oracle/product/crs/bin/racgwrap(check) timed out for ora.p520a.vip! (timeout=60)2014-08-22 17:03:03.498: [  CRSAPP][11205]32CheckResource error for ora.p520a.vip error code = -2<strong><span style="color:#ff0000;">2014-08-24 17:02:53.137: [  CRSEVT][11072]32CAAMonitorHandler :: 0:Could not join /oracle/product/crs/bin/racgwrap(check)category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child</span></strong></p><p>2014-08-24 17:02:54.229: [  CRSEVT][11072]32CAAMonitorHandler :: 0:Action Script /oracle/product/crs/bin/racgwrap(check) timed out for ora.p520a.vip! (timeout=60)2014-08-24 17:02:54.229: [  CRSAPP][11072]32CheckResource error for ora.p520a.vip error code = -22014-08-26 17:02:51.141: [  CRSEVT][11199]32CAAMonitorHandler :: 0:Could not join /oracle/product/crs/bin/racgwrap(check)category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child</p><p>2014-08-26 17:02:52.131: [  CRSEVT][11199]32CAAMonitorHandler :: 0:Action Script /oracle/product/crs/bin/racgwrap(check) timed out for ora.p520a.vip! (timeout=60)2014-08-26 17:02:52.132: [  CRSAPP][11199]32CheckResource error for ora.p520a.vip error code = -22014-08-27 17:03:05.161: [  CRSEVT][11259]32CAAMonitorHandler :: 0:Could not join /oracle/product/crs/bin/racgwrap(check)category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child</p><p>2014-08-27 17:03:06.859: [  CRSEVT][11259]32CAAMonitorHandler :: 0:Action Script /oracle/product/crs/bin/racgwrap(check) timed out for ora.p520a.vip! (timeout=60)2014-08-27 17:03:06.859: [  CRSAPP][11259]32CheckResource error for ora.p520a.vip error code = -22014-08-28 17:03:37.069: [  CRSEVT][11067]32CAAMonitorHandler :: 0:Could not join /oracle/product/crs/bin/racgwrap(check)category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child</p><p>2014-08-28 17:03:40.451: [  CRSEVT][11067]32CAAMonitorHandler :: 0:Action Script /oracle/product/crs/bin/racgwrap(check) timed out for ora.p520a.vip! (timeout=60)2014-08-28 17:03:40.451: [  CRSAPP][11067]32CheckResource error for ora.p520a.vip error code = -22014-09-01 17:03:08.068: [  CRSEVT][11061]32CAAMonitorHandler :: 0:Could not join /oracle/product/crs/bin/racgwrap(check)category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child</p>

三、解决方法

通过METALINK搜索，以上报错信息与BUG 8572205症状相吻合，不一定需要打补丁也可以通过修改配置文件来规避

该BUG的现象是,BUG发作的时间很短,但如果没有配置FAF,会话将中止而且新会话无法连接p520a.vip

BUG的详细描述如下：

===================================================================================

Hdr: 8572205 10.2.0.4 PCW 10.2.0.4 RACG PRODID-5 PORTID-23Abstract: CHILDCRASH, OS ERROR: 0, OTHER: ABNORMAL TERMINATION OF CHILD *** 06/03/09 10:03 am ***TAR:----7537479.993 PROBLEM:--------complete outage in the 4 instances out of 6 because of the following issue:=====        2009-06-02 02:26:00.470: [  CRSEVT][911170] CAAMonitorHandler :: 0:Could not join    /opt/oracle/product/10.2.0/crs/bin/racgwrap(check)    category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal    termination of the child        2009-06-02 02:26:00.470: [  CRSEVT][911170] CAAMonitorHandler :: 0:Action Script    /opt/oracle/product/10.2.0/crs/bin/racgwrap(check) timed out for ora.eprvd4244.vip! (timeout=60)    2009-06-02 02:26:00.470: [  CRSAPP][911170] CheckResource error for ora.eprvd4244.vip error code =    -2    2009-06-02 02:35:22.561: [  CRSEVT][911158] CAAMonitorHandler :: 0:Could not join    /opt/oracle/product/10.2.0/racdb_04/bin/racgwrap(check)    category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal    termination of the child        2009-06-02 02:35:22.561: [  CRSEVT][911158] CAAMonitorHandler :: 0:Action Script    /opt/oracle/product/10.2.0/racdb_04/bin/racgwrap(check) timed out for ora.se001p.se001p5.inst!    (timeout=600)    2009-06-02 02:35:22.561: [  CRSAPP][911158] CheckResource error for ora.se001p.se001p5.inst error    code = -2    2009-06-02 02:35:23.101: [  CRSEVT][911159] CAAMonitorHandler :: 0:Could not join    /opt/oracle/product/10.2.0/racdb_04/bin/racgwrap(check)    category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal    termination of the child        2009-06-02 02:35:23.101: [  CRSEVT][911159] CAAMonitorHandler :: 0:Action Script    /opt/oracle/product/10.2.0/racdb_04/bin/racgwrap(check) timed out for    ora.eprvd4244.LISTENER_OFAC0P_EPRVD4244.lsnr! (timeout=600)    2009-06-02 02:35:23.101: [  CRSAPP][911159] CheckResource error for    ora.eprvd4244.LISTENER_OFAC0P_EPRVD4244.lsnr error code = -2    2009-06-02 02:35:23.691: [  CRSEVT][911162] CAAMonitorHandler :: 0:Could not join    /opt/oracle/product/10.2.0/racdb_04/bin/racgwrap(check)    category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal    termination of the child======= DIAGNOSTIC ANALYSIS:--------------------This issue is already addressed in bug:6196746.  This bug is fixed in 10.2.0.5. and    11.1.0.7.The Workaround is as follows:    =====        1. Stop CRS on the Node.        2. Make a copy of racgwrap located under $ORACLE_HOME/bin and $CRS_HOME/bin on the Node        3. Edit the file racgwrap and modify the last 3 lines from:        $ORACLE_HOME/bin/racgmain "$"    status=$?    exit $status        to:        # Line added to test fix for Bug 6196746    exec $ORACLE_HOME/bin/racgmain "$"        4. Restart CRS and make sure that all the resources are starts.    ===== WORKAROUND:-----------The Workaround is NOT working in rolling way. customer CAN NOT have the complete outage in the cluster as its their vital business generating system RELATED BUGS:-------------Bug 6196746 - HUGE AND GROWING LIST OF RACG CHECK VIP PROCESSES, TIMEOUT 

===================================================================================

 

-------------------------------------------------------------------------------------------------

本文来自于我的技术博客 http://blog.csdn.net/robo23

转载请标注源文链接,否则追究法律责任!