数据库实例节点重启问题分析:
来源:互联网 发布:淘宝优惠券秒杀技巧 编辑:程序博客网 时间:2024/05/26 12:53
alertoa-db2.log
2014-07-13 02:29:09.132
[cssd(2752638)]CRS-1612:Network communication with node oa-db1 (1) missing for 50% of timeout interval. Removal of this node from cluster in 14.443 seconds
2014-07-13 02:29:09.723
[cssd(2752638)]CRS-1604:CSSD voting file is offline: /dev/asmdisk5; details at (:CSSNM00058:) in /oracle/product/11.2.0/grid11g/log/oa-db2/cssd/ocssd.log.
2014-07-13 02:29:09.724
[cssd(2752638)]CRS-1606:The number of voting files available, 0, is less than the minimum number of voting files required, 1, resulting in CSSD termination
to ensure data integrity; details at (:CSSNM00018:) in /oracle/product/11.2.0/grid11g/log/oa-db2/cssd/ocssd.log
2014-07-13 02:29:09.724
[cssd(2752638)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /oracle/product/11.2.0/grid11g/log/oa-db2/cssd/ocss
d.
--磁盘心跳出现问题:
2014-07-13 02:29:05.193: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2014-07-13 02:29:09.133: [ CSSD][3605]clssnmPollingThread: node oa-db1 (1) at 50% heartbeat fatal, removal in 14.443 seconds
2014-07-13 02:29:09.133: [ CSSD][3605]clssnmPollingThread: node oa-db1 (1) is impending reconfig, flag 2229260, misstime 15557
2014-07-13 02:29:09.133: [ CSSD][3605]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status
(1)
2014-07-13 02:29:09.210: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2014-07-13 02:29:09.210: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2014-07-13 02:29:09.723: [ CSSD][2834](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for 32528 ms for voting file /dev/asmdisk5)
2014-07-13 02:29:09.724: [ CSSD][2834]clssnmvDiskAvailabilityChange: voting file /dev/asmdisk5 now offline
2014-07-13 02:29:09.724: [ CSSD][2834](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
2014-07-13 02:29:09.724: [ CSSD][2834]###################################
2014-07-13 02:29:09.724: [ CSSD][2834]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
2014-07-13 02:29:09.724: [ CSSD][2834]###################################
2014-07-13 02:29:09.724: [ CSSD][2834](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2014-07-13 02:29:09.724: [ CSSD][2834]
----- Call Stack Trace -----
2014-07-13 02:29:09.728: [ CSSD][2834]calling call entry argument values in hex
2014-07-13 02:29:09.728: [ CSSD][2834]location type point (? means dubious value)
2014-07-13 02:29:09.728: [ CSSD][2834]-------------------- -------- -------------------- ----------------------------
2014-07-13 02:29:09.729: [ CSSD][2834]clssscExit()+708 bl 10006ddd4 67800000678 ? 1000F3828 ?
2014-07-13 02:29:09.729: [ CSSD][2834] 11008F270 ? 900000000AB3B94 ?
2014-07-13 02:29:09.729: [ CSSD][2834] 000000000 ? 01105008D ?
2014-07-13 02:29:09.729: [ CSSD][2834] 01105008D ? 000000000 ?
2014-07-13 02:29:09.729: [ CSSD][2834]clssnmvDiskCheck()+ call clssscExit() 1115D74F0 ? 200000002 ?
2014-07-13 02:29:09.729: [ CSSD][2834]2744 000000001 ? 1000FDD98 ?
2014-07-13 02:29:09.729: [ CSSD][2834] 11008F270 ? 01105008D ?
2014-07-13 02:29:09.729: [ CSSD][2834] 01105008D ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834]clssnmvDiskPingMoni call clssnmvDiskCheck() 1115D74F0 ? 1111AF850 ?
2014-07-13 02:29:09.730: [ CSSD][2834]torThread()+416 112584680 ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834] 01105008D ? 0137F0440 ?
2014-07-13 02:29:09.730: [ CSSD][2834]clssscthrdmain()+20 call clssnmvDiskPingMoni 1115D74F0 ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834]8 torThread() 000000000 ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834]_pthread_body()+240 call clssscthrdmain() 1115D74F0 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834]fffffffffffffffc call _pthread_body() 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.732: [ CSSD][2834] 000000000 ? 000000000 ?
[oracle@oa2 /home/oracle]$errpt -aj A924A5FC|more
---------------------------------------------------------------------------
LABEL: CORE_DUMP
IDENTIFIER: A924A5FC
Date/Time: Sun Jul 13 02:29:22 GMT+08:00 2014
Sequence Number: 35060
Machine Id: 00C7BA7D4C00
Node Id: OA-DB2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSPROC
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SOFTWARE PROGRAM
User Causes
USER GENERATED SIGNAL
Recommended Actions
CORRECT THEN RETRY
Failure Causes
SOFTWARE PROGRAM
Recommended Actions
RERUN THE APPLICATION PROGRAM
IF PROBLEM PERSISTS THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE
Detail Data
SIGNAL NUMBER
6
USER'S PROCESS ID:
2752638
FILE SYSTEM SERIAL NUMBER
13
INODE NUMBER
0 12354
CORE FILE NAME
/oracle/product/11.2.0/grid11g/log/oa-db2/cssd/core
PROGRAM NAME
ocssd.bin
STACK EXECUTION DISABLED
[oracle@oa2 /oracle/product/11.2.0/grid11g/log/oa-db2/cssd]$pwd
2014-07-13 02:29:09.132
[cssd(2752638)]CRS-1612:Network communication with node oa-db1 (1) missing for 50% of timeout interval. Removal of this node from cluster in 14.443 seconds
2014-07-13 02:29:09.723
[cssd(2752638)]CRS-1604:CSSD voting file is offline: /dev/asmdisk5; details at (:CSSNM00058:) in /oracle/product/11.2.0/grid11g/log/oa-db2/cssd/ocssd.log.
2014-07-13 02:29:09.724
[cssd(2752638)]CRS-1606:The number of voting files available, 0, is less than the minimum number of voting files required, 1, resulting in CSSD termination
to ensure data integrity; details at (:CSSNM00018:) in /oracle/product/11.2.0/grid11g/log/oa-db2/cssd/ocssd.log
2014-07-13 02:29:09.724
[cssd(2752638)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /oracle/product/11.2.0/grid11g/log/oa-db2/cssd/ocss
d.
--磁盘心跳出现问题:
2014-07-13 02:29:05.193: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2014-07-13 02:29:09.133: [ CSSD][3605]clssnmPollingThread: node oa-db1 (1) at 50% heartbeat fatal, removal in 14.443 seconds
2014-07-13 02:29:09.133: [ CSSD][3605]clssnmPollingThread: node oa-db1 (1) is impending reconfig, flag 2229260, misstime 15557
2014-07-13 02:29:09.133: [ CSSD][3605]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status
(1)
2014-07-13 02:29:09.210: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2014-07-13 02:29:09.210: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2014-07-13 02:29:09.723: [ CSSD][2834](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for 32528 ms for voting file /dev/asmdisk5)
2014-07-13 02:29:09.724: [ CSSD][2834]clssnmvDiskAvailabilityChange: voting file /dev/asmdisk5 now offline
2014-07-13 02:29:09.724: [ CSSD][2834](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
2014-07-13 02:29:09.724: [ CSSD][2834]###################################
2014-07-13 02:29:09.724: [ CSSD][2834]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
2014-07-13 02:29:09.724: [ CSSD][2834]###################################
2014-07-13 02:29:09.724: [ CSSD][2834](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2014-07-13 02:29:09.724: [ CSSD][2834]
----- Call Stack Trace -----
2014-07-13 02:29:09.728: [ CSSD][2834]calling call entry argument values in hex
2014-07-13 02:29:09.728: [ CSSD][2834]location type point (? means dubious value)
2014-07-13 02:29:09.728: [ CSSD][2834]-------------------- -------- -------------------- ----------------------------
2014-07-13 02:29:09.729: [ CSSD][2834]clssscExit()+708 bl 10006ddd4 67800000678 ? 1000F3828 ?
2014-07-13 02:29:09.729: [ CSSD][2834] 11008F270 ? 900000000AB3B94 ?
2014-07-13 02:29:09.729: [ CSSD][2834] 000000000 ? 01105008D ?
2014-07-13 02:29:09.729: [ CSSD][2834] 01105008D ? 000000000 ?
2014-07-13 02:29:09.729: [ CSSD][2834]clssnmvDiskCheck()+ call clssscExit() 1115D74F0 ? 200000002 ?
2014-07-13 02:29:09.729: [ CSSD][2834]2744 000000001 ? 1000FDD98 ?
2014-07-13 02:29:09.729: [ CSSD][2834] 11008F270 ? 01105008D ?
2014-07-13 02:29:09.729: [ CSSD][2834] 01105008D ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834]clssnmvDiskPingMoni call clssnmvDiskCheck() 1115D74F0 ? 1111AF850 ?
2014-07-13 02:29:09.730: [ CSSD][2834]torThread()+416 112584680 ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834] 01105008D ? 0137F0440 ?
2014-07-13 02:29:09.730: [ CSSD][2834]clssscthrdmain()+20 call clssnmvDiskPingMoni 1115D74F0 ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834]8 torThread() 000000000 ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834]_pthread_body()+240 call clssscthrdmain() 1115D74F0 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834]fffffffffffffffc call _pthread_body() 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.732: [ CSSD][2834] 000000000 ? 000000000 ?
[oracle@oa2 /home/oracle]$errpt -aj A924A5FC|more
---------------------------------------------------------------------------
LABEL: CORE_DUMP
IDENTIFIER: A924A5FC
Date/Time: Sun Jul 13 02:29:22 GMT+08:00 2014
Sequence Number: 35060
Machine Id: 00C7BA7D4C00
Node Id: OA-DB2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSPROC
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SOFTWARE PROGRAM
User Causes
USER GENERATED SIGNAL
Recommended Actions
CORRECT THEN RETRY
Failure Causes
SOFTWARE PROGRAM
Recommended Actions
RERUN THE APPLICATION PROGRAM
IF PROBLEM PERSISTS THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE
Detail Data
SIGNAL NUMBER
6
USER'S PROCESS ID:
2752638
FILE SYSTEM SERIAL NUMBER
13
INODE NUMBER
0 12354
CORE FILE NAME
/oracle/product/11.2.0/grid11g/log/oa-db2/cssd/core
PROGRAM NAME
ocssd.bin
STACK EXECUTION DISABLED
[oracle@oa2 /oracle/product/11.2.0/grid11g/log/oa-db2/cssd]$pwd
/oracle/product/11.2.0/grid11g/log/oa-db2/cssd/core 需要分析 /dev/asmdisk5 offline原因
由于权限原因无法查询asm运行日志,不过猜测应该没问题。
CSSD 磁盘心跳异常 引起crs重启!
0 0
- 数据库实例节点重启问题分析:
- 如何诊断节点重启问题
- 如何诊断节点重启问题
- Oracle数据库实例及监听重启
- rac集群节点级联重启故障分析
- elasticsearch 重启节点
- 主机硬件问题导致rac节点重启
- 11gR2 如何诊断节点重启问题
- RAC+DG 单节点重启DG失效问题
- 重构数据库分析
- 分析重启问题一种方法--------sc8810
- 【问题】sql数据库安装重启失败
- 数据库去重问题
- rac 实例被莫名重启的案例分析
- 关于oracle启用实例,以及数据库重启
- Greenplum 数据库初始化总是出现几个节点实例无法启动的问题
- 重启webshpere节点 脚本
- hadoop节点重启步骤
- 中国经济的在发展,可是一部分坏境也受到了污染
- 疯狂Java和Android面试题大全(Android版)
- 浮点数在计算机中的表示
- MySQL 存储数据时 乱码问题
- struts2如何访问servlet API
- 数据库实例节点重启问题分析:
- Drools 入门例子
- Struts2.x教程(三) Struts2拦截器
- 小火来讲---C++中的运算符重载
- Oracle笔记之登录认证
- 船业是啊是打飞机快乐阿双方的快乐阿斯蒂芬但是分开
- eclipse 生成javadoc打开时乱码的问题
- 黑马程序员_java高新之反射
- DROOLS语法详解