Oracle CRS 不能启动，日志报错： "has a disk HB, but no network HB, DHB has rcfg..."

现象：
--查看crs状态
#/u01/app/11.2.0/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager


[root@ntrac1 ~]# /u01/app/oracle/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  OFFLINE                               Instance Shutdown   
ora.cluster_interconnect.haip
      1        ONLINE  OFFLINE                                                   
ora.crf
      1        ONLINE  OFFLINE                                                   
ora.crsd
      1        ONLINE  OFFLINE                                                   
ora.cssd
      1        ONLINE  OFFLINE                               STARTING            
ora.cssdmonitor
      1        ONLINE  ONLINE       ntrac1                                       
ora.ctssd
      1        ONLINE  OFFLINE                                                   
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.evmd
      1        ONLINE  OFFLINE                                                   
ora.gipcd
      1        ONLINE  ONLINE       ntrac1                                       
ora.gpnpd
      1        ONLINE  ONLINE       ntrac1                                       
ora.mdnsd
      1        ONLINE  ONLINE       ntrac1                     


--查看grid日志
#tail -100f $GRID_HOME/log/ntrac1/alertntrac1.log
2014-08-06 12:29:59.627: 
[/u01/app/oracle/grid/bin/cssdagent(32145)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/oracle/grid/log/ntrac1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2014-08-06 12:29:59.628: 
[cssd(32230)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/oracle/grid/log/ntrac1/cssd/ocssd.log
2014-08-06 12:29:59.628: 
[cssd(32230)]CRS-1603:CSSD on node ntrac1 shutdown by user.
2014-08-06 12:30:04.791: 
[ohasd(20111)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'ntrac1'.
2014-08-06 12:30:06.569: 
[cssd(36385)]CRS-1713:CSSD daemon is started in clustered mode
2014-08-06 12:30:08.191: 
[ohasd(20111)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2014-08-06 12:30:22.814: 
[cssd(36385)]CRS-1707:Lease acquisition for node ntrac1 number 1 completed
2014-08-06 12:30:24.103: 
[cssd(36385)]CRS-1605:CSSD voting file is online: /dev/mapper/CML_OCR02; details in /u01/app/oracle/grid/log/ntrac1/cssd/ocssd.log.
2014-08-06 12:30:24.111: 
[cssd(36385)]CRS-1605:CSSD voting file is online: /dev/mapper/CML_OCR03; details in /u01/app/oracle/grid/log/ntrac1/cssd/ocssd.log.
2014-08-06 12:30:24.122: 
[cssd(36385)]CRS-1605:CSSD voting file is online: /dev/mapper/CML_OCR01; details in /u01/app/oracle/grid/log/ntrac1/cssd/ocssd.log.


--查看ocssd日志
#tail -100f $GRID_HOME/log/ntrac1/cssd/ocssd.log
2014-08-06 14:45:04.140: [    CSSD][483813120]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
2014-08-06 14:45:04.623: [    CSSD][488544000]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0
2014-08-06 14:45:04.968: [    CSSD][502802176]clssnmvDHBValidateNcopy: node 2, ntrac2, has a disk HB, but no network HB, DHB has rcfg 301688209, wrtcnt, 3361203, LATS 5085774, lastSeqNo 3361200, uniqueness 1406193376, timestamp 1407307504/1113874084
2014-08-06 14:45:04.968: [    CSSD][502802176]clssnmvDHBValidateNcopy: node 3, ntrac3, has a disk HB, but no network HB, DHB has rcfg 301688209, wrtcnt, 3360733, LATS 5085774, lastSeqNo 3360730, uniqueness 1406193385, timestamp 1407307504/1113864924
2014-08-06 14:45:05.105: [    CSSD][498054912]clssnmvDHBValidateNcopy: node 2, ntrac2, has a disk HB, but no network HB, DHB has rcfg 301688209, wrtcnt, 3361205, LATS 5085904, lastSeqNo 3361202, uniqueness 1406193376, timestamp 1407307504/1113874544
2014-08-06 14:45:05.105: [    CSSD][498054912]clssnmvDHBValidateNcopy: node 3, ntrac3, has a disk HB, but no network HB, DHB has rcfg 301688209, wrtcnt, 3360735, LATS 5085904, lastSeqNo 3360732, uniqueness 1406193385, timestamp 1407307504/1113864974


解决：

--查看网络连接，发现问题
从其它节点ping故障节点的私网IP地址，发现是ping不通的。

初步确定是网络原因，可能是网卡的问题，后来发现是有故障节点的有一根私有网卡上的一根网线没有插上，插上就重新启动crs就没问题了。

参考：

http://sqlsewer.blogspot.com/2013/07/oracle-crs-is-not-starting-has-disk-hb.html

http://t.askmaclean.com/thread-3709-1-1.html