rac 实例不能mount

今天维护人员告知有个9i的rac实例无法正常启动，一直处于mount状态，也没有报错信息，一直hang在lmon registered with NM - instance id 2 (internal mem no 1)这里不动。

Sat Sep 10 14:00:33 2011
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2 Version 1.0
PMON started with pid=2
DIAG started with pid=3
LMON started with pid=4
LMD0 started with pid=5
LMS0 started with pid=6
LMS1 started with pid=7
DBW0 started with pid=8
LGWR started with pid=9
CKPT started with pid=10
SMON started with pid=11
RECO started with pid=12
CJQ0 started with pid=13
Sat Sep 10 14:00:36 2011
ARCH: STARTING ARCH PROCESSES
ARC0 started with pid=14
ARC0: Archival started
ARC1 started with pid=15
ARC1: Archival started
Sat Sep 10 14:00:37 2011
ARCH: STARTING ARCH PROCESSES COMPLETE
Sat Sep 10 14:00:37 2011
ARC1: Thread not mounted
Sat Sep 10 14:00:37 2011
ARC0: Thread not mounted
Sat Sep 10 14:00:51 2011
alter database mount
Sat Sep 10 14:00:51 2011
lmon registered with NM - instance id 2 (internal mem no 1)

开始怀疑是HA有问题，导致无法读取共享资源hang住，查看两边的pv属性，发现都是正常的concurrent状态，重启了HA后故障依旧。

后通过查看集群状态发现有两个心跳ip状态是down，两个节点无相ping果然无法ping通，通过询问维护人员知道因换过电源，初步怀疑是换电源的时候把网线给碰松或者碰掉了导致，去机房查看状态发现果然两块网卡指示灯都不亮，接实所有网线后状态指示灯呈正常绿色闪烁状态，再次启动数据库后正常启动。

                clstat - HACMP Cluster Status Monitor
                -------------------------------------


Cluster: kkdb_cluster        (1149099876)
Sat Sep 10 16:49:01 BEIST 2011
                State: UP               Nodes: 2
                SubState: STABLE


        Node: kkdb1          State: UP
           Interface: kkdb1_boot1 (1)        Address: 192.168.0.21
                                                State:   UP
           Interface: kkdb1_boot2 (1)        Address: 192.168.1.21
                                                State:   UP
           Interface: kkdb1_hb (0)           Address: 192.168.3.21
                                                State:   DOWN
           Interface: kkdb1_svc (1)          Address: 10.10.13.62
                                                State:   UP
           Interface: kkdb1_rac (2)          Address: 192.168.2.21
                                                State:   DOWN
           Resource Group: kkdb1_rg                  State:  On line
           Resource Group: kkdb_share_rg                     State:  On line


        Node: kkdb2          State: UP
           Interface: kkdb2_boot1 (1)        Address: 192.168.0.23
                                                State:   UP
           Interface: kkdb2_boot2 (1)        Address: 192.168.1.23
                                                State:   UP
           Interface: kkdb2_hb (0)           Address: 192.168.3.23
                                                State:   DOWN
           Interface: kkdb2_svc (1)          Address: 10.10.13.64
                                                State:   UP
           Interface: kkdb2_rac (2)          Address: 192.168.2.23
                                                State:   DOWN
           Resource Group: kkdb2_rg                  State:  On line
           Resource Group: kkdb_share_rg                     State:  On line