Oracle 11203 GRID在CRS磁盘组损坏后的处理

11.2.0.3重建CRS磁盘，by ORA老菜，转载请注明作者和出处

本文主要是处理CRS磁盘组(ocr和votedisk所在磁盘组)损坏的情况下，对11.2.0.3 GRID重建CRS磁盘的处理过程

一、故障模拟

$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3012
         Available space (kbytes) :     259108
         ID                       : 1676783525
         Device/File Name         :    +asmcrs
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user


SQL>  select inst_name from v$active_instances;

INST_NAME
--------------------------------------------------------------------------------
dbrac1:rac1
dbrac2:rac2



SQL> col name format a30
SQL> col path format a40
SQL> select name,path from v$asm_disk;

NAME                           PATH
------------------------------ ----------------------------------------
                               /dev/rhdisk10
                               /dev/rhdisk11
                               /dev/rhdisk2
                               /dev/rhdisk3
                               /dev/rhdisk9
ASMCRS_0000                    /dev/rhdisk4
ASMVG1_0000                    /dev/rhdisk5
ASMVG1_0001                    /dev/rhdisk6
ASMVG1_0002                    /dev/rhdisk7
ASMVG1_0003                    /dev/rhdisk8

10 rows selected.


/oracle/asm/cdata/dbrac-cluster
#
#
#
#
# ls -lrt
total 86144
-rw-------    1 root     system      7331840 May 17 02:07 week_.ocr
-rw-------    1 root     system      7331840 May 18 10:11 day.ocr
-rw-------    1 root     system      7331840 May 19 10:11 day_.ocr
-rw-------    1 root     system      7331840 May 19 22:11 backup02.ocr
-rw-------    1 root     system      7331840 May 20 02:11 backup01.ocr
-rw-------    1 root     system      7331840 May 20 06:11 backup00.ocr
-rw-------    1 root     system       105559 May 31 02:48 ocr0531.ocr


dd if=/dev/zero of=/dev/rhdisk4 bs=1024 count=2560



# dd if=/dev/zero of=/dev/rhdisk4 bs=1024 count=2560
2560+0 records in.
2560+0 records out.
#
# ./srvctl stop database -d rac

# ./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dbrac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'dbrac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dbrac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.ASMCRS.dg' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.ASMVG1.dg' on 'dbrac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'dbrac1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.dbrac1.vip' on 'dbrac1'
CRS-2677: Stop of 'ora.dbrac1.vip' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.dbrac1.vip' on 'dbrac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'dbrac2'
CRS-2677: Stop of 'ora.ASMVG1.dg' on 'dbrac1' succeeded
CRS-2676: Start of 'ora.dbrac1.vip' on 'dbrac2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'dbrac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'dbrac2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'dbrac2' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'dbrac1' succeeded
CRS-2677: Stop of 'ora.cvu' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'dbrac2'
CRS-2677: Stop of 'ora.oc4j' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'dbrac2'
CRS-2676: Start of 'ora.cvu' on 'dbrac2' succeeded
CRS-2676: Start of 'ora.oc4j' on 'dbrac2' succeeded
CRS-2677: Stop of 'ora.ASMCRS.dg' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dbrac1'
CRS-2677: Stop of 'ora.asm' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dbrac1'
CRS-2677: Stop of 'ora.ons' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dbrac1'
CRS-2677: Stop of 'ora.net1.network' on 'dbrac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dbrac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.crf' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.asm' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dbrac1'
CRS-2677: Stop of 'ora.mdnsd' on 'dbrac1' succeeded
CRS-2677: Stop of 'ora.crf' on 'dbrac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'dbrac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'dbrac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dbrac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dbrac1'
CRS-2677: Stop of 'ora.cssd' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dbrac1'
CRS-2677: Stop of 'ora.gipcd' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dbrac1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'dbrac1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'dbrac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dbrac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

二、处理过程如下：
在node1启动crs
# ./crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'dbrac1'
CRS-2676: Start of 'ora.mdnsd' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dbrac1'
CRS-2676: Start of 'ora.gpnpd' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbrac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'dbrac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'dbrac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dbrac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'dbrac1'
CRS-2676: Start of 'ora.diskmon' on 'dbrac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'dbrac1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'dbrac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'dbrac1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dbrac1'
CRS-2676: Start of 'ora.ctssd' on 'dbrac1' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'dbrac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'dbrac1'
CRS-2676: Start of 'ora.asm' on 'dbrac1' succeeded
# ps -ef|grep smon
    root 2228290 7536750   0 02:56:04  pts/5  0:00 grep smon
    grid 4063380       1   0 02:54:49      -  0:00 asm_smon_+ASM1


su - grid

为OCR和votedisk创建ASM磁盘组
$ id
uid=301(grid) gid=300(dba) groups=1(staff),600(hagsuser)
$
$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Thu May 31 02:55:52 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL>
SQL>
SQL>
SQL>
SQL> create diskgroup ASMCRS external redundancy disk '/dev/rhdisk4' attribute 'COMPATIBLE.ASM' = '11.2';

Diskgroup created

# id
uid=0(root) gid=0(system) groups=2(bin),3(sys),7(security),8(cron),10(audit),11(lp),600(hagsuser)
# pwd
/oracle/asm/cdata/dbrac-cluster
#

# ls -lrt
total 86136
-rw-------    1 root     system      7331840 May 17 02:07 week_.ocr
-rw-------    1 root     system      7331840 May 18 10:11 day.ocr
-rw-------    1 root     system      7331840 May 19 10:11 day_.ocr
-rw-------    1 root     system      7331840 May 19 22:11 backup02.ocr
-rw-------    1 root     system      7331840 May 20 02:11 backup01.ocr
-rw-------    1 root     system      7331840 May 20 06:11 backup00.ocr
恢复ocr
#/oracle/asm/bin/ocrconfig -restore backup00.ocr

恢复vote disk
# /oracle/asm/bin/crsctl replace votedisk +ASMCRS
Successful addition of voting disk 986d9f44180e4f5cbf1c328c2c7d727f.
Successfully replaced voting disk group with +ASMCRS.
CRS-4266: Voting file(s) successfully replaced




关闭crs
./crsctl stop crs -f

启动crs
# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
检查资源情况
# ./crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.ASMCRS.dg  ora....up.type ONLINE    ONLINE    dbrac1      
ora.ASMVG1.dg  ora....up.type ONLINE    ONLINE    dbrac1      
ora....ER.lsnr ora....er.type ONLINE    ONLINE    dbrac1      
ora....N1.lsnr ora....er.type ONLINE    ONLINE    dbrac1      
ora.asm        ora.asm.type   ONLINE    ONLINE    dbrac1      
ora.cvu        ora.cvu.type   ONLINE    ONLINE    dbrac1      
ora....SM1.asm application    ONLINE    ONLINE    dbrac1      
ora....C1.lsnr application    ONLINE    ONLINE    dbrac1      
ora.dbrac1.gsd application    OFFLINE   OFFLINE               
ora.dbrac1.ons application    ONLINE    ONLINE    dbrac1      
ora.dbrac1.vip ora....t1.type ONLINE    ONLINE    dbrac1      
ora....SM2.asm application    ONLINE    ONLINE    dbrac2      
ora....C2.lsnr application    ONLINE    ONLINE    dbrac2      
ora.dbrac2.gsd application    OFFLINE   OFFLINE               
ora.dbrac2.ons application    ONLINE    ONLINE    dbrac2      
ora.dbrac2.vip ora....t1.type ONLINE    ONLINE    dbrac2      
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    dbrac1      
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    dbrac1      
ora.ons        ora.ons.type   ONLINE    ONLINE    dbrac1      
ora.rac.db     ora....se.type ONLINE    ONLINE    dbrac1      
ora....ry.acfs ora....fs.type ONLINE    ONLINE    dbrac1      
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    dbrac1



# ./crsctl check cluster -all
**************************************************************
dbrac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
dbrac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


OK，收工