ORACLE 11G RAC ASM磁盘全部丢失后的恢复

一、环境描述

（1）Oracle 11.2.0.3 RAC ON Oracle Linux 6 x86_64，只有一个ASM外部冗余磁盘组——DATA；
（2）OCR，VOTEDISK，DATAFILE，CONTROLFILE，SPFILE全部位于这个磁盘组上；

 

二、故障描述

（1）存储故障导致ASM磁盘丢失。
（2）CRS因为OCR和VOTEDISK的丢失，除了OHAS还联机外，CLUSTERWARE服务都已经停止。

 

三、备份情况

（1）RMAN备份：包括controlfile，database，spfile，archivelog，
（2）OCR备份：没有进行过人工备份，在$CRS_HOME/cdata目录下有CRS自动备份文件。

 

四、操作步骤

说明：准使用CRS自动备份的文件恢复OCR，使用RMAN备份来恢复数据库；准备恢复数据的同时，调整ASM磁盘组，将OCR，VOTEDISK同数据库文件分开存放。

 

4.1 恢复OCR和VOTEDISK

(1) 在所有RAC节点上停止CRS服务

[root@rac1 ~]# crsctl stop has -fCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'CRS-2673: Attempting to stop 'ora.crf' on 'rac1'CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeededCRS-2677: Stop of 'ora.crf' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeededCRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completedCRS-4133: Oracle High Availability Services has been stopped.

[root@rac2 ~]# crsctl stop has -fCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'CRS-2673: Attempting to stop 'ora.crf' on 'rac2'CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeededCRS-2677: Stop of 'ora.crf' on 'rac2' succeededCRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeededCRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeededCRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completedCRS-4133: Oracle High Availability Services has been stopped.

 

(2) 在一个节点上以NOCRS方式启动CRS，此操作会启动ASM实例。

[root@rac1 ~]# crsctl start crs -excl -nocrsCRS-4123: Oracle High Availability Services has been started.CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeededCRS-2676: Start of 'ora.gipcd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.cssd' on 'rac1'CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'CRS-2676: Start of 'ora.diskmon' on 'rac1' succeededCRS-2676: Start of 'ora.cssd' on 'rac1' succeededCRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac1'CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac1' succeededCRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'CRS-2676: Start of 'ora.ctssd' on 'rac1' succeededCRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeededCRS-2672: Attempting to start 'ora.asm' on 'rac1'CRS-2676: Start of 'ora.asm' on 'rac1' succeeded

 

(3) 新添加了三块磁盘，已经使用UDEV进行了绑定，查看磁盘状态。

[root@rac1 ~]# su - grid[grid@rac1 ~]$ sqlplus / as sysasmSQL*Plus: Release 11.2.0.3.0 Production on Fri Jul 5 17:41:49 2013Copyright (c) 1982, 2011, Oracle.  All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management optionsSQL> select group_number group#, disk_number disk#, OS_MB, state, path, header_status from v$asm_disk order by 1,2;    GROUP#      DISK#      OS_MB STATE      PATH                 HEADER_STATUS---------- ---------- ---------- ---------- -------------------- ----------------------         0          0       1024 NORMAL     /dev/asm-diskc       CANDIDATE         0          1       5120 NORMAL     /dev/asm-diskd       CANDIDATE         0          2      20480 NORMAL     /dev/asm-diskb       CANDIDATE

 

(4) 创建三个磁盘组，SYSTEMDG给CRS使用，用于存放OCR，VOTEDISK和ASM实例的SPFILE。其余两个给ORACLE使用，DATADG用于存放datafile，controlfile，redolog，spfile；ARCLOGDG存放archivelog。

SQL> create diskgroup SYSTEMDG external redundancy  2  disk '/dev/asm-diskc'  3  ATTRIBUTE 'compatible.rdbms' = '11.2','compatible.asm' = '11.2';Diskgroup created.SQL> create diskgroup DATADG external redundancy  2  disk '/dev/asm-diskb'  3  ATTRIBUTE 'compatible.rdbms' = '11.2','compatible.asm' = '11.2';Diskgroup created.SQL> create diskgroup ARCLOGDG external redundancy  2  disk '/dev/asm-diskd'  3  ATTRIBUTE 'compatible.rdbms' = '11.2','compatible.asm' = '11.2';Diskgroup created.

 

(5) 准备恢复OCR和VOTEDISK，/etc/oracle/ocr.loc中记录了OCR路径，修改ocrconfig_loc的值，以便将OCR恢复到新的磁盘组中。

[root@rac1 ~]# more /etc/oracle/ocr.lococrconfig_loc=+DATAlocal_only=FALSE[root@rac1 ~]# vi /etc/oracle/ocr.lococrconfig_loc=+SYSTEMDGlocal_only=FALSE

 

(6) 恢复OCR

[root@rac1 ~]# ocrconfig -showbackupPROT-26: Oracle Cluster Registry backup locations were retrieved from a local copyrac1     2013/07/05 12:30:00     /u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocrrac1     2013/07/05 08:30:00     /u01/app/11.2.0/grid/cdata/rac-cluster/backup01.ocrrac1     2013/07/05 04:30:00     /u01/app/11.2.0/grid/cdata/rac-cluster/backup02.ocrrac1     2013/07/05 00:29:59     /u01/app/11.2.0/grid/cdata/rac-cluster/day.ocrrac1     2013/07/05 00:29:59     /u01/app/11.2.0/grid/cdata/rac-cluster/week.ocrPROT-25: Manual backups for the Oracle Cluster Registry are not available[root@rac1 ~]# ocrconfig -restore /u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr[root@rac1 ~]#[root@rac1 ~]# ocrcheckStatus of Oracle Cluster Registry is as follows :         Version                  :          3         Total space (kbytes)     :     262120         Used space (kbytes)      :       2840         Available space (kbytes) :     259280         ID                       :   59415097         Device/File Name         :  +SYSTEMDG                                    Device/File integrity check succeeded                                    Device/File not configured                                    Device/File not configured                                    Device/File not configured                                    Device/File not configured         Cluster registry integrity check succeeded         Logical corruption check succeeded

 

(7) 创建VOTEDISK

[root@rac1 ~]#  crsctl replace votedisk +SYSTEMDGCRS-4602: Failed 27 to add voting file afb0ca0f35684f1abfd43d5ec2dc1123.Failed to replace voting disk group with +SYSTEMDG.CRS-4000: Command Replace failed, or completed with errors.

  以上报错是因为使用UDEV绑定ASM磁盘时需要更改默认磁盘搜索路径为/dev/asm*，修改ASM磁盘搜索路径

[root@rac1 ~]# su - grid[grid@rac1 ~]$ sqlplus / as sysasmSQL*Plus: Release 11.2.0.3.0 Production on Fri Jul 5 19:03:25 2013Copyright (c) 1982, 2011, Oracle.  All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management optionsSQL> show parameter asm_diskstringNAME                                 TYPE        VALUE------------------------------------ ----------- ------------------------------asm_diskstring                       stringSQL> SQL> SQL> alter system set asm_diskstring = '/dev/asm*';System altered.SQL> create spfile from memory;create spfile from memory*ERROR at line 1:ORA-00349: failure obtaining block size for'+DATA/rac-cluster/asmparameterfile/registry.253.819922365'ORA-15001: diskgroup "DATA" does not exist or is not mountedSQL> create spfile='+SYSTEMDG' from memory;File created.SQL> startup force mount;ORA-32004: obsolete or deprecated parameter(s) specified for ASM instanceASM instance startedTotal System Global Area  283930624 bytesFixed Size                  2227664 bytesVariable Size             256537136 bytesASM Cache                  25165824 bytesASM diskgroups mounted

 

在次创建VOTEDISK，成功。

[root@rac1 init]# crsctl replace votedisk +SYSTEMDGSuccessful addition of voting disk 8ebb7a63accb4fa8bfa7ab65df7a8c8a.Successfully replaced voting disk group with +SYSTEMDG.CRS-4266: Voting file(s) successfully replaced

 

(8)  OCR和VOTEDISK都恢复完成后，重启CRS到正常模式。

[root@rac1 ~]# crsctl stop has -fCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'CRS-2673: Attempting to stop 'ora.asm' on 'rac1'CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeededCRS-2677: Stop of 'ora.asm' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeededCRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.cssd' on 'rac1'CRS-2677: Stop of 'ora.cssd' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeededCRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completedCRS-4133: Oracle High Availability Services has been stopped.[root@rac1 ~]# crsctl start crsCRS-4123: Oracle High Availability Services has been started.[root@rac1 ~]# crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online[root@rac1 ~]#

 

 4.2 修改CRS注册表中相关配置信息

(1) 挂载新的ASM磁盘组

[grid@rac1 ~]$ sqlplus / as sysasmSQL*Plus: Release 11.2.0.3.0 Production on Sat Jul 6 00:16:05 2013Copyright (c) 1982, 2011, Oracle.  All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management optionsSQL> SQL> select name,state from v$asm_diskgroup;NAME                           STATE------------------------------ -----------SYSTEMDG                       MOUNTEDARCLOGDG                       DISMOUNTEDDATADG                         DISMOUNTEDSQL> alter diskgroup ARCLOGDG,DATADG mount;Diskgroup altered.

 

(2) 更改CRS配置文件中数据库的磁盘组为DATADG和ARCLOGDG

[root@rac1 ~]# srvctl modify database -d csdb -a "DATADG,ARCLOGDG"

 

(3) 禁用并删除原来的磁盘组DATA

[root@rac1 ~]# srvctl disable diskgroup -g DATA[root@rac1 ~]# srvctl remove diskgroup -g DATA[root@rac1 rac-cluster]# crs_stat -t -vName           Type           R/RA   F/FT   Target    State     Host        ----------------------------------------------------------------------ora....OGDG.dg ora....up.type 0/5    0/     ONLINE    ONLINE    rac1        ora.DATADG.dg  ora....up.type 0/5    0/     ONLINE    ONLINE    rac1        ora....ER.lsnr ora....er.type 0/5    0/     ONLINE    ONLINE    rac1        ora....N1.lsnr ora....er.type 0/5    0/0    ONLINE    ONLINE    rac1        ora.asm        ora.asm.type   0/5    0/     ONLINE    ONLINE    rac1        ora.csdb.db    ora....se.type 0/2    0/1    ONLINE    OFFLINE               ora.cvu        ora.cvu.type   0/5    0/0    ONLINE    ONLINE    rac1        ora.gsd        ora.gsd.type   0/5    0/     OFFLINE   OFFLINE               ora....network ora....rk.type 0/5    0/     ONLINE    ONLINE    rac1        ora.oc4j       ora.oc4j.type  0/1    0/2    ONLINE    ONLINE    rac1        ora.ons        ora.ons.type   0/3    0/     ONLINE    ONLINE    rac1        ora....SM1.asm application    0/5    0/0    ONLINE    ONLINE    rac1        ora....C1.lsnr application    0/5    0/0    ONLINE    ONLINE    rac1        ora.rac1.gsd   application    0/5    0/0    OFFLINE   OFFLINE               ora.rac1.ons   application    0/3    0/0    ONLINE    ONLINE    rac1        ora.rac1.vip   ora....t1.type 0/0    0/0    ONLINE    ONLINE    rac1        ora.rac2.vip   ora....t1.type 0/0    0/0    ONLINE    ONLINE    rac1        ora.scan1.vip  ora....ip.type 0/0    0/0    ONLINE    ONLINE    rac1

 

(4) 在OCR注册表中修改Oracle数据库参数文件的位置

[root@rac1 ~]# srvctl modify database -d csdb -p +DATADG/csdb/spfilecsdb.ora

 

4.3 恢复数据库

(1) 查看备份文件路径和名称

[root@rac1 ~]# su - oracle[oracle@rac1 ~]$ [oracle@rac1 ~]$ cd /u01/app/oracle/backup[oracle@rac1 backup]$ lltotal 221796-rw-r----- 1 oracle asmadmin   5357568 Jul  5 15:19 arc_819991156_9.bk-rw-r----- 1 oracle asmadmin      2560 Jul  5 15:19 arc_819991158_11.bk-rw-r----- 1 oracle asmadmin 203104256 Jul  5 15:18 CSDB_819991120_5.bk-rw-r----- 1 oracle asmadmin  18546688 Jul  5 15:19 ctl_file_0coe04jq_1_1_20130705.ctl-rw-r----- 1 oracle asmadmin     98304 Jul  5 15:19 spfile_0doe04js_1_1_20130705[oracle@rac1 backup]$

 

(2) 创建一个基本的启动参数文件，以便启动数据库到nomout状态恢复spfile

[oracle@rac1 ~]$ touch /u01/app/oracle/backup/init.ora[oracle@rac1 ~]$ vi /u01/app/oracle/backup/init.ora*.db_name='csdb'*.remote_login_passwordfile='exclusive'

 

(3) 使用刚创建的参数文件将数据库启动到nomount状态

[oracle@rac1 ~]$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.3.0 Production on Sat Jul 6 13:56:06 2013Copyright (c) 1982, 2011, Oracle.  All rights reserved.Connected to an idle instance.SQL> startup nomount pfile='/u01/app/oracle/backup/init.ora';ORACLE instance started.Total System Global Area  238034944 bytesFixed Size                  2227136 bytesVariable Size             180356160 bytesDatabase Buffers           50331648 bytesRedo Buffers                5120000 bytesSQL>

 

(4) 使用RMAN恢复SPFILE到ASM磁盘组DATADG

[oracle@rac1 ~]$ rman target /Recovery Manager: Release 11.2.0.3.0 - Production on Sat Jul 6 13:59:26 2013Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.connected to target database: CSDB (not mounted)RMAN> restore spfile to '+DATADG/csdb/spfilecsdb.ora' from '/u01/app/oracle/backup/spfile_0doe04js_1_1_20130705';Starting restore at 06-JUL-13using channel ORA_DISK_1channel ORA_DISK_1: restoring spfile from AUTOBACKUP /u01/app/oracle/backup/spfile_0doe04js_1_1_20130705channel ORA_DISK_1: SPFILE restore from AUTOBACKUP completeFinished restore at 06-JUL-13

 

(5) 使用恢复后spfile启动数据库，并修改control_files，db_recovery_file_dest，log_archive_dest等存在旧路径的参数值。

[oracle@rac1 ~]$ vi $ORACLE_HOME/dbs/initcsdb1.ora SPFILE='+DATADG/csdb/spfilecsdb.ora'[oracle@rac1 ~]$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.3.0 Production on Sat Jul 6 14:11:58 2013Copyright (c) 1982, 2011, Oracle.  All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, OLAP, Data Miningand Real Application Testing optionsSQL> startup nomount forceORACLE instance started.Total System Global Area 1653518336 bytesFixed Size                  2228904 bytesVariable Size            1073745240 bytesDatabase Buffers          570425344 bytesRedo Buffers                7118848 bytesSQL>SQL> show parameter control_filesNAME                                 TYPE        VALUE------------------------------------ ----------- ------------------------------control_files                        string      +DATA/csdb/control01.ctl, +DAT                                                 A/csdb/control02.ctlSQL>SQL> alter system set control_files='+DATADG/csdb/control01.ctl','+DATADG/csdb/control02.ctl' scope=spfileSystem altered.SQL> alter system set db_recovery_file_dest='+DATADG' scope=spfile;System altered.SQL> alter system set log_archive_dest_1='LOCATION=+ARCLOGDG' scope=spfile;System altered.SQL> startup force nomount;ORACLE instance started.Total System Global Area 1653518336 bytesFixed Size                  2228904 bytesVariable Size            1073745240 bytesDatabase Buffers          570425344 bytesRedo Buffers                7118848 bytes

 

(6) 查看数据库的DBID

[oracle@rac1 ~]$ strings /u01/app/oracle/backup/CSDB_819991120_5.bk | grep MAXVALUE,  返回的值类似下面的例子，其中那一窜数字即为DBID。...MAXVALUE, MAXVALUE!3042905279, MAXVALUE,3042905279, MAXVALUE,...

 

(7) 恢复控制文件到新的ASM磁盘组DATADG

[oracle@rac1 ~]$ rman target /Recovery Manager: Release 11.2.0.3.0 - Production on Sat Jul 6 14:28:28 2013Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.connected to target database: CSDB (not mounted)RMAN> set dbid=3042905279executing command: SET DBIDRMAN> restore controlfile from '/u01/app/oracle/backup/ctl_file_0coe04jq_1_1_20130705.ctl';Starting restore at 06-JUL-13using target database control file instead of recovery catalogallocated channel: ORA_DISK_1channel ORA_DISK_1: SID=18 instance=csdb1 device type=DISKchannel ORA_DISK_1: restoring control filechannel ORA_DISK_1: restore complete, elapsed time: 00:00:01output file name=+DATADG/csdb/control01.ctloutput file name=+DATADG/csdb/control02.ctlFinished restore at 06-JUL-13

 

(8) 进入SQLPLUS，查看旧数据文件信息

[oracle@rac1 ~]$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.3.0 Production on Sat Jul 6 14:41:12 2013Copyright (c) 1982, 2011, Oracle.  All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Data Mining and Real Application Testing optionsSQL> alter database mount;Database altered. SQL> col name format a50SQL> select file#,name from v$datafile;     FILE# NAME---------- --------------------------------------------------         1 +DATA/csdb/datafile/system.260.819979847         2 +DATA/csdb/datafile/sysaux.261.819979871         3 +DATA/csdb/datafile/undotbs1.262.819979889         4 +DATA/csdb/datafile/undotbs2.264.819979905         5 +DATA/csdb/datafile/users.265.819979913

 

(9) 使用RMAN恢复数据库

RMAN> run{2> set newname for datafile 1 to '+DATADG/csdb/datafile/system.260.819979847';3> set newname for datafile 2 to '+DATADG/csdb/datafile/sysaux.261.819979871';4> set newname for datafile 3 to '+DATADG/csdb/datafile/undotbs1.262.819979889';5> set newname for datafile 4 to '+DATADG/csdb/datafile/undotbs2.264.819979905';6> set newname for datafile 5 to '+DATADG/csdb/datafile/users.265.819979913';7> restore database;8> switch datafile all;9> recover database;10> }executing command: SET NEWNAMEreleased channel: ORA_DISK_1executing command: SET NEWNAMEexecuting command: SET NEWNAMEexecuting command: SET NEWNAMEexecuting command: SET NEWNAMEStarting restore at 07-JUL-13Starting implicit crosscheck backup at 07-JUL-13allocated channel: ORA_DISK_1Crosschecked 10 objectsFinished implicit crosscheck backup at 07-JUL-13Starting implicit crosscheck copy at 07-JUL-13using channel ORA_DISK_1Finished implicit crosscheck copy at 07-JUL-13searching for all files in the recovery areacataloging files...no files catalogedusing channel ORA_DISK_1channel ORA_DISK_1: starting datafile backup set restorechannel ORA_DISK_1: specifying datafile(s) to restore from backup setchannel ORA_DISK_1: restoring datafile 00002 to +DATADG/csdb/datafile/sysaux.261.819979871channel ORA_DISK_1: restoring datafile 00003 to +DATADG/csdb/datafile/undotbs1.262.819979889channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/CSDB_819991120_6.bkchannel ORA_DISK_1: piece handle=/u01/app/oracle/backup/CSDB_819991120_6.bk tag=ORCL_HOT_DB_BKchannel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:46channel ORA_DISK_1: starting datafile backup set restorechannel ORA_DISK_1: specifying datafile(s) to restore from backup setchannel ORA_DISK_1: restoring datafile 00001 to +DATADG/csdb/datafile/system.260.819979847channel ORA_DISK_1: restoring datafile 00004 to +DATADG/csdb/datafile/undotbs2.264.819979905channel ORA_DISK_1: restoring datafile 00005 to +DATADG/csdb/datafile/users.265.819979913channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/CSDB_819991120_5.bkchannel ORA_DISK_1: piece handle=/u01/app/oracle/backup/CSDB_819991120_5.bk tag=ORCL_HOT_DB_BKchannel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:35Finished restore at 07-JUL-13datafile 1 switched to datafile copyinput datafile copy RECID=6 STAMP=820112831 file name=+DATADG/csdb/datafile/system.284.820112797datafile 2 switched to datafile copyinput datafile copy RECID=7 STAMP=820112831 file name=+DATADG/csdb/datafile/sysaux.282.820112751datafile 3 switched to datafile copyinput datafile copy RECID=8 STAMP=820112831 file name=+DATADG/csdb/datafile/undotbs1.283.820112751datafile 4 switched to datafile copyinput datafile copy RECID=9 STAMP=820112831 file name=+DATADG/csdb/datafile/undotbs2.285.820112797datafile 5 switched to datafile copyinput datafile copy RECID=10 STAMP=820112831 file name=+DATADG/csdb/datafile/users.286.820112797Starting recover at 07-JUL-13using channel ORA_DISK_1starting media recoverychannel ORA_DISK_1: starting archived log restore to default destinationchannel ORA_DISK_1: restoring archived logarchived log thread=1 sequence=15channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/arc_819991156_9.bkchannel ORA_DISK_1: piece handle=/u01/app/oracle/backup/arc_819991156_9.bk tag=TAG20130705T151916channel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:01archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_1_seq_15.256.820112833 thread=1 sequence=15channel ORA_DISK_1: starting archived log restore to default destinationchannel ORA_DISK_1: restoring archived logarchived log thread=2 sequence=2channel ORA_DISK_1: restoring archived logarchived log thread=1 sequence=16channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/arc_819991156_10.bkchannel ORA_DISK_1: piece handle=/u01/app/oracle/backup/arc_819991156_10.bk tag=TAG20130705T151916channel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:01archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_2_seq_2.257.820112835 thread=2 sequence=2channel default: deleting archived log(s)archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_1_seq_15.256.820112833 RECID=5 STAMP=820112832archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_1_seq_16.258.820112835 thread=1 sequence=16channel default: deleting archived log(s)archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_2_seq_2.257.820112835 RECID=7 STAMP=820112834channel ORA_DISK_1: starting archived log restore to default destinationchannel ORA_DISK_1: restoring archived logarchived log thread=2 sequence=3channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/arc_819991158_11.bkchannel ORA_DISK_1: piece handle=/u01/app/oracle/backup/arc_819991158_11.bk tag=TAG20130705T151916channel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:01archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_2_seq_3.257.820112837 thread=2 sequence=3channel default: deleting archived log(s)archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_2_seq_3.257.820112837 RECID=8 STAMP=820112835unable to find archived logarchived log thread=2 sequence=4RMAN-00571: ===========================================================RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============RMAN-00571: ===========================================================RMAN-03002: failure of recover command at 07/07/2013 01:07:16RMAN-06054: media recovery requesting unknown archived log for thread 2 with sequence 4 and starting SCN of 323980

(10) 更改REDO LOG位置信息

SQL> select member from v$logfile;MEMBER--------------------------------------------------------------------------------+DATA/csdb/redo01.log+DATA/csdb/redo02.log+DATA/csdb/redo03.log+DATA/csdb/redo04.logSQL> alter database rename file '+DATA/csdb/redo01.log' to '+DATADG/csdb/redo01.log';Database altered.SQL> alter database rename file '+DATA/csdb/redo02.log' to '+DATADG/csdb/redo02.log';Database altered.SQL> alter database rename file '+DATA/csdb/redo03.log' to '+DATADG/csdb/redo03.log';Database altered.SQL> alter database rename file '+DATA/csdb/redo04.log' to '+DATADG/csdb/redo04.log';Database altered.

(11) 打开数据库

SQL> alter database open resetlogs;Database altered.

(12) 更改TEMP表空间文件位置

SQL> select name from v$tempfile;NAME--------------------------------------------------------------------------------+DATA/csdb/tempfile/temp.263.819979895SQL> alter tablespace temp add tempfile '+DATADG';Tablespace altered.SQL> alter tablespace temp drop tempfile '+DATA/csdb/tempfile/temp.263.819979895';Tablespace altered

 

4.4 完成恢复操作

(1) 在其他RAC节点上更改OCR路径

 [root@rac2 ~]# vi /etc/oracle/ocr.lococrconfig_loc=+SYSTEMDGlocal_only=FALSE

(2) 在恢复节点上重启CRS

[root@rac1 ~]# crsctl stop crs[root@rac1 ~]# crsctl start has

 

(3) 在其他节点上启动CRS

[root@rac2 ~]# crsctl start crs