ORACLE 使用rman备份通过restore、recover恢复standby库ORA-10877实战

1、备库standby异常报错

昨天凌晨磁盘空间突然暴涨导致oracle备库异常，报警，后了过来清理掉磁盘的备份文件，去了之后，归档日志能同步过来了，但是启动备库standby发现mrp没有启动，后台报错如下，google了，说这种情况要重新再做备库（不知道是否还有其它修复办法呢？）：错误日志报错信息如下：

Completed: alter database recover managed standby database using current logfile disconnect from sessionWed May 31 10:44:48 2017Dumping diagnostic data in directory=[cdmp_20170531104448], requested by (instance=1, osid=43411 (PR00)), summary=[incident=600239].Use ADRCI or Support Workbench to package the incident.See Note 411.1 at My Oracle Support for error and packaging details.Errors with log /oracle/app/oracle/archivelogs/1_18799_940127349.dbfMRP0: Background Media Recovery terminated with error 600Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_pr00_43411.trc:ORA-00600: internal error code, arguments: [2619], [18799], [], [], [], [], [], [], [], [], [], []Managed Standby Recovery not using Real Time ApplyRecovery interrupted!Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_pr00_43411.trc:ORA-00600: internal error code, arguments: [2619], [18799], [], [], [], [], [], [], [], [], [], []Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_mrp0_43408.trc  (incident=600231):ORA-00600: internal error code, arguments: [2619], [18799], [], [], [], [], [], [], [], [], [], []ORA-10877: error signaled in parallel recovery slave ORA-10877: error signaled in parallel recovery slave ORA-10877: error signaled in parallel recovery slave ORA-10877: error signaled in parallel recovery slave ORA-10877: error signaled in parallel recovery slave 

2、使用duplicate target 进行恢复备库

duplicate target database for standby nofilenamecheck dorecover;Starting Duplicate Db at 05-JUN-17release channel c1;release channel c2;using target database control file instead of recovery catalogallocated channel: ORA_AUX_DISK_1channel ORA_AUX_DISK_1: SID=4708 device type=DISKcontents of Memory Script:{   set until scn  14430447592;   restore clone standby controlfile;}executing Memory Scriptexecuting command: SET until clauseStarting restore at 05-JUN-17using channel ORA_AUX_DISK_1channel ORA_AUX_DISK_1: starting datafile backup set restorechannel ORA_AUX_DISK_1: restoring control filechannel ORA_AUX_DISK_1: reading from backup piece /oracle/backup/data/controlfiles/c-3391761643-20170605-00channel ORA_AUX_DISK_1: ORA-19870: error while restoring backup piece /oracle/backup/data/controlfiles/c-3391761643-20170605-00ORA-19505: failed to identify file "/oracle/backup/data/controlfiles/c-3391761643-20170605-00"ORA-27037: unable to obtain file statusLinux-x86_64 Error: 2: No such file or directoryAdditional information: 3

这里恢复失败，google了下，没有找到合适的办法，咋办呢？需要换种思路去恢复了，duplicate此路不通了。

3、尝试使用最原始的rman备份直接restore、recover的方式

copy控制文件、备份文件，备份文件已经有了，所以不需要再次生成，但是控制文件需要再次生成一下，于是在主库上生成最新的备份的控制文件：

RMAN> backup current controlfile for standby format '/home/oracle/ctlfile4.bak';Starting backup at 05-JUN-17using target database control file instead of recovery catalogallocated channel: ORA_DISK_1channel ORA_DISK_1: SID=948 device type=DISKchannel ORA_DISK_1: starting full datafile backup setchannel ORA_DISK_1: specifying datafile(s) in backup setincluding standby control file in backup setchannel ORA_DISK_1: starting piece 1 at 05-JUN-17channel ORA_DISK_1: finished piece 1 at 05-JUN-17piece handle=/home/oracle/ctlfile4.bak tag=TAG20170605T223329 comment=NONEchannel ORA_DISK_1: backup set complete, elapsed time: 00:00:01Finished backup at 05-JUN-17Starting Control File and SPFILE Autobackup at 05-JUN-17piece handle=/oracle/backup/data/controlfiles/c-3391761643-20170605-02 comment=NONEFinished Control File and SPFILE Autobackup at 05-JUN-17RMAN> 

scp将控制文件和备份文件copy到备库上去

scp -r 2017-06-05 101.251.31.131:/oracle/scp /home/oracle/ctlfile4.bak 101.251.31.131:/home/oracle/

接下来基本全在备库上进行操作，进行restore standby controlfile恢复控制文件

RMAN> restore standby controlfile from '/home/oracle/ctlfile4.bak';Starting restore at 05-JUN-17using target database control file instead of recovery catalogallocated channel: ORA_DISK_1channel ORA_DISK_1: SID=4708 device type=DISKchannel ORA_DISK_1: restoring control filechannel ORA_DISK_1: restore complete, elapsed time: 00:00:01output file name=/oracle/app/oracle/oradata/powerdes/control01.ctloutput file name=/oracle/app/oracle/fast_recovery_area/powerdes/control02.ctlFinished restore at 05-JUN-17RMAN> 

启动到mount状态

RMAN> alter database mount;database mountedreleased channel: ORA_DISK_1RMAN

注册备份集合

RMAN> catalog start with '/oracle/2017-06-05';Starting implicit crosscheck backup at 05-JUN-17allocated channel: ORA_DISK_1channel ORA_DISK_1: SID=4709 device type=DISKCrosschecked 147 objectsFinished implicit crosscheck backup at 05-JUN-17Starting implicit crosscheck copy at 05-JUN-17using channel ORA_DISK_1Crosschecked 2 objectsFinished implicit crosscheck copy at 05-JUN-17searching for all files in the recovery areacataloging files...no files catalogedsearching for all files that match the pattern /oracle/2017-06-05List of Files Unknown to the Database=====================================File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9901.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9899.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9910.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9912.bakFile Name: /oracle/2017-06-05/rman_backup.logFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9908.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9904.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9905.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9906.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9909.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9903.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9902.bakFile Name: /oracle/2017-06-05/full_POWERDES_20170605_9911.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9900.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9907.bakDo you really want to catalog the above files (enter YES or NO)? YEScataloging files...cataloging doneList of Cataloged Files=======================File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9901.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9899.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9910.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9912.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9908.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9904.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9905.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9906.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9909.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9903.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9902.bakFile Name: /oracle/2017-06-05/full_POWERDES_20170605_9911.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9900.bakFile Name: /oracle/2017-06-05/arch_POWERDES_20170605_9907.bakList of Files Which Where Not Cataloged=======================================File Name: /oracle/2017-06-05/rman_backup.log  RMAN-07517: Reason: The file header is corruptedRMAN>

博客作者黄杉（mchdba），博客源地址：http://blog.csdn.net/mchdba/article/details/72874642，谢绝转载。


开始restore database操作

RMAN> restore database;Starting restore at 05-JUN-17using channel ORA_DISK_1channel ORA_DISK_1: starting datafile backup set restorechannel ORA_DISK_1: specifying datafile(s) to restore from backup setchannel ORA_DISK_1: restoring datafile 00001 to /home/oradata/powerdes/system01.dbfchannel ORA_DISK_1: restoring datafile 00002 to /home/oradata/powerdes/sysaux01.dbfchannel ORA_DISK_1: restoring datafile 00003 to /home/oradata/powerdes/undotbs01.dbfchannel ORA_DISK_1: restoring datafile 00004 to /home/oradata/powerdes/users01.dbfchannel ORA_DISK_1: restoring datafile 00005 to /home/oradata/powerdes/powerdesk01.dbfchannel ORA_DISK_1: restoring datafile 00006 to /home/oradata/powerdes/plas01.dbfchannel ORA_DISK_1: restoring datafile 00007 to /home/oradata/powerdes/pl01.dbfchannel ORA_DISK_1: restoring datafile 00008 to /home/oradata/powerdes/help01.dbfchannel ORA_DISK_1: restoring datafile 00009 to /home/oradata/powerdes/adobelc01.dbfchannel ORA_DISK_1: restoring datafile 00010 to /home/oradata/powerdes/sms01.dbfchannel ORA_DISK_1: restoring datafile 00011 to /home/oradata/powerdes/plcrm01.dbfchannel ORA_DISK_1: restoring datafile 00012 to /home/oradata/powerdes/powerdesk02.dbfchannel ORA_DISK_1: restoring datafile 00013 to /home/oradata/powerdes/datagm01.dbfchannel ORA_DISK_1: restoring datafile 00014 to /home/oradata/powerdes/plimp01.DBFchannel ORA_DISK_1: restoring datafile 00015 to /home/oradata/powerdes/dwetl01.DBFchannel ORA_DISK_1: restoring datafile 00016 to /home/oradata/powerdes/dw02.DBFchannel ORA_DISK_1: restoring datafile 00017 to /home/oradata/powerdes/timdba01.DBFchannel ORA_DISK_1: restoring datafile 00018 to /home/oradata/powerdes/users02.dbfchannel ORA_DISK_1: restoring datafile 00019 to /home/oradata/powerdes/system02.dbfchannel ORA_DISK_1: restoring datafile 00020 to /home/oradata/powerdes/powerdesk03.dbfchannel ORA_DISK_1: restoring datafile 00021 to /home/oradata/powerdes/timdba02.dbfchannel ORA_DISK_1: reading from backup piece /oracle/2017-06-05/full_POWERDES_20170605_9911.bakchannel ORA_DISK_1: piece handle=/oracle/2017-06-05/full_POWERDES_20170605_9911.bak tag=TAG20170605T033303channel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 01:29:37Finished restore at 05-JUN-17RMAN> 

开始rocover database操作

RMAN> recover database;Starting recover at 05-JUN-17using channel ORA_DISK_1starting media recoveryarchived log for thread 1 with sequence 20421 is already on disk as file /oracle/app/oracle/archivelogs/1_20421_940127349.dbfarchived log for thread 1 with sequence 20422 is already on disk as file /oracle/app/oracle/archivelogs/1_20422_940127349.dbfarchived log for thread 1 with sequence 20423 is already on disk as file /oracle/app/oracle/archivelogs/1_20423_940127349.dbfarchived log for thread 1 with sequence 20424 is already on disk as file /oracle/app/oracle/archivelogs/1_20424_940127349.dbfarchived log for thread 1 with sequence 20425 is already on disk as file /oracle/app/oracle/archivelogs/1_20425_940127349.dbfarchived log for thread 1 with sequence 20426 is already on disk as file /oracle/app/oracle/archivelogs/1_20426_940127349.dbfarchived log for thread 1 with sequence 20427 is already on disk as file /oracle/app/oracle/archivelogs/1_20427_940127349.dbfarchived log for thread 1 with sequence 20428 is already on disk as file /oracle/app/oracle/archivelogs/1_20428_940127349.dbfarchived log for thread 1 with sequence 20429 is already on disk as file /oracle/app/oracle/archivelogs/1_20429_940127349.dbfarchived log for thread 1 with sequence 20430 is already on disk as file /oracle/app/oracle/archivelogs/1_20430_940127349.dbfarchived log for thread 1 with sequence 20431 is already on disk as file /oracle/app/oracle/archivelogs/1_20431_940127349.dbfarchived log for thread 1 with sequence 20432 is already on disk as file /oracle/app/oracle/archivelogs/1_20432_940127349.dbfarchived log for thread 1 with sequence 20433 is already on disk as file /oracle/app/oracle/archivelogs/1_20433_940127349.dbfarchived log for thread 1 with sequence 20434 is already on disk as file /oracle/app/oracle/archivelogs/1_20434_940127349.dbfchannel ORA_DISK_1: starting archived log restore to default destinationchannel ORA_DISK_1: restoring archived logarchived log thread=1 sequence=20357channel ORA_DISK_1: restoring archived logarchived log thread=1 sequence=20358channel ORA_DISK_1: restoring archived logarchived log thread=1 sequence=20359channel ORA_DISK_1: restoring archived logarchived log thread=1 sequence=20360channel ORA_DISK_1: reading from backup piece /oracle/2017-06-05/arch_POWERDES_20170605_9912.bakchannel ORA_DISK_1: piece handle=/oracle/2017-06-05/arch_POWERDES_20170605_9912.bak tag=TAG20170605T051030channel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:15archived log file name=/oracle/app/oracle/archivelogs/1_20357_940127349.dbf thread=1 sequence=20357archived log file name=/oracle/app/oracle/archivelogs/1_20358_940127349.dbf thread=1 sequence=20358archived log file name=/oracle/app/oracle/archivelogs/1_20359_940127349.dbf thread=1 sequence=20359archived log file name=/oracle/app/oracle/archivelogs/1_20360_940127349.dbf thread=1 sequence=20360unable to find archived logarchived log thread=1 sequence=20361RMAN-00571: ===========================================================RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============RMAN-00571: ===========================================================RMAN-03002: failure of recover command at 06/05/2017 22:17:07RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 20361 and starting SCN of 14425991740RMAN>  recover database until scn 14425991740;Starting recover at 05-JUN-17using channel ORA_DISK_1starting media recoveryOracle Error: ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error belowORA-01152: file 1 was not restored from a sufficiently old backup ORA-01110: data file 1: '/home/oradata/powerdes/system01.dbf'media recovery complete, elapsed time: 00:00:01Finished recover at 05-JUN-17RMAN>

最后开始启动传输并且应用归档日志

SQL> alter database recover managed standby database using current logfile disconnect from session;Database altered.SQL> 

4、开始检查主库备库数据一致性

查看归档日志应用情况：

SQL> select sequence#,applied from v$archived_log order by sequence# asc; SEQUENCE# APPLIED---------- ---------     20379 YES     20380 YES     20381 YES     20382 YES     20383 YES     20384 YES     20385 YES     20386 YES     20387 YES     20388 YES     20389 YES SEQUENCE# APPLIED---------- ---------     20390 YES     20391 YES     20392 YES     20393 YES     20394 YES     20395 YES     20396 YES     20397 YES     20398 NO     20399 NO     20400 NO

查看后台alert日志，正在进行应用

Media Recovery Log /oracle/app/oracle/archivelogs/1_20391_940127349.dbfMon Jun 05 22:24:00 2017Media Recovery Log /oracle/app/oracle/archivelogs/1_20392_940127349.dbfMon Jun 05 22:24:10 2017Media Recovery Log /oracle/app/oracle/archivelogs/1_20393_940127349.dbfMon Jun 05 22:24:22 2017Media Recovery Log /oracle/app/oracle/archivelogs/1_20394_940127349.dbfMon Jun 05 22:24:33 2017Media Recovery Log /oracle/app/oracle/archivelogs/1_20395_940127349.dbfMon Jun 05 22:24:47 2017Media Recovery Log /oracle/app/oracle/archivelogs/1_20396_940127349.dbfMon Jun 05 22:25:00 2017Media Recovery Log /oracle/app/oracle/archivelogs/1_20397_940127349.dbf

查看备库主库log信息是保持一致的，如下所示：

SQL> archive log list;Database log mode          Archive ModeAutomatic archival         EnabledArchive destination        /oracle/app/oracle/archivelogsOldest online log sequence     20430Next log sequence to archive   20435Current log sequence           20435SQL> SQL> archive log list;Database log mode          Archive ModeAutomatic archival         EnabledArchive destination        /oracle/app/oracle/archivelogsOldest online log sequence     20430Next log sequence to archive   0Current log sequence           20435SQL>

至此，利用rman备份直接restore、recover的方式恢复了dataguard的standby库。


仔细思考下，这个问题以前一直没有遇到过，磁盘满了后，按照道理来说，清理磁盘后，可以直接恢复得，但是却不能恢复，原因猜测在于启用了实时应用归档日志导致。这样就导致了重新再次传输加载的时候，混乱了。去查看以前的dataguard搭建记录http://blog.csdn.net/mchdba/article/details/71036947，发现有类似的操作记录：

（2）添加current logfile参数，使得应用当前正在读写，还没有完成归档的日志
SQL> alter database recover managedstandby database using current logfile disconnect from session;