ORACLE11g 云上Data Guard环境备库down机恢复实战过程

 

 

 

1、Oracle备库down机

微软云上的oracle备库意外ping不通了，各种报警上来了，联系微软云后台工程师，他们看了下，大概是说文件损坏了，信息如下：

 Checking all file systems.

 [/sbin/fsck.ext4 (1) -- /] fsck.ext4 -a /dev/sda1

 /dev/sda1: clean, 134941/1905008 files, 2416626/7608064 blocks

 [/sbin/fsck.ext4 (1) -- /oracle] fsck.ext4 -a /dev/sdc1

 fsck.ext4: No such file or directory while trying to open /dev/sdc1

 /dev/sdc1:

 The superblock could not be read or does not describe a correct ext2

 filesystem.  If the device is valid and it really contains an ext2

 filesystem (and not swap or ufs or something else), then the superblock

 is corrupt, and you might try running e2fsck with an alternate superblock:

 e2fsck -b 8193 <device> 

[FAILED]

*** An error occurred during the file system check.

*** Dropping you to a shell; the system will reboot

*** when you leave the shell.

Give root password for maintenance

(or type Control-D to continue):

 系统在启动的过程中无法识别/dev/sdc1 的superblock，未能正常启动到login界面，sdc1的superblock损坏。

 

 

 

 

2、尝试界面恢复

给azure云的工程师建议，直接在管理界面关闭，然后再启动，试试看。不知道他们是否采纳了，过了3个小时候后，他们跟我说一句恢复了，oracle备库服务器已经启动起来了。

 

然后我远程登录进去，一看数据文件目录还在，就准备startup，不过startup报错，如下所示

SQL> startup;

ORACLE instance started.

 

Total System Global Area 1.1825E+10 bytes

Fixed Size     2217024 bytes

Variable Size      6106909632 bytes

Database Buffers    5704253440 bytes

Redo Buffers        11849728 bytes

Database mounted.

ORA-10458: standby database requires recovery

ORA-01196: file 1 is inconsistent due to a failed media recovery session

ORA-01110: data file 1: '/home/oradata/powerdes/system01.dbf'

 

 

SQL>

 

 

看到是书籍不一致，所以在启动的时候，尝试介质恢复失败，在system01.dbf这个数据文件上面恢复失败了。

 

 

 

3、启动归档传输

启动归档，必须在mount状态下执行：

SQL> alter database recover managed standby database using current logfile disconnect from session;

 

Database altered.

 

SQL>

 

 

 

 

4、实时观察日志应用

实时观察，查看主库备库归档传输完成：

 

# 主库查看：

SQL> select sequence#,applied from v$archived_log order by sequence#;

......

 SEQUENCE# APPLIED

---------- ---------

     16850 NO

     16850 NO

     16851 NO

     16851 NO

     16852 NO

     16852 NO

     16853 NO

     16853 NO

     16854 NO

     16854 NO

     16855 NO

 

 SEQUENCE# APPLIED

---------- ---------

     16855 NO

 

5600 rows selected.

 

SQL>

 

 

# 备库查看：

SQL> select sequence#,applied from v$archived_log order by sequence#;

......

 

---------- ---------

     16844 NO

     16845 NO

     16846 NO

     16847 NO

     16848 NO

     16849 NO

     16850 NO

     16851 NO

     16852 NO

     16853 NO

     16854 NO

 

 SEQUENCE# APPLIED

---------- ---------

     16855 NO

 

5600 rows selected.

 

SQL>

 

 

 

但是归档传输完成了，只是表面从主库的归档日志已经完全拉到备库了，但是日志应用还没有结束，还在执行中，去alert后台日志就可以看到信息：

Mon Nov 21 19:35:31 2016

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16715_906253421.dbf

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16716_906253421.dbf

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16717_906253421.dbf

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16718_906253421.dbf

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16719_906253421.dbf

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16720_906253421.dbf

Mon Nov 21 19:35:41 2016

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16721_906253421.dbf

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16722_906253421.dbf

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16723_906253421.dbf

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16724_906253421.dbf

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16725_906253421.dbf

Mon Nov 21 19:35:58 2016

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16726_906253421.dbf

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16727_906253421.dbf

Mon Nov 21 19:36:09 2016

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16728_906253421.dbf

 

 

也可以查看dg状态，看到正在应用日志中

SQL> col value for a80

SQL> set linesize 2000

SQL> col name for a50

SQL> col value for a80

SQL> select name,value from v$dataguard_stats;

 

NAME                                                        VALUE

-------------------------------------------------- --------------------------------------------------------------------------------

transport lag                                            +00 00:00:00

apply lag                                          +02 01:23:55

apply finish time                                      +00 00:06:31.000

estimated startup time                                  11

 

SQL>

 

 

 

等日志应用完成后，再查看后台alert log可以看到：

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16853_906253421.dbf

Mon Nov 21 19:50:21 2016

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16854_906253421.dbf

Mon Nov 21 19:50:31 2016

Media Recovery Log /oracle/app/oracle/flash_recovery_area/archivelog1_16855_906253421.dbf

Media Recovery Waiting for thread 1 sequence 16856 (in transit)

Recovery of Online Redo Log: Thread 1 Group 4 Seq 16856 Reading mem 0

  Mem# 0: /home/oradata/powerdes/redo_dg_021.log

 

 

 

5、打开数据库

SQL> alter database open;

alter database open

*

ERROR at line 1:

ORA-10456: cannot open standby database; media recovery session may be in

progress

 

 

SQL> select name,open_mode from v$database;

 

NAME         OPEN_MODE

--------- --------------------

POWERDES  MOUNTED

 

SQL> alter database recover managed standby database cancel;

 

Database altered.

 

SQL> alter database open;

 

Database altered.

 

SQL> alter database recover managed standby database using current logfile disconnect;

 

Database altered.

 

SQL>

 

 

至此，oracle备库完全修复完成，主备数据库开始保持一致保持同步。