DataGuard主备服务器网络暂时中断

在为客户进行巡检时发现在上周五DataGuard发生过异常。
客户环境为Oracle 10.2.0.5 RAC，单实例备库，默认最大性能模式，实时应用。主备库服务器不在同一物理地点，之间为专线联接，
具体描述为：
查询主备库的v$archived_log中显示归档日志正常，主备库归档一致，全部正常应用。
查询主备库的v$archive_gap中无任何记录，无丢失的归档。
查询主备库的v$managed_standby无异常，正常启用实时应用。

查询主库的v$dataguard_status发现有错误记录：

SQL> select facility,severity,timestamp,message from v$dataguard_status where severity = 'Error';FACILITY                 SEVERITY      TIMESTAMP           MESSAGE------------------------ ------------- ------------------- --------------------------------------------------------------------------------Log Transport Services   Error         2013-03-22 15:22:48 LGWR: I/O error 3135 archiving log 1 to 'ORCLBK'Log Transport Services   Error         2013-03-22 15:23:27 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 15:23:27 ARC0: Error 12571 attaching to RFS for reconnectLog Transport Services   Error         2013-03-22 15:23:27 PING[ARC0]: Error 3113 when pinging standby ORCLBK.Log Transport Services   Error         2013-03-22 15:32:12 Error 12170 received logging on to the standbyLog Transport Services   Error         2013-03-22 15:32:12 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12170.Log Transport Services   Error         2013-03-22 15:38:28 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 15:38:28 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12571.Log Transport Services   Error         2013-03-22 15:47:13 Error 12170 received logging on to the standbyLog Transport Services   Error         2013-03-22 15:47:13 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12170.Log Transport Services   Error         2013-03-22 15:53:28 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 15:53:28 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12571.Log Transport Services   Error         2013-03-22 16:02:13 Error 12170 received logging on to the standbyLog Transport Services   Error         2013-03-22 16:02:13 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12170.Log Transport Services   Error         2013-03-22 16:08:29 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 16:08:29 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12571.Log Transport Services   Error         2013-03-22 16:17:14 Error 12170 received logging on to the standbyLog Transport Services   Error         2013-03-22 16:17:14 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12170.Log Transport Services   Error         2013-03-22 16:23:29 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 16:23:29 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12571.Log Transport Services   Error         2013-03-22 16:32:14 Error 12170 received logging on to the standbyLog Transport Services   Error         2013-03-22 16:32:14 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12170.Log Transport Services   Error         2013-03-22 16:38:29 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 16:38:29 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12571.Log Transport Services   Error         2013-03-22 16:47:15 Error 12170 received logging on to the standbyLog Transport Services   Error         2013-03-22 16:47:15 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12170.Log Transport Services   Error         2013-03-22 16:53:30 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 16:53:30 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12571.Log Transport Services   Error         2013-03-22 17:02:15 Error 12170 received logging on to the standbyLog Transport Services   Error         2013-03-22 17:02:15 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12170.Log Transport Services   Error         2013-03-22 17:08:30 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 17:08:30 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12571.Log Transport Services   Error         2013-03-22 17:17:15 Error 12170 received logging on to the standbyLog Transport Services   Error         2013-03-22 17:17:15 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12170.Log Transport Services   Error         2013-03-22 17:23:31 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 17:23:31 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12571.Log Transport Services   Error         2013-03-22 17:32:16 Error 12170 received logging on to the standbyLog Transport Services   Error         2013-03-22 17:32:16 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12170.Log Transport Services   Error         2013-03-22 17:38:31 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 17:38:31 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12571.Log Transport Services   Error         2013-03-22 17:47:17 Error 12170 received logging on to the standbyLog Transport Services   Error         2013-03-22 17:47:17 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12170.Log Transport Services   Error         2013-03-22 17:53:32 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 17:53:32 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12571.Log Transport Services   Error         2013-03-22 18:02:17 Error 12170 received logging on to the standbyLog Transport Services   Error         2013-03-22 18:02:17 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12170.Log Transport Services   Error         2013-03-22 18:08:32 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 18:08:32 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12571.Log Transport Services   Error         2013-03-22 18:17:17 Error 12170 received logging on to the standbyLog Transport Services   Error         2013-03-22 18:17:17 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12170.Log Transport Services   Error         2013-03-22 18:23:33 Error 12571 received logging on to the standbyLog Transport Services   Error         2013-03-22 18:23:33 PING[ARC0]: Heartbeat failed to connect to standby 'ORCLBK'. Error is 12571.已选择52行。

说明当时主库向备库传输归档日志时，因无法连接到备库而产生错误。而现在主备库的归档日志已经一致且都正常应用，说明当时主备库之间的连接是暂时断开，之后又恢复了
检查备库的告警日志，发现如下信息：

Fri Mar 22 14:09:40 CST 2013Media Recovery Waiting for thread 2 sequence 8737 (in transit)Fri Mar 22 14:09:40 CST 2013Recovery of Online Redo Log: Thread 2 Group 16 Seq 8737 Reading mem 0  Mem# 0: /export/home/oracle/oradata/orclbk/onlinelog/group_16.968.792854599Fri Mar 22 17:19:06 CST 2013RFS[13]: Possible network disconnect with primary databaseFri Mar 22 17:19:20 CST 2013RFS[14]: Possible network disconnect with primary databaseFri Mar 22 17:19:52 CST 2013RFS[10]: Possible network disconnect with primary databaseFri Mar 22 17:19:54 CST 2013RFS[12]: Possible network disconnect with primary databaseFri Mar 22 18:26:03 CST 2013Redo Shipping Client Connected as PUBLIC-- Connected User is ValidRFS[15]: Assigned to RFS process 10111RFS[15]: Identified database type as 'physical standby'Fri Mar 22 18:26:13 CST 2013RFS[15]: Archived Log: '/export/home/oracle/oradata/orclbk/archivelog/2_8737_705419065.dbf'RFS[15]: Archived Log: '/export/home/oracle/oradata/orclbk/archivelog/2_8738_705419065.dbf'Fri Mar 22 18:26:25 CST 2013Redo Shipping Client Connected as PUBLIC-- Connected User is ValidRFS[16]: Assigned to RFS process 10113RFS[16]: Identified database type as 'physical standby'Fri Mar 22 18:26:28 CST 2013RFS[15]: Archived Log: '/export/home/oracle/oradata/orclbk/archivelog/2_8739_705419065.dbf'Fri Mar 22 18:26:34 CST 2013RFS[16]: Archived Log: '/export/home/oracle/oradata/orclbk/archivelog/1_9187_705419065.dbf'RFS[16]: Archived Log: '/export/home/oracle/oradata/orclbk/archivelog/1_9188_705419065.dbf'Fri Mar 22 18:26:36 CST 2013RFS[15]: Archived Log: '/export/home/oracle/oradata/orclbk/archivelog/2_8740_705419065.dbf'RFS[15]: Archived Log: '/export/home/oracle/oradata/orclbk/archivelog/2_8741_705419065.dbf'Fri Mar 22 18:26:51 CST 2013RFS[15]: Archived Log: '/export/home/oracle/oradata/orclbk/archivelog/2_8742_705419065.dbf'Fri Mar 22 18:26:57 CST 2013Media Recovery Log /export/home/oracle/oradata/orclbk/archivelog/2_8737_705419065.dbfMedia Recovery Log /export/home/oracle/oradata/orclbk/archivelog/1_9187_705419065.dbfFri Mar 22 18:26:58 CST 2013RFS[15]: Archived Log: '/export/home/oracle/oradata/orclbk/archivelog/2_8743_705419065.dbf'Fri Mar 22 18:27:03 CST 2013Media Recovery Log /export/home/oracle/oradata/orclbk/archivelog/2_8738_705419065.dbfMedia Recovery Log /export/home/oracle/oradata/orclbk/archivelog/2_8739_705419065.dbfMedia Recovery Log /export/home/oracle/oradata/orclbk/archivelog/1_9188_705419065.dbfFri Mar 22 18:27:23 CST 2013Media Recovery Log /export/home/oracle/oradata/orclbk/archivelog/2_8740_705419065.dbfFri Mar 22 18:27:33 CST 2013Redo Shipping Client Connected as PUBLIC-- Connected User is ValidRFS[17]: Assigned to RFS process 10117

可以看到，备库在等待日志“thread 2 sequence 8737”时发现一直未接收到，提示可能发生了网络中断。六点多恢复正常后，自动将未接收的几个归档日志一起传输过来，并正常在备库恢复。
之后询问客户，得知当时确实因运营商维护原因，专线暂时中断。