HP-UX+11.2.0.3RAC因裸设备权限不一致导致RAC一个节点报错的解决

故障情况：
HP-UX+11.2.0.3RAC；未使用ASM使用了裸设备；--这里就不讨论11gR2使用裸设备的坑了，历史问题了。。
在存储划盘过来后，A节点的裸设备权限不正确，B节点权限正确；
然后，将裸设备做为数据文件加入到表空间后，很快，遇到日志切换后开始报错(因为检查点)；
此时，已经排查到磁盘权限问题并修改为正确的权限；但是A节点ALERT日志中仍然在报错。
报错信息：

Mon Jan 11 09:41:10 2016Thread 1 advanced to log sequence 169313 (LGWR switch)  Current log# 1 seq# 169313 mem# 0: /dev/vgpAAAdb11/rrredo_11_2048Mon Jan 11 09:41:20 2016Archived Log entry 240885 added for thread 1 sequence 169312 ID 0x360aa448 dest 1:Mon Jan 11 09:42:08 2016Errors in file /app/oracle/diag/rdbms/p1AAAdb/P1AAADB1/trace/P1AAADB1_m002_8386.trcORA-01157: cannot identify/lock data file 743 - see DBWR trace fileORA-01110: data file 743: '/dev/vgpAAAdb17/rLV_AAA_L_D01'Errors in file /app/oracle/diag/rdbms/p1AAAdb/P1AAADB1/trace/P1AAADB1_m002_8386.trc:ORA-01157: cannot identify/lock data file 744 - see DBWR trace fileORA-01110: data file 744: '/dev/vgpAAAdb18/rLV_AAA_M_D01'Mon Jan 11 09:42:28 2016Thread 1 advanced to log sequence 169314 (LGWR switch)  Current log# 2 seq# 169314 mem# 0: /dev/vgpAAAdb11/rrredo_12_2048Mon Jan 11 09:42:46 2016

解决方法：
此时需要在问题节点上将报错的数据文件OFFLINE，然后再对数据文件进行RECOVER；然后重新ONLINE数据文件即可；
未报错的节点上可以不操作；
---------
解决过程：

SQL> select file_name,status from dba_data_files where file_id=743;select file_name,status from dba_data_files where file_id=743                             *ERROR at line 1:ORA-01157: cannot identify/lock data file 743 - see DBWR trace fileORA-01110: data file 743: '/dev/vgpAAAdb17/rLV_AAA_L_D01'

----offline操作

SQL> alter database datafile '/dev/vgpAAAdb17/rLV_AAA_L_D01' offline;Database altered.SQL> alter database datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01' offline;Database altered.----online操作SQL> alter database datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01' online;alter database datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01' online*ERROR at line 1:ORA-01113: file 744 needs media recoveryORA-01110: data file 744: '/dev/vgpAAAdb18/rLV_AAA_M_D01'

----recover数据文件操作

SQL> recover datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01';Media recovery complete.SQL> alter database datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01' online;Database altered.SQL> alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' online;alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' online*ERROR at line 1:ORA-01113: file 743 needs media recoveryORA-01110: data file 743: '/dev/vgpAAAdb17/rLV_AAA_L_D01'SQL> recover datafile '/dev/vgpAAAdb17/rLV_AAA_L_D01';Media recovery complete.SQL> alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' online;Database altered.

----进行检查点及日志切换操作，均可以正常进行。--稍后业务部门对表等数据查询验证均正常。

SQL> alter system checkpoint;System altered.SQL> alter system switch logfile;System altered.SQL> select file_name,status from dba_data_files where file_id=743;FILE_NAME--------------------------------------------------------------------------------STATUS---------/dev/vgpAAAdb17/rLV_AAA_L_D01AVAILABLESQL> select file_name,status from dba_data_files where file_id=744;FILE_NAME--------------------------------------------------------------------------------STATUS---------/dev/vgpAAAdb18/rLV_AAA_M_D01AVAILABLE

------------------
对应的ALERT日志：

Mon Jan 11 10:01:56 2016Recovery of Online Redo Log: Thread 2 Group 15 Seq 71551 Reading mem 0  Mem# 0: /dev/vgpAAAdb11/rrredo_27_2048Mon Jan 11 10:02:08 2016Recovery of Online Redo Log: Thread 1 Group 6 Seq 169318 Reading mem 0  Mem# 0: /dev/vgpAAAdb11/rrredo_16_2048Mon Jan 11 10:02:09 2016Errors in file /app/oracle/diag/rdbms/p1AAAdb/P1AAADB1/trace/P1AAADB1_m005_11643.trc:ORA-01135: file 743 accessed for DML/query is offlineORA-01110: data file 743: '/dev/vgpAAAdb17/rLV_AAA_L_D01'Errors in file /app/oracle/diag/rdbms/p1AAAdb/P1AAADB1/trace/P1AAADB1_m005_11643.trc:ORA-01135: file 744 accessed for DML/query is offlineORA-01110: data file 744: '/dev/vgpAAAdb18/rLV_AAA_M_D01'Mon Jan 11 10:02:22 2016Recovery of Online Redo Log: Thread 1 Group 7 Seq 169319 Reading mem 0  Mem# 0: /dev/vgpAAAdb11/rrredo_17_2048Media Recovery Complete (P1AAADB1)Completed: ALTER DATABASE RECOVER  datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01'  Mon Jan 11 10:02:37 2016alter database datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01' onlineCompleted: alter database datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01' onlineMon Jan 11 10:03:00 2016alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' onlineORA-1113 signalled during: alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' online...Mon Jan 11 10:03:14 2016ALTER DATABASE RECOVER  datafile '/dev/vgpAAAdb17/rLV_AAA_L_D01'  Media Recovery StartSerial Media Recovery startedRecovery of Online Redo Log: Thread 1 Group 4 Seq 169316 Reading mem 0  Mem# 0: /dev/vgpAAAdb11/rrredo_14_2048Recovery of Online Redo Log: Thread 2 Group 14 Seq 71550 Reading mem 0  Mem# 0: /dev/vgpAAAdb11/rrredo_26_2048Recovery of Online Redo Log: Thread 1 Group 5 Seq 169317 Reading mem 0  Mem# 0: /dev/vgpAAAdb11/rrredo_15_2048Recovery of Online Redo Log: Thread 2 Group 15 Seq 71551 Reading mem 0  Mem# 0: /dev/vgpAAAdb11/rrredo_27_2048Mon Jan 11 10:03:32 2016Recovery of Online Redo Log: Thread 1 Group 6 Seq 169318 Reading mem 0  Mem# 0: /dev/vgpAAAdb11/rrredo_16_2048Mon Jan 11 10:03:45 2016Recovery of Online Redo Log: Thread 1 Group 7 Seq 169319 Reading mem 0  Mem# 0: /dev/vgpAAAdb11/rrredo_17_2048Media Recovery Complete (P1AAADB1)Completed: ALTER DATABASE RECOVER  datafile '/dev/vgpAAAdb17/rLV_AAA_L_D01'  Mon Jan 11 10:03:58 2016alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' onlineCompleted: alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' onlineMon Jan 11 10:04:42 2016Thread 1 advanced to log sequence 169320 (LGWR switch)  Current log# 8 seq# 169320 mem# 0: /dev/vgpAAAdb11/rrredo_18_2048