TSM备份Oracle报错（rman-06012,ANR1401W）

问题：

Oracle归档日志备份成功，但2条channel有一个报错。

环境：

TSM服务器：2个Oracle node，4个DB2 node，共享8个drive。其中，两个oracle node分别共享其中4个drive。
报错的oracle node为RFIDP。

tsm: TSM>q pathSession established with server TSM: Linux/s390x  Server Version 6, Release 2, Level 4.0  Server date/time: 09/28/2015 14:26:33  Last access: 09/28/2015 13:06:41Source Name     Source Type     Destination     Destination     On-Line                                Name            Type            -----------     -----------     -----------     -----------     -------TSM             SERVER          TS3500LIB1      LIBRARY         Yes    TSM             SERVER          DRIVER10        DRIVE           Yes    TSM             SERVER          DRIVER11        DRIVE           Yes    TSM             SERVER          DRIVER12        DRIVE           Yes    TSM             SERVER          DRIVER13        DRIVE           Yes    TSM             SERVER          DRIVER14        DRIVE           Yes    TSM             SERVER          DRIVER15        DRIVE           Yes    TSM             SERVER          DRIVER16        DRIVE           Yes    TSM             SERVER          DRIVER17        DRIVE           Yes    JTFXPSTA1       SERVER          DRIVER14        DRIVE           Yes    JTFXPSTA1       SERVER          DRIVER15        DRIVE           Yes    JTFXPSTA1       SERVER          DRIVER16        DRIVE           Yes    JTFXPSTA1       SERVER          DRIVER17        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER14        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER15        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER16        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER17        DRIVE           Yes    

排查过程：

检查TSM actlog

q actlog search=rfidp begint=11:00

从日志中发现， 有磁带mount失败。

进一步检查磁带mount失败的原因：

q actlog search=nj0174 begint=11:00

从日志中看到，driver14和driver16都无法mount这个磁带。

那么问题是在磁带上还是driver上呢？

首先检查改磁带状态：

tsm: TSM>q vol nj0174 f=d                   Volume Name: NJ0174             Storage Pool Name: ORACLE_POOL             Device Class Name: 3592DEVC            Estimated Capacity: 12.0 T       Scaled Capacity Applied: 100                      Pct Util: 22.9                 Volume Status: Filling                        Access: Read/Write        Pct. Reclaimable Space: 0.0               Scratch Volume?: Yes               In Error State?: No      Number of Writable Sides: 1       Number of Times Mounted: 138             Write Pass Number: 1     Approx. Date Last Written: 09/28/2015 13:50:38        Approx. Date Last Read: 08/10/2015 01:10:30           Date Became Pending:         Number of Write Errors: 0         Number of Read Errors: 0               Volume Location: Volume is MVS Lanfree Capable : NoLast Update by (administrator):          Last Update Date/Time: 08/10/2015 01:07:44          Begin Reclaim Period:             End Reclaim Period:   Drive Encryption Key Manager: None       Logical Block Protected: Notsm: TSM>

从上面的输出可以看到磁带状态正常。那么问题有可能在drive上。

继续查看actlog，关于driver14和driver16的记录：

tsm: TSM>q actlog search=driver14 begind=-1

tsm: TSM>q actlog search=driver16 begind=-1

这两个drive一直都无法mount磁带。问题可以确定就出在这两个drive上。

检查操作系统，带库所有drive状态。(drive过多，省略了若干)
发现driver14正常，driver16有一条路径是offline。

[root@TSM /]# lstapeFICON/ESCON tapes (found 0):TapeNo  BusID      CuType/Model DevType/Model   BlkSize State   Op      MedState...                  sg54    IBMtape13     31           13:0:1:1 IBM              03592E07 tapedrvrunning                                                               sg55    IBMtape14     32           14:0:0:1 IBM              03592E07 tapedrvrunning                                                               sg56    IBMchanger1   3            14:0:1:1 IBM              03584L22 changerrunning                                                               sg57    IBMtape15     33           14:0:2:1 IBM              03592E07 tapedrvrunning                                                               sg58    IBMtape16     34           15:0:0:1 IBM              03592E07 tapedrvrunning                                                               sg59    IBMtape17     35           15:0:1:1 IBM              03592E07 tapedrvrunning                                                               ...                 sg35    IBMtape14     32           8:0:0:1  IBM              03592E07 tapedrvrunning                                                               sg36    IBMchanger1   3            8:0:1:1  IBM              03584L22 changerrunning                                                               sg37    IBMtape15     33           8:0:2:1  IBM              03592E07 tapedrvrunning                                                               sg38    N/A           9:0:0:1      IBM      03592E07         tapedrv  offlinesg39    IBMtape17     35           9:0:1:1  IBM              03592E07 tapedrvrunning                                                               [root@TSM /]# 

登陆带库web控制台检查drive状态：

两个红框中的drive就是driver14和driver16。表面上看，状态正常。

对这两个drive做一次power cycle试试。

power cycle成功，在TSM server操作系统中，再次查看drive设备状态：driver16之前offline的路径已经变成running状态。

测试：

rman备份发起2个channel，该数据库节点在TSM server中配置了4条path，手动offline 2条，这样可以测试driver14和driver16的path。

tsm: TSM>q pathSession established with server TSM: Linux/s390x  Server Version 6, Release 2, Level 4.0  Server date/time: 09/28/2015 14:26:33  Last access: 09/28/2015 13:06:41Source Name     Source Type     Destination     Destination     On-Line                                Name            Type            -----------     -----------     -----------     -----------     -------...RFIDPSTA1       SERVER          DRIVER14        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER15        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER16        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER17        DRIVE           Yes  

手动offline driver15和driver17

update path RFIDPSTA1 driver15 srctype=server  desttype=drive DEVICE=/dev/IBMtape1 LIBRARY=ts3500lib1 online=noupdate path RFIDPSTA1 driver17 srctype=server  desttype=drive DEVICE=/dev/IBMtape3 LIBRARY=ts3500lib1 online=notsm: TSM>update path RFIDPSTA1 driver15 srctype=server  desttype=drive DEVICE=/dev/IBMtape1 LIBRARY=ts3500lib1 online=noANR1726E A operation using the path from RFIDPSTA1 to TS3500LIB1 DRIVER15 is currently being used.ANS8001I Return code 12.

driver15 in use，需要手动dismount磁带出来再进行offline

tsm: TSM>q moANR8329I 3592 volume NJ0100 is mounted R/W in drive DRIVER10 (/dev/IBMtape10), status: IDLE.ANR8329I 3592 volume NJ0080 is mounted R/W in drive DRIVER17 (/dev/IBMtape17), status: IDLE.ANR8329I 3592 volume NJ0054 is mounted R/W in drive DRIVER13 (/dev/IBMtape13), status: IDLE.ANR8329I 3592 volume NJ0034 is mounted R/W in drive DRIVER12 (/dev/IBMtape12), status: IDLE.ANR8329I 3592 volume NJ0035 is mounted R/W in drive DRIVER15 (/dev/IBMtape15), status: IDLE.ANR8334I         5 matches found.tsm: TSM>dismount vol nj0035ANR8499I Command accepted.

手动dismount磁带成功。

再次查看mount状态：

tsm: TSM>q moANR8329I 3592 volume NJ0100 is mounted R/W in drive DRIVER10 (/dev/IBMtape10), status: IDLE.ANR8329I 3592 volume NJ0080 is mounted R/W in drive DRIVER17 (/dev/IBMtape17), status: IDLE.ANR8329I 3592 volume NJ0054 is mounted R/W in drive DRIVER13 (/dev/IBMtape13), status: IDLE.ANR8329I 3592 volume NJ0034 is mounted R/W in drive DRIVER12 (/dev/IBMtape12), status: IDLE.ANR8331I 3592 volume NJ0035 is mounted R/W in drive DRIVER15 (/dev/IBMtape15), status: DISMOUNTING.ANR8334I         5 matches found.

dismount中

tsm: TSM>q moANR8329I 3592 volume NJ0100 is mounted R/W in drive DRIVER10 (/dev/IBMtape10), status: IDLE.ANR8329I 3592 volume NJ0054 is mounted R/W in drive DRIVER13 (/dev/IBMtape13), status: IDLE.ANR8329I 3592 volume NJ0034 is mounted R/W in drive DRIVER12 (/dev/IBMtape12), status: IDLE.ANR8334I         4 matches found.

mount列表中已无driver15

再次手动offline driver15

tsm: TSM>update path RFIDPSTA1 driver15 srctype=server  desttype=drive DEVICE=/dev/IBMtape1 LIBRARY=ts3500lib1 online=noANR1722I A path from RFIDPSTA1 to TS3500LIB1 DRIVER15 has been updated.

offline成功

tsm: TSM>q pathSource Name     Source Type     Destination     Destination     On-Line                                Name            Type            -----------     -----------     -----------     -----------     -------TSM             SERVER          TS3500LIB1      LIBRARY         Yes    TSM             SERVER          DRIVER10        DRIVE           Yes    TSM             SERVER          DRIVER11        DRIVE           Yes    TSM             SERVER          DRIVER12        DRIVE           Yes    TSM             SERVER          DRIVER13        DRIVE           Yes    TSM             SERVER          DRIVER14        DRIVE           Yes    TSM             SERVER          DRIVER15        DRIVE           Yes    TSM             SERVER          DRIVER16        DRIVE           Yes    TSM             SERVER          DRIVER17        DRIVE           Yes    JTFXPSTA1       SERVER          DRIVER14        DRIVE           Yes    JTFXPSTA1       SERVER          DRIVER15        DRIVE           Yes    JTFXPSTA1       SERVER          DRIVER16        DRIVE           Yes    JTFXPSTA1       SERVER          DRIVER17        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER14        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER15        DRIVE           No    RFIDPSTA1       SERVER          DRIVER16        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER17        DRIVE           No    

重新运行归档日志备份脚本，测试driver14和driver16是否可用。

查看磁带mount情况，可以看到driver14和driver16已成功mount磁带，并且状态为R/W in use，说明备份正在写入磁带中。可以确定这两个drive已经可以正常使用了。

tsm: TSM>tsm: TSM>q moANR8329I 3592 volume NJ0100 is mounted R/W in drive DRIVER10 (/dev/IBMtape10), status: IDLE.ANR8329I 3592 volume NJ0080 is mounted R/W in drive DRIVER13 (/dev/IBMtape13), status: IDLE.ANR8330I 3592 volume NJ0035 is mounted R/W in drive DRIVER16 (/dev/IBMtape16), status: IN USE.ANR8329I 3592 volume NJ0034 is mounted R/W in drive DRIVER12 (/dev/IBMtape12), status: IDLE.ANR8330I 3592 volume NJ0174 is mounted R/W in drive DRIVER14 (/dev/IBMtape14), status: IN USE.ANR8334I         6 matches found.

继续等待，直到脚本执行完毕，检查脚本日志。备份成功，无报错。

[oracle@RFIDP log]$ tail  /oracle/rmanscript/log/arch_092813.logarchive log filename=/oradata/arch/1_166187_777424034.dbf recid=166178 stamp=891610878archive log filename=/oradata/arch/1_166188_777424034.dbf recid=166179 stamp=891611321archive log filename=/oradata/arch/1_166189_777424034.dbf recid=166180 stamp=891611495Finished backup at 28-SEP-15released channel: t1released channel: t2Recovery Manager complete.

最后不要忘记把刚才offline的path重新online。

tsm: TSM>update path RFIDPSTA1 driver15 srctype=server  desttype=drive DEVICE=/dev/IBMtape1 LIBRARY=ts3500lib1 online=yesANR1722I A path from RFIDPSTA1 to TS3500LIB1 DRIVER15 has been updated.tsm: TSM>update path RFIDPSTA1 driver17 srctype=server  desttype=drive DEVICE=/dev/IBMtape3 LIBRARY=ts3500lib1 online=yesANR1722I A path from RFIDPSTA1 to TS3500LIB1 DRIVER17 has been updated.tsm: TSM>tsm: TSM>q pathSource Name     Source Type     Destination     Destination     On-Line                                Name            Type            -----------     -----------     -----------     -----------     -------TSM             SERVER          TS3500LIB1      LIBRARY         Yes    TSM             SERVER          DRIVER10        DRIVE           Yes    TSM             SERVER          DRIVER11        DRIVE           Yes    TSM             SERVER          DRIVER12        DRIVE           Yes    TSM             SERVER          DRIVER13        DRIVE           Yes    TSM             SERVER          DRIVER14        DRIVE           Yes    TSM             SERVER          DRIVER15        DRIVE           Yes    TSM             SERVER          DRIVER16        DRIVE           Yes    TSM             SERVER          DRIVER17        DRIVE           Yes    JTFXPSTA1       SERVER          DRIVER14        DRIVE           Yes    JTFXPSTA1       SERVER          DRIVER15        DRIVE           Yes    JTFXPSTA1       SERVER          DRIVER16        DRIVE           Yes    JTFXPSTA1       SERVER          DRIVER17        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER14        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER15        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER16        DRIVE           Yes    RFIDPSTA1       SERVER          DRIVER17        DRIVE           Yes