Oracle 11.2.0.4.5 RAC升级OJVM_11.2.0.4.160419_psu出现的问题

以下报错仅限RAC环境，单机非ASM环境下安装此类psu不存在如下问题。

单机ASM环境待测试。。。

环境：

# cat /etc/issue
Red Hat Enterprise Linux Server release 6.5 (Santiago)
Kernel \r on an \m

SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE    11.2.0.4.0      Production
TNS for Linux: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production

Oracle 11.2.0.4.5 RAC升级OJVM_11.2.0.4.160419_psu_p22674697_112040_Linux-x86-64，执行完opatch appply后，启动数据库报错如下：
SQL> startup
ORA-01565: error in identifying file '+DATA/orcl/spfileorcl.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/orcl/spfileorcl.ora
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15040: diskgroup is incomplete


数据库alert日志记录如下：
lmon registered with NM - instance number 2 (internal mem no 1)
Reconfiguration started (old inc 0, new inc 2)
List of instances:
 2 (myinst: 2) 
 Global Resource Directory frozen
* allocate domain 0, invalid = TRUE 
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info 
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Post SMON to start 1st pass IR
 Submitted all GCS remote-cache requests
 Post SMON to start 1st pass IR
 Fix write in gcs resources
Reconfiguration complete
Tue May 10 06:15:54 2016
LCK0 started with pid=30, OS id=14830 
Starting background process RSMN
Tue May 10 06:15:54 2016
RSMN started with pid=31, OS id=14832 
ORACLE_BASE from environment = /u02/app/oracle
NOTE: Loaded library: System 
ORA-15025: could not open disk "/dev/raw/raw4"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
Tue May 10 06:15:56 2016
SUCCESS: diskgroup DATA was dismounted
ERROR: diskgroup DATA was not mounted
ORA-01565: Unable to open Spfile +DATA/orcl/spfileorcl.ora.
Tue May 10 06:15:56 2016
ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.121)(PORT=1521))' SCOPE=MEMORY SID='orcl2';
USER (ospid: 14743): terminating the instance due to error 1565
Instance terminated by USER, pid = 14743

查询磁盘权限及ASM磁盘组挂载情况，均正常，如下：
[root@rac2 bin]# ll /dev/raw/raw*
crw-rw---- 1 grid asmadmin 162, 1 May 10 06:18 /dev/raw/raw1
crw-rw---- 1 grid asmadmin 162, 2 May 10 06:18 /dev/raw/raw2
crw-rw---- 1 grid asmadmin 162, 3 May 10 06:18 /dev/raw/raw3
crw-rw---- 1 grid asmadmin 162, 4 May 10 05:50 /dev/raw/raw4
crw-rw---- 1 grid asmadmin 162, 5 May 10 05:50 /dev/raw/raw5
crw-rw---- 1 root disk     162, 0 May 10 05:02 /dev/raw/rawctl
[grid@rac2 ~]$ asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576      8192     6209                0            6209              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576       512       20                0              20              0             N  FRA/
MOUNTED  NORMAL  N         512   4096  1048576      3072     2146             1024             561              0             Y  OCR/
[grid@rac2 ~]$ sqlplus / as sysasm
SQL> select name,state from v$asm_disk;
NAME                           STATE
------------------------------ --------
OCR_0000                       NORMAL
OCR_0002                       NORMAL
DATA_0000                      NORMAL
OCR_0001                       NORMAL
FRA_0000                       NORMAL
SQL> select name,state from v$asm_diskgroup;
NAME                           STATE
------------------------------ -----------
DATA                           MOUNTED
FRA                            MOUNTED
OCR                            MOUNTED

usermod -G oinstall,asmadmin,asmdba,asmoper,dba grid

[oracle@rac2 ~]$ sqlplus / as sysdba
SQL> startup
ORACLE instance started.
Total System Global Area 1152450560 bytes
Fixed Size                  2252584 bytes
Variable Size             436207832 bytes
Database Buffers          704643072 bytes
Redo Buffers                9347072 bytes
Database mounted.
Database opened.


Tue May 10 06:23:34 2016
minact-scn: master continuing after IR
Starting background process GTX0
Tue May 10 06:23:38 2016
GTX0 started with pid=41, OS id=16476 
Starting background process RCBG
Tue May 10 06:23:38 2016
RCBG started with pid=42, OS id=16481 
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Tue May 10 06:23:39 2016
QMNC started with pid=43, OS id=16485 
Tue May 10 06:23:44 2016
Errors in file /u02/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_16499.trc:
ORA-15025: could not open disk "/dev/raw/raw4"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
Errors in file /u02/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_16499.trc:
ORA-15025: could not open disk "/dev/raw/raw4"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 23 logical extent 0 of file 256 in group [1.2811931592] from disk DATA_0000  allocation unit 125 reason error; if possible, will try another mirror side

检查$ORACLE_HOME/bin/oracle文件权限，如下：
[oracle@rac1 bin]$ ll oracle
-rwxr-x--x 1 oracle oinstall 239867342 May 10 05:20 oracle
修改如下：
[oracle@rac1 bin]$ chmod 6751 oracle              
[oracle@rac1 bin]$ ll oracle
-rwsr-s--x 1 oracle oinstall 239867342 May 10 05:20 oracle

[oracle@rac1 trace]$ id
uid=1200(oracle) gid=1000(oinstall) groups=1000(oinstall),1100(asmadmin),1200(dba),1201(oper),1300(asmdba)
[root@rac1 ~]# id grid
uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1100(asmadmin),1200(dba),1300(asmdba),1301(asmoper)

[oracle@rac1 ~]$ ll $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 oracle oinstall 239867342 May 10 05:53 /u02/app/oracle/product/11.2.0_home/bin/oracle
[grid@rac1 ~]$ ll $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 grid oinstall 209914471 Apr 29 01:55 /u01/app/11.2.0/grid/bin/oracle

[oracle@rac2 trace]$ ll $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 oracle oinstall 239867342 May 10 05:55 /u02/app/oracle/product/11.2.0_home/bin/oracle
[grid@rac2 ~]$ ll $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 grid oinstall 209914471 Apr 29 01:58 /u01/app/11.2.0/grid/bin/oracle


#################################################
对比正常的RAC信息，如下：
[root@normal_rac1 ~]# id oracle
uid=1102(oracle) gid=1000(oinstall) groups=1000(oinstall),1201(asmadmin),1202(asmdba),1301(dba),1302(oper)
[root@normal_rac1 ~]# id grid
uid=1101(grid) gid=1000(oinstall) groups=1000(oinstall),1201(asmadmin),1202(asmdba),1203(asmoper),1301(dba)


[grid@normal_rac1 ~]$ ll $ORACLE_HOME/bin/oracle
-rwsr-s--x. 1 grid oinstall 209914471 Apr 11 15:39 /u01/app/11.2.0/grid/bin/oracle*
[oracle@normal_rac1 ~]$ ll $ORACLE_HOME/bin/oracle
-rwsr-s--x. 1 oracle asmadmin 239815157 Apr 11 19:15 /u01/app/oracle/product/11.2.0_1/bin/oracle*
#################################################

参考Database Will NotMount: ORA-15025, ORA-27041, 'Permission denied', ORA-15081 (文档 ID 1378747.1)，执行如下：
[grid@rac1 bin]$ ./setasmgidwrap o=/u02/app/oracle/product/11.2.0home_1/bin/oracle
###该步骤相当于执行chmod 6751 oracle和chown oracle:asmadmin oracle
[oracle@rac2 trace]$ ll $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 oracle asmadmin 239867342 May 10 05:55 /u02/app/oracle/product/11.2.0_home/bin/oracle

至此startup正常。


#################################################
#################################################
再次验证：
[root@rac2 ~]# cd /u02/app/oracle/product/11.2.0_home/bin/
执行opatch apply前及执行opatch apply过程中：
[root@rac2 bin]# ll oracle
-rwsr-s--x 1 oracle asmadmin 239789549 May 10 05:39 oracle
-rwsr-s--x 1 oracle asmadmin 239789549 May 10 05:39 oracle
[root@rac2 bin]# ll oracle
-rwsr-s--x 1 oracle asmadmin 239789549 May 10 05:39 oracle
执行opatch apply后：
[root@rac2 bin]# ll oracle
-rwsr-s--x 1 oracle oinstall 239867342 May 10 06:31 oracle
[root@rac2 bin]# ll oracle
-rwsr-s--x 1 oracle oinstall 239867342 May 10 06:31 oracle


解决办法参考：
Database Will NotMount: ORA-15025, ORA-27041, 'Permission denied', ORA-15081 (文档 ID 1378747.1)
[grid@rac1 bin]$ ./setasmgidwrap o=/u02/app/oracle/product/11.2.0home_1/bin/oracle


同样在执行psu回退操作时，同样存在该问题。

总结：安装11.2.0.4.8以后带有11.2.0.4.yymmdd格式的psu时，执行完opatch apply后oracle软件安装路径下的$ORACLE_HOME/bin/oracle文件（非grid软件安装路径下的$ORACLE_HOME/bin/oracle）属组会改变（此前RAC安装11.2.0.4.160119也出现此问题：http://blog.csdn.net/lk_db/article/details/50962910），解决方案参考（文档 ID 1378747.1）。