Upgrade Oracle RAC 11.2.0.3 to 11.2.0.4 on Linux x86_64

Section 1: upgrade GI from 11.2.0.3 to 11.2.0.4

Step 1:

[grid@rac1 grid]$ runcluvfy.sh stage -pre crsinst -upgrade -n rac1,rac2 -rolling -src_crshome /u01/app/11.2.0/grid -dest_crshome /u01/app/11.2.0.4/grid -dest_version 11.2.0.4.0


Performing pre-checks for cluster services setup 


Checking node reachability...
Node reachability check passed from node "rac1"




Checking user equivalence...
User equivalence check passed for user "grid"


Checking CRS user consistency
CRS user consistency check successful


Checking node connectivity...


Checking hosts config file...


Verification of the hosts config file successful


Check: Node connectivity for interface "eth0"
Node connectivity passed for interface "eth0"
TCP connectivity check passed for subnet "10.0.0.0"




Check: Node connectivity for interface "eth1"
Node connectivity passed for interface "eth1"
TCP connectivity check passed for subnet "192.168.8.0"


Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "10.0.0.0".
Subnet mask consistency check passed for subnet "192.168.8.0".
Subnet mask consistency check passed.


Node connectivity check passed


Checking multicast communication...


Checking subnet "10.0.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "10.0.0.0" for multicast communication with multicast group "230.0.1.0" passed.


Checking subnet "192.168.8.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.8.0" for multicast communication with multicast group "230.0.1.0" passed.


Check of multicast communication passed.


Checking OCR integrity...


OCR integrity check passed


Checking ASMLib configuration.
Check for ASMLib configuration passed.
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "rac2:/u01/app/11.2.0.4/grid,rac2:/tmp"
Free disk space check passed for "rac1:/u01/app/11.2.0.4/grid,rac1:/tmp"
Check for multiple users with UID value 501 passed 
User existence check passed for "grid"
Group existence check passed for "oinstall"
Membership check for user "grid" in group "oinstall" [as Primary] passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
There are no oracle patches required for home "/u01/app/11.2.0/grid".
There are no oracle patches required for home "/u01/app/11.2.0.4/grid".
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "binutils"
Package existence check passed for "compat-libcap1"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check passed for "ksh"
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Check for multiple users with UID value 0 passed 
Current group ID check passed


Starting check for consistency of primary group of root user


Check for consistency of root user's primary group passed
Package existence check passed for "cvuqdisk"


Starting Clock synchronization checks using Network Time Protocol(NTP)...


NTP Configuration file check started...
NTP Configuration file check passed
No NTP Daemons or Services were found to be running
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
rac2,rac1
Clock synchronization check using Network Time Protocol(NTP) failed


Core file name pattern consistency check passed.


User "grid" is not part of "root" group. Check passed
Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes


File "/etc/resolv.conf" does not have both domain and search entries defined
domain entry in file "/etc/resolv.conf" is consistent across nodes
search entry in file "/etc/resolv.conf" is consistent across nodes
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: rac2,rac1


File "/etc/resolv.conf" is not consistent across nodes




UDev attributes check for OCR locations started...
UDev attributes check passed for OCR locations 




UDev attributes check for Voting Disk locations started...
UDev attributes check passed for Voting Disk locations 


Time zone consistency check passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.


Checking Oracle Cluster Voting Disk configuration...


ASM Running check passed. ASM is running on all specified nodes


Oracle Cluster Voting Disk configuration check passed


Clusterware version consistency passed


Starting check for Reverse path filter setting ...


Check for Reverse path filter setting passed


Pre-check for cluster services setup was unsuccessful on all the nodes. 

Step 2:

./runInstaller

*upgrade oracle grid infrastructure or oracle automatic storage managment

installation location : /u01/app/11.2.0.4/grid (must different with 11.2.0.3)

Step 3:

run following script on RAC1&RAC2 

/u01/app/11.2.0.4/grid/rootupgrade.sh 

Performing root user operation for Oracle 11g 


The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.4/grid


Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.4/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer


ASM upgrade has started on first node.




CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.sdb.db' on 'rac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.cvu' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'
CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2676: Start of 'ora.cvu' on 'rac2' succeeded
CRS-2677: Stop of 'ora.sdb.db' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1'
CRS-2677: Stop of 'ora.DATA.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rac2' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac1'
CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'
CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in upstart
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Step 4:

confirm the GI current version:

[grid@rac1 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]

Step 5:

change user grid .bash_profile

Section 2: upgrade database from 11.2.0.3 to 11.2.0.4

Step 1:

comment out the ORACLE_SID ORACLE_BASE ORACLE_HOME in .bash_profile

Step 2:

*install database software only

software location:/u01/app/oracle/product/11.2.0.4/db_1

Step 3:

run following script on RAC1&RAC2:

/u01/app/oracle/product/11.2.4/db_1/root.sh 

Step 4:

run utlu112i.sql before upgrade

[oracle@rac1 ~]$ export ORACLE_SID=sdb1
[oracle@rac1 ~]$ export ORACLE_BASE=/u01/app/oracle
[oracle@rac1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1

[oracle@rac1 ~]$ $ORACLE_HOME/bin/sqlplus "/as sysdba"


SQL*Plus: Release 11.2.0.3.0 Production on Thu Feb 20 16:58:41 2014


Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options


SQL> @/u01/app/oracle/product/11.2.4/db_1/rdbms/admin/utlu112i.sql
Oracle Database 11.2 Pre-Upgrade Information Tool 02-20-2014 16:59:50
Script Version: 11.2.0.4.0 Build: 001
.
**********************************************************************
Database:
**********************************************************************
--> name:          SDB
--> version:       11.2.0.3.0
--> compatible:    11.2.0.0.0
--> blocksize:     8192
--> platform:      Linux x86 64-bit
--> timezone file: V14
.
**********************************************************************
Tablespaces: [make adjustments in the current environment]
**********************************************************************
--> SYSTEM tablespace is adequate for the upgrade.
.... minimum required size: 919 MB
--> SYSAUX tablespace is adequate for the upgrade.
.... minimum required size: 633 MB
--> UNDOTBS1 tablespace is adequate for the upgrade.
.... minimum required size: 400 MB
--> TEMP tablespace is adequate for the upgrade.
.... minimum required size: 60 MB
.
**********************************************************************
Flashback: OFF
**********************************************************************
**********************************************************************
Update Parameters: [Update Oracle Database 11.2 init.ora or spfile]
Note: Pre-upgrade tool was run on a lower version 64-bit database.
**********************************************************************
--> If Target Oracle is 32-Bit, refer here for Update Parameters:
-- No update parameter changes are required.
.


--> If Target Oracle is 64-Bit, refer here for Update Parameters:
-- No update parameter changes are required.
.
**********************************************************************
Renamed Parameters: [Update Oracle Database 11.2 init.ora or spfile]
**********************************************************************
-- No renamed parameters found. No changes are required.
.
**********************************************************************
Obsolete/Deprecated Parameters: [Update Oracle Database 11.2 init.ora or spfile]
**********************************************************************
-- No obsolete parameters found. No changes are required
.


**********************************************************************
Components: [The following database components will be upgraded or installed]
**********************************************************************
--> Oracle Catalog Views         [upgrade]  VALID
--> Oracle Packages and Types    [upgrade]  VALID
--> JServer JAVA Virtual Machine [upgrade]  VALID
--> Oracle XDK for Java          [upgrade]  VALID
--> Real Application Clusters    [upgrade]  VALID
--> Oracle Workspace Manager     [upgrade]  VALID
--> OLAP Analytic Workspace      [upgrade]  VALID
--> OLAP Catalog                 [upgrade]  VALID
--> EM Repository                [upgrade]  VALID
--> Oracle Text                  [upgrade]  VALID
--> Oracle XML Database          [upgrade]  VALID
--> Oracle Java Packages         [upgrade]  VALID
--> Oracle interMedia            [upgrade]  VALID
--> Spatial                      [upgrade]  VALID
--> Expression Filter            [upgrade]  VALID
--> Rule Manager                 [upgrade]  VALID
--> Oracle Application Express   [upgrade]  VALID
... APEX will only be upgraded if the version of APEX in
... the target Oracle home is higher than the current one.
--> Oracle OLAP API              [upgrade]  VALID
.
**********************************************************************
Miscellaneous Warnings
**********************************************************************
WARNING: --> The "cluster_database" parameter is currently "TRUE"
.... and must be set to "FALSE" prior to running a manual upgrade.
WARNING: --> Your recycle bin is turned on and currently contains no objects.
.... Because it is REQUIRED that the recycle bin be empty prior to upgrading
.... and your recycle bin is turned on, you may need to execute the command:
        PURGE DBA_RECYCLEBIN
.... prior to executing your upgrade to confirm the recycle bin is empty.
WARNING: --> Database contains schemas with objects dependent on DBMS_LDAP package.
.... Refer to the 11g Upgrade Guide for instructions to configure Network ACLs.
.... USER APEX_030200 has dependent objects.
.
**********************************************************************
Recommendations
**********************************************************************
Oracle recommends gathering dictionary statistics prior to
upgrading the database.
To gather dictionary statistics execute the following command
while connected as SYSDBA:


    EXECUTE dbms_stats.gather_dictionary_stats;


**********************************************************************
Oracle recommends reviewing any defined events prior to upgrading.


To view existing non-default events execute the following commands
while connected AS SYSDBA:
  Events:
    SELECT (translate(value,chr(13)||chr(10),' ')) FROM sys.v$parameter2
      WHERE  UPPER(name) ='EVENT' AND  isdefault='FALSE'


  Trace Events:
    SELECT (translate(value,chr(13)||chr(10),' ')) from sys.v$parameter2
      WHERE UPPER(name) = '_TRACE_EVENTS' AND isdefault='FALSE'


Changes will need to be made in the init.ora or spfile.


**********************************************************************

Step 5:

upgrade database by DBUA

/u01/app/oracle/product/11.2.4/db_1/bin/dbua

Step 6:

do some verify

SQL> run
  1* select comp_name,version,status,schema from dba_registry


COMP_NAME                 VERSION         STATUS     SCHEMA
------------------------- --------------- ---------- ----------
OWB                       11.2.0.3.0      VALID      OWBSYS
Oracle Application Expres 3.2.1.00.12     VALID      APEX_03020
s                                                    0


Oracle Enterprise Manager 11.2.0.4.0      VALID      SYSMAN
OLAP Catalog              11.2.0.4.0      VALID      OLAPSYS
Spatial                   11.2.0.4.0      VALID      MDSYS
Oracle Multimedia         11.2.0.4.0      VALID      ORDSYS
Oracle XML Database       11.2.0.4.0      VALID      XDB
Oracle Text               11.2.0.4.0      VALID      CTXSYS
Oracle Expression Filter  11.2.0.4.0      VALID      EXFSYS


COMP_NAME                 VERSION         STATUS     SCHEMA
------------------------- --------------- ---------- ----------
Oracle Rules Manager      11.2.0.4.0      VALID      EXFSYS
Oracle Workspace Manager  11.2.0.4.0      VALID      WMSYS
Oracle Database Catalog V 11.2.0.4.0      VALID      SYS
iews


Oracle Database Packages  11.2.0.4.0      VALID      SYS
and Types


JServer JAVA Virtual Mach 11.2.0.4.0      VALID      SYS
ine




COMP_NAME                 VERSION         STATUS     SCHEMA
------------------------- --------------- ---------- ----------
Oracle XDK                11.2.0.4.0      VALID      SYS
Oracle Database Java Pack 11.2.0.4.0      VALID      SYS
ages


OLAP Analytic Workspace   11.2.0.4.0      VALID      SYS
Oracle OLAP API           11.2.0.4.0      VALID      SYS
Oracle Real Application C 11.2.0.4.0      VALID      SYS
lusters




19 rows selected.


SQL> 
SQL> select owner,object_name,object_type,status from dba_objects where status<>'VALID';


no rows selected

refer doc:

http://www.oracle.com/technetwork/cn/articles/chan-upgrade-084454-zhs.html

https://blogs.oracle.com/Database4CN/entry/%E5%A6%82%E4%BD%95%E5%8D%87%E7%BA%A7oracle_grid_infrastructure%E5%92%8Crac%E4%BB%8E11_2_0

http://docs.oracle.com/cd/E11882_01/install.112/e41961/procstop.htm#CEGGHCJG

 ------------------

11.2.0.4 runInstaller: [INS-06006] Passwordless SSH connectivity not set up between the following nodes(s) (文档 ID 1597212.1)

check /u01/app/oraInventory/Contentsxml to verify if the node name has been updated.