Last login: Mon Mar 11 01:30:25 2013 from 196.128.1.234[root@node1 ~]# [root@node1 ~]# [root@node1 ~]# uname -aLinux node1 2.6.32-200.13.1.el5uek #1 SMP Wed Jul 27 21:02:33 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux[root@node1 ~]# [root@node1 ~]# dateMon Mar 11 07:17:35 CST 2013[root@node1 ~]# which crsctl/oracle/app/grid/product/11.2.0/bin/crsctl[root@node1 ~]# /oracle/app/grid/product/11.2.0/bin/crsctl stop cluster -allCRS-2673: Attempting to stop 'ora.crsd' on 'node1'CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node1'CRS-2673: Attempting to stop 'ora.oc4j' on 'node1'CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node1'CRS-2673: Attempting to stop 'ora.racdb.racdb1srv.svc' on 'node1'CRS-2673: Attempting to stop 'ora.racdb.racdb2srv.svc' on 'node1'CRS-2673: Attempting to stop 'ora.racdb.racdbsrv.svc' on 'node1'CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'node1'CRS-2673: Attempting to stop 'ora.cvu' on 'node1'CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node1' succeededCRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'node1' succeededCRS-2673: Attempting to stop 'ora.scan1.vip' on 'node1'CRS-2677: Stop of 'ora.scan1.vip' on 'node1' succeededCRS-2677: Stop of 'ora.racdb.racdb2srv.svc' on 'node1' succeededCRS-2677: Stop of 'ora.racdb.racdb1srv.svc' on 'node1' succeededCRS-2677: Stop of 'ora.racdb.racdbsrv.svc' on 'node1' succeededCRS-2673: Attempting to stop 'ora.node1.vip' on 'node1'CRS-2672: Attempting to start 'ora.racdb.racdb1srv.svc' on 'node2'CRS-2672: Attempting to start 'ora.racdb.racdb2srv.svc' on 'node2'CRS-2673: Attempting to stop 'ora.crsd' on 'node2'CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node2'CRS-2677: Stop of 'ora.node1.vip' on 'node1' succeededCRS-2676: Start of 'ora.racdb.racdb2srv.svc' on 'node2' succeededCRS-2676: Start of 'ora.racdb.racdb1srv.svc' on 'node2' succeededCRS-2673: Attempting to stop 'ora.racdb.db' on 'node1'CRS-2673: Attempting to stop 'ora.racdb.racdb2srv.svc' on 'node2'CRS-2673: Attempting to stop 'ora.racdb.racdb1srv.svc' on 'node2'CRS-2677: Stop of 'ora.racdb.racdb2srv.svc' on 'node2' succeededCRS-2677: Stop of 'ora.racdb.racdb1srv.svc' on 'node2' succeededCRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node2'CRS-2673: Attempting to stop 'ora.racdb.racdbsrv.svc' on 'node2'CRS-2677: Stop of 'ora.racdb.racdbsrv.svc' on 'node2' succeededCRS-2673: Attempting to stop 'ora.racdb.db' on 'node2'CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node2' succeededCRS-2673: Attempting to stop 'ora.node2.vip' on 'node2'CRS-2677: Stop of 'ora.node2.vip' on 'node2' succeededCRS-2677: Stop of 'ora.racdb.db' on 'node2' succeededCRS-2673: Attempting to stop 'ora.DATA.dg' on 'node2'CRS-2677: Stop of 'ora.racdb.db' on 'node1' succeededCRS-2673: Attempting to stop 'ora.DATA.dg' on 'node1'CRS-2677: Stop of 'ora.oc4j' on 'node1' succeededCRS-2677: Stop of 'ora.cvu' on 'node1' succeededCRS-2677: Stop of 'ora.DATA.dg' on 'node2' succeededCRS-2673: Attempting to stop 'ora.asm' on 'node2'CRS-2677: Stop of 'ora.asm' on 'node2' succeededCRS-2673: Attempting to stop 'ora.ons' on 'node2'CRS-2677: Stop of 'ora.ons' on 'node2' succeededCRS-2673: Attempting to stop 'ora.net1.network' on 'node2'CRS-2677: Stop of 'ora.net1.network' on 'node2' succeededCRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node2' has completedCRS-2677: Stop of 'ora.crsd' on 'node2' succeededCRS-2673: Attempting to stop 'ora.ctssd' on 'node2'CRS-2673: Attempting to stop 'ora.evmd' on 'node2'CRS-2673: Attempting to stop 'ora.asm' on 'node2'CRS-2677: Stop of 'ora.evmd' on 'node2' succeededCRS-2677: Stop of 'ora.asm' on 'node2' succeededCRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'node2'CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'node2' succeededCRS-2677: Stop of 'ora.DATA.dg' on 'node1' succeededCRS-2673: Attempting to stop 'ora.asm' on 'node1'CRS-2677: Stop of 'ora.ctssd' on 'node2' succeededCRS-2673: Attempting to stop 'ora.cssd' on 'node2'CRS-2677: Stop of 'ora.asm' on 'node1' succeededCRS-2673: Attempting to stop 'ora.ons' on 'node1'CRS-2677: Stop of 'ora.cssd' on 'node2' succeededCRS-2677: Stop of 'ora.ons' on 'node1' succeededCRS-2673: Attempting to stop 'ora.net1.network' on 'node1'CRS-2677: Stop of 'ora.net1.network' on 'node1' succeededCRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node1' has completedCRS-2677: Stop of 'ora.crsd' on 'node1' succeededCRS-2673: Attempting to stop 'ora.ctssd' on 'node1'CRS-2673: Attempting to stop 'ora.evmd' on 'node1'CRS-2673: Attempting to stop 'ora.asm' on 'node1'CRS-2677: Stop of 'ora.evmd' on 'node1' succeededCRS-2677: Stop of 'ora.asm' on 'node1' succeededCRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'node1'CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'node1' succeededCRS-2677: Stop of 'ora.ctssd' on 'node1' succeededCRS-2673: Attempting to stop 'ora.cssd' on 'node1'CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded[root@node1 ~]# /oracle/app/grid/product/11.2.0/bin/crsctl stop crs -fCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1'CRS-2673: Attempting to stop 'ora.mdnsd' on 'node1'CRS-2673: Attempting to stop 'ora.crf' on 'node1'CRS-2677: Stop of 'ora.crf' on 'node1' succeededCRS-2673: Attempting to stop 'ora.gipcd' on 'node1'CRS-2677: Stop of 'ora.mdnsd' on 'node1' succeededCRS-2677: Stop of 'ora.gipcd' on 'node1' succeededCRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeededCRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completedCRS-4133: Oracle High Availability Services has been stopped.<<< node2[root@node2 ~]# /oracle/app/grid/product/11.2.0/bin/crsctl stop crs -fCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node2'CRS-2673: Attempting to stop 'ora.mdnsd' on 'node2'CRS-2673: Attempting to stop 'ora.crf' on 'node2'CRS-2677: Stop of 'ora.mdnsd' on 'node2' succeededCRS-2677: Stop of 'ora.crf' on 'node2' succeededCRS-2673: Attempting to stop 'ora.gipcd' on 'node2'CRS-2677: Stop of 'ora.gipcd' on 'node2' succeededCRS-2673: Attempting to stop 'ora.gpnpd' on 'node2'CRS-2677: Stop of 'ora.gpnpd' on 'node2' succeededCRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node2' has completedCRS-4133: Oracle High Availability Services has been stopped.[root@node2 ~]# dateMon Mar 11 07:21:39 CST 2013[root@node2 ~]#<<< node1[root@node1 ~]# dateMon Mar 11 07:21:42 CST 2013[root@node1 ~]# [root@node1 ~]# date --set 03/10/13 +%D;date --set 16:37:00 +%T03/10/1316:37:00[root@node1 ~]# dateSun Mar 10 16:37:08 CST 2013[root@node1 ~]# [root@node1 ~]# hwclock --systohc[root@node1 ~]#<<< node2[root@node2 ~]# date --set 03/10/13 +%D;date --set 16:37:00 +%T03/10/1316:37:00[root@node2 ~]# dateSun Mar 10 16:37:01 CST 2013[root@node2 ~]# hwclock --systohc[root@node2 ~]# <<< node1[root@node1 ~]# crsctl start crsCRS-4123: Oracle High Availability Services has been started.[root@node1 ~]# <<< node2[root@node2 ~]# /oracle/app/grid/product/11.2.0/bin/crsctl start crsCRS-4123: Oracle High Availability Services has been started.[root@node2 ~]# [root@node1 ~]# crsctl stat res -t--------------------------------------------------------------------------------NAME TARGET STATE SERVER STATE_DETAILS --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.DATA.dg ONLINE ONLINE node1 ONLINE ONLINE node2 ora.LISTENER.lsnr ONLINE ONLINE node1 ONLINE ONLINE node2 ora.asm ONLINE ONLINE node1 Started ONLINE ONLINE node2 Started ora.gsd OFFLINE OFFLINE node1 OFFLINE OFFLINE node2 ora.net1.network ONLINE ONLINE node1 ONLINE ONLINE node2 ora.ons ONLINE ONLINE node1 ONLINE ONLINE node2 --------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE node2 ora.cvu 1 ONLINE ONLINE node2 ora.node1.vip 1 ONLINE ONLINE node1 ora.node2.vip 1 ONLINE ONLINE node2 ora.oc4j 1 ONLINE ONLINE node2 ora.racdb.db 1 ONLINE ONLINE node1 Open 2 ONLINE ONLINE node2 Open ora.racdb.racdb1srv.svc 1 ONLINE ONLINE node1 ora.racdb.racdb2srv.svc 1 ONLINE ONLINE node1 ora.racdb.racdbsrv.svc 1 ONLINE ONLINE node1 2 ONLINE ONLINE node2 ora.scan1.vip 1 ONLINE ONLINE node2 [root@node1 ~]# Shutting down instance (abort)License high water mark = 16USER (ospid: 8666): terminating the instanceMon Mar 11 07:18:37 2013Instance terminated by USER, pid = 8666Mon Mar 11 07:18:38 2013Instance shutdown completeSun Mar 10 16:42:06 2013Starting ORACLE instance (normal)Specified value of sga_max_size is too small, bumping to 545259520****************** Large Pages Information ***************** Total Shared Global Region in Large Pages = 0 KB (0%) Large Pages used by this instance: 0 (0 KB)Large Pages unused system wide = 0 (0 KB) (alloc incr 4096 KB)Large Pages configured system wide = 0 (0 KB)Large Page size = 2048 KB RECOMMENDATION: Total Shared Global Region size is 522 MB. For optimal performance, prior to the next instance restart increase the number of unused Large Pages by atleast 261 2048 KB Large Pages (522 MB) system wide to get 100% of the Shared Global Region allocated with Large pages***********************************************************LICENSE_MAX_SESSION = 0LICENSE_SESSIONS_WARNING = 0Private Interface 'eth1:1' configured from GPnP for use as a private interconnect. [name='eth1:1', type=1, ip=169.254.87.133, mac=08-00-27-3b-29-cb, net=169.254.0.0/17, mask=255.255.128.0, use=haip:cluster_interconnect/62]Private Interface 'eth2:1' configured from GPnP for use as a private interconnect. [name='eth2:1', type=1, ip=169.254.224.212, mac=08-00-27-bb-f2-79, net=169.254.128.0/17, mask=255.255.128.0, use=haip:cluster_interconnect/62]Public Interface 'eth0' configured from GPnP for use as a public interface. [name='eth0', type=1, ip=196.128.4.101, mac=08-00-27-0d-ed-71, net=196.128.0.0/16, mask=255.255.0.0, use=public/1]Public Interface 'eth0:1' configured from GPnP for use as a public interface. [name='eth0:1', type=1, ip=196.128.4.201, mac=08-00-27-0d-ed-71, net=196.128.0.0/16, mask=255.255.0.0, use=public/1]Picked latch-free SCN scheme 3Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0SYS auditing is disabledStarting up:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, OLAP, Data Miningand Real Application Testing options.ORACLE_HOME = /oracle/app/oracle/product/11.2.0System name: LinuxNode name: node1Release: 2.6.32-200.13.1.el5uekVersion: #1 SMP Wed Jul 27 21:02:33 EDT 2011Machine: x86_64Using parameter settings in server-side pfile /oracle/app/oracle/product/11.2.0/dbs/initracdb1.oraSystem parameters with non-default values: processes = 1000 sessions = 1522 instance_groups = "racdb" instance_groups = "racdb1" sga_max_size = 520M shared_pool_size = 304M large_pool_size = 4M java_pool_size = 4M streams_pool_size = 0 spfile = "+DATA/racdb/spfileracdb.ora" _trace_events = "10000-10999:255:2,20,45" sga_target = 0 memory_target = 0 control_files = "+DATA/racdb/control01.ctl" control_files = "+DATA/racdb/control02.ctl" db_block_size = 8192 db_cache_size = 200M _shared_io_pool_size = 0 compatible = "11.2.0.0.0" log_archive_dest_1 = "location=/home/oracle/arch" db_files = 2000 cluster_database = TRUE thread = 1 undo_tablespace = "UNDOTBS1" instance_number = 1 remote_login_passwordfile= "EXCLUSIVE" db_domain = "" service_names = "dball" service_names = "db1" dispatchers = "(PROTOCOL=TCP) (SERVICE=racdbXDB)" remote_listener = "racscan:1530" parallel_instance_group = "racdb" audit_file_dest = "/oracle/app/oracle/admin/racdb/adump" audit_trail = "DB" db_name = "racdb" open_cursors = 300 pga_aggregate_target = 280M sec_max_failed_login_attempts= 1 diagnostic_dest = "/oracle/app/oracle"Deprecated system parameters with specified values: instance_groups End of deprecated system parameter listingCluster communication is configured to use the following interface(s) for this instance 169.254.87.133 169.254.224.212cluster interconnect IPC version:Oracle UDP/IP (generic)IPC Vendor 1 proto 2Sun Mar 10 16:42:15 2013PMON started with pid=2, OS id=28733 Sun Mar 10 16:42:15 2013PSP0 started with pid=3, OS id=28737 Sun Mar 10 16:42:16 2013VKTM started with pid=4, OS id=28746 at elevated priorityVKTM running at (1)millisec precision with DBRM quantum (100)msSun Mar 10 16:42:16 2013GEN0 started with pid=5, OS id=28753 Sun Mar 10 16:42:16 2013DIAG started with pid=6, OS id=28758 Sun Mar 10 16:42:16 2013DBRM started with pid=7, OS id=28765 Sun Mar 10 16:42:16 2013PING started with pid=8, OS id=28769 Sun Mar 10 16:42:16 2013ACMS started with pid=9, OS id=28773 Sun Mar 10 16:42:16 2013DIA0 started with pid=10, OS id=28777 Sun Mar 10 16:42:16 2013LMON started with pid=11, OS id=28781 Sun Mar 10 16:42:16 2013LMD0 started with pid=12, OS id=28785 * Load Monitor used for high load check * New Low - High Load Threshold Range = [960 - 1280] Sun Mar 10 16:42:16 2013LMS0 started with pid=13, OS id=28789 at elevated prioritySun Mar 10 16:42:16 2013RMS0 started with pid=14, OS id=28795 Sun Mar 10 16:42:16 2013LMHB started with pid=15, OS id=28799 Sun Mar 10 16:42:16 2013MMAN started with pid=16, OS id=28803 Sun Mar 10 16:42:16 2013DBW0 started with pid=17, OS id=28807 Sun Mar 10 16:42:16 2013LGWR started with pid=18, OS id=28811 Sun Mar 10 16:42:16 2013CKPT started with pid=19, OS id=28815 Sun Mar 10 16:42:16 2013SMON started with pid=20, OS id=28819 Sun Mar 10 16:42:16 2013RECO started with pid=21, OS id=28823 Sun Mar 10 16:42:16 2013RBAL started with pid=22, OS id=28827 Sun Mar 10 16:42:17 2013ASMB started with pid=23, OS id=28831 Sun Mar 10 16:42:17 2013MMON started with pid=24, OS id=28835 Sun Mar 10 16:42:17 2013starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...Sun Mar 10 16:42:17 2013MMNL started with pid=25, OS id=28841 starting up 1 shared server(s) ...NOTE: initiating MARK startup Starting background process MARKStarting background process MARKSun Mar 10 16:42:17 2013MARK started with pid=28, OS id=28855 lmon registered with NM - instance number 1 (internal mem no 0)NOTE: MARK has subscribed Reconfiguration started (old inc 0, new inc 2)List of instances: 1 (myinst: 1) Global Resource Directory frozen* allocate domain 0, invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resourcesReconfiguration completeSun Mar 10 16:42:17 2013LCK0 started with pid=30, OS id=28872 Starting background process RSMNSun Mar 10 16:42:17 2013RSMN started with pid=31, OS id=28876 ORACLE_BASE not set in environment. It is recommendedthat ORACLE_BASE be set in the environmentReusing ORACLE_BASE from an earlier startup = /oracle/app/oracleSun Mar 10 16:42:18 2013ALTER SYSTEM SET local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=196.128.4.201)(PORT=1523))))' SCOPE=MEMORY SID='racdb1';ALTER DATABASE MOUNT /* db agent *//* {2:45663:2} */This instance was first to mountReconfiguration started (old inc 2, new inc 4)List of instances: 1 2 (myinst: 1) Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resourcesReconfiguration completeNOTE: Loaded library: System SUCCESS: diskgroup DATA was mountedNOTE: dependency between database racdb and diskgroup resource ora.DATA.dg is establishedSuccessful mount of redo thread 1, with mount id 807368170Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)Lost write protection disabledCompleted: ALTER DATABASE MOUNT /* db agent *//* {2:45663:2} */ALTER DATABASE OPEN /* db agent *//* {2:45663:2} */This instance was first to openSun Mar 10 16:45:51 2013Beginning crash recovery of 1 threadsStarted redo scanCompleted redo scan read 330 KB redo, 68 data blocks need recoveryStarted redo application at Thread 1: logseq 48, block 13367Recovery of Online Redo Log: Thread 1 Group 2 Seq 48 Reading mem 0 Mem# 0: +DATA/racdb/redo02.logCompleted redo application of 0.22MBCompleted crash recovery at Thread 1: logseq 48, block 14027, scn 12902091574043 68 data blocks read, 68 data blocks written, 330 redo k-bytes readARCH: STARTING ARCH PROCESSESSun Mar 10 16:45:52 2013ARC0 started with pid=36, OS id=29872 ARC0: Archival startedARCH: STARTING ARCH PROCESSES COMPLETEARC0: STARTING ARCH PROCESSESSun Mar 10 16:45:54 2013ARC1 started with pid=37, OS id=29879 Picked broadcast on commit scheme to generate SCNsSun Mar 10 16:45:54 2013ARC2 started with pid=38, OS id=29883 ARC1: Archival startedARC2: Archival startedARC2: Becoming the 'no FAL' ARCHARC2: Becoming the 'no SRL' ARCHARC1: Becoming the heartbeat ARCHSun Mar 10 16:45:54 2013ARC3 started with pid=39, OS id=29887 ARC3: Archival startedARC0: STARTING ARCH PROCESSES COMPLETESun Mar 10 16:45:55 2013Thread 1 advanced to log sequence 49 (thread open)Thread 1 opened at log sequence 49 Current log# 1 seq# 49 mem# 0: +DATA/racdb/redo01.logSuccessful open of redo thread 1MTTR advisory is disabled because FAST_START_MTTR_TARGET is not setSun Mar 10 16:45:57 2013SMON: enabling cache recoveryArchived Log entry 734 added for thread 1 sequence 48 ID 0x30038e22 dest 1:[28885] Successfully onlined Undo Tablespace 2.Undo initialization finished serial:0 start:1588520374 end:1588520984 diff:610 (6 seconds)Verifying file header compatibility for 11g tablespace encryption..Verifying 11g file header compatibility for tablespace encryption completedSMON: enabling tx recoverySun Mar 10 16:45:59 2013minact-scn: Inst 1 is now the master inc#:4 mmon proc-id:28835 status:0x7minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.00000000 gcalc-scn:0x0000.00000000minact-scn: Master returning as live inst:2 has inc# mismatch instinc:0 cur:4 errcnt:0Sun Mar 10 16:46:03 2013Database Characterset is AL32UTF8No Resource Manager plan activeStarting background process GTX0Sun Mar 10 16:46:05 2013GTX0 started with pid=41, OS id=29903 Starting background process RCBGSun Mar 10 16:46:05 2013RCBG started with pid=42, OS id=29907 replication_dependency_tracking turned off (no async multimaster replication found)Starting background process QMNCSun Mar 10 16:46:06 2013QMNC started with pid=43, OS id=29911 Completed: ALTER DATABASE OPEN /* db agent *//* {2:45663:2} */ALTER SYSTEM SET service_names='db1','dball','racdb1srv' SCOPE=MEMORY SID='racdb1';ALTER SYSTEM SET service_names='racdb1srv','db1','dball','racdbsrv' SCOPE=MEMORY SID='racdb1';Sun Mar 10 16:46:09 2013ALTER SYSTEM SET service_names='racdbsrv','racdb1srv','db1','dball','racdb2srv' SCOPE=MEMORY SID='racdb1';
