ORA-03113: 通信通道的文件结尾 进程 ID: 3949 会话 ID: 1 序列号: 3

一套11.2.0.3的RAC拆分为单节点。更新完最新Bundle&PSU，test了下自动回滚。当回滚成功后。

[oracle@testora dbs]$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.3.0 Production on 星期一 12月 23 10:51:36 2013Copyright (c) 1982, 2011, Oracle.  All rights reserved.已连接到空闲例程。SQL> startup ORACLE 例程已经启动。Total System Global Area  609624064 bytesFixed Size    2230792 bytesVariable Size  226493944 bytesDatabase Buffers  377487360 bytesRedo Buffers    3411968 bytesORA-03113: 通信通道的文件结尾进程 ID: 3949会话 ID: 1 序列号: 3SQL> exit

这个时候dg的状态为挂载了

[grid@testora ~]$ sqlplus / as sysasmSQL*Plus: Release 11.2.0.3.0 Production on Mon Dec 23 12:48:28 2013Copyright (c) 1982, 2011, Oracle.  All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Automatic Storage Management optionSQL> select status from v$instance;STATUS------------STARTED

alter 告警日志：/u01/app/oracle/diag/rdbms/testora/testora/trace/alert_testora.log

Mon Dec 23 10:51:55 2013Starting ORACLE instance (normal)****************** Large Pages Information *****************Total Shared Global Region in Large Pages = 0 KB (0%)Large Pages used by this instance: 0 (0 KB)Large Pages unused system wide = 0 (0 KB) (alloc incr 4096 KB)Large Pages configured system wide = 0 (0 KB)Large Page size = 2048 KBRECOMMENDATION:  Total Shared Global Region size is 586 MB. For optimal performance,  prior to the next instance restart increase the number  of unused Large Pages by atleast 293 2048 KB Large Pages (586 MB)  system wide to get 100% of the Shared  Global Region allocated with Large pages***********************************************************LICENSE_MAX_SESSION = 0LICENSE_SESSIONS_WARNING = 0Picked latch-free SCN scheme 3Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.2.0/dbhome_1/dbs/archAutotune of undo retention is turned on.IMODE=BRILAT =27LICENSE_MAX_USERS = 0SYS auditing is disabledStarting up:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing options.ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_1System name:    LinuxNode name:      testoraRelease:        2.6.18-194.el5Version:        #1 SMP Tue Mar 16 21:52:39 EDT 2010Machine:        x86_64Using parameter settings in client-side pfile /u01/app/oracle/product/11.2.0/dbhome_1/dbs/inittestora.ora on machine testoraSystem parameters with non-default values:  processes                = 150  nls_language             = "SIMPLIFIED CHINESE"  sga_target               = 584M  control_files            = "+DATA/testora/controlfile/current.260.822235297"  db_block_size            = 8192  compatible               = "11.2.0.0.0"  db_create_file_dest      = "+DATA"  db_recovery_file_dest_size= 41222905K  undo_tablespace          = "UNDOTBS1"  remote_login_passwordfile= "EXCLUSIVE"  db_domain                = ""  dispatchers              = "(PROTOCOL=TCP) (SERVICE=testoraXDB)"  audit_file_dest          = "/u01/app/oracle/admin/testora/adump"  audit_trail              = "DB"  db_name                  = "testora"  open_cursors             = 300  diagnostic_dest          = "/u01/app/oracle"Mon Dec 23 10:51:56 2013PMON started with pid=2, OS id=3902Mon Dec 23 10:51:56 2013PSP0 started with pid=3, OS id=3904Mon Dec 23 10:51:57 2013VKTM started with pid=4, OS id=3906 at elevated priorityVKTM running at (1)millisec precision with DBRM quantum (100)msMon Dec 23 10:51:57 2013GEN0 started with pid=5, OS id=3910Mon Dec 23 10:51:57 2013DIAG started with pid=6, OS id=3912Mon Dec 23 10:51:57 2013DBRM started with pid=7, OS id=3914Mon Dec 23 10:51:57 2013DIA0 started with pid=8, OS id=3916Mon Dec 23 10:51:57 2013MMAN started with pid=9, OS id=3918Mon Dec 23 10:51:57 2013DBW0 started with pid=10, OS id=3920Mon Dec 23 10:51:57 2013LGWR started with pid=11, OS id=3922Mon Dec 23 10:51:57 2013CKPT started with pid=12, OS id=3924Mon Dec 23 10:51:57 2013SMON started with pid=13, OS id=3926Mon Dec 23 10:51:57 2013RECO started with pid=14, OS id=3928Mon Dec 23 10:51:57 2013RBAL started with pid=15, OS id=3930Mon Dec 23 10:51:58 2013ASMB started with pid=16, OS id=3932Mon Dec 23 10:51:58 2013MMON started with pid=17, OS id=3934NOTE: initiating MARK startupstarting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...Mon Dec 23 10:51:58 2013MMNL started with pid=18, OS id=3938Starting background process MARKMon Dec 23 10:51:58 2013Mon Dec 23 10:30:23 2013System state dump requested by (instance=1, osid=3733 (PMON)), summary=[abnormal instance termination].System State dumped to trace file /u01/app/oracle/diag/rdbms/testora/testora/trace/testora_diag_3743.trcDumping diagnostic data in directory=[cdmp_20131223103023], requested by (instance=1, osid=3733 (PMON)), summary=[abnormal instance termination].Instance terminated by PMON, pid = 3733Mon Dec 23 10:51:55 2013Starting ORACLE instance (normal)****************** Large Pages Information *****************Total Shared Global Region in Large Pages = 0 KB (0%)Large Pages used by this instance: 0 (0 KB)Large Pages unused system wide = 0 (0 KB) (alloc incr 4096 KB)Large Pages configured system wide = 0 (0 KB)Large Page size = 2048 KBRECOMMENDATION:  Total Shared Global Region size is 586 MB. For optimal performance,  prior to the next instance restart increase the number  of unused Large Pages by atleast 293 2048 KB Large Pages (586 MB)  system wide to get 100% of the Shared  Global Region allocated with Large pages***********************************************************LICENSE_MAX_SESSION = 0LICENSE_SESSIONS_WARNING = 0Picked latch-free SCN scheme 3Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.2.0/dbhome_1/dbs/archAutotune of undo retention is turned on.IMODE=BRILAT =27LICENSE_MAX_USERS = 0SYS auditing is disabledStarting up:Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing options.ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_1System name:    LinuxNode name:      testoraRelease:        2.6.18-194.el5Version:        #1 SMP Tue Mar 16 21:52:39 EDT 2010Machine:        x86_64Using parameter settings in client-side pfile /u01/app/oracle/product/11.2.0/dbhome_1/dbs/inittestora.ora on machine testoraSystem parameters with non-default values:  processes                = 150  nls_language             = "SIMPLIFIED CHINESE"  sga_target               = 584M  control_files            = "+DATA/testora/controlfile/current.260.822235297"  db_block_size            = 8192  compatible               = "11.2.0.0.0"  db_create_file_dest      = "+DATA"  db_recovery_file_dest_size= 41222905K  undo_tablespace          = "UNDOTBS1"  remote_login_passwordfile= "EXCLUSIVE"  db_domain                = ""  dispatchers              = "(PROTOCOL=TCP) (SERVICE=testoraXDB)"  audit_file_dest          = "/u01/app/oracle/admin/testora/adump"  audit_trail              = "DB"  db_name                  = "testora"  open_cursors             = 300  diagnostic_dest          = "/u01/app/oracle"Mon Dec 23 10:51:56 2013PMON started with pid=2, OS id=3902Mon Dec 23 10:51:56 2013PSP0 started with pid=3, OS id=3904Mon Dec 23 10:51:57 2013VKTM started with pid=4, OS id=3906 at elevated priorityVKTM running at (1)millisec precision with DBRM quantum (100)msMon Dec 23 10:51:57 2013GEN0 started with pid=5, OS id=3910Mon Dec 23 10:51:57 2013DIAG started with pid=6, OS id=3912Mon Dec 23 10:51:57 2013DBRM started with pid=7, OS id=3914Mon Dec 23 10:51:57 2013DIA0 started with pid=8, OS id=3916Mon Dec 23 10:51:57 2013MMAN started with pid=9, OS id=3918Mon Dec 23 10:51:57 2013DBW0 started with pid=10, OS id=3920Mon Dec 23 10:51:57 2013LGWR started with pid=11, OS id=3922Mon Dec 23 10:51:57 2013CKPT started with pid=12, OS id=3924Mon Dec 23 10:51:57 2013SMON started with pid=13, OS id=3926Mon Dec 23 10:51:57 2013RECO started with pid=14, OS id=3928Mon Dec 23 10:51:57 2013RBAL started with pid=15, OS id=3930Mon Dec 23 10:51:58 2013ASMB started with pid=16, OS id=3932Mon Dec 23 10:51:58 2013MMON started with pid=17, OS id=3934NOTE: initiating MARK startupstarting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...Mon Dec 23 10:51:58 2013MMNL started with pid=18, OS id=3938Starting background process MARKMon Dec 23 10:51:58 2013ORACLE_BASE from environment = /u01/app/oracleMon Dec 23 10:51:58 2013ALTER DATABASE   MOUNTMon Dec 23 10:52:01 2013Sweep [inc][132146]: completedSweep [inc][132145]: completedSweep [inc2][132146]: completedSweep [inc2][132145]: completedNOTE: Loaded library: SystemORA-15025: ?????? "/dev/raw/raw1"ORA-27041: ??????Linux-x86_64 Error: 13: Permission deniedAdditional information: 9SUCCESS: diskgroup DATA was mountedNOTE: dependency between database testora and diskgroup resource ora.DATA.dg is establishedErrors in file /u01/app/oracle/diag/rdbms/testora/testora/trace/testora_ckpt_3924.trc  (incident=133345):ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []Incident details in: /u01/app/oracle/diag/rdbms/testora/testora/incident/incdir_133345/testora_ckpt_3924_i133345.trcUse ADRCI or Support Workbench to package the incident.See Note 411.1 at My Oracle Support for error and packaging details.Errors in file /u01/app/oracle/diag/rdbms/testora/testora/trace/testora_ckpt_3924.trc  (incident=133346):ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []Incident details in: /u01/app/oracle/diag/rdbms/testora/testora/incident/incdir_133346/testora_ckpt_3924_i133346.trcDumping diagnostic data in directory=[cdmp_20131223105205], requested by (instance=1, osid=3924 (CKPT)), summary=[incident=133345].Use ADRCI or Support Workbench to package the incident.See Note 411.1 at My Oracle Support for error and packaging details.ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 3924Dumping diagnostic data in directory=[cdmp_20131223105206], requested by (instance=1, osid=3924 (CKPT)), summary=[incident=133346].Mon Dec 23 10:52:07 2013PMON (ospid: 3902): terminating the instance due to error 469Mon Dec 23 10:52:08 2013System state dump requested by (instance=1, osid=3902 (PMON)), summary=[abnormal instance termination].System State dumped to trace file /u01/app/oracle/diag/rdbms/testora/testora/trace/testora_diag_3912.trcDumping diagnostic data in directory=[cdmp_20131223105208], requested by (instance=1, osid=3902 (PMON)), summary=[abnormal instance termination].Instance terminated by PMON, pid = 3902

cd $ORACLE_HOME/bin
-rwsr-s--x 1 oracle oinstall oracle （错误）


正常 
-rwsr-s--x 1 oracle asmadmin 232656133 Dec 19 11:07 oracle


解决：
 chown  oracle:asmadmin oracle
 chmod 6751 oracle




验证：
[oracle@testora ~]$ sqlplus / as sysdba


SQL*Plus: Release 11.2.0.3.0 Production on 星期一 12月 23 12:57:54 2013


Copyright (c) 1982, 2011, Oracle.  All rights reserved.


已连接到空闲例程。


idle> startup
ORACLE 例程已经启动。


Total System Global Area  609624064 bytes
Fixed Size    2230792 bytes
Variable Size  226493944 bytes
Database Buffers  377487360 bytes
Redo Buffers    3411968 bytes
数据库装载完毕。
数据库已经打开。




[oracle@testora ~]$ lsnrctl status


LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 23-12月-2013 12:58:53


Copyright (c) 1991, 2011, Oracle.  All rights reserved.


正在连接到 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.150)(PORT=1521)))
LISTENER 的 STATUS
------------------------
别名                      LISTENER
版本                      TNSLSNR for Linux: Version 11.2.0.3.0 - Production
启动日期                  23-12月-2013 11:22:54
正常运行时间              0 天 1 小时 35 分 59 秒
跟踪级别                  off
安全性                    ON: Local OS Authentication
SNMP                      OFF
监听程序参数文件          /u01/app/grid/product/11.2.0/grid/network/admin/listener.ora
监听程序日志文件          /u01/app/grid/diag/tnslsnr/testora/listener/alert/log.xml
监听端点概要...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=testora)(PORT=1521)))
服务摘要..
服务 "+ASM" 包含 1 个实例。
  实例 "+ASM", 状态 READY, 包含此服务的 1 个处理程序...
服务 "testora" 包含 1 个实例。
  实例 "testora", 状态 READY, 包含此服务的 1 个处理程序...
服务 "testoraXDB" 包含 1 个实例。
  实例 "testora", 状态 READY, 包含此服务的 1 个处理程序...
命令执行成功
[oracle@testora ~]$ sqlplus test/test@192.168.1.150:1521/testora


SQL*Plus: Release 11.2.0.3.0 Production on 星期一 12月 23 12:59:30 2013


Copyright (c) 1982, 2011, Oracle.  All rights reserved.




连接到: 
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options


test@TESTORA>