数据库故障处理的案例-归档目录空间满（一）

某单位机房突然掉电，导致数据库异常，是否发现存储也有异常以及数据库管理的不善，导致一序列的问题接踵而来，还好本次的故障虽然问题好几次，但幸运的是数据库还是救了回来。
    数据库环境：Oracle 10.2.0.5 Rac For Linux X86_64，使用OCFS+ASM方式， 下面请看日志：

ORA-06512: at line 1

Tue May 13 10:59:26 2014

Thread 2 advanced to log sequence 11976 (LGWR switch)

  Current log# 4 seq# 11976 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083

Tue May 13 13:09:24 2014

Thread 2 advanced to log sequence 11977 (LGWR switch)

  Current log# 3 seq# 11977 mem# 0: +DG/dqb/onlinelog/group_3.268.803845083

Tue May 13 16:30:39 2014

Starting ORACLE instance (normal)

LICENSE_MAX_SESSION = 0

LICENSE_SESSIONS_WARNING = 0

Interface type 1 eth1 192.168.1.0 configured from OCR for use as a cluster interconnect

Interface type 1 eth0 10.177.1.0 configured from OCR for use as  a public interface

Picked latch-free SCN scheme 3

Autotune of undo retention is turned on. 

LICENSE_MAX_USERS = 0

SYS auditing is disabled

ksdpec: called for event 13740 prior to event group initialization

Starting up ORACLE RDBMS Version: 10.2.0.4.0.

System parameters with non-default values:

  processes                = 300

  sessions                 = 335

  __shared_pool_size       = 637534208

  __large_pool_size        = 16777216

  __java_pool_size         = 16777216

  __streams_pool_size      = 0

  spfile                   = +DG/dqb/spfiledqb.ora

  sga_target               = 1610612736

  control_files            = +DG/dqb/controlfile/current.272.803844991

  control_file_record_keep_time= 8

  db_block_size            = 8192

  __db_cache_size          = 922746880

  compatible               = 10.2.0.3.0

  log_archive_dest_1       = LOCATION=+dg/dqb/arch/arch1

  db_file_multiblock_read_count= 16

  cluster_database         = TRUE

  cluster_database_instances= 2

  db_create_file_dest      = +DG

  thread                   = 2

  instance_number          = 2

  undo_management          = AUTO

  undo_tablespace          = UNDOTBS2

  remote_login_passwordfile= EXCLUSIVE

  db_domain                = 

  dispatchers              = (PROTOCOL=TCP) (SERVICE=dqbXDB)

  remote_listener          = LISTENERS_DQB

  job_queue_processes      = 10

  background_dump_dest     = /oracle/product/admin/dqb/bdump

  user_dump_dest           = /oracle/product/admin/dqb/udump

  core_dump_dest           = /oracle/product/admin/dqb/cdump

  audit_file_dest          = /oracle/product/admin/dqb/adump

  db_name                  = dqb

  open_cursors             = 300

  pga_aggregate_target     = 3363831808

Cluster communication is configured to use the following interface(s) for this instance

  192.168.1.2

Tue May 13 16:30:39 2014

cluster interconnect IPC version:Oracle UDP/IP (generic)

IPC Vendor 1 proto 2

PMON started with pid=4, OS id=13983

DIAG started with pid=8, OS id=13985

PSP0 started with pid=12, OS id=13987

LMON started with pid=16, OS id=13989

LMD0 started with pid=2, OS id=13991

LMS0 started with pid=6, OS id=13993

LMS1 started with pid=3, OS id=13997

LMS2 started with pid=20, OS id=14001

LMS3 started with pid=5, OS id=14005

LMS4 started with pid=10, OS id=14009

LMS5 started with pid=7, OS id=14014

LMS6 started with pid=24, OS id=14022

LMS7 started with pid=9, OS id=14026

MMAN started with pid=28, OS id=14030

DBW0 started with pid=14, OS id=14032

DBW1 started with pid=11, OS id=14034

DBW2 started with pid=32, OS id=14036

DBW3 started with pid=13, OS id=14038

LGWR started with pid=15, OS id=14040

CKPT started with pid=36, OS id=14042

SMON started with pid=40, OS id=14044

RECO started with pid=44, OS id=14046

CJQ0 started with pid=18, OS id=14048

MMON started with pid=48, OS id=14050

Tue May 13 16:30:40 2014

starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...

MMNL started with pid=52, OS id=14052

Tue May 13 16:30:40 2014

starting up 1 shared server(s) ...

Tue May 13 16:30:40 2014

lmon registered with NM - instance id 2 (internal mem no 1)

Tue May 13 16:30:40 2014

Reconfiguration started (old inc 0, new inc 2)

List of nodes:

 1

 Global Resource Directory frozen

* allocate domain 0, invalid = TRUE 

 Communication channels reestablished

 Master broadcasted resource hash value bitmaps

 Non-local Process blocks cleaned out

Tue May 13 16:30:40 2014

 LMS 1: 0 GCS shadows cancelled, 0 closed

Tue May 13 16:30:40 2014

 LMS 2: 0 GCS shadows cancelled, 0 closed

Tue May 13 16:30:40 2014

 LMS 6: 0 GCS shadows cancelled, 0 closed

Tue May 13 16:30:40 2014

 LMS 3: 0 GCS shadows cancelled, 0 closed

Tue May 13 16:30:40 2014

 LMS 0: 0 GCS shadows cancelled, 0 closed

Tue May 13 16:30:40 2014

 LMS 4: 0 GCS shadows cancelled, 0 closed

Tue May 13 16:30:40 2014

 LMS 5: 0 GCS shadows cancelled, 0 closed

Tue May 13 16:30:40 2014

 LMS 7: 0 GCS shadows cancelled, 0 closed

 Set master node info 

 Submitted all remote-enqueue requests

 Dwn-cvts replayed, VALBLKs dubious

 All grantable enqueues granted

 Post SMON to start 1st pass IR

Tue May 13 16:30:40 2014

 LMS 2: 0 GCS shadows traversed, 0 replayed

Tue May 13 16:30:40 2014

 LMS 3: 0 GCS shadows traversed, 0 replayed

Tue May 13 16:30:40 2014

 LMS 0: 0 GCS shadows traversed, 0 replayed

Tue May 13 16:30:40 2014

 LMS 7: 0 GCS shadows traversed, 0 replayed

Tue May 13 16:30:40 2014

 LMS 1: 0 GCS shadows traversed, 0 replayed

Tue May 13 16:30:40 2014

 LMS 5: 0 GCS shadows traversed, 0 replayed

Tue May 13 16:30:40 2014

 LMS 4: 0 GCS shadows traversed, 0 replayed

Tue May 13 16:30:40 2014

 LMS 6: 0 GCS shadows traversed, 0 replayed

Tue May 13 16:30:40 2014

 Submitted all GCS remote-cache requests

 Fix write in gcs resources

Reconfiguration complete

LCK0 started with pid=30, OS id=14059

Tue May 13 16:30:41 2014

ALTER DATABASE   MOUNT

Tue May 13 16:30:41 2014

This instance was first to mount

Tue May 13 16:30:41 2014

Starting background process ASMB

ASMB started with pid=56, OS id=14070

Starting background process RBAL

RBAL started with pid=60, OS id=14074

Tue May 13 16:30:44 2014

SUCCESS: diskgroup DG was mounted

Tue May 13 16:30:49 2014

Setting recovery target incarnation to 2

Tue May 13 16:30:49 2014

Successful mount of redo thread 2, with mount id 4126418993

Tue May 13 16:30:49 2014

Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)

Completed: ALTER DATABASE   MOUNT

Tue May 13 16:30:49 2014

ALTER DATABASE OPEN

This instance was first to open

Tue May 13 16:30:53 2014

Beginning crash recovery of 2 threads

 parallel recovery started with 16 processes

Tue May 13 16:30:55 2014

Started redo scan

Tue May 13 16:30:55 2014

Completed redo scan

 0 redo blocks read, 0 data blocks need recovery

Tue May 13 16:30:55 2014

Started redo application at

 Thread 1: logseq 15107, block 142214

 Thread 2: logseq 11977, block 43783

Tue May 13 16:30:55 2014

Recovery of Online Redo Log: Thread 1 Group 1 Seq 15107 Reading mem 0

  Mem# 0: +DG/dqb/onlinelog/group_1.271.803844993

Tue May 13 16:30:55 2014

Recovery of Online Redo Log: Thread 2 Group 3 Seq 11977 Reading mem 0

  Mem# 0: +DG/dqb/onlinelog/group_3.268.803845083

Tue May 13 16:30:55 2014

Completed redo application

Tue May 13 16:30:55 2014

Completed crash recovery at

 Thread 1: logseq 15107, block 142214, scn 13881050933462

 Thread 2: logseq 11977, block 43783, scn 13881050931946

 0 data blocks read, 0 data blocks written, 0 redo blocks read

Tue May 13 16:30:55 2014

Thread 1 advanced to log sequence 15108 (thread recovery)

Picked broadcast on commit scheme to generate SCNs

Tue May 13 16:30:55 2014

LGWR: STARTING ARCH PROCESSES

ARC0 started with pid=58, OS id=14186

Tue May 13 16:30:55 2014

ARC0: Archival started

ARC1: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

ARC1 started with pid=39, OS id=14188

Tue May 13 16:30:55 2014

Thread 2 advanced to log sequence 11978 (thread open)

Thread 2 opened at log sequence 11978

  Current log# 4 seq# 11978 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083

Successful open of redo thread 2

Tue May 13 16:30:55 2014

MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set

Tue May 13 16:30:55 2014

ARC1: Becoming the 'no FAL' ARCH

ARC1: Becoming the 'no SRL' ARCH

Tue May 13 16:30:55 2014

ARC0: Becoming the heartbeat ARCH

Tue May 13 16:30:55 2014

SMON: enabling cache recovery

Tue May 13 16:30:56 2014

ARC1: Log corruption near block 43776 change 13881059254746 time ?

Tue May 13 16:30:56 2014

Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_14188.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059254746 time 05/13/2014 13:09:24

ORA-00312: online log 3 thread 2: '+DG/dqb/onlinelog/group_3.268.803845083'

ARC1: All Archive destinations made inactive due to error 354

Tue May 13 16:30:56 2014

ARC1: Closing local archive destination LOG_ARCHIVE_DEST_1: '+DG/dqb/arch/arch1/2_11977_803844992.dbf' (error 354)

 (dqb2)

Committing creation of archivelog '+DG/dqb/arch/arch1/2_11977_803844992.dbf' (error 354)

ARCH: Archival stopped, error occurred. Will continue retrying

Tue May 13 16:30:56 2014

ORACLE Instance dqb2 - Archival Error

Tue May 13 16:30:56 2014

ORA-16038: log 3 sequence# 11977 cannot be archived

ORA-00354: corrupt redo log block header

ORA-00312: online log 3 thread 2: '+DG/dqb/onlinelog/group_3.268.803845083'

Tue May 13 16:30:56 2014

Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_14188.trc:

ORA-16038: log 3 sequence# 11977 cannot be archived

ORA-00354: corrupt redo log block header

ORA-00312: online log 3 thread 2: '+DG/dqb/onlinelog/group_3.268.803845083'

Tue May 13 16:30:56 2014

Successfully onlined Undo Tablespace 5.

Tue May 13 16:30:56 2014

SMON: enabling tx recovery

Tue May 13 16:30:56 2014

Database Characterset is ZHS16GBK

Opening with internal Resource Manager plan

where NUMA PG = 4, CPUs = 20

Tue May 13 16:30:57 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35

ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'

Tue May 13 16:30:57 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:

ORA-00600: internal error code, arguments: [4194], [65], [47], [], [], [], [], []

Tue May 13 16:30:59 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35

ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'

ORA-00600: internal error code, arguments: [4194], [65], [47], [], [], [], [], []

Tue May 13 16:31:00 2014

Trace dumping is performing id=[cdmp_20140513163100]

Tue May 13 16:31:00 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35

ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'

ORA-00600: internal error code, arguments: [4194], [65], [47], [], [], [], [], []

Tue May 13 16:31:00 2014

Doing block recovery for file 6 block 4405

Block recovery from logseq 11978, block 89 to scn 13881066792885

Tue May 13 16:31:00 2014

Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0

  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083

Block recovery stopped at EOT rba 11978.91.16

Block recovery completed at rba 11978.91.16, scn 3231.4027457777

Doing block recovery for file 6 block 121

Block recovery from logseq 11978, block 89 to scn 13881066791152

Tue May 13 16:31:00 2014

Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0

  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083

Block recovery completed at rba 11978.91.16, scn 3231.4027457777

Tue May 13 16:31:00 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35

ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'

Tue May 13 16:31:00 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:

ORA-00600: internal error code, arguments: [4193], [25330], [25385], [], [], [], [], []

Tue May 13 16:31:00 2014

Errors in file /oracle/product/admin/dqb/bdump/dqb2_smon_14044.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35

ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'

Tue May 13 16:31:00 2014

Errors in file /oracle/product/admin/dqb/bdump/dqb2_smon_14044.trc:

ORA-00600: internal error code, arguments: [4194], [43], [26], [], [], [], [], []

Tue May 13 16:31:01 2014

Errors in file /oracle/product/admin/dqb/bdump/dqb2_smon_14044.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35

ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'

ORA-00600: internal error code, arguments: [4194], [43], [26], [], [], [], [], []

Tue May 13 16:31:01 2014

Doing block recovery for file 6 block 4718

Block recovery from logseq 11978, block 100 to scn 13881066792905

Tue May 13 16:31:01 2014

Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0

  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083

Block recovery stopped at EOT rba 11978.101.16

Block recovery completed at rba 11978.101.16, scn 3231.4027459528

Doing block recovery for file 6 block 105

Block recovery from logseq 11978, block 100 to scn 13881066792903

Tue May 13 16:31:01 2014

Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0

  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083

Block recovery completed at rba 11978.101.16, scn 3231.4027459528

Tue May 13 16:31:01 2014

Errors in file /oracle/product/admin/dqb/bdump/dqb2_smon_14044.trc:

ORA-01595: error freeing extent (4) of rollback segment (17))

ORA-00600: internal error code, arguments: [4194], [43], [26], [], [], [], [], []

Tue May 13 16:31:03 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35

ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'

ORA-00600: internal error code, arguments: [4193], [25330], [25385], [], [], [], [], []

Tue May 13 16:31:03 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35

ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'

ORA-00600: internal error code, arguments: [4193], [25330], [25385], [], [], [], [], []

Tue May 13 16:31:03 2014

Doing block recovery for file 6 block 301

Block recovery from logseq 11978, block 91 to scn 13881066792908

Tue May 13 16:31:03 2014

Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0

  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083

Block recovery stopped at EOT rba 11978.101.16

Block recovery completed at rba 11978.101.16, scn 3231.4027459528

Doing block recovery for file 6 block 153

Block recovery from logseq 11978, block 91 to scn 13881066792887

Tue May 13 16:31:03 2014

Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0

  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083

Block recovery completed at rba 11978.93.16, scn 3231.4027459513

replication_dependency_tracking turned off (no async multimaster replication found)

Starting background process QMNC

QMNC started with pid=66, OS id=14288

Tue May 13 16:31:04 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35

ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'

Tue May 13 16:31:04 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:

ORA-00600: internal error code, arguments: [4193], [25330], [25385], [], [], [], [], []

Tue May 13 16:31:07 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35

ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'

ORA-00600: internal error code, arguments: [4193], [25330], [25385], [], [], [], [], []

Tue May 13 16:31:07 2014

Doing block recovery for file 6 block 301

Block recovery from logseq 11978, block 91 to scn 13881066792908

Tue May 13 16:31:07 2014

Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0

  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083

Tue May 13 16:31:07 2014

Trace dumping is performing id=[cdmp_20140513163107]

Tue May 13 16:31:07 2014

Block recovery completed at rba 11978.101.16, scn 3231.4027459534

Doing block recovery for file 6 block 153

Block recovery from logseq 11978, block 91 to scn 13881066792913

Tue May 13 16:31:07 2014

Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0

  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083

Block recovery completed at rba 11978.104.16, scn 3231.4027459538

Error 0 in kwqmnpartition(), aborting txn 

Tue May 13 16:31:07 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35

ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'

Tue May 13 16:31:07 2014

ksdpec: called for event 13740 prior to event group initialization

Starting up ORACLE RDBMS Version: 10.2.0.4.0.

System parameters with non-default values:

  processes                = 300

  sessions                 = 335

  __shared_pool_size       = 637534208

  __large_pool_size        = 16777216

  __java_pool_size         = 16777216

  __streams_pool_size      = 0

  spfile                   = +DG/dqb/spfiledqb.ora

  sga_target               = 1610612736

  control_files            = +DG/dqb/controlfile/current.272.803844991

  control_file_record_keep_time= 8

  db_block_size            = 8192

  __db_cache_size          = 922746880

  compatible               = 10.2.0.3.0

  log_archive_dest_1       = LOCATION=+dg/dqb/arch/arch1

  db_file_multiblock_read_count= 16

  cluster_database         = TRUE

  cluster_database_instances= 2

  db_create_file_dest      = +DG

  db_recovery_file_dest_size= 10737418240

  thread                   = 2

  instance_number          = 2

  undo_management          = AUTO

  undo_tablespace          = UNDOTBS2

  remote_login_passwordfile= EXCLUSIVE

  db_domain                = 

  dispatchers              = (PROTOCOL=TCP) (SERVICE=dqbXDB)

  remote_listener          = LISTENERS_DQB

  job_queue_processes      = 10

  background_dump_dest     = /oracle/product/admin/dqb/bdump

  user_dump_dest           = /oracle/product/admin/dqb/udump

  core_dump_dest           = /oracle/product/admin/dqb/cdump

  audit_file_dest          = /oracle/product/admin/dqb/adump

  db_name                  = dqb

  open_cursors             = 300

  pga_aggregate_target     = 3363831808

Cluster communication is configured to use the following interface(s) for this instance

  192.168.1.2

Wed May 14 11:31:48 2014

cluster interconnect IPC version:Oracle UDP/IP (generic)

IPC Vendor 1 proto 2

PMON started with pid=2, OS id=32760

DIAG started with pid=6, OS id=32762

PSP0 started with pid=10, OS id=32764

LMON started with pid=14, OS id=32766

LMD0 started with pid=18, OS id=300

LMS0 started with pid=22, OS id=302

LMS1 started with pid=3, OS id=306

LMS2 started with pid=4, OS id=310

LMS3 started with pid=5, OS id=314

LMS4 started with pid=26, OS id=318

LMS5 started with pid=7, OS id=322

LMS6 started with pid=8, OS id=326

LMS7 started with pid=9, OS id=330

MMAN started with pid=30, OS id=334

DBW0 started with pid=34, OS id=336

DBW1 started with pid=11, OS id=338

DBW2 started with pid=12, OS id=340

DBW3 started with pid=13, OS id=342

LGWR started with pid=15, OS id=344

CKPT started with pid=38, OS id=346

SMON started with pid=42, OS id=348

RECO started with pid=46, OS id=350

CJQ0 started with pid=50, OS id=395

MMON started with pid=54, OS id=397

Wed May 14 11:31:49 2014

starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...

MMNL started with pid=58, OS id=399

Wed May 14 11:31:49 2014

starting up 1 shared server(s) ...

Wed May 14 11:31:49 2014

lmon registered with NM - instance id 2 (internal mem no 1)

Wed May 14 11:31:49 2014

Reconfiguration started (old inc 0, new inc 2)

List of nodes:

 1

 Global Resource Directory frozen

* allocate domain 0, invalid = TRUE 

 Communication channels reestablished

 Master broadcasted resource hash value bitmaps

 Non-local Process blocks cleaned out

Wed May 14 11:31:49 2014

 LMS 0: 0 GCS shadows cancelled, 0 closed

Wed May 14 11:31:49 2014

 LMS 1: 0 GCS shadows cancelled, 0 closed

Wed May 14 11:31:49 2014

 LMS 3: 0 GCS shadows cancelled, 0 closed

Wed May 14 11:31:49 2014

 LMS 2: 0 GCS shadows cancelled, 0 closed

Wed May 14 11:31:49 2014

 LMS 4: 0 GCS shadows cancelled, 0 closed

Wed May 14 11:31:49 2014

 LMS 5: 0 GCS shadows cancelled, 0 closed

Wed May 14 11:31:49 2014

 LMS 6: 0 GCS shadows cancelled, 0 closed

Wed May 14 11:31:49 2014

 LMS 7: 0 GCS shadows cancelled, 0 closed

 Set master node info 

 Submitted all remote-enqueue requests

 Dwn-cvts replayed, VALBLKs dubious

 All grantable enqueues granted

 Post SMON to start 1st pass IR

Wed May 14 11:31:49 2014

 LMS 1: 0 GCS shadows traversed, 0 replayed

Wed May 14 11:31:49 2014

 LMS 0: 0 GCS shadows traversed, 0 replayed

Wed May 14 11:31:49 2014

 LMS 6: 0 GCS shadows traversed, 0 replayed

Wed May 14 11:31:49 2014

 LMS 2: 0 GCS shadows traversed, 0 replayed

Wed May 14 11:31:49 2014

 LMS 7: 0 GCS shadows traversed, 0 replayed

Wed May 14 11:31:49 2014

 LMS 3: 0 GCS shadows traversed, 0 replayed

Wed May 14 11:31:49 2014

 LMS 5: 0 GCS shadows traversed, 0 replayed

Wed May 14 11:31:49 2014

 LMS 4: 0 GCS shadows traversed, 0 replayed

Wed May 14 11:31:49 2014

 Submitted all GCS remote-cache requests

 Fix write in gcs resources

Reconfiguration complete

LCK0 started with pid=70, OS id=405

Wed May 14 11:31:50 2014

ALTER DATABASE   MOUNT

Wed May 14 11:31:50 2014

This instance was first to mount

Wed May 14 11:31:50 2014

Starting background process ASMB

ASMB started with pid=78, OS id=416

Starting background process RBAL

RBAL started with pid=82, OS id=420

Wed May 14 11:31:53 2014

SUCCESS: diskgroup DG was mounted

Wed May 14 11:31:57 2014

Setting recovery target incarnation to 2

Wed May 14 11:31:57 2014

Successful mount of redo thread 2, with mount id 4126510502

Wed May 14 11:31:57 2014

Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)

Completed: ALTER DATABASE   MOUNT

Wed May 14 11:31:57 2014

ALTER DATABASE OPEN

This instance was first to open

Picked broadcast on commit scheme to generate SCNs

Wed May 14 11:31:58 2014

LGWR: STARTING ARCH PROCESSES

ARC0 started with pid=90, OS id=475

Wed May 14 11:31:58 2014

ARC0: Archival started

ARC1 started with pid=23, OS id=477

Wed May 14 11:31:58 2014

ARC1: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

Wed May 14 11:31:58 2014

ARC1: Becoming the 'no FAL' ARCH

ARC1: Becoming the 'no SRL' ARCH

Wed May 14 11:31:58 2014

ARC0: Becoming the heartbeat ARCH

Wed May 14 11:31:58 2014

ARCH: Log corruption near block 43776 change 13881059254746 time ?

Wed May 14 11:31:58 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_466.trc:

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 43776 change 13881059254746 time 05/13/2014 13:09:24

ORA-00312: online log 3 thread 2: '+DG/dqb/onlinelog/group_3.268.803845083'

ARCH: All Archive destinations made inactive due to error 354

Wed May 14 11:32:01 2014

ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '+DG/dqb/arch/arch1/2_11977_803844992.dbf' (error 354)

 (dqb2)

Committing creation of archivelog '+DG/dqb/arch/arch1/2_11977_803844992.dbf' (error 354)

Wed May 14 11:32:01 2014

Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_466.trc:

ORA-16038: log 3 sequence# 11977 cannot be archived

ORA-00354: corrupt redo log block header

ORA-00312: online log 3 thread 2: '+DG/dqb/onlinelog/group_3.268.803845083'

USER: terminating instance due to error 16038

Wed May 14 11:32:01 2014

Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms1_306.trc:

ORA-16038: log  sequence#  cannot be archived

Wed May 14 11:32:01 2014

Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms2_310.trc:

ORA-16038: log  sequence#  cannot be archived

Wed May 14 11:32:01 2014

Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms3_314.trc:

ORA-16038: log  sequence#  cannot be archived

Wed May 14 11:32:01 2014

Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms6_326.trc:

ORA-16038: log  sequence#  cannot be archived

Wed May 14 11:32:01 2014

System state dump is made for local instance

System State dumped to trace file /oracle/product/admin/dqb/bdump/dqb2_diag_32762.trc

Wed May 14 11:32:01 2014

Trace dumping is performing id=[cdmp_20140514113201]

Wed May 14 11:32:04 2014

Instance terminated by USER, pid = 466

Wed May 14 11:32:53 2014

出现上面的解决办法当然是删除无用的过期归档日志，腾出归档空间，以便数据库启动后执行归档进程恢复动作。
1、登陆到数据库服务器,并切换到oracle安装用户
#su - ora10g
$export ORACLE_SID=+ASM1   输出环境变量进入asm实例命令行
$asmcmd
asmcmd>pwd
+DG
asmcmd>cd +DG/.....
.......