ORA-08103: object no longer exists

昨天恢复完测试库ORA-600(3020)错误后，数据库可以正常open
但是使用过程中又出现了新的错误
http://blog.csdn.net/goolenblog/article/details/17798171

SQL> create table goolen as select * from dba_objects;
Table created.

SQL> insert into  goolen  select * from dba_objects;
72219 rows created.

SQL> insert into  goolen  select * from dba_objects;
insert into  goolen  select * from dba_objects
             *
ERROR at line 1:
ORA-08103: object no longer exists

+++设置8103事件
SQL> ALTER SYSTEM SET  EVENTS  '8103 TRACE NAME ERRORSTACK LEVEL 3';
System altered.

SQL> insert into  goolen  select * from dba_objects;
insert into  goolen  select * from dba_objects
*
ERROR at line 1:
ORA-08103: object no longer exists

+++alter信息如下：
Sat Jan 04 09:05:39 2014
OS Pid: 13825 executed alter system set events '8103 TRACE NAME ERRORSTACK LEVEL 3'
Errors in file /opt/app/oracle/diag/rdbms/goolen/goolen/trace/goolen_ora_13825.trc:
ORA-08103: object no longer exists
Sat Jan 04 09:05:44 2014
Trace dumping is performing id=[cdmp_20140104090544]

+++trace信息：
Incident 16187 created, dump file: /opt/app/oracle/diag/rdbms/goolen/goolen/incident/incdir_16187/goolen_ora_13825_i16187.trc
ORA-00600: internal error code, arguments: [ktssinseg3], [6], [25165882], [], [], [], [], [], [], [], [], []
kcbzib: dump suspect buffer
*** 2014-01-04 09:03:17.331
buffer tsn: 6 rdba: 0x0180003a (6/58)
scn: 0x0000.001f7093 seq: 0x01 flg: 0x04 tail: 0x70930601
frmt: 0x02 chkval: 0xeb30 type: 0x06=trans data
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x000000008E416000 to 0x000000008E418000
08E416000 0000A206 0180003A 001F7093 04010000  [....:....p......]
08E416010 0000EB30 00000001 000124CD 001F708C  [0........$...p..]

+++从trace信息可以看出，rdba: 0x0180003a (6/58)，file 6 block 58这个块有问题
+++检测一下数据文件：
[oracle@localhost bbed]$ dbv file=/opt/app/oracle/oradata/goolen/goolen01.dbf
DBVERIFY: Release 11.2.0.1.0 - Production on Sat Jan 4 09:11:12 2014
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
DBVERIFY - Verification starting : FILE = /opt/app/oracle/oradata/goolen/goolen01.dbf

DBVERIFY - Verification complete
Total Pages Examined         : 2480
Total Pages Processed (Data) : 2275
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 2
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 193
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 10
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 2060751 (0.2060751)

RMAN> validate datafile 6 block 58;
Starting validate at 04-JAN-14
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=35 device type=DISK
channel ORA_DISK_1: starting validation of datafile
channel ORA_DISK_1: specifying datafile(s) for validation
input datafile file number=00006 name=/opt/app/oracle/oradata/goolen/goolen01.dbf
channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
List of Datafiles
=================
File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
6    OK     0              0            1               2060435   
  File Name: /opt/app/oracle/oradata/goolen/goolen01.dbf
  Block Type Blocks Failing Blocks Processed
  ---------- -------------- ----------------
  Data       0              1               
  Index      0              0               
  Other      0              0               
Finished validate at 04-JAN-14

+++试着做一下块恢复
RMAN> blockrecover datafile 6 block 58;
Starting recover at 04-JAN-14
using channel ORA_DISK_1
starting media recovery
media recovery complete, elapsed time: 00:00:00
Finished recover at 04-JAN-14

+++还是报错
SQL> insert into  goolen  select * from dba_objects;
insert into  goolen  select * from dba_objects
*
ERROR at line 1:
ORA-08103: object no longer exists

+++因为这个是在insert的时候，分配extent的时候发现数据块有问题，而报的错，也就是说这个块是没有数据的
我们把这个块clear一下
RMAN> recover datafile 6 block 58 clear;
Starting recover at 04-JAN-14
using channel ORA_DISK_1
Finished recover at 04-JAN-14

+++insert还是报错，但是已经不在是ora-8103
SQL> insert into  goolen  select * from dba_objects;
insert into  goolen  select * from dba_objects
*
ERROR at line 1:
ORA-01578: ORACLE data block corrupted (file # 6, block # 58)
ORA-01110: data file 6: '/opt/app/oracle/oradata/goolen/goolen01.dbf'

+++尝试用一个新块去替换这个有问题的块
SQL> create table goolen1 as select * from goolen where rownum <=1;
Table created.

SQL> select 
  2  dbms_rowid.rowid_relative_fno(rowid) fno,
  3  dbms_rowid.rowid_block_number(rowid) bno
  4  from goolen1;
       FNO        BNO
---------- ----------
         6        179

+++使用bbed来copy这个新块
+++之后再更改一下file 6 block 58这个块的rdba 和data_object_id
BBED> copy file 6 block 179 to file 6 block 58;
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y
 File: /opt/app/oracle/oradata/goolen/goolen01.dbf (6)
 Block: 58               Offsets:    0 to  511           Dba:0x0180003a
------------------------------------------------------------------------
 06a20000 b3008001 c47a1f00 00000106 7eac0000 01000000 ce240100 e2791f00 
 00000000 03003200 b0008001 ffff0000 00000000 00000000 00000000 00800000 
 e2791f00 06000000 4b040000 b102c000 90011000 01204b00 c47a1f00 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00010100 
 ffff1400 331f1f1f 6c1f0000 0100331f 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

BBED> set file 6 block 58
        FILE#           6
        BLOCK#          58

BBED> map  
 File: /opt/app/oracle/oradata/goolen/goolen01.dbf (6)
 Block: 58                                    Dba:0x0180003a
------------------------------------------------------------
 KTB Data Block (Table/Cluster)
 struct kcbh, 20 bytes                      @0       
 struct ktbbh, 96 bytes                     @20      
 struct kdbh, 14 bytes                      @124     
 struct kdbt[1], 4 bytes                    @138     
 sb2 kdbr[1]                                @142     
 ub1 freespace[7967]                        @144     
 ub1 rowdata[77]                            @8111    
 ub4 tailchk                                @8188    

BBED> p kcbh
struct kcbh, 20 bytes                       @0       
   ub1 type_kcbh                            @0        0x06
   ub1 frmt_kcbh                            @1        0xa2
   ub1 spare1_kcbh                          @2        0x00
   ub1 spare2_kcbh                          @3        0x00
   ub4 rdba_kcbh                            @4        0x018000b3
   ub4 bas_kcbh                             @8        0x001f7ac4
   ub2 wrp_kcbh                             @12       0x0000
   ub1 seq_kcbh                             @14       0x01
   ub1 flg_kcbh                             @15       0x06 (KCBHFDLC, KCBHFCKV)
   ub2 chkval_kcbh                          @16       0xac7e
   ub2 spare3_kcbh                          @18       0x0000

BBED> m /x 3a offset 4
 File: /opt/app/oracle/oradata/goolen/goolen01.dbf (6)
 Block: 58               Offsets:    4 to  515           Dba:0x0180003a
------------------------------------------------------------------------
 3a008001 c47a1f00 00000106 7eac0000 01000000 ce240100 e2791f00 00000000 
 03003200 b0008001 ffff0000 00000000 00000000 00000000 00800000 e2791f00 
 06000000 4b040000 b102c000 90011000 01204b00 c47a1f00 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00010100 ffff1400 
 331f1f1f 6c1f0000 0100331f 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes pe line>

BBED> p offset 4
kcbh.rdba_kcbh
--------------
ub4 rdba_kcbh                               @4        0x0180003a

BBED> p ktbbh
struct ktbbh, 96 bytes                      @20      
   ub1 ktbbhtyp                             @20       0x01 (KDDBTDATA)
   union ktbbhsid, 4 bytes                  @24      
      ub4 ktbbhsg1                          @24       0x000124ce
      ub4 ktbbhod1                          @24       0x000124ce
   struct ktbbhcsc, 8 bytes                 @28      
      ub4 kscnbas                           @28       0x001f79e2
      ub2 kscnwrp                           @32       0x0000
   sb2 ktbbhict                             @36       3
   ub1 ktbbhflg                             @38       0x32 (NONE)
   ub1 ktbbhfsl                             @39       0x00
   ub4 ktbbhfnx                             @40       0x018000b0
   struct ktbbhitl[0], 24 bytes             @44      
      struct ktbitxid, 8 bytes              @44      
         ub2 kxidusn                        @44       0xffff
         ub2 kxidslt                        @46       0x0000
         ub4 kxidsqn                        @48       0x00000000
      struct ktbituba, 8 bytes              @52      
         ub4 kubadba                        @52       0x00000000
         ub2 kubaseq                        @56       0x0000
         ub1 kubarec                        @58       0x00
      ub2 ktbitflg                          @60       0x8000 (KTBFCOM)
      union _ktbitun, 2 bytes               @62      
         sb2 _ktbitfsc                      @62       0
         ub2 _ktbitwrp                      @62       0x0000
      ub4 ktbitbas                          @64       0x001f79e2
   struct ktbbhitl[1], 24 bytes             @68      
      struct ktbitxid, 8 bytes              @68      
         ub2 kxidusn                        @68       0x0006
         ub2 kxidslt                        @70       0x0000
         ub4 kxidsqn                        @72       0x0000044b
      struct ktbituba, 8 bytes              @76      
         ub4 kubadba                        @76       0x00c002b1
         ub2 kubaseq                        @80       0x0190
         ub1 kubarec                        @82       0x10
      ub2 ktbitflg                          @84       0x2001 (KTBFUPB)
      union _ktbitun, 2 bytes               @86      
         sb2 _ktbitfsc                      @86       75
         ub2 _ktbitwrp                      @86       0x004b
      ub4 ktbitbas                          @88       0x001f7ac4
   struct ktbbhitl[2], 24 bytes             @92      
      struct ktbitxid, 8 bytes              @92      
         ub2 kxidusn                        @92       0x0000
         ub2 kxidslt                        @94       0x0000
         ub4 kxidsqn                        @96       0x00000000
      struct ktbituba, 8 bytes              @100     
         ub4 kubadba                        @100      0x00000000
         ub2 kubaseq                        @104      0x0000
         ub1 kubarec                        @106      0x00
      ub2 ktbitflg                          @108      0x0000 (NONE)
      union _ktbitun, 2 bytes               @110     
         sb2 _ktbitfsc                      @110      0
         ub2 _ktbitwrp                      @110      0x0000
      ub4 ktbitbas                          @112      0x00000000

BBED> m /x cd offset 24
 File: /opt/app/oracle/oradata/goolen/goolen01.dbf (6)
 Block: 58               Offsets:   24 to  535           Dba:0x0180003a
------------------------------------------------------------------------
 cd240100 e2791f00 00000000 03003200 b0008001 ffff0000 00000000 00000000 
 00000000 00800000 e2791f00 06000000 4b040000 b102c000 90011000 01204b00 
 c47a1f00 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00010100 ffff1400 331f1f1f 6c1f0000 0100331f 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

BBED> p offset 24
ktbbh.ktbbhsid.ktbbhod1
-----------------------
ub4 ktbbhod1                                @24       0x000124cd

BBED> sum apply;
Check value for File 6, Block 58:
current = 0xacf4, required = 0xacf4

BBED> exit

+++dbv检测一下，发现已经没有坏块
[oracle@localhost bbed]$ dbv file=/opt/app/oracle/oradata/goolen/goolen01.dbf
DBVERIFY: Release 11.2.0.1.0 - Production on Sat Jan 4 09:39:15 2014
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
DBVERIFY - Verification starting : FILE = /opt/app/oracle/oradata/goolen/goolen01.dbf

DBVERIFY - Verification complete
Total Pages Examined         : 2480
Total Pages Processed (Data) : 2276
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 2
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 192
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 10
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 2063096 (0.2063096)

SQL> alter system flush buffer_cache;
System altered.

+++但是在insert的时候还是报错
SQL> insert into  goolen  select * from dba_objects;
insert into  goolen  select * from dba_objects
*
ERROR at line 1:
ORA-08103: object no longer exists

SQL> select header_file,header_block from dba_segments where segment_name='GOOLEN';
HEADER_FILE HEADER_BLOCK
----------- ------------
          6           50

+++然后查询发现这个块被segment_name为6.58的占着
SQL> col segment_name for a35
SQL> select header_file,header_block,segment_name,segment_type from dba_segments where header_block=58;
HEADER_FILE HEADER_BLOCK SEGMENT_NAME                        SEGMENT_TYPE
----------- ------------ ----------------------------------- ------------------
          6           58 6.58                                TEMPORARY

SQL> select index_name from dba_indexes where table_name='GOOLEN';
no rows selected

+++直接从seg$里删除这条信息，再次insert已经OK
SQL> delete seg$ where file#=6 and block#=58;
1 row deleted.

SQL> commit;
Commit complete.

SQL> select header_file,header_block,segment_name,segment_type from dba_segments where header_block=58;

no rows selected

SQL> conn scott/tiger
Connected.
SQL> insert into scott.goolen select * from dba_objects;
72220 rows created.

SQL> commit;

Commit complete.