ORA-00600 [504]

ALERT日志：
Wed Sep 10 09:00:53 2014
Errors in file /u01/app/oracle/diag/rdbms/trsendb/trsendb2/trace/trsendb2_ora_40371414.trc  (incident=821340):
ORA-00600: internal error code, arguments: [504], [0x700005DEC63C610], [8], [2], [ges process parent latch], [2062], [0], [0x000000000], [], [], [], []
ORA-28000: the account is locked
Wed Sep 10 09:00:53 2014
Errors in file /u01/app/oracle/diag/rdbms/trsendb/trsendb2/trace/trsendb2_ora_35128316.trc  (incident=821539):
ORA-00600: internal error code, arguments: [504], [0x700005DEC41FF50], [8], [2], [ges process parent latch], [550], [0], [0x000000000], [], [], [], []
ORA-28000: the account is locked
Incident details in: /u01/app/oracle/diag/rdbms/trsendb/trsendb2/incident/incdir_821340/trsendb2_ora_40371414_i821340.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
......

trc文件的堆栈调用信息
......
dbgexExplicitEndInc  call     dbgexPhaseII()       616C6572745F7374 ?
()+476                                             646462322E6C6F67 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
dbgeEndDDEInvocatio  call     dbgexExplicitEndInc  1108B5D00 ? 110A630A8 ?
nImpl()+544                   ()                   FFFFFFFFFFF3AE0 ? 110454630 ?
                                                   1108B5D00 ? 110104D80 ?
                                                   FFFFFFFFFFF3C00 ? 1108B5D00 ?
dbgeEndDDEInvocatio  call     dbgeEndDDEInvocatio  1108B5D00 ? 110A630A8 ?
n()+48                        nImpl()              109C0B42C ? 000000000 ?
                                                   000000281 ? 700005DEC57DDF8 ?
                                                   000000002 ? 1108B5D00 ?
ksl_level_check()+1  call     dbgeEndDDEInvocatio  1F8000001F8 ? 000000002 ?
356                           n()                  700005DEC57DDF8 ? 000000000 ?
                                                   000000008 ? 000000000 ?
                                                   000000002 ? 000000001 ?
kslgetl()+852        call     ksl_level_check()    000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   FFFFFFFFFFF4890 ? 000000000 ?
                                                   000000000 ? 000000001 ?
kjucll()+876         call     kslgetl()            000000001 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   700005DE9BD7900 ?
                                                   FFFFFFFFFFF4BF4 ? 000000001 ?
                                                   110A3057E ?
kjuscl()+2956        call     kjucll()             0100004B0 ? 0D622D858 ?
                                                   000000000 ? 000000003 ?
                                                   000000026 ? 700005E0B8D15B8 ?
                                                   FFFFFFFFFFF4B80 ? 000000023 ?
ksiprls()+608        call     kjuscl()             700005EEFA20260 ? 000000000 ?
                                                   000000000 ? 1000000000 ?
                                                   70000578F855DC8 ? 000000003 ?
                                                   700005D9FE42E20 ? 000000002 ?
kqlmLock()+2220      call     ksiprls()            000000000 ? 000000000 ?
                                                   700005C8E81A798 ? 000000002 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ? 100000000 ?
kqlmClusterLock()+2  call     kqlmLock()           101255454 ?
56                                                 340C13193150B2D1 ?
                                                   FFFFFFFFFFF52B0 ? 110454630 ?
                                                   1001FC5F8 ? 700005EEFA201A8 ?
                                                   FFFFFFFFFFF52F0 ? 109B2D820 ?
kgllkdl()+2716       call     kqlmClusterLock()    000000000 ? 000000000 ?
                                                   FFFFFFFFFFF5410 ? 1099CD7D0 ?
                                                   FFFFFFFFFFF5520 ?
                                                   700005E69643608 ?
                                                   FFFFFFFFFFF5430 ?
                                                   700005EEFA200A8 ?
kgllkds()+104        call     kgllkdl()            E00000000000000 ? 000000000 ?
                                                   100000001 ? 110454630 ?
                                                   70000578F855DC8 ?
                                                   FFFFFFFFFFF5D18 ? 110000478 ?
                                                   7000055763ED758 ?
kglUnLock()+368      call     kgllkds()            FFFFFFFFFFF5D18 ?
                                                   7000000000330D8 ?
                                                   FFFFFFFFFFF5970 ? 000000000 ?
......
                                        
MOS文档:
Following error occurred :

ORA-00600: internal error code, arguments: [504], [0x09B667498], [8], [2], [ges process parent latch], [55], [0], [0x000000000], [], [], [], []

----- Beginning of Customized Incident Dump(s) -----
*** LATCH HIERARCHY ERROR ***
An attempt to get the 'ges process parent latch' latch (child #55) at level 2 (address = 0x9b667498)
from location 'kjp.h LINE:854 ID:kjucll: closing lock' (also see call stack below)
conflicts with the following held latch(es):
Level 3: 'Read Only Database Account Status' (address = 0x60033f10)
gotten from location 'kzs.h LINE:1494 ID:kpolnb: failed login count'
----- End of Customized Incident Dump(s) -----

*** 2012-09-25 11:55:04.985
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- SQL Statement (None) -----
Current SQL information unavailable - no cursor.

----- Call Stack Trace -----

ksl_level_check kslgetl kjucll kjuscl ksiprls kqlmLock kqlmClusterLock kgllkdl

[03]: ksl_level_check [VOS]<-- Signaling
[04]: kslgetl [VOS]
[05]: kjucll []
[06]: kjuscl []
[07]: ksiprls [KSI]
[08]: kqlmLock [LIBCACHE]
[09]: kqlmClusterLock [LIBCACHE]
[10]: kgllkdl [LIBCACHE]

MOS文档上给出的解决方案：
The issue is investigated though bug 11901263 which closed as duplicate of bug 14508968

从堆栈的信息比对来看，就是此bug 14508968，因此，可以到mos上去获得此bug的补丁，打上即可

虽然是触发bug，从alert日志及dba_users视图来看，有个用户的密码被锁，再从aud$视图发现有1017和28000；发现有个同事，用错误密码尝试10次将用户锁住了
select userid,userhost,terminal,returncode,comment$text, CAST ((FROM_TZ(ntimestamp#,'00:00') AT LOCAL) AS DATE)
from sys.aud$ where returncode<>0 and userid='trsen' ;

总结：
这次的bug，可能是用户被锁，高并发访问此用户的数据，导致bug出现，当发现实例down以后，我立刻去起instance和service，发现还是down instance，并且日志有如下信息，
当我们解锁用户后，就没有触发此bug了，建议打补丁。

ORA-28000: the account is locked
......
opiodr aborting process unknown ospid (10027970) as a result of ORA-1092
Wed Sep 10 09:58:13 2014
ORA-1092 : opitsk aborting process
Wed Sep 10 09:58:13 2014
ORA-1092 : opitsk aborting process
Wed Sep 10 09:58:13 2014
License high water mark = 1610
Wed Sep 10 09:58:21 2014
Instance terminated by PMON, pid = 45613456
USER (ospid: 49349490): terminating the instance
Instance terminated by USER, pid = 49349490
Wed Sep 10 09:58:27 2014
Starting ORACLE instance (normal)