使用oradebug ipc来检查cluster interconnect

来源:互联网 发布:自动化学报 软件学报 编辑:程序博客网 时间:2024/05/18 02:19
参考文章:
How to Use Oradebug to Get Trace File Name and Location (文档 ID 310830.1)
Tuning Inter-Instance Performance in RAC and OPS (文档 ID 181489.1)
Instances Unable To Start If MTU Size Is Different for Cluster_interconnect (文档 ID 300388.1)


[oracle@rac1 ~]$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.4.0 Production on Fri Jun 23 11:57:51 2017Copyright (c) 1982, 2013, Oracle.  All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Data Mining and Real Application Testing optionsSQL> oradebug setmypidStatement processed.SQL> oradebug ipcIPC information written to the trace fileSQL> oradebug tracefile_name/u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_17205.trcSQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Data Mining and Real Application Testing options[oracle@rac1 ~]$ 







/u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_17205.trc文件的内容如下:


Trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_17205.trcOracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Data Mining and Real Application Testing optionsORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1System name:LinuxNode name:rac1Release:3.10.0-327.el7.x86_64Version:#1 SMP Thu Oct 29 17:29:29 EDT 2015Machine:x86_64Instance name: orcl1Redo thread mounted by this instance: 1Oracle process number: 34Unix process pid: 17205, image: oracle@rac1 (TNS V1-V3)*** 2017-06-23 12:03:14.891*** SESSION ID:(1274.497) 2017-06-23 12:03:14.891*** CLIENT ID:() 2017-06-23 12:03:14.891*** SERVICE NAME:(SYS$USERS) 2017-06-23 12:03:14.891*** MODULE NAME:(sqlplus@rac1 (TNS V1-V3)) 2017-06-23 12:03:14.891*** ACTION NAME:() 2017-06-23 12:03:14.891 Processing Oradebug command 'setmypid'*** 2017-06-23 12:03:14.891Oradebug command 'setmypid' console output: <none>*** 2017-06-23 12:03:20.882Processing Oradebug command 'ipc'Dump of unix-generic skgm contextareaflags            000000f7realmflags           0000001fmapsize              00000800protectsize          00001000lcmsize              00001000seglen               00200000largestsize  0000000fba082000smallestsize 0000000008000000stacklimit     0x7ffee85188d0stackdir                   -1mode                      640magic                acc01adeHandle:        0x7f71abcd40b0 `/u01/app/oracle/product/11.2.0/db_1orcl1'Dump of unix-generic realm handle `/u01/app/oracle/product/11.2.0/db_1orcl1', flags = 00000000 Area #0 `Fixed Size' containing Subareas 0-0  Total size 0000000000229ae0 Minimum Subarea size 00000000   Area  Subarea    Shmid      Stable Addr      Actual Addr      0        0   327686 0x00000060000000 0x00000060000000                              Subarea size     Segment size                          000000000022a000 0000000018000000 Area #1 `Variable Size' containing Subareas 4-4  Total size 0000000df8000000 Minimum Subarea size 08000000   Area  Subarea    Shmid      Stable Addr      Actual Addr      1        4   360455 0x00000078000000 0x00000078000000                              Subarea size     Segment size                          0000000df8000000 0000000df8000000 Area #2 `Redo Buffers' containing Subareas 2-2  Total size 0000000007caf000 Minimum Subarea size 00000000   Area  Subarea    Shmid      Stable Addr      Actual Addr      2        2   327686 0x0000007034f000 0x0000007034f000                              Subarea size     Segment size                          0000000007caf000 0000000018000000 Area #3 `Base Allocator Control' containing Subareas 3-3  Total size 0000000000002000 Minimum Subarea size 00000000   Area  Subarea    Shmid      Stable Addr      Actual Addr      3        3   327686 0x00000077ffe000 0x00000077ffe000                              Subarea size     Segment size                          0000000000002000 0000000018000000 Area #4 `Slab Allocator Control' containing Subareas 1-1  Total size 0000000010125000 Minimum Subarea size 00000000   Area  Subarea    Shmid      Stable Addr      Actual Addr      4        1   327686 0x0000006022a000 0x0000006022a000                              Subarea size     Segment size                          0000000010125000 0000000018000000 Area #5 `skgm overhead' containing Subareas 5-5  Total size 0000000000005000 Minimum Subarea size 00000000   Area  Subarea    Shmid      Stable Addr      Actual Addr      5        5   393224 0x00000e70000000 0x00000e70000000                              Subarea size     Segment size                          0000000000005000 0000000000200000Dump of Linux-specific skgm contextsharedmmu 00000001shareddec        0used region        0: start 0000000060000000 length 000001c208000000Maximum processes:               = 1000Number of semaphores per set:    = 125Semaphores key overhead per set: = 4User Semaphores per set:         = 121Number of semaphore sets:        = 9Semaphore identifiers:           = 9Semaphore List=524298557067589836622605655374688143720912753681786450-------------- system semaphore information ------------------- Message Queues --------key        msqid      owner      perms      used-bytes   messages    ------ Shared Memory Segments --------key        shmid      owner      perms      bytes      nattch     status      0x00000000 0          root       644        80         2                       0x00000000 32769      root       644        16384      2                       0x00000000 65538      root       644        280        2                       0x00000000 196611     grid       640        4096       0                       0x00000000 229380     grid       640        4096       0                       0xa8c203cc 262149     grid       640        4096       0                       0x00000000 327686     oracle     640        402653184  43                      0x00000000 360455     oracle     640        59995324416 43                      0x9f8b9fa4 393224     oracle     640        2097152    43                      ------ Semaphore Arrays --------key        semid      owner      perms      nsems     0x000000a7 0          root       600        1         0x1fe737a8 131073     grid       640        125       0x1fe737a9 163842     grid       640        125       0x1fe737aa 196611     grid       640        125       0x1fe737ab 229380     grid       640        125       0x1fe737ac 262149     grid       640        125       0x1fe737ad 294918     grid       640        125       0x1fe737ae 327687     grid       640        125       0x1fe737af 360456     grid       640        125       0x1fe737b0 393225     grid       640        125       0xe7ec25d0 524298     oracle     640        125       0xe7ec25d1 557067     oracle     640        125       0xe7ec25d2 589836     oracle     640        125       0xe7ec25d3 622605     oracle     640        125       0xe7ec25d4 655374     oracle     640        125       0xe7ec25d5 688143     oracle     640        125       0xe7ec25d6 720912     oracle     640        125       0xe7ec25d7 753681     oracle     640        125       0xe7ec25d8 786450     oracle     640        125       ksxpdmp:  facility 0 (?) (0x0, (nil)) counts 0, 0Dumping the osd state  Dumping the osd context (brief)SKGXP:[7f71abce3c48.0]{ctx}: SKGXPCTX: 0x0x7f71abce3c48 ctx   SKGXP:[7f71abce3c48.1]{ctx}: Dump start: 1498190600906194 (gtod usec)SKGXP:[7f71abce3c48.2]{ctx}: Compatible: 0 0.0 0SKGXP:[7f71abce3c48.3]{ctx}: SKGXP Versions: 3.3.4.2.2.c1.7(3.2.4.1)SKGXP:[7f71abce3c48.4]{ctx}: SKGXP:[7f71abce3c48.5]{ctx}: WAIT HISTORYSKGXP:[7f71abce3c48.6]{ctx}: Wait Time  Time since     Fast reaps  Wait Type  Return CodeSKGXP:[7f71abce3c48.7]{ctx}:   (ms)     prev wait(ms)  before                           SKGXP:[7f71abce3c48.8]{ctx}: ---------  -------------- ----------- ---------  -----------SKGXP:[7f71abce3c48.9]{ctx}: wait delta 321 sec (321964 msec) ctx ts 0 sec (1 msec) last ts 0 msec twait secs 5 (blocked 0) curts = 321964SKGXP:[7f71abce3c48.10]{ctx}: SKGXP:[7f71abce3c48.11]{ctx}: EXTENDED WAIT HISTORY STATSSKGXP:[7f71abce3c48.12]{ctx}: Time       Prev      Ncomp     Nwait     Nloop     Nsel      Select    No Svc     Events      SKGXP:[7f71abce3c48.13]{ctx}: (-ms)      (ms)                                              (ms)      (ms)                   SKGXP:[7f71abce3c48.14]{ctx}: ---------- --------- --------- --------- --------- --------- --------- ---------- ----------- SKGXP:[7f71abce3c48.15]{ctx}: current (0)SKGXP:[7f71abce3c48.16]{ctx}:     321964         0         0         0         0         0         0          0           0 SKGXP:[7f71abce3c48.17]{ctx}: SKGXP:[7f71abce3c48.18]{ctx}: user cpu time since last wait 0 sec 140728898420736 ticksSKGXP:[7f71abce3c48.19]{ctx}: system cpu time since last wait 0 sec 140728898420736 ticksSKGXP:[7f71abce3c48.20]{ctx}: locked 2SKGXP:[7f71abce3c48.21]{ctx}: blocked 0SKGXP:[7f71abce3c48.22]{ctx}: timed wait receives 0SKGXP:[7f71abce3c48.23]{ctx}: fast reaps since last wait 0SKGXP:[7f71abce3c48.24]{ctx}: context timestamp 0x1SKGXP:[7f71abce3c48.25]{ctx}: flags=1 flags1=30 zcpyflg=5824 iflags=40820 aflags=80 antopts 0 trcflags=0 trclevel=3SKGXP:[7f71abce3c48.26]{ctx}: rcvbuf=131072 sndbuf=262144SKGXP:[7f71abce3c48.27]{ctx}: expire idle_ach=120 def_reap=60 if_detect=60 hb=180 nopath=5 vrpc=30 (on=300, off=30)SKGXP:[7f71abce3c48.28]{ctx}: mcph alloc=0 inuse=0 inactive=0 thresh=0 hashtbl=(nil)SKGXP:[7f71abce3c48.29]{ctx}: SKGXP:[7f71abce3c48.30]{ctx}: admno 0x3d9849db admport:SKGXP:[7f71abce3c48.31]{ctx}: SSKGXPT 0x7f71abce51b8 flags 0x5 { READPENDING } sockno 4 IP 169.254.9.62 UDP 62213 lerr 0SKGXP:[7f71abce3c48.32]{ctx}: SKGXP:[7f71abce3c48.33]{ctx}: post port: SKGXP:[7f71abce3c48.34]{ctx}: SSKGXPT 0x7f71abce5680 flags 0x0  sockno 7 IP 127.0.0.1 UDP 62213 lerr 0SKGXP:[7f71abce3c48.35]{ctx}: SKGXP:[7f71abce3c48.36]{ctx}: sconno [6eb2aebf, 6eb2aebf]=0 aconno [504ea0a2, 504ea0a2]=0SKGXP:[7f71abce3c48.37]{ctx}: SKGXP:[7f71abce3c48.38]{ctx}: SSKGXPCTX 0x7f71abce3c48 flags=0x0 npt=3 zcomp=0 lerr=0SKGXP:[7f71abce3c48.39]{ctx}: SKGXP:[7f71abce3c48.40]{ctx}: time handleSKGXP:[7f71abce3c48.41]{ctx}: SSKGXPTH 0x7f71abce4c68 start 1463076436732 last 1463076758696 count 0 thresh 0 tcb 1SKGXP:[7f71abce3c48.42]{ctx}: SKGXP:[7f71abce3c48.43]{ctx}: memory (native) allocs 6 free 1 lost 0 out 8376 max 32768SKGXP:[7f71abce3c48.44]{ctx}: SKGXP:[7f71abce3c48.45]{ctx}: ctx(10176) sctx(1960) rqh(1064) cid(24)SKGXP:[7f71abce3c48.46]{ctx}: prt(480) cnh(1984/15088) ach(1568/14672) rpc(2448) pid(64)SKGXP:[7f71abce3c48.47]{ctx}: rgnst(472) rgnpt(296) rgn(15784) rbd(120) bid(64) mcph(8416/21520) mupdh(1168/14272)SKGXP:[7f71abce3c48.48]{ctx}: mcb(852008) smap(4184) nsm(2084) me(1152) mcbe(48) mcbi(0) epid(40)SKGXP:[7f71abce3c48.49]{ctx}: mhdr(76) fhdr(28) madm(272) aggpid(32)SKGXP:[7f71abce3c48.50]{ctx}: sta msc(448) cnh(344) prt(216) ep(48)SKGXP:[7f71abce3c48.51]{ctx}: aggpt(208) swin(384) finfo(176) segf(200)SKGXP:[7f71abce3c48.52]{ctx}: SKGXP:[7f71abce3c48.53]{ctx}: done QueueSKGXP:[7f71abce3c48.54]{ctx}: no completed requestsSKGXP:[7f71abce3c48.55]{ctx}: SKGXP:[7f71abce3c48.56]{ctx}: connection QueueSKGXP:[7f71abce3c48.57]{ctx}: no pending connect disconnect operationsSKGXP:[7f71abce3c48.58]{ctx}: SKGXP:[7f71abce3c48.59]{ctx}: sends waiting to be transmittedSKGXP:[7f71abce3c48.60]{ctx}: no sends waiting to be transmittedSKGXP:[7f71abce3c48.61]{ctx}: SKGXP:[7f71abce3c48.62]{ctx}: pending ack QueueSKGXP:[7f71abce3c48.63]{ctx}: no send requests pending ackSKGXP:[7f71abce3c48.64]{ctx}: SKGXP:[7f71abce3c48.65]{ctx}: Vrpc requests pending replySKGXP:[7f71abce3c48.66]{ctx}: SKGXP:[7f71abce3c48.67]{ctx}: Port queueSKGXP:[7f71abce3c48.68]{ctx}: SKGXP:[7f71abce3c48.69]{ctx}: Region queueSKGXP:[7f71abce3c48.70]{ctx}: SKGXPRGN: 0x7f71a8a3c258SKGXP:[7f71abce3c48.71]{ctx}:    region id 0x23ec rgn idx: 0 rgn key: 1329307295SKGXP:[7f71abce3c48.72]{ctx}:    base address 0x78000000 size: 59995324416SKGXP:[7f71abce3c48.73]{ctx}:    flags 0x0 state: 0x2 largest buffer: 0SKGXP:[7f71abce3c48.74]{ctx}:    num bids prepared: 0SKGXP:[7f71abce3c48.75]{ctx}: SKGXPRGNSTATE: 0x7f71abce4fd8SKGXP:[7f71abce3c48.76]{ctx}: rgns mapped: 1 num shared: 1 max: 1SKGXP:[7f71abce3c48.77]{ctx}: flags: 0 bseqno: 11886 nxtrgnno: 9196 mmapcnt: 1SKGXP:[7f71abce3c48.78]{ctx}: shared ports open: 1 limits (1,16)SKGXP:[7f71abce3c48.79]{ctx}: shared bids prepared: 0 hiwat: 0SKGXP:[7f71abce3c48.80]{ctx}: space - used: 0 max possible: 4194304 sock limit: 524288SKGXP:[7f71abce3c48.81]{ctx}: rgnarr[0x7f71abced860(32)] rgnpts[0]=0x7f71abced980SKGXP:[7f71abce3c48.82]{ctx}:   REGION ARRAYSKGXP:[7f71abce3c48.83]{ctx}: rgn mapped[0] = 0x7f71a8a3c258SKGXP:[7f71abce3c48.84]{ctx}:   SHARED PORT ARRAYSKGXP:[7f71abce3c48.85]{ctx}:    rgn port[0]=0x7f71abced980SKGXP:[7f71abce3c48.86]{ctx}: SKGXPRGNPORT: 0x7f71abced980SKGXP:[7f71abce3c48.87]{ctx}: nbids: 0 space: 0 used: 0SKGXP:[7f71abce3c48.88]{ctx}: SSKGXPT 0x7f71abced980 flags 0x0  sockno 10 IP 169.254.9.62 UDP 42117 lerr 0 ---------->>>此处SKGXP:[7f71abce3c48.89]{ctx}:   SKGXPGPID Internet address 169.254.9.62 UDP port number 42117, mask 16 ---------->>>此处SKGXP:[7f71abce3c48.90]{ctx}: SKGXP:[7f71abce3c48.91]{ctx}: Dumping Connection Handle Table (0x7f71abce5918)SKGXP:[7f71abce3c48.92]{ctx}:                hdl     sconno     aconno      admno   RmtPid  state   seq#       msgs     rtrans    credits      rtt   last ack         id               ip SKGXP:[7f71abce3c48.93]{ctx}: SKGXP:[7f71abce3c48.94]{ctx}: Dumping Accept Handle Table ((nil))SKGXP:[7f71abce3c48.95]{ctx}: SKGXP:[7f71abce3c48.96]{ctx}: Dump end: 1498190600906893 (gtod usec)SKGXP:[7f71abce3c48.97]{ctx}: SKGXPCTX: 0x0x7f71abce3c48 completeSKGXP:[7f71abce3c48.98]{ctx}: Done dumping the osd stateDumping ksxp state  ksxppg=0x7f71a8cb47a8 ksxpsg=0xe49972028 ksxpsg_a=0xe49972028ksxpssg=0xe49971d70 rm=0xe69745528proc state: (pid: 34) [flg: 1 sg: 1] curts 1498190600 wtctr 0  Dumping ksxp contexts    Context[2] 0x7f71a8a31fb0 GES state 1      Dumping region queue       region count: 0      Dumping connection queue       connection count: 0      Dumping done requests queue       done requests count: 0      Dumping mupq requests queue       mupq requests count: 0      Dumping zombie requests queue       zombie requests count: 0      Dumping delayed callback requests queue       delayed callback requests count: 0    Context[1] 0x7f71abce3798 gc ksxp component context state 1      Dumping region queue       region count: 1      Dumping connection queue       connection count: 0      Dumping done requests queue       done requests count: 0      Dumping mupq requests queue       mupq requests count: 0      Dumping zombie requests queue       zombie requests count: 0      Dumping delayed callback requests queue       delayed callback requests count: 0  Dumping pending request queue   pending request count: 0*** 2017-06-23 12:03:20.907Oradebug command 'ipc' console output: IPC information written to the trace file*** 2017-06-23 12:03:40.517Processing Oradebug command 'tracefile_name'*** 2017-06-23 12:03:40.517Oradebug command 'tracefile_name' console output: /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_17205.trc

查看该主机的IP地址情况:

[oracle@rac1 ~]$ ip addr show1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host        valid_lft forever preferred_lft forever2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000    link/ether 0c:c4:7a:ac:3c:c8 brd ff:ff:ff:ff:ff:ff    inet 192.168.100.101/24 brd 192.168.100.255 scope global enp5s0f0       valid_lft forever preferred_lft forever    inet 192.168.100.193/24 brd 192.168.100.255 scope global secondary enp5s0f0:1       valid_lft forever preferred_lft forever    inet 192.168.100.191/24 brd 192.168.100.255 scope global secondary enp5s0f0:3       valid_lft forever preferred_lft forever    inet6 fe80::ec4:7aff:feac:3cc8/64 scope link        valid_lft forever preferred_lft forever3: enp5s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000    link/ether 0c:c4:7a:ac:3c:c9 brd ff:ff:ff:ff:ff:ff    inet 172.17.0.1/24 brd 172.17.0.255 scope global enp5s0f1       valid_lft forever preferred_lft forever    inet 169.254.9.62/16 brd 169.254.255.255 scope global enp5s0f1:1 --------->>>>注意此处.       valid_lft forever preferred_lft forever    inet6 fe80::ec4:7aff:feac:3cc9/64 scope link        valid_lft forever preferred_lft forever4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7f:11 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff    inet 10.10.10.101/24 brd 10.10.10.255 scope global ib0       valid_lft forever preferred_lft forever    inet6 fe80::7efe:9003:91:7f11/64 scope link        valid_lft forever preferred_lft forever5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7f:12 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff6: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP qlen 1024    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7e:d1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff    inet 10.10.11.102/24 brd 10.10.11.255 scope global ib2       valid_lft forever preferred_lft forever    inet6 fe80::7efe:9003:91:7ed1/64 scope link        valid_lft forever preferred_lft forever7: ib3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN qlen 1024    link/infiniband a0:00:03:00:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:91:7e:d2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN     link/ether 52:54:00:8f:5a:c4 brd ff:ff:ff:ff:ff:ff    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0       valid_lft forever preferred_lft forever9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500    link/ether 52:54:00:8f:5a:c4 brd ff:ff:ff:ff:ff:ff[oracle@rac1 ~]$