如何分析 kernel panic

转载：http://blog.chinaunix.net/space.php?uid=22415790&do=blog&id=1774104

the below is a example of kernel panic log.

[  875.497504] Unable to handle kernel paging request at virtual address 4042a00c
[  875.499110] pgd = c0004000
[  875.501797] [4042a00c] *pgd=00000000
[  875.505364] Internal error: Oops: 5 [#1] PREEMPT
[  875.509957] Modules linked in: dhd
[  875.513350] CPU: 0    Not tainted  (2.6.29 #21)
[  875.517877] PC is at lock_acquire+0x60/0x74
[  875.522024] LR is at 0x0
[  875.524544] pc : [<c0090e10>]    lr : [<00000000>]    psr: 60000093
[  875.524557] sp : c042bea0  ip : c042a000  fp : c042becc
[  875.536004] r10: 0000e0cd  r9 : 00000000  r8 : 16027ae8
[  875.541210] r7 : 0000036b  r6 : 00000000  r5 : 4042a000  r4 : 60000093
[  875.547717] r3 : 00000000  r2 : 00000000  r1 : 00000000  r0 : 00000000
[  875.554230] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[  875.561610] Control: 10c5787d  Table: 1c0ec019  DAC: 00000017
[  875.567337] Process swapper (pid: 0, stack limit = 0xc042a2e0)
[  875.573157] Stack: (0xc042bea0 to 0xc042c000)
[  875.577490] bea0: 00000002 00000080 00000000 c00735e4 c0036c9c c04d4b40 c00735e4 1d34ce80
[  875.585737] bec0: c042bef4 c042bed0 c02fcef8 c0090dbc 00000002 00000000 c00735e4 c04d6010
[  875.593984] bee0: 0000e0cd c04d4b40 c042bf3c c042bef8 c00735e4 c02fcebc c0082d34 c006d870
[  875.602230] bf00: c042bf60 00000001 c002e518 c045e1a8 c042bf3c 0000e0cd c04d49a0 1d34ce80
[  875.610477] bf20: 0000036b 16027ae8 00000000 0000001f c042bf94 c042bf40 c008b1f8 c00735cc
[  875.618724] bf40: 00000000 c999a0a0 1d9e6627 0000036b 0000e0cd c00622a4 60000013 c042f318
[  875.626970] bf60: 1d9e6627 0000036b 0000001f c002e518 c042a000 c002e518 c045e1a8 16027ae8
[  875.635217] bf80: 510f00f2 0000001f c042bfb4 c042bf98 c002e418 c008b028 c0889a90 c045dd64
[  875.643464] bfa0: c0029ea4 c042ec08 c042bfc4 c042bfb8 c02f76a4 c002e3f0 c042bff4 c042bfc8
[  875.651710] bfc0: c0008bc0 c02f7644 c0008514 00000000 00000000 c0029ea4 00000000 10c5387d
[  875.659957] bfe0: c045e25c c002a2a8 00000000 c042bff8 16008034 c000894c 00000000 00000000
[  875.668204] Backtrace:
[  875.670637] [<c0090db0>] (lock_acquire+0x0/0x74) from [<c02fcef8>] (_spin_lock+0x48/0x58)
[  875.678797]  r6:1d34ce80 r5:c00735e4 r4:c04d4b40
[  875.683397] [<c02fceb0>] (_spin_lock+0x0/0x58) from [<c00735e4>] (get_next_timer_interrupt+0x24/0x224)
[  875.692684]  r5:c04d4b40 r4:0000e0cd
[  875.696244] [<c00735c0>] (get_next_timer_interrupt+0x0/0x224) from [<c008b1f8>] (tick_nohz_stop_sched_tick+0x1dc/0x434)
[  875.707004] [<c008b01c>] (tick_nohz_stop_sched_tick+0x0/0x434) from [<c002e418>] (cpu_idle+0x34/0x6c)
[  875.716204] [<c002e3e4>] (cpu_idle+0x0/0x6c) from [<c02f76a4>] (rest_init+0x6c/0x80)
[  875.723937]  r7:c042ec08 r6:c0029ea4 r5:c045dd64 r4:c0889a90
[  875.729577] [<c02f7638>] (rest_init+0x0/0x80) from [<c0008bc0>] (start_kernel+0x280/0x2e4)
[  875.737824] [<c0008940>] (start_kernel+0x0/0x2e4) from [<16008034>] (0x16008034)
[  875.745197] Code: e58dc008 e59bc00c e58dc00c ebfffdc6 (e595300c)
[  875.751446] Kernel panic - not syncing: Fatal exception

1.
the below is the analysis result:
[  875.497504] Unable to handle kernel paging request at virtual address 4042a00c
the whole description of the panic.

2.
Modules linked in: dhd
it just means there is one module whose name is dhd.ko while panic happens.
not means the panic is in this module.

3.
Process swapper (pid: 0, stack limit = 0xc042a2e0)
pid of the panic process is 0.

4.
[  875.517877] PC is at lock_acquire+0x60/0x74
[  875.522024] LR is at 0x0
[  875.524544] pc : [<c0090e10>]    lr : [<00000000>]    psr: 60000093
[  875.524557] sp : c042bea0  ip : c042a000  fp : c042becc
[  875.536004] r10: 0000e0cd  r9 : 00000000  r8 : 16027ae8
[  875.541210] r7 : 0000036b  r6 : 00000000  r5 : 4042a000  r4 : 60000093
[  875.547717] r3 : 00000000  r2 : 00000000  r1 : 00000000  r0 : 0000000

the log shows the value of ARM registers.
0x74 means the whole length of lock_acquire() function in disassemble format.
0x60 means the key point while panic happens.
gdb the output file which include lock_acquire() function and then disassemble the function to make sure which line has issue.
for example.
arm-linux-gdb lockdep.o
disassemble lock_acquire

5.
[  875.573157] Stack: (0xc042bea0 to 0xc042c000)
[  875.577490] bea0: 00000002 00000080 00000000 c00735e4 c0036c9c c04d4b40 c00735e4 1d34ce80
[  875.585737] bec0: c042bef4 c042bed0 c02fcef8 c0090dbc 00000002 00000000 c00735e4 c04d6010
[  875.593984] bee0: 0000e0cd c04d4b40 c042bf3c c042bef8 c00735e4 c02fcebc c0082d34 c006d870
[  875.602230] bf00: c042bf60 00000001 c002e518 c045e1a8 c042bf3c 0000e0cd c04d49a0 1d34ce80
[  875.610477] bf20: 0000036b 16027ae8 00000000 0000001f c042bf94 c042bf40 c008b1f8 c00735cc
[  875.618724] bf40: 00000000 c999a0a0 1d9e6627 0000036b 0000e0cd c00622a4 60000013 c042f318
[  875.626970] bf60: 1d9e6627 0000036b 0000001f c002e518 c042a000 c002e518 c045e1a8 16027ae8
[  875.635217] bf80: 510f00f2 0000001f c042bfb4 c042bf98 c002e418 c008b028 c0889a90 c045dd64
[  875.643464] bfa0: c0029ea4 c042ec08 c042bfc4 c042bfb8 c02f76a4 c002e3f0 c042bff4 c042bfc8
[  875.651710] bfc0: c0008bc0 c02f7644 c0008514 00000000 00000000 c0029ea4 00000000 10c5387d
[  875.659957] bfe0: c045e25c c002a2a8 00000000 c042bff8 16008034 c000894c 00000000 00000000
it contains the context of the Stack from 0xc042bea0 to 0xc042c000.

6.
[  875.668204] Backtrace:
[  875.670637] [<c0090db0>] (lock_acquire+0x0/0x74) from [<c02fcef8>] (_spin_lock+0x48/0x58)
[  875.678797]  r6:1d34ce80 r5:c00735e4 r4:c04d4b40
[  875.683397] [<c02fceb0>] (_spin_lock+0x0/0x58) from [<c00735e4>] (get_next_timer_interrupt+0x24/0x224)
[  875.692684]  r5:c04d4b40 r4:0000e0cd
[  875.696244] [<c00735c0>] (get_next_timer_interrupt+0x0/0x224) from [<c008b1f8>] (tick_nohz_stop_sched_tick+0x1dc/0x434)
[  875.707004] [<c008b01c>] (tick_nohz_stop_sched_tick+0x0/0x434) from [<c002e418>] (cpu_idle+0x34/0x6c)
[  875.716204] [<c002e3e4>] (cpu_idle+0x0/0x6c) from [<c02f76a4>] (rest_init+0x6c/0x80)
[  875.723937]  r7:c042ec08 r6:c0029ea4 r5:c045dd64 r4:c0889a90
[  875.729577] [<c02f7638>] (rest_init+0x0/0x80) from [<c0008bc0>] (start_kernel+0x280/0x2e4)
[  875.737824] [<c0008940>] (start_kernel+0x0/0x2e4) from [<16008034>] (0x16008034)
it means function call stack. from the above log, the call stack is:
start_kernel() -> rest_init() -> cpu_idle() -> tick_nohz_stop_sched_tick() -> get_next_timer_interrupt() -> _spin_lock() -> lock_acquire().

7.
[  875.745197] Code: e58dc008 e59bc00c e58dc00c ebfffdc6 (e595300c)
it means the latest instruction.
e58dc008     str    ip, [sp, #8]
e59bc00c     ldr    ip, [fp, #12]
e58dc00c     str    ip, [sp, #12]
ebfffdc6        ......