调研内核调用栈方便的工具 kmalloc-top

原创文章，转载请注明：转载自Erlang非业余研究

本文链接地址:调研内核调用栈方便的工具 kmalloc-top

我们在研究内核的时候，看了内核代码后，就想着某个函数被谁谁调用。 调用路径有很多条，有热门的，有偏门的，但从代码不大容易看出。 如果我们能和gdb那样在函数上设个断点，看下内核函数的调用栈就清楚了。 但是如何统计热门路线呢？用systemtap就可以，参看这里，这里。

但是用systemtap写统计的时候，用到统计功能的话，如果你的采样点非常多，超过systemtap规定的上线，systemtap会选择罢工，直接推出，很不爽。

kmalloc-top就是为了解决这个问题写的一个perl脚本，原本用来调查内核中kmalloc的使用情况的，在一个繁忙的内核中,kmallo每秒会被调用成千上万次，明显会超过处理的上限。 所以kmalloc-top的方法是stap部分只负责收集堆栈信息，收集一个就写到标准输出一个，然后由perl脚本来进一步分析统计。

脚本位于：/usr/local/share/doc/systemtap/examples/memory/kmalloc-top

#The systemtap script that instruments the kmalloc
$script=”
global kmalloc_stack

probe kernel.function(\”__kmalloc\”) { kmalloc_stack[backtrace()]++ }

probe timer.ms(100), end
{
foreach (stack in kmalloc_stack) {
printf(\”\\n\”)
print_stack(stack)
printf(\”\\n\”)
printf(\”%d\\n\”, kmalloc_stack[stack])
}
delete kmalloc_stack
}
“;

我们使用的时候把probe点用自己感兴趣的替换掉就好。

它支持的功能见注释部分：

#!/usr/bin/perl
#
# This script accumulates the execution paths of all calls to kmalloc
# in the kernel. On Ctrl-C (or exit of the command provided by -c option),
# it sorts, filters and displays them on
# stdout.
#
# The -e (exclude) option can be used to specify a comma-separated list
# – any stack with contents matching any of the items in the list will
# be excluded from the output.
#
# The -m (min) option can be used to specify the minimum number of
# occurrences a stack needs to be included in the output.
#
# The -t (top) option can be used to specify printing only the
# top N stack traces.
#
# The -c (command) option runs starts the command and script
# only runs for the duration of the command.
#
# The -o (options) option passes the options to systemap.
#
# Usage: ./kmalloc-top [-m min] [-e exclude list] [-t top_n] [-c command]
# [-o options]
# Ctrl-c

在实际使用中我们会加入 -o –all-modules, 用于打印全调用栈信息，详细见这里

典型使用如下：
./kmalloc-top -o –all-modules -m 128 -t 10
取top 10, 每个至少被调用10次。

演示下：

$uname -r2.6.18-164.el5$sudo  /usr/local/share/doc/systemtap/examples/memory/kmalloc-top -o --all-modules  -m 10 -t 3Will print only the top 3 stacks.Will print stacks with counts >= 10.Press Ctrl-C to stop.ERROR: Skipped too many probes, check MAXSKIPPED or try again with stap -t for more details.WARNING: Number of errors: 0, skipped probes: 101This path seen 31 times: 0xffffffff800d9d3b : __kmalloc+0x0/0x9f [kernel] 0xffffffff8002e1e9 : __alloc_skb+0x5c/0x12d [kernel] 0xffffffff802214a3 : sock_alloc_send_pskb+0x7d/0x282 [kernel] 0xffffffff8004a71d : unix_stream_sendmsg+0x15f/0x346 [kernel] 0xffffffff80055261 : sock_sendmsg+0xf8/0x14a [kernel] 0xffffffff80220056 : sys_sendto+0x11c/0x14f [kernel] 0xffffffff8005d116 : system_call+0x7e/0x83 [kernel] 0x246 : __mod_error_mask1type56+0x16/0x30 [bnx2i] 0x2 : __mod_version79+0x2/0x10 [cnic] 0x4000000000000000 0x40000000000000 0x400000000000 0x4000000000 0x40000000 0x400000 0x4000 0x40 : __mod_description77+0x20/0x40 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic]This path seen 22 times: 0xffffffff800d9d3b : __kmalloc+0x0/0x9f [kernel] 0xffffffff8002e1e9 : __alloc_skb+0x5c/0x12d [kernel] 0xffffffff80025c28 : tcp_sendmsg+0x184/0xb0e [kernel] 0xffffffff80055261 : sock_sendmsg+0xf8/0x14a [kernel] 0xffffffff8021fb48 : sys_sendmsg+0x217/0x28a [kernel] 0xffffffff8005d116 : system_call+0x7e/0x83 [kernel] 0x202 : __mod_error_mask157+0x2/0x30 [bnx2i] 0x2 : __mod_version79+0x2/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x0 : __mod_version79+0x0/0x10 [cnic] 0x700000000000000 0x7000000000000 0x70000000000 0x700000000 0x7000000 0x70000This path seen 14 times: 0xffffffff800d9d3b : __kmalloc+0x0/0x9f [kernel] 0xffffffff8002e1e9 : __alloc_skb+0x5c/0x12d [kernel] 0xffffffff8003a7e8 : tcp_send_ack+0x20/0xf1 [kernel] 0xffffffff80252178 : tcp_delack_timer+0x180/0x1ed [kernel] 0xffffffff800968be : run_timer_softirq+0x133/0x1af [kernel] 0xffffffff8001235a : __do_softirq+0x89/0x133 [kernel] 0xffffffff8005e2fc : call_softirq+0x1c/0x28 [kernel] 0xffff81034ab93f98 0x20ffff81034ab93f 0xfe20ffff81034ab9 0xb8fe20ffff81034a 0x4ab8fe20ffff8103 0x34ab8fe20ffff81 0x81034ab8fe20ffff 0xff81034ab8fe20ff 0xffff81034ab8fe20 0x14ffff81034ab8fe 0xcb14ffff81034ab8 0x6cb14ffff81034a 0x8006cb14ffff8103 0xff8006cb14ffff81Num stacks before filtering: 271Num stacks after filtering: 3Total kmallocs (before filtering): 536Total kmallocs (after filtering): 67The filter stacks have 12.5 of the total kmallocs