Linux Debugging 2 - System Profiling

1. The /proc filesystem

 

一个表示进程的虚拟文件系统，每个进程一个目录，名字是进程号，里面存有关于一个进程的各种信息，可以监管一个进程，依据map和fd信息做出诊断。同时/proc也可用于控制和诊断核心以及硬件。

 

1) kernel and hardware 诊断信息

refer : http://blog.chinaunix.net/u/31/showart_602243.html

例如： APM(advanced power management), execdomains(registered execution domains), fb(frame buffer information).

2) kernel tunable parameters /proc/sys 例如，

/home/a/j/nomad2:pwd/proc/sys/net/ipv4/home/a/j/nomad2:lsconf neigh tcp_memicmp_echo_ignore_all route tcp_moderate_rcvbuficmp_echo_ignore_broadcasts tcp_abc tcp_mtu_probingicmp_errors_use_inbound_ifaddr tcp_abort_on_overflow tcp_no_metrics_saveicmp_ignore_bogus_error_responses tcp_adv_win_scale tcp_orphan_retriesicmp_ratelimit tcp_allowed_congestion_control tcp_reorderingicmp_ratemask tcp_app_win tcp_retrans_collapseigmp_max_memberships tcp_available_congestion_control tcp_retries1igmp_max_msf tcp_base_mss tcp_retries2inet_peer_gc_maxtime tcp_congestion_control tcp_rfc1337inet_peer_gc_mintime tcp_dma_copybreak tcp_rmeminet_peer_maxttl tcp_dsack tcp_sackinet_peer_minttl tcp_ecn tcp_slow_start_after_idleinet_peer_threshold tcp_fack tcp_stdurgip_default_ttl tcp_fin_timeout tcp_syn_retriesip_dynaddr tcp_frto tcp_synack_retriesip_forward tcp_frto_response tcp_syncookiesip_local_port_range tcp_keepalive_intvl tcp_timestampsip_no_pmtu_disc tcp_keepalive_probes tcp_tso_win_divisorip_nonlocal_bind tcp_keepalive_time tcp_tw_recycleipfrag_high_thresh tcp_low_latency tcp_tw_reuseipfrag_low_thresh tcp_max_orphans tcp_window_scalingipfrag_max_dist tcp_max_ssthresh tcp_wmemipfrag_secret_interval tcp_max_syn_backlog tcp_workaround_signed_windowsipfrag_time tcp_max_tw_buckets/home/a/j/nomad2:echo 1 >ip_forward/home/a/j/nomad2:sysctl -w net.ipv4.ip_forward=1

3) 进程诊断信息

/home/a/j/nomad2:pwd/proc/22034/home/a/j/nomad2:ls -Fattr/ clear_refs cpuset exe@ io maps mountstats oom_score smaps statusauxv cmdline cwd@ fd/ limits mem numa_maps root@ stat task/cgroup coredump_filter environ fdinfo/ loginuid mounts oom_adj sched statm wchan

 

2. 中断

the following table shows the IRQ#, number of interrupts, responsible PIC, and registered handler.

/home/a/j/nomad2:pwd/proc/home/a/j/nomad2:cat interrupts CPU0 CPU1 0: 342 0 IO-APIC-edge timer 1: 2 0 IO-APIC-edge i8042 4: 2 0 IO-APIC-edge 8: 1 0 IO-APIC-edge rtc 9: 0 0 IO-APIC-fasteoi acpi 12: 4 0 IO-APIC-edge i8042 14: 459271 0 IO-APIC-edge libata 15: 0 0 IO-APIC-edge libata 16: 7635628 0 IO-APIC-fasteoi eth0 20: 338070 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb4, libata 21: 0 0 IO-APIC-fasteoi uhci_hcd:usb2 22: 0 0 IO-APIC-fasteoi uhci_hcd:usb3NMI: 0 0 Non-maskable interruptsLOC: 8867982 8281590 Local timer interruptsRES: 182963 469095 Rescheduling interruptsCAL: 242 90 function call interruptsTLB: 7039 16037 TLB shootdownsTRM: 0 0 Thermal event interruptsTHR: 0 0 Threshold APIC interruptsSPU: 0 0 Spurious interruptsERR: 0

 

/proc/stat的内容如下，

 

/home/a/j/nomad2:cat /proc/statcpu 3987145 0 1306724 6475556 8213 4534 21324 0 0cpu0 1991828 0 660103 3214978 7115 4534 12830 0 0cpu1 1995316 0 646621 3260578 1098 0 8494 0 0intr 8452956 343 2 0 0 2 0 0 0 1 0 0 0 4 0 461229 0 7652206 0 0 0 339169 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0ctxt 23576136btime 1291929146processes 231620procs_running 3procs_blocked 0

 

interrupts are per-process, and in SMP, multiple interrupts can be handled concurrently by the different CPUs.

 

3. /proc/kallsyms : all expoted module symbols from the kernel, and module name is in []

 

/home/a/j/nomad2:head /proc/kallsymsffffffff80200000 A _textffffffff80200000 T startup_64ffffffff802000b7 t ident_completeffffffff80200110 T secondary_startup_64ffffffff802001a8 T initial_codeffffffff802001b0 T init_rspffffffff802001b8 t bad_addressffffffff802001c0 T early_idt_handlerffffffff8020020b t early_recursion_flagffffffff8020020f t early_idt_msg

 

the symbols are entry points, which are used primarily by insmod when modules are inserted, and another use of ksyms is to map kernel structures when peeking into the kernel memory, by using /dev/kmem.

 

4. Dmesg (print or control the kernel ring buffer)

 

kernel messages may be rerouted to user mode via klogd.

/home/a/j/nomad2:dmesg |more[ 0.000000] Initializing cgroup subsys cpuset[ 0.000000] Initializing cgroup subsys cpu[ 0.000000] Linux version 2.6.24-22-generic (buildd@crested) (gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)) #1 SMP Mon Nov 24 19:35:06 UTC 2008 (Ubuntu 2.6.24-22.45-generic)[ 0.000000] Command line: root=UUID=e021d2fd-f6b7-40fa-b6b3-3d086aa71a80 ro quiet splash[ 0.000000] BIOS-provided physical RAM map:[ 0.000000] BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)[ 0.000000] BIOS-e820: 0000000000100000 - 00000000dffc0000 (usable)[ 0.000000] BIOS-e820: 00000000dffc0000 - 00000000dffcfc00 (ACPI data)[ 0.000000] BIOS-e820: 00000000dffcfc00 - 00000000dffff000 (reserved)[ 0.000000] BIOS-e820: 00000000f0000000 - 00000000f4000000 (reserved)[ 0.000000] BIOS-e820: 00000000fec00000 - 00000000fed00400 (reserved)[ 0.000000] BIOS-e820: 00000000fed13000 - 00000000feda0000 (reserved)[ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)[ 0.000000] BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)[ 0.000000] BIOS-e820: 0000000100000000 - 00000001ffffe000 (usable)[ 0.000000] BIOS-e820: 00000001ffffe000 - 0000000200000000 (reserved)[ 0.000000] BIOS-e820: 0000000200000000 - 0000000220000000 (usable)[ 0.000000] Entering add_active_range(0, 0, 160) 0 entries of 3200 used[ 0.000000] Entering add_active_range(0, 256, 917440) 1 entries of 3200 used[ 0.000000] Entering add_active_range(0, 1048576, 2097150) 2 entries of 3200 used[ 0.000000] Entering add_active_range(0, 2097152, 2228224) 3 entries of 3200 used

 

in /var/log,

/home/a/j/nomad2:pwd/var/log/home/a/j/nomad2:lsBastille daemon.log.2.gz dpkg.log.12.gz kern.log.3.gz mysql.log.2.gz syslog.6.gzaccount daemon.log.3.gz dpkg.log.2.gz lastlog mysql.log.3.gz sysstatapache2 daemon.log.4.gz dpkg.log.3.gz lastlog.1 mysql.log.4.gz udevapparmor debug dpkg.log.4.gz lpr.log mysql.log.5.gz user.logapt debug.0 dpkg.log.5.gz mail.err mysql.log.6.gz user.log.0auth.log debug.1.gz dpkg.log.6.gz mail.info mysql.log.7.gz user.log.1.gzauth.log.0 debug.2.gz dpkg.log.7.gz mail.log news user.log.2.gzauth.log.1.gz debug.3.gz dpkg.log.8.gz mail.warn pycentral.log user.log.3.gzauth.log.2.gz dist-upgrade dpkg.log.9.gz messages scrollkeeper.log user.log.4.gzauth.log.3.gz dmesg faillog messages.0 scrollkeeper.log.1 uucp.logauth.log.4.gz dmesg.0 fontconfig.log messages.1.gz scrollkeeper.log.2 uucp.log.0auth.log.5.gz dmesg.1.gz fsck messages.2.gz sulog vsftpd.logauth.log.6.gz dmesg.2.gz gdm messages.3.gz syslog vsftpd.log.1boot dmesg.3.gz installer messages.4.gz syslog.0 vsftpd.log.2btmp dmesg.4.gz iptraf messages.5.gz syslog.1.gz vsftpd.log.3btmp.1 dpkg.log kern.log mysql syslog.2.gz vsftpd.log.4daemon.log dpkg.log.1 kern.log.0 mysql.err syslog.3.gz wtmpdaemon.log.0 dpkg.log.10.gz kern.log.1.gz mysql.log syslog.4.gz wtmp.1daemon.log.1.gz dpkg.log.11.gz kern.log.2.gz mysql.log.1.gz syslog.5.gz wtmp.report

 

5. Devices

/sys for devices/modules just like /proc for processes.

/home/a/j/nomad2:pwd/sys/module/home/a/j/nomad2:ls -F8250/ battery/ fan/ jbd/ nfs_acl/ pty/ sr_mod/ac/ bitblit/ fbcon/ keyboard/ parport/ rcupdate/ sunrpc/acpi/ button/ font/ libata/ parport_pc/ rd/ tg3/aerdriver/ capability/ fuse/ lockd/ pata_acpi/ scsi_mod/ thermal/amd64_agp/ cdrom/ i8042/ lp/ pci_hotplug/ sd_mod/ tileblit/apparmor/ dcdbas/ iTCO_vendor_support/ mbcache/ pcspkr/ serio_raw/ uhci_hcd/ata_generic/ ehci_hcd/ iTCO_wdt/ mousedev/ printk/ sg/ usbcore/ata_piix/ evdev/ ip_tables/ mtrr/ processor/ shpchp/ vt/atkbd/ ext3/ iptable_filter/ nfs/ psmouse/ softcursor/ x_tables/

 

6. 2 kind processes

migration(runqueue -> idle) and watchdog(make sure cpu active)