linux 性能分析常规逻辑和手段总结

一. 追查cpu占用较高的进程(线程)
  1 . 如何查找出当前系统中占用cpu或者内存最高的进程?   

ps aux |sort -rn -k 3 |head -n3  查找出当前系统中cpu资源占用前三名的进程, -k 3 表示用ps结果中的第三列排序,ps aux输出结果列顺序为:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND,因此要是按照mem排序使用 k 4

[root@b28-1330 ~]# ps aux |sort -rn -k 3 |head -n3root      7855 21.1  8.1 8618980 2696340 ?     Sl   Apr14 595:14 /export/servers/jdk1.6.0_25/bin/java -Djava.util.logging.config.file=/export/home/tomcat/domains/item.soa.360buy.com/server1/conf/logging.properties -Djava.library.path=/usr/local/lib -server -Xms2048m -Xmx2048m -XX:MaxPermSize=256m -Djava.awt.headless=true -Dsun.net.client.defaultConnectTimeout=60000 -Dsun.net.client.defaultReadTimeout=60000 -Djmagick.systemclassloader=no -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.ttl=300 -Dcom.sun.management.jmxremote.port=10004 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=192.168.13.30 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/export/servers/tomcat6.0.33/endorsed -classpath /export/servers/tomcat6.0.33/bin/bootstrap.jar -Dcatalina.base=/export/home/tomcat/domains/item.soa.360buy.com/server1 -Dcatalina.home=/export/servers/tomcat6.0.33 -Djava.io.tmpdir=/export/home/tomcat/domains/item.soa.360buy.com/server1/temp org.apache.catalina.startup.Bootstrap -config /export/home/tomcat/domains/item.soa.360buy.com/server1/conf/server.xml startroot      8268 20.9  8.5 8613152 2801708 ?     Sl   Apr14 587:22 /export/servers/jdk1.6.0_25/bin/java -Djava.util.logging.config.file=/export/home/tomcat/domains/item.soa.360buy.com/server2/conf/logging.properties -Djava.library.path=/usr/local/lib -server -Xms2048m -Xmx2048m -XX:MaxPermSize=256m -Djava.awt.headless=true -Dsun.net.client.defaultConnectTimeout=60000 -Dsun.net.client.defaultReadTimeout=60000 -Djmagick.systemclassloader=no -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.ttl=300 -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/export/servers/tomcat6.0.33/endorsed -classpath /export/servers/tomcat6.0.33/bin/bootstrap.jar -Dcatalina.base=/export/home/tomcat/domains/item.soa.360buy.com/server2 -Dcatalina.home=/export/servers/tomcat6.0.33 -Djava.io.tmpdir=/export/home/tomcat/domains/item.soa.360buy.com/server2/temp org.apache.catalina.startup.Bootstrap -config /export/home/tomcat/domains/item.soa.360buy.com/server2/conf/server.xml startUSER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

 

  2. 通过第一步我们确定了占用cpu最高的进程的pid是7855,那么如何确定该进程的线程cpu占用情况呢?     

top -Hp 7855    -H 显示线程 -p 指定pid   如果执行上述命令之后,发现还有一个进程,那么按下 shift+H, 即可显示threads

root@b28-1330 ~]# top -Hp 7855top - 14:09:41 up 330 days, 22:33,  4 users,  load average: 0.73, 0.81, 0.71Tasks: 5416 total,   0 running, 5416 sleeping,   0 stopped,   0 zombieCpu(s):  2.9%us,  0.6%sy,  0.0%ni, 96.4%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%stMem:  32933092k total, 30577956k used,  2355136k free,   438232k buffersSwap: 16779884k total,        0k used, 16779884k free, 22525628k cached  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                 7906 root      15   0 8416m 2.6g  11m S  3.9  8.2  16:58.92 java                                                                   21006 root      15   0 8416m 2.6g  11m S  3.2  8.2   9:59.56 java                                                                   21019 root      15   0 8416m 2.6g  11m S  2.9  8.2   9:55.70 java        

   经过排查,我们发现 tid(Thread Id)为7906线程占用cpu最高

  3. 上一步我们已经找到了占用cpu最高的线程,那么我们下面要确定一下这个线程的详细信息

  我们使用 jstack来进行深度追踪:

     将tid 7906转换为十六进制: printf "%x\n"  --> 1ee2

     使用命令 jstack 7855 |grep -i 1ee2 -A 20  来查询进程详细信息:

  

[root@b28-1330 ~]#  jstack 7855 |grep -i 1ee2 -A 20 "New I/O client worker #1-1" daemon prio=10 tid=0x000000005ad02000 nid=0x1ee2 runnable [0x0000000043972000]   java.lang.Thread.State: RUNNABLE        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)        - locked <0x00000007801074a0> (a sun.nio.ch.Util$2)        - locked <0x0000000780107490> (a java.util.Collections$UnmodifiableSet)        - locked <0x00000007800fbcc0> (a sun.nio.ch.EPollSelectorImpl)        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)        at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:163)        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)        at java.lang.Thread.run(Thread.java:662)

 

   ok这个线程的堆栈信息就出来了...

二.  如何查看占用JVM内存最大的前20个class实例对象  
 主要用到jmap 和sort命令  

[root@b28-1330 ~]# jmap -histo 7855 |sort -rn -k3|head -n11  Total       6221631      455243264   1:        111761       95387048  [B   2:        884396       63940104  [C   3:         52573       52139904  [I   4:        680476       21775232  java.lang.String   5:        120038       17978192  <constMethodKlass>   6:        120038       16335920  <methodKlass>   7:         11558       12871496  <constantPoolKlass>   8:        103599       12431880  java.net.SocksSocketImpl   9:        350199       11206368  java.util.HashMap$Entry  10:        167532        9294056  <symbolKlass>