阿里云服务器cpu连续n天使用率为100%问题解决方案！
来源：互联网 发布：本地mysql如何配host 编辑：程序博客网 时间：2024/06/15 17:41
硬件配置:阿里云服务器(CPU:4核 内存:4GB 数据盘:450G 带宽:5Mbps).系统配置: iluckysi@ILUCKYSI-PC:/etc# cat issueUbuntu 12.04.5 LTS \n \l异常信息:cpu连续n天使用率为100%.查找原因:查看系统cpu和memory使用情况.iluckysi@ILUCKYSI-PC:/etc# toptop - 11:27:43 up 303 days, 3:31, 2 users, load average: 5.06, 5.03, 5.05Tasks: 180 total, 6 running, 174 sleeping, 0 stopped, 0 zombieCpu(s): 69.8%us, 30.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%stMem: 4051260k total, 3592692k used, 458568k free, 227332k buffersSwap: 2187260k total, 70940k used, 2116320k free, 2351156k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19608 daemon 20 0 29948 2768 1060 R 97 0.1 160766:24 perl 26598 daemon 20 0 23740 1964 708 R 85 0.0 168071:12 perl 4020 daemon 20 0 832 8 0 R 80 0.0 237454:42 pdflush 19612 daemon 20 0 29948 2524 864 R 72 0.1 177923:31 perl 31348 daemon 20 0 23740 728 400 R 62 0.0 167897:47 perl 25444 root 20 0 3321m 550m 12m S 2 13.9 45:24.95 java 11739 daemon 20 0 30216 344 192 S 1 0.0 24:40.73 perl 然后按c,查看COMMAND对应的实际进程.iluckysi@ILUCKYSI-PC:/etc# toptop - 11:29:20 up 303 days, 3:32, 2 users, load average: 5.01, 5.02, 5.05Tasks: 185 total, 6 running, 179 sleeping, 0 stopped, 0 zombieCpu(s): 65.3%us, 34.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.2%hi, 0.2%si, 0.0%stMem: 4051260k total, 3594584k used, 456676k free, 227332k buffersSwap: 2187260k total, 70940k used, 2116320k free, 2351272k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4020 daemon 20 0 832 8 0 R 99 0.0 237456:05 [pdflush] 31348 daemon 20 0 23740 728 400 R 78 0.0 167898:59 klogd -x 19608 daemon 20 0 29948 2768 1060 R 76 0.1 160767:42 /usr/sbin/sshd -D 19612 daemon 20 0 29948 2524 864 R 74 0.1 177924:48 /usr/sbin/acpid 26598 daemon 20 0 23740 1964 708 R 68 0.0 168072:28 klogd -x 25444 root 20 0 3321m 550m 12m S 2 13.9 45:25.68 /usr/lib/jdk/jdk1.7.0_71/jre/bin/java -Djava.util.logging.config.file=/usr/lib/tomcat/apache-tomcat-7.0.32/conf/logging.prope 398 root 20 0 679m 12m 8224 S 1 0.3 405:19.89 /usr/local/aegis/aegis_client/aegis_00_73/AliYunDun 12731 root 20 0 882m 9000 6284 S 1 0.2 124:16.60 /usr/local/aegis/alihids/AliHids 更多关于top指令的使用方法,请上网查询......异常分析:查看cpu使用率过高的这几个进程关联的操作:ls -al:查看某个进程打开的文件的权限.iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/4020/fd/total 0dr-x------ 2 daemon daemon 0 Jun 12 16:45 .dr-xr-xr-x 7 daemon daemon 0 May 1 09:56 ..lr-x------ 1 daemon daemon 64 Jun 12 16:45 0 -> pipe:[68038509]l-wx------ 1 daemon daemon 64 Jun 12 16:45 1 -> /dev/null (deleted)l-wx------ 1 daemon daemon 64 Jun 12 16:45 2 -> /dev/null (deleted)lrwx------ 1 daemon daemon 64 Jun 12 16:45 3 -> /var/tmp/.fontUnix (deleted)iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/31348/fd/total 0dr-x------ 2 daemon daemon 0 Jun 12 16:45 .dr-xr-xr-x 7 daemon daemon 0 Jan 23 16:02 ..lr-x------ 1 daemon daemon 64 Jun 12 16:45 0 -> pipe:[634666134]l-wx------ 1 daemon daemon 64 Jun 12 16:45 1 -> pipe:[634666138]l-wx------ 1 daemon daemon 64 Jun 12 16:45 11 -> /opt/httpd-2.2.21/logs/mod_jk.loglrwx------ 1 daemon daemon 64 Jun 12 16:45 12 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)lrwx------ 1 daemon daemon 64 Jun 12 16:45 13 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)l-wx------ 1 daemon daemon 64 Jun 12 16:45 2 -> pipe:[634666141]lrwx------ 1 daemon daemon 64 Jun 12 16:45 3 -> socket:[637677664]iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/19608/fd/total 0dr-x------ 2 daemon daemon 0 May 6 00:44 .dr-xr-xr-x 7 daemon daemon 0 May 1 09:56 ..lr-x------ 1 daemon daemon 64 May 6 00:44 0 -> pipe:[1849120182]l-wx------ 1 daemon daemon 64 May 6 00:44 1 -> pipe:[1849120186]l-wx------ 1 daemon daemon 64 May 6 00:44 11 -> /opt/httpd-2.2.21/logs/mod_jk.loglrwx------ 1 daemon daemon 64 May 6 00:44 12 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)lrwx------ 1 daemon daemon 64 May 6 00:44 13 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)l-wx------ 1 daemon daemon 64 May 6 00:44 2 -> pipe:[1849120188]lrwx------ 1 daemon daemon 64 May 6 00:44 3 -> socket:[1335075729]iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/19612/fd/total 0dr-x------ 2 daemon daemon 0 May 6 00:44 .dr-xr-xr-x 7 daemon daemon 0 May 1 09:56 ..lr-x------ 1 daemon daemon 64 May 6 00:44 0 -> pipe:[1849120182]l-wx------ 1 daemon daemon 64 May 6 00:44 1 -> pipe:[1849120186]l-wx------ 1 daemon daemon 64 May 6 00:44 11 -> /opt/httpd-2.2.21/logs/mod_jk.loglrwx------ 1 daemon daemon 64 May 6 00:44 12 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)lrwx------ 1 daemon daemon 64 May 6 00:44 13 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)l-wx------ 1 daemon daemon 64 May 6 00:44 2 -> pipe:[1849120188]lrwx------ 1 daemon daemon 64 May 6 00:44 3 -> socket:[2163495078]iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/26598/fd/total 0dr-x------ 2 daemon daemon 0 Jun 12 16:45 .dr-xr-xr-x 7 daemon daemon 0 Jan 23 16:02 ..lr-x------ 1 daemon daemon 64 Jun 12 16:45 0 -> pipe:[3453697476]l-wx------ 1 daemon daemon 64 Jun 12 16:45 1 -> pipe:[3453697477]l-wx------ 1 daemon daemon 64 Jun 12 16:45 11 -> /opt/httpd-2.2.21/logs/mod_jk.loglrwx------ 1 daemon daemon 64 Jun 12 16:45 12 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)lrwx------ 1 daemon daemon 64 Jun 12 16:45 13 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)l-wx------ 1 daemon daemon 64 Jun 12 16:45 2 -> pipe:[3453697478]lrwx------ 1 daemon daemon 64 Jun 12 16:45 3 -> socket:[525581618]iluckysi@ILUCKYSI-PC:/var/tmp# 同时可以使用如下指令进行分析: lsof -p pid:查看正在运行中的进程打开了哪些文件,目录和套接字,是系统监测工具之一.iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 4020COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEpdflush 4020 daemon cwd DIR 3,1 12288 1835016 /usr/sbinpdflush 4020 daemon rtd DIR 3,1 4096 2 /pdflush 4020 daemon txt REG 3,1 562008 264410 /var/tmp/pdflush (deleted)pdflush 4020 daemon 0r FIFO 0,8 0t0 68038509 pipepdflush 4020 daemon 1w CHR 1,3 0t0 4782 /dev/null (deleted)pdflush 4020 daemon 2w CHR 1,3 0t0 4782 /dev/null (deleted)pdflush 4020 daemon 3u REG 3,1 0 263762 /var/tmp/.fontUnix (deleted)iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 31348COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEperl 31348 daemon cwd DIR 3,1 4096 2097153 /tmpperl 31348 daemon rtd DIR 3,1 4096 2 /perl 31348 daemon txt REG 3,1 10352 1841631 /usr/bin/perlperl 31348 daemon mem REG 3,1 26968 1841662 /usr/lib/perl/5.10.1/auto/Socket/Socket.soperl 31348 daemon mem REG 3,1 22840 1841660 /usr/lib/perl/5.10.1/auto/IO/IO.soperl 31348 daemon mem REG 3,1 43288 1314543 /lib/x86_64-linux-gnu/libcrypt-2.15.soperl 31348 daemon mem REG 3,1 1811128 1314563 /lib/x86_64-linux-gnu/libc-2.15.soperl 31348 daemon mem REG 3,1 135366 1314539 /lib/x86_64-linux-gnu/libpthread-2.15.soperl 31348 daemon mem REG 3,1 1030512 1314566 /lib/x86_64-linux-gnu/libm-2.15.soperl 31348 daemon mem REG 3,1 14768 1314577 /lib/x86_64-linux-gnu/libdl-2.15.soperl 31348 daemon mem REG 3,1 1479112 1841632 /usr/lib/libperl.so.5.10.1perl 31348 daemon mem REG 3,1 149280 1310795 /lib/x86_64-linux-gnu/ld-2.15.soperl 31348 daemon 0r FIFO 0,8 0t0 634666134 pipeperl 31348 daemon 1w FIFO 0,8 0t0 634666138 pipeperl 31348 daemon 2w FIFO 0,8 0t0 634666141 pipeperl 31348 daemon 3u IPv4 637677664 0t0 TCP 110.76.39.140:44833->209.92.176.14:http (ESTABLISHED)perl 31348 daemon 11w REG 3,1 138468379 661852 /opt/httpd-2.2.21/logs/mod_jk.logperl 31348 daemon 12u REG 3,1 448 661951 /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)perl 31348 daemon 13u REG 3,1 1 688338 /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 19608COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEperl 19608 daemon cwd DIR 3,1 4096 2097153 /tmpperl 19608 daemon rtd DIR 3,1 4096 2 /perl 19608 daemon txt REG 3,1 10352 1841631 /usr/bin/perlperl 19608 daemon mem REG 3,1 105288 1314564 /lib/x86_64-linux-gnu/libresolv-2.15.soperl 19608 daemon mem REG 3,1 31104 1314576 /lib/x86_64-linux-gnu/libnss_dns-2.15.soperl 19608 daemon mem REG 3,1 52120 1314561 /lib/x86_64-linux-gnu/libnss_files-2.15.soperl 19608 daemon mem REG 3,1 26968 1841662 /usr/lib/perl/5.10.1/auto/Socket/Socket.soperl 19608 daemon mem REG 3,1 22840 1841660 /usr/lib/perl/5.10.1/auto/IO/IO.soperl 19608 daemon mem REG 3,1 43288 1314543 /lib/x86_64-linux-gnu/libcrypt-2.15.soperl 19608 daemon mem REG 3,1 1811128 1314563 /lib/x86_64-linux-gnu/libc-2.15.soperl 19608 daemon mem REG 3,1 135366 1314539 /lib/x86_64-linux-gnu/libpthread-2.15.soperl 19608 daemon mem REG 3,1 1030512 1314566 /lib/x86_64-linux-gnu/libm-2.15.soperl 19608 daemon mem REG 3,1 14768 1314577 /lib/x86_64-linux-gnu/libdl-2.15.soperl 19608 daemon mem REG 3,1 1479112 1841632 /usr/lib/libperl.so.5.10.1perl 19608 daemon mem REG 3,1 149280 1310795 /lib/x86_64-linux-gnu/ld-2.15.soperl 19608 daemon 0r FIFO 0,8 0t0 1849120182 pipeperl 19608 daemon 1w FIFO 0,8 0t0 1849120186 pipeperl 19608 daemon 2w FIFO 0,8 0t0 1849120188 pipeperl 19608 daemon 3u IPv4 1335075729 0t0 TCP 110.76.39.140:53110->119.68.205.1:smtp (ESTABLISHED)perl 19608 daemon 11w REG 3,1 138471419 661852 /opt/httpd-2.2.21/logs/mod_jk.logperl 19608 daemon 12u REG 3,1 448 661951 /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)perl 19608 daemon 13u REG 3,1 1 688338 /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 19612COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEperl 19612 daemon cwd DIR 3,1 4096 2097153 /tmpperl 19612 daemon rtd DIR 3,1 4096 2 /perl 19612 daemon txt REG 3,1 10352 1841631 /usr/bin/perlperl 19612 daemon mem REG 3,1 105288 1314564 /lib/x86_64-linux-gnu/libresolv-2.15.soperl 19612 daemon mem REG 3,1 31104 1314576 /lib/x86_64-linux-gnu/libnss_dns-2.15.soperl 19612 daemon mem REG 3,1 52120 1314561 /lib/x86_64-linux-gnu/libnss_files-2.15.soperl 19612 daemon mem REG 3,1 26968 1841662 /usr/lib/perl/5.10.1/auto/Socket/Socket.soperl 19612 daemon mem REG 3,1 22840 1841660 /usr/lib/perl/5.10.1/auto/IO/IO.soperl 19612 daemon mem REG 3,1 43288 1314543 /lib/x86_64-linux-gnu/libcrypt-2.15.soperl 19612 daemon mem REG 3,1 1811128 1314563 /lib/x86_64-linux-gnu/libc-2.15.soperl 19612 daemon mem REG 3,1 135366 1314539 /lib/x86_64-linux-gnu/libpthread-2.15.soperl 19612 daemon mem REG 3,1 1030512 1314566 /lib/x86_64-linux-gnu/libm-2.15.soperl 19612 daemon mem REG 3,1 14768 1314577 /lib/x86_64-linux-gnu/libdl-2.15.soperl 19612 daemon mem REG 3,1 1479112 1841632 /usr/lib/libperl.so.5.10.1perl 19612 daemon mem REG 3,1 149280 1310795 /lib/x86_64-linux-gnu/ld-2.15.soperl 19612 daemon 0r FIFO 0,8 0t0 1849120182 pipeperl 19612 daemon 1w FIFO 0,8 0t0 1849120186 pipeperl 19612 daemon 2w FIFO 0,8 0t0 1849120188 pipeperl 19612 daemon 3u IPv4 2163495078 0t0 TCP 110.76.39.140:43416->119.68.205.1:smtp (ESTABLISHED)perl 19612 daemon 11w REG 3,1 138471723 661852 /opt/httpd-2.2.21/logs/mod_jk.logperl 19612 daemon 12u REG 3,1 448 661951 /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)perl 19612 daemon 13u REG 3,1 1 688338 /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 26598COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEperl 26598 daemon cwd DIR 3,1 4096 2097153 /tmpperl 26598 daemon rtd DIR 3,1 4096 2 /perl 26598 daemon txt REG 3,1 10352 1841631 /usr/bin/perlperl 26598 daemon mem REG 3,1 26968 1841662 /usr/lib/perl/5.10.1/auto/Socket/Socket.soperl 26598 daemon mem REG 3,1 22840 1841660 /usr/lib/perl/5.10.1/auto/IO/IO.soperl 26598 daemon mem REG 3,1 43288 1314543 /lib/x86_64-linux-gnu/libcrypt-2.15.soperl 26598 daemon mem REG 3,1 1811128 1314563 /lib/x86_64-linux-gnu/libc-2.15.soperl 26598 daemon mem REG 3,1 135366 1314539 /lib/x86_64-linux-gnu/libpthread-2.15.soperl 26598 daemon mem REG 3,1 1030512 1314566 /lib/x86_64-linux-gnu/libm-2.15.soperl 26598 daemon mem REG 3,1 14768 1314577 /lib/x86_64-linux-gnu/libdl-2.15.soperl 26598 daemon mem REG 3,1 1479112 1841632 /usr/lib/libperl.so.5.10.1perl 26598 daemon mem REG 3,1 149280 1310795 /lib/x86_64-linux-gnu/ld-2.15.soperl 26598 daemon 0r FIFO 0,8 0t0 3453697476 pipeperl 26598 daemon 1w FIFO 0,8 0t0 3453697477 pipeperl 26598 daemon 2w FIFO 0,8 0t0 3453697478 pipeperl 26598 daemon 3u IPv4 525581618 0t0 TCP 110.76.39.140:34176->210.253.114.69:81 (ESTABLISHED)perl 26598 daemon 11w REG 3,1 138472179 661852 /opt/httpd-2.2.21/logs/mod_jk.logperl 26598 daemon 12u REG 3,1 448 661951 /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)perl 26598 daemon 13u REG 3,1 1 688338 /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)iluckysi@ILUCKYSI-PC:/var/tmp# 由上面的分析得知,除了4020进程,其余进程都和httpd有关系.并且除了4020进程,其余所有的进程都对外有一个TCP连接,查看连接的ip,都是来自国外.深入分析:在上面的输出中,我们看到了.fontUnix,查询apache的error.log,看到如下记录.[Sat Nov 22 22:31:30 2014] [error] [client 166.78.138.102] Connecting to 85.236.52.116:80... [Sat Nov 22 22:31:30 2014] [error] [client 166.78.138.102] connected.[Sat Nov 22 22:31:30 2014] [error] [client 166.78.138.102] HTTP request sent, awaiting response... [Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] 200 OK[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] Length: 1018 [text/plain][Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] Saving to: `/var/tmp/.font-unix.sh'[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] [Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] 0K 100% 199M=0s[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] [Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] 2014-11-22 22:31:33 (199 MB/s) - `/var/tmp/.font-unix.sh' saved [1018/1018]\解决方法:将上面分析出的的有问题的进程强制杀死.iluckysi@ILUCKYSI-PC:~# kill -9 4020iluckysi@ILUCKYSI-PC:~# kill -9 31348iluckysi@ILUCKYSI-PC:~# kill -9 26598iluckysi@ILUCKYSI-PC:~# kill -9 19612iluckysi@ILUCKYSI-PC:/tmp# kill -9 19608此时到阿里云平台查看服务器cpu使用率,发现cpu使用率降下来了.最终方案:阿里云Linux Bash严重漏洞修复紧急通知:http://bbs.aliyun.com/read/176977.html升级系统版本,升级bash版本.
0 0
- 阿里云服务器cpu连续n天使用率为100%问题解决方案！
- 数据库服务器CPU占满100%的问题解决方案列表！
- 阿里云服务器centos6.5 安装docker报错的问题解决方案
- 配置Maven阿里云仓库和添加依赖时搜索为空问题解决方案
- 提高CPU性能问题解决方案
- poj-2140-Herd Sums- 数学规律-连续和为N的方案数
- 连续整数之和为n
- 阿里云服务器MySQL无法连接问题解决纪实
- SecureCRT连接阿里云ECS服务器，经常掉线。问题解决
- 服务器病毒问题解决- 阿里云 挖矿病毒，Circle_MI.png
- 阿里云windows服务器突然远程不了问题解决
- 阿里云，云服务器 ECS配制IIS全套方案-踩坑记
- 阿里云服务器部署方案(nginx+tomcat+mysql)
- 阿里云服务器部署方案(nginx+tomcat+mysql)
- Linux服务器搭建（以阿里云服务器为例）
- 阿里云服务器（centos）安装node和n
- 云服务器Ubuntu系统下中文乱码问题解决方案
- 【Codeforces Round 271 (Div 2)D】【DP】Flowers 黑色必须连续摆放k，长度为n的摆放方案数
- 黑马程序员--基本数据类型
- Hash树
- 44444
- 使用C/C++编写PHP Extension
- GRE词汇记忆捷径
- 阿里云服务器cpu连续n天使用率为100%问题解决方案！
- 2232
- mac 启动php-fpm
- STL学习笔记之容器--优先队列
- 8种Nosql数据库系统对比
- selenium学习之路
- Vmware vSphere 5.0系列教程之六 虚拟机及主机配置文件的创建和模板部署
- MySQL数据库备份命令
- vim速查
