我们服务器搭建了cacti进行监控，一次通过查看磁盘IO图像的时候，发现每天凌晨3：20-3：35的时候磁盘IO一下飙的很高，然后想知道到底是哪个进程占用那么高的IO，下面是解决方法：
写个检测脚本check_io_process.sh,当磁盘IO占用高的时候进行磁盘IO读写进程次数的检测:
#!/bin/bash
# Date: 2013/8/20
# Author: zhangkai
# Description: This script is used to check IO higher process.
# History:
iostat_log=/data/logs/iostat/iostat.log
dmesg_log=/data/logs/iostat/dmesg.log
dstat_log=/data/logs/iostat/dstat.log
if [ ! -d /data/logs/iostat ];then
mkdir -p /data/logs/iostat
fi
add(){
str=$@
sum=`echo ${str// /+}|bc -l`
}
iostat -x 1 5 > $iostat_log
idle_percent=`cat $iostat_log | awk 'BEGIN{flag=0} {if(flag ==1){print $12; flag=0;} if (index($0, "%util" )) {flag = 1;}}'`
add $idle_percent
#求5次查询IO占用率的平均值
avg=`echo $sum/5|bc`
if [[ $avg -ge 70 ]];then
echo 1 > /proc/sys/vm/block_dump
echo "-----------------------------------------------" >> $dmesg_log
echo `date "+%Y-%m-%d %H:%M:%S"` >> $dmesg_log
python /data/dmesg_io.py >> $dmesg_log
echo "-----------------------------------------------" >> $dstat_log
echo `date "+%Y-%m-%d %H:%M:%S"` >> $dstat_log
dstat -d --top-bio 1 10 >> $dstat_log
echo 0 > /proc/sys/vm/block_dump
fi
其中该shell脚本调用了检测磁盘IO读写进程次数的python脚本,下面是dmesg_io.py的代码：
#!/usr/bin/python
# Monitoring per-process disk I/O activity
# written by http://www.vpsee.com
import sys, os, time, signal, re
class DiskIO:
def __init__(self, pname=None, pid=None, reads=0, writes=0):
self.pname = pname
self.pid = pid
self.reads = 0
self.writes = 0
def main():
argc = len(sys.argv)
if argc != 1:
print "usage: ./iotop"
sys.exit(0)
if os.getuid() != 0:
print "must be run as root"
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
os.system('echo 1 > /proc/sys/vm/block_dump')
print "TASK PID READ WRITE"
# while True:
os.system('dmesg -c > /tmp/diskio.log')
l = []
f = open('/tmp/diskio.log', 'r')
line = f.readline()
while line:
m = re.match(\
'^(\S+)\((\d+)\): (READ|WRITE) block (\d+) on (\S+)', line)
if m != None:
if not l:
l.append(DiskIO(m.group(1), m.group(2)))
line = f.readline()
continue
found = False
for item in l:
if item.pid == m.group(2):
found = True
if m.group(3) == "READ":
item.reads = item.reads + 1
elif m.group(3) == "WRITE":