宕机日记:swap交换分区和并发测试ab工具的使用

昨天我一朋友跟我说他的网站跪了，网站是用cms套的商城系统，服务器也是单点的，可能随着用户量增长，不堪重负，终于倒下。
我登陆上服务器看了下 发现是数据库跪了 我登进数据库看了下数据库错误日志路径

SHOW GLOBAL VARIABLES LIKE '%log%';

/var/log/mariadb/mariadb.log
我把日志打开看了下 发现做完18点发生了crash 原因是out of memory

170613 18:34:13  InnoDB: Database was not shut down normally!InnoDB: Starting crash recovery.InnoDB: Reading tablespace information from the .ibd files...InnoDB: Restoring possible half-written data pages from the doublewriteInnoDB: buffer...InnoDB: Doing recovery: scanned up to log sequence number 51339298170613 18:34:13  InnoDB: Starting an apply batch of log records to the database...InnoDB: Progress in percents: 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed170613 18:34:14  InnoDB: Waiting for the background threads to start170613 18:34:15 Percona XtraDB (http://www.percona.com) 5.5.49-MariaDB-38.0 started; log sequence number 51339298170613 18:34:15 [Note] Plugin 'FEEDBACK' is disabled.170613 18:34:15 [Note] Server socket created on IP: '0.0.0.0'.170613 18:34:15 [Note] Event Scheduler: Loaded 0 events170613 18:34:15 [Note] /usr/libexec/mysqld: ready for connections.Version: '5.5.52-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server170613 18:34:15 [ERROR] mysqld: Table './mycms/config' is marked as crashed and should be repaired170613 18:34:15 [ERROR] mysqld: Table './mycms/config' is marked as crashed and should be repaired170613 18:34:15 [ERROR] mysqld: Table './mycms/config' is marked as crashed and should be repaired170613 18:34:15 [Warning] Checking table:   './mycms/config'170613 18:35:38 [Warning] IP address '*******' could not be resolved: Name or service not known170613 18:36:11 mysqld_safe Number of processes running now: 0170613 18:36:11 mysqld_safe mysqld restarted170613 18:36:12 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 2134 ...170613 18:36:12 InnoDB: The InnoDB memory heap is disabled170613 18:36:12 InnoDB: Mutexes and rw_locks use GCC atomic builtins170613 18:36:12 InnoDB: Compressed tables use zlib 1.2.7170613 18:36:12 InnoDB: Using Linux native AIO170613 18:36:13 InnoDB: Initializing buffer pool, size = 128.0MInnoDB: mmap(137756672 bytes) failed; errno 12170613 18:36:13 InnoDB: Completed initialization of buffer pool170613 18:36:13 InnoDB: Fatal error: cannot allocate memory for the buffer pool170613 18:36:13 [ERROR] Plugin 'InnoDB' init function returned error.170613 18:36:13 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.170613 18:36:13 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)170613 18:36:13 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)170613 18:36:13 [ERROR] mysqld: Out of memory (Needed 72499200 bytes)170613 18:36:13 [Note] Plugin 'FEEDBACK' is disabled.170613 18:36:13 [ERROR] Unknown/unsupported storage engine: InnoDB170613 18:36:13 [ERROR] Aborting

Google搜了下 发现有人说是swap没有利用的原因 看了下 还真的swap为0

Swap的调整对Linux服务器，特别是Web服务器的性能至关重要。通过调整Swap，有时可以越过系统性能瓶颈，节省系统升级费用。Swap空间的作用可简单描述为：当系统的物理内存不够用的时候，就需要将物理内存中的一部分空间释放出来，以供当前运行的程序使用。那些被释放的空间可能来自一些很长时间没有什么操作的程序，这些被释放的空间被临时保存到Swap空间中，等到那些程序要运行时，再从Swap中恢复保存的数据到内存中。这样，系统总是在物理内存不够时，才进行Swap交换。
Linux 中的 SWAP（交换分区），类似于 Windows 的虚拟内存。系统会把一部分硬盘空间虚拟成内存使用，将系统内非活动内存换页到 SWAP，以提高系统可用内存。

[root@centos /]# free -m              total        used        free      shared  buff/cache   availableMem:           1839         713         123          12        1002         962Swap:             0           0           0

我用另一台服务器的ab来并发测试下 100次请求10次并发

[root@centos ~]# ab -n 100 -c 10 http://******/This is ApacheBench, Version 2.3 <$Revision: 1430300 $>Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.******.net/Licensed to The Apache Software Foundation, http://www.apache.org/Benchmarking ****** (be patient).....doneServer Software:        Apache/2.4.6Server Hostname:        www.******.comServer Port:            80Document Path:          /Document Length:        52881 bytesConcurrency Level:      10Time taken for tests:   316.242 secondsComplete requests:      100Failed requests:        99   (Connect: 0, Receive: 0, Length: 99, Exceptions: 0)Write errors:           0Total transferred:      5290724 bytesHTML transferred:       5271624 bytes#吞吐率，相当于 LR 中的每秒事务数，后面括号中的 mean 表示这是一个平均值Requests per second:    0.32 [#/sec] (mean)#用户平均请求等待时间Time per request:       31624.187 [ms] (mean)#服务器平均请求处理时间Time per request:       3162.419 [ms] (mean, across all concurrent requests)Transfer rate:          16.34 [Kbytes/sec] receivedConnection Times (ms)              min  mean[+/-sd] median   maxConnect:        3    3   0.3      3       4Processing: 16882 29781 3926.3  30144   37067Waiting:    11866 24743 3922.3  25136   32043Total:      16885 29784 3926.3  30147   37070#这段是每个请求处理时间的分布情况，50%的处理时间在4930ms内，66%的处理时间在5008ms内...，重要的是看90%的处理时间Percentage of the requests served within a certain time (ms)  50%  30147  66%  31511  75%  32243  80%  32936  90%  34668  95%  35557  98%  35989  99%  37070 100%  37070 (longest request)

这个cms有点恶心 用户平均等待时间需要30s
28个并发的时候成功压垮数据库 -s表示响应时间 防止响应过久 ab不等待了

这是没有加-s的情况

[root@VM_219_199_centos ~]# ab -n 50 -c 20 http://www.*****.com/Benchmarking www.*****.com (be patient)...apr_pollset_poll: The timeout specified has expired (70007)Total of 1 requests completed

#接下来开始调整swap#创建用于交换分区的文件：dd if=/dev/zero of=/mnt/swap bs=1M count=1024#设置交换分区文件：mkswap /mnt/swap#立即启用交换分区文件 这里一开始会出现swapon: /mnt/swap: read swap header failed: Invalid argument 后来却自己好了 我也很绝望。。swapon /mnt/swap#这时的free -m 如下free -m              total        used        free      shared  buff/cache   availableMem:           1839         710         955          12         173         981Swap:          1023           0        1023#设置开机时自启用 SWAP 分区：#需要修改文件 /etc/fstab 中的 SWAP 行，添加/mnt/swap swap swap defaults 0 0

更改好swap后 我们再并发测试一下
我们可以看到中途系统使用了swap区 并且测试结束后数据库不会宕机 将并发量调到45 也照样撑得住

ab -n 50 -c 28 -s 500 http://www.******.com/

调整了swap让服务器能撑的住更多并发数
其实在测试的过程中 CPU也飙到了99 100但导致数据库crash的却是内存不足 看来我还需要继续深入理解操作系统才行。

参考链接：
http://blog.chriscabin.com/others/wordpress/1502.html
https://help.aliyun.com/knowledge_detail/42534.html
http://www.jianshu.com/p/43d04d8baaf7
https://www.douban.com/note/501373268/