MHA+Keepalived

一：MHA的部署参考：简单粗暴的MHA部署

二：MHA结合Keepalived以使MySQL高可用对应用透明
1.Keepalived的Download&INSTALL
Download Keepalived

INSTALL For Keepalivedtar zxvf keepalived.xxxcd keepalivedxxx./configure --prefix=/usr/local/keepalivedmake && make install configure: error:  !!! OpenSSL is not properly installed on your system. !!!  !!! Can not include OpenSSL headers files.            !!!解决：yum install openssl-devel

2.Keepalived的configuration

[root@db-two keepalived-1.2.20]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived  /etc/rc.d/init.d/[root@db-two keepalived-1.2.20]# cp /usr/local/keepalived/etc/sysconfig/keepalived  /etc/sysconfig/[root@db-two keepalived-1.2.20]# mkdir /etc/keepalived[root@db-two keepalived-1.2.20]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf  /etc/keepalived/[root@db-two keepalived-1.2.20]# cp /usr/local/keepalived/sbin/keepalived   /usr/sbin/vim /etc/keepalived/keepalived.confMASTER[root@db-three keepalived-1.2.20]# cat  /etc/keepalived/keepalived.confglobal_defs { router_id mysql-master #修改为自己的主机名 notification_email { [e-mail]   #接收邮件，可以有多个，一行一个} #当主、备份设备发生改变时，通过邮件通知 notification_email_from [e-mail] #发送邮箱服务器 smtp_server stmp.163.com #发送邮箱超时时间 smtp_connect_timeout 30 }##################第一部分###################vrrp_instance VI_1 {     state BACKUP    #都修改成BACKUP     interface eth0    #绑定的网卡     virtual_router_id 60 #默认51 主从都修改为60     priority 100            #优先级，在mysql-slave上LVS上修改成80，即master>slave     advert_int 1     nopreempt #不抢占资源，意思就是它活了之后也不会再把主抢回来     authentication {     # 认证方式，可以是PASS或AH两种认证方式     auth_type PASS     # 认证密码     auth_pass 1111     }virtual_ipaddress {     192.168.1.100     }}##################第二部分###################virtual_server 192.168.1.100 3306 {     delay_loop 6     lb_algo wrr     lb_kind DR     nat_mask 255.255.255.0     persistence_timeout 50     protocol TCP    real_server 192.168.1.30 3306 {     weight 1     TCP_CHECK {         connect_timeout 10         nb_get_retry 3         connect_port 3306         }     }}SLAVE[root@db-two keepalived-1.2.20]# cat  /etc/keepalived/keepalived.confglobal_defs { router_id mysql-master #修改为自己的主机名 notification_email { [e-mail]   #接收邮件，可以有多个，一行一个} #当主、备份设备发生改变时，通过邮件通知 notification_email_from [e-mail] #发送邮箱服务器 smtp_server stmp.163.com #发送邮箱超时时间 smtp_connect_timeout 30 }##################第一部分###################vrrp_instance VI_1 {     state BACKUP    #都修改成BACKUP     interface eth0    #绑定的网卡     virtual_router_id 60 #默认51 主从都修改为60     priority 80            #优先级，在mysql-slave上LVS上修改成80     advert_int 1     nopreempt #不抢占资源，意思就是它活了之后也不会再把主抢回来     authentication {     # 认证方式，可以是PASS或AH两种认证方式     auth_type PASS     # 认证密码     auth_pass 1111     }virtual_ipaddress {     192.168.1.100     }}##################第二部分###################virtual_server 192.168.1.100 3306 {     delay_loop 6     lb_algo wrr     lb_kind DR     nat_mask 255.255.255.0     persistence_timeout 50     protocol TCP    real_server 192.168.1.20 3306 {     weight 1     TCP_CHECK {         connect_timeout 10         nb_get_retry 3         connect_port 3306         }     }}Manager For Keepalived/etc/init.d/keepalived start/etc/init.d/keepalived stop/etc/init.d/keepalived statusKeepalived的日志在/var/log/messages注意：Keepalived的启动顺序对VIP绑定的影响?

3.对简单粗暴的MHA部署 mha.conf修改如下

[server default]manager_workdir=/usr/local/mhamaster_ip_failover_script=/usr/local/mha/script/master_ip_failover     #master_ip_failover、master_ip_online_change、send_report脚本是由mha4mysql-manager源码包中sample/script下拷贝的master_ip_online_change_script=/usr/local/mha/script/master_ip_online_changepassword=123remote_workdir=/usr/local/mharepl_password=passwdrepl_user=slavereport_script=/usr/local/mha/script/send_reportshutdown_script=""ssh_user=rootuser=root[server1]candidate_master=1check_repl_delay=0hostname=192.168.1.20[server2]hostname=192.168.1.30[server3]hostname=192.168.1.40

4.对mha.conf中提到的master_ip_failover脚本修改如下

#!/usr/bin/env perluse strict;use warnings FATAL => 'all';use Getopt::Long;my (    $command,          $ssh_user,        $orig_master_host, $orig_master_ip,    $orig_master_port, $new_master_host, $new_master_ip,    $new_master_port);my $vip = '192.168.1.100';my $ssh_start_vip = "/etc/init.d/keepalived start";my $ssh_stop_vip = "/etc/init.d/keepalived stop";GetOptions(    'command=s'          => \$command,    'ssh_user=s'         => \$ssh_user,    'orig_master_host=s' => \$orig_master_host,    'orig_master_ip=s'   => \$orig_master_ip,    'orig_master_port=i' => \$orig_master_port,    'new_master_host=s'  => \$new_master_host,    'new_master_ip=s'    => \$new_master_ip,    'new_master_port=i'  => \$new_master_port,);exit &main();sub main {    print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";    if ( $command eq "stop" || $command eq "stopssh" ) {        my $exit_code = 1;        eval {            print "Disabling the VIP on old master: $orig_master_host \n";            &stop_vip();            $exit_code = 0;        };        if ($@) {            warn "Got Error: $@\n";            exit $exit_code;        }        exit $exit_code;    }    elsif ( $command eq "start" ) {        my $exit_code = 10;        eval {            print "Enabling the VIP - $vip on the new master - $new_master_host \n";            &start_vip();            $exit_code = 0;        };        if ($@) {            warn $@;            exit $exit_code;        }        exit $exit_code;    }    elsif ( $command eq "status" ) {        print "Checking the Status of the script.. OK \n";        #`ssh $ssh_user\@cluster1 \" $ssh_start_vip \"`;        exit 0;    }    else {        &usage();        exit 1;    }}# A simple system call that enable the VIP on the new mastersub start_vip() {    `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;}# A simple system call that disable the VIP on the old_mastersub stop_vip() {     return 0  unless  ($ssh_user);    `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;}sub usage {    print    "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";}

三：MHA+Keepalived的Failover测试以及相关日志信息如下

[root@db-three keepalived-1.2.20]# /etc/init.d/mysqld  stopStopping mysqld:                                           [  OK  ][root@db-three keepalived-1.2.20]# tail -f /var/log/messagesJul  7 11:43:39 db-three Keepalived_vrrp[8272]: VRRP_Instance(VI_1) Entering MASTER STATEJul  7 11:43:39 db-three Keepalived_vrrp[8272]: VRRP_Instance(VI_1) setting protocol VIPs.Jul  7 11:43:39 db-three Keepalived_vrrp[8272]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.100Jul  7 11:43:39 db-three Keepalived_healthcheckers[8271]: Netlink reflector reports IP 192.168.1.100 addedJul  7 11:43:44 db-three Keepalived_vrrp[8272]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.100Jul  7 11:56:23 db-three Keepalived_healthcheckers[8271]: TCP connection to [192.168.1.30]:3306 failed.Jul  7 11:56:24 db-three Keepalived_healthcheckers[8271]: TCP connection to [192.168.1.30]:3306 failed.Jul  7 11:56:24 db-three Keepalived_healthcheckers[8271]: Check on service [192.168.1.30]:3306 failed after 1 retry.Jul  7 11:56:24 db-three Keepalived_healthcheckers[8271]: Removing service [192.168.1.30]:3306 from VS [192.168.1.100]:3306Jul  7 11:56:24 db-three Keepalived_healthcheckers[8271]: Lost quorum 1-0=1 > 0 for VS [192.168.1.100]:3306Jul  7 11:56:38 db-three Keepalived[8269]: StoppingJul  7 11:56:38 db-three Keepalived_vrrp[8272]: VRRP_Instance(VI_1) sent 0 priorityJul  7 11:56:38 db-three Keepalived_vrrp[8272]: VRRP_Instance(VI_1) removing protocol VIPs.Jul  7 11:56:38 db-three Keepalived_healthcheckers[8271]: Netlink reflector reports IP 192.168.1.100 removedJul  7 11:56:38 db-three Keepalived_healthcheckers[8271]: StoppedJul  7 11:56:39 db-three Keepalived_vrrp[8272]: StoppedJul  7 11:56:39 db-three Keepalived[8269]: Stopped Keepalived v1.2.20 (07/07,2016)slave的keepalived日志信息[root@db-two keepalived-1.2.20]# tail -f /var/log/messagesJun 21 15:05:37 db-two Keepalived_healthcheckers[10358]: Unknown keyword 'nat_mask'Jun 21 15:05:37 db-two Keepalived_healthcheckers[10358]: Unknown keyword 'nb_get_retry'Jun 21 15:05:37 db-two Keepalived_healthcheckers[10358]: Using LinkWatch kernel netlink reflector...Jun 21 15:05:37 db-two Keepalived_healthcheckers[10358]: Activating healthchecker for service [192.168.1.20]:3306Jun 21 15:15:13 db-two Keepalived_vrrp[10359]: VRRP_Instance(VI_1) Transition to MASTER STATEJun 21 15:15:14 db-two Keepalived_vrrp[10359]: VRRP_Instance(VI_1) Entering MASTER STATEJun 21 15:15:14 db-two Keepalived_vrrp[10359]: VRRP_Instance(VI_1) setting protocol VIPs.Jun 21 15:15:14 db-two Keepalived_vrrp[10359]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.100Jun 21 15:15:14 db-two Keepalived_healthcheckers[10358]: Netlink reflector reports IP 192.168.1.100 addedJun 21 15:15:19 db-two Keepalived_vrrp[10359]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.100mha的manager日志信息Tue Jun 21 22:56:55 2016 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)Tue Jun 21 22:56:55 2016 - [info] Executing SSH check script: exit 0Tue Jun 21 22:56:55 2016 - [info] HealthCheck: SSH to 192.168.1.30 is reachable.Tue Jun 21 22:56:58 2016 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)Tue Jun 21 22:56:58 2016 - [warning] Connection failed 2 time(s)..Tue Jun 21 22:57:01 2016 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)Tue Jun 21 22:57:01 2016 - [warning] Connection failed 3 time(s)..Tue Jun 21 22:57:04 2016 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)Tue Jun 21 22:57:04 2016 - [warning] Connection failed 4 time(s)..Tue Jun 21 22:57:04 2016 - [warning] Master is not reachable from health checker!Tue Jun 21 22:57:04 2016 - [warning] Master 192.168.1.30(192.168.1.30:3306) is not reachable!Tue Jun 21 22:57:04 2016 - [warning] SSH is reachable.Tue Jun 21 22:57:04 2016 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mha/mha.conf again, and trying to connect to all serverrver status..Tue Jun 21 22:57:04 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Tue Jun 21 22:57:04 2016 - [info] Reading application default configuration from /etc/mha/mha.conf..Tue Jun 21 22:57:04 2016 - [info] Reading server configuration from /etc/mha/mha.conf..Tue Jun 21 22:57:04 2016 - [info] GTID failover mode = 1Tue Jun 21 22:57:04 2016 - [info] Dead Servers:Tue Jun 21 22:57:04 2016 - [info]   192.168.1.30(192.168.1.30:3306)Tue Jun 21 22:57:04 2016 - [info] Alive Servers:Tue Jun 21 22:57:04 2016 - [info]   192.168.1.20(192.168.1.20:3306)Tue Jun 21 22:57:04 2016 - [info]   192.168.1.40(192.168.1.40:3306)Tue Jun 21 22:57:04 2016 - [info] Alive Slaves:Tue Jun 21 22:57:04 2016 - [info]   192.168.1.20(192.168.1.20:3306)  Version=5.6.31-log (oldest major version between slaves) log-bin:enabledTue Jun 21 22:57:04 2016 - [info]     GTID ONTue Jun 21 22:57:04 2016 - [info]     Replicating from 192.168.1.30(192.168.1.30:3306)Tue Jun 21 22:57:04 2016 - [info]     Primary candidate for the new Master (candidate_master is set)Tue Jun 21 22:57:04 2016 - [info]   192.168.1.40(192.168.1.40:3306)  Version=5.6.31-log (oldest major version between slaves) log-bin:enabledTue Jun 21 22:57:04 2016 - [info]     GTID ONTue Jun 21 22:57:04 2016 - [info]     Replicating from 192.168.1.30(192.168.1.30:3306)Tue Jun 21 22:57:04 2016 - [info] Checking slave configurations..Tue Jun 21 22:57:04 2016 - [info]  read_only=1 is not set on slave 192.168.1.20(192.168.1.20:3306).Tue Jun 21 22:57:04 2016 - [info]  read_only=1 is not set on slave 192.168.1.40(192.168.1.40:3306).Tue Jun 21 22:57:04 2016 - [info] Checking replication filtering settings..Tue Jun 21 22:57:04 2016 - [info]  Replication filtering check ok.Tue Jun 21 22:57:04 2016 - [info] Master is down!Tue Jun 21 22:57:04 2016 - [info] Terminating monitoring script.Tue Jun 21 22:57:04 2016 - [info] Got exit code 20 (Master dead).Tue Jun 21 22:57:04 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Tue Jun 21 22:57:04 2016 - [info] Reading application default configuration from /etc/mha/mha.conf..Tue Jun 21 22:57:04 2016 - [info] Reading server configuration from /etc/mha/mha.conf..Tue Jun 21 22:57:04 2016 - [info] MHA::MasterFailover version 0.56.Tue Jun 21 22:57:04 2016 - [info] Starting master failover.Tue Jun 21 22:57:04 2016 - [info]Tue Jun 21 22:57:04 2016 - [info] * Phase 1: Configuration Check Phase..Tue Jun 21 22:57:04 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] GTID failover mode = 1Tue Jun 21 22:57:05 2016 - [info] Dead Servers:Tue Jun 21 22:57:05 2016 - [info]   192.168.1.30(192.168.1.30:3306)Tue Jun 21 22:57:05 2016 - [info] Checking master reachability via MySQL(double check)...Tue Jun 21 22:57:05 2016 - [info]  ok.Tue Jun 21 22:57:05 2016 - [info] Alive Servers:Tue Jun 21 22:57:05 2016 - [info]   192.168.1.20(192.168.1.20:3306)Tue Jun 21 22:57:05 2016 - [info]   192.168.1.40(192.168.1.40:3306)Tue Jun 21 22:57:05 2016 - [info] Alive Slaves:Tue Jun 21 22:57:05 2016 - [info]   192.168.1.20(192.168.1.20:3306)  Version=5.6.31-log (oldest major version between slaves) log-bin:enabledTue Jun 21 22:57:05 2016 - [info]     GTID ONTue Jun 21 22:57:05 2016 - [info]     Replicating from 192.168.1.30(192.168.1.30:3306)Tue Jun 21 22:57:05 2016 - [info]     Primary candidate for the new Master (candidate_master is set)Tue Jun 21 22:57:05 2016 - [info]   192.168.1.40(192.168.1.40:3306)  Version=5.6.31-log (oldest major version between slaves) log-bin:enabledTue Jun 21 22:57:05 2016 - [info]     GTID ONTue Jun 21 22:57:05 2016 - [info]     Replicating from 192.168.1.30(192.168.1.30:3306)Tue Jun 21 22:57:05 2016 - [info] Starting GTID based failover.Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] ** Phase 1: Configuration Check Phase completed.Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] * Phase 2: Dead Master Shutdown Phase..Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] Forcing shutdown so that applications never connect to the current master..Tue Jun 21 22:57:05 2016 - [info] Executing master IP deactivation script:Tue Jun 21 22:57:05 2016 - [info]   /usr/local/mha/script/master_ip_failover --orig_master_host=192.168.1.30 --orig_master_ip=192.168.1.30 --orig_master_port=3306 --command=stopssh --ssh_usIN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===Disabling the VIP on old master: 192.168.1.30Tue Jun 21 22:57:05 2016 - [info]  done.Tue Jun 21 22:57:05 2016 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.Tue Jun 21 22:57:05 2016 - [info] * Phase 2: Dead Master Shutdown Phase completed.Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] * Phase 3: Master Recovery Phase..Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] * Phase 3.1: Getting Latest Slaves Phase..Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] The latest binary log file/position on all slaves is mysql-bin.000002:373Tue Jun 21 22:57:05 2016 - [info] Latest slaves (Slaves that received relay log files to the latest):Tue Jun 21 22:57:05 2016 - [info]   192.168.1.20(192.168.1.20:3306)  Version=5.6.31-log (oldest major version between slaves) log-bin:enabledTue Jun 21 22:57:05 2016 - [info]     GTID ONTue Jun 21 22:57:05 2016 - [info]     Replicating from 192.168.1.30(192.168.1.30:3306)Tue Jun 21 22:57:05 2016 - [info]     Primary candidate for the new Master (candidate_master is set)Tue Jun 21 22:57:05 2016 - [info]   192.168.1.40(192.168.1.40:3306)  Version=5.6.31-log (oldest major version between slaves) log-bin:enabledTue Jun 21 22:57:05 2016 - [info]     GTID ONTue Jun 21 22:57:05 2016 - [info]     Replicating from 192.168.1.30(192.168.1.30:3306)Tue Jun 21 22:57:05 2016 - [info] The oldest binary log file/position on all slaves is mysql-bin.000002:373Tue Jun 21 22:57:05 2016 - [info] Oldest slaves:Tue Jun 21 22:57:05 2016 - [info]   192.168.1.20(192.168.1.20:3306)  Version=5.6.31-log (oldest major version between slaves) log-bin:enabledTue Jun 21 22:57:05 2016 - [info]     GTID ONTue Jun 21 22:57:05 2016 - [info]     Replicating from 192.168.1.30(192.168.1.30:3306)Tue Jun 21 22:57:05 2016 - [info]     Primary candidate for the new Master (candidate_master is set)Tue Jun 21 22:57:05 2016 - [info]   192.168.1.40(192.168.1.40:3306)  Version=5.6.31-log (oldest major version between slaves) log-bin:enabledTue Jun 21 22:57:05 2016 - [info]     GTID ONTue Jun 21 22:57:05 2016 - [info]     Replicating from 192.168.1.30(192.168.1.30:3306)Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] * Phase 3.3: Determining New Master Phase..Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] Searching new master from slaves..Tue Jun 21 22:57:05 2016 - [info]  Candidate masters from the configuration file:Tue Jun 21 22:57:05 2016 - [info]   192.168.1.20(192.168.1.20:3306)  Version=5.6.31-log (oldest major version between slaves) log-bin:enabledTue Jun 21 22:57:05 2016 - [info]     GTID ONTue Jun 21 22:57:05 2016 - [info]     Replicating from 192.168.1.30(192.168.1.30:3306)Tue Jun 21 22:57:05 2016 - [info]     Primary candidate for the new Master (candidate_master is set)Tue Jun 21 22:57:05 2016 - [info]  Non-candidate masters:Tue Jun 21 22:57:05 2016 - [info]  Searching from candidate_master slaves which have received the latest relay log events..Tue Jun 21 22:57:05 2016 - [info] New master is 192.168.1.20(192.168.1.20:3306)Tue Jun 21 22:57:05 2016 - [info] Starting master failover..Tue Jun 21 22:57:05 2016 - [info]From:192.168.1.30(192.168.1.30:3306) (current master) +--192.168.1.20(192.168.1.20:3306) +--192.168.1.40(192.168.1.40:3306)To:192.168.1.20(192.168.1.20:3306) (new master) +--192.168.1.40(192.168.1.40:3306)Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] * Phase 3.3: New Master Recovery Phase..Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info]  Waiting all logs to be applied..Tue Jun 21 22:57:05 2016 - [info]   done.Tue Jun 21 22:57:05 2016 - [info] Getting new master's binlog name and position..Tue Jun 21 22:57:05 2016 - [info]  master-bin.000004:231Tue Jun 21 22:57:05 2016 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.20', MASTER_PORT=3306, MASTER_AUTO_POSIT_USER='slave', MASTER_PASSWORD='xxx';Tue Jun 21 22:57:05 2016 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: master-bin.000004, 231, 252b2aad-4399-11e6-8ba8-000c29ca702d:2,4a81183e-3722-11e6-ba60-000c29443328:1-3Tue Jun 21 22:57:05 2016 - [info] Executing master IP activate script:Tue Jun 21 22:57:05 2016 - [info]   /usr/local/mha/script/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.1.30 --orig_master_ip=192.168.1.30 --orig_master_portaster_host=192.168.1.20 --new_master_ip=192.168.1.20 --new_master_port=3306 --new_master_user='root' --new_master_password='123'Unknown option: new_master_userUnknown option: new_master_passwordIN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===Enabling the VIP - 192.168.1.100 on the new master - 192.168.1.20Tue Jun 21 22:57:05 2016 - [info]  OK.Tue Jun 21 22:57:05 2016 - [info] ** Finished master recovery successfully.Tue Jun 21 22:57:05 2016 - [info] * Phase 3: Master Recovery Phase completed.Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] * Phase 4: Slaves Recovery Phase..Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] * Phase 4.1: Starting Slaves in parallel..Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] -- Slave recovery on host 192.168.1.40(192.168.1.40:3306) started, pid: 37397. Check tmp log /usr/local/mha/192.168.1.40_3306_20160621225704.log if it takeTue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] Log messages from 192.168.1.40 ...Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info]  Resetting slave 192.168.1.40(192.168.1.40:3306) and starting replication from the new master 192.168.1.20(192.168.1.20:3306)..Tue Jun 21 22:57:05 2016 - [info]  Executed CHANGE MASTER.Tue Jun 21 22:57:05 2016 - [info]  Slave started.Tue Jun 21 22:57:05 2016 - [info]  gtid_wait(252b2aad-4399-11e6-8ba8-000c29ca702d:2,4a81183e-3722-11e6-ba60-000c29443328:1-3) completed on 192.168.1.40(192.168.1.40:3306). Executed 0 events.Tue Jun 21 22:57:05 2016 - [info] End of log messages from 192.168.1.40.Tue Jun 21 22:57:05 2016 - [info] -- Slave on host 192.168.1.40(192.168.1.40:3306) started.Tue Jun 21 22:57:05 2016 - [info] All new slave servers recovered successfully.Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] * Phase 5: New master cleanup phase..Tue Jun 21 22:57:05 2016 - [info]Tue Jun 21 22:57:05 2016 - [info] Resetting slave info on the new master..Tue Jun 21 22:57:05 2016 - [info]  192.168.1.20: Resetting slave info succeeded.Tue Jun 21 22:57:05 2016 - [info] Master failover to 192.168.1.20(192.168.1.20:3306) completed successfully.Tue Jun 21 22:57:05 2016 - [info] Deleted server2 entry from /etc/mha/mha.conf .Tue Jun 21 22:57:05 2016 - [info]----- Failover Report -----mha: MySQL Master failover 192.168.1.30(192.168.1.30:3306) to 192.168.1.20(192.168.1.20:3306) succeededMaster 192.168.1.30(192.168.1.30:3306) is down!Check MHA Manager logs at db-one for details.Started automated(non-interactive) failover.Invalidated master IP address on 192.168.1.30(192.168.1.30:3306)Selected 192.168.1.20(192.168.1.20:3306) as a new master.192.168.1.20(192.168.1.20:3306): OK: Applying all logs succeeded.192.168.1.20(192.168.1.20:3306): OK: Activated master IP address.192.168.1.40(192.168.1.40:3306): OK: Slave started, replicating from 192.168.1.20(192.168.1.20:3306)192.168.1.20(192.168.1.20:3306): Resetting slave info succeeded.Master failover to 192.168.1.20(192.168.1.20:3306) completed successfully.Tue Jun 21 22:57:05 2016 - [info] Sending mail..

（over）