mysql高可用MHA配置

MHA配置mysql高可用

本次实验的环境为rhel6.5
确保主从复制环境已经ok！！！
monitor 172.25.78.7 server7
master 172.25.78.2 server2
slave 172.25.78.3 server3（备主）
slave2 172.25.78.4 server4

MHA工作原理总结为以下几条：
（1）从宕机崩溃的master保存二进制日志事件（binlog events）;
（2）识别含有最新更新的slave;
（3）应用差异的中继日志(relay log) 到其他slave;
（4）应用从master保存的二进制日志事件(binlog events);
（5）提升一个slave为新master;
（6）使用其他的slave连接新的master进行复制。

MySQL MHA架构介绍：
MHA（Master High Availability）目前在MySQL高可用方面是一个相对成熟的解决方案，它由日本DeNA公司youshimaton（现就职于Facebook公司）开发，是一套优秀的作为MySQL高可用性环境下故障切换和主从提升的高可用软件。在MySQL故障切换过程中，MHA能做到在0~30秒之内自动完成数据库的故障切换操作，并且在进行故障切换的过程中，MHA能在最大程度上保证数据的一致性，以达到真正意义上的高可用。
该软件由两部分组成：MHA Manager（管理节点）和MHA Node（数据节点）。MHA Manager可以单独部署在一台独立的机器上管理多个master-slave集群，也可以部署在一台slave节点上。MHA Node运行在每台MySQL服务器上，MHA Manager会定时探测集群中的master节点，当master出现故障时，它可以自动将最新数据的slave提升为新的master，然后将所有其他的slave重新指向新的master。整个故障转移过程对应用程序完全透明。
在MHA自动故障切换过程中，MHA试图从宕机的主服务器上保存二进制日志，最大程度的保证数据的不丢失，但这并不总是可行的。例如，如果主服务器硬件故障或无法通过ssh访问，MHA没法保存二进制日志，只进行故障转移而丢失了最新的数据。使用MySQL 5.5的半同步复制，可以大大降低数据丢失的风险。MHA可以与半同步复制结合起来。如果只有一个slave已经收到了最新的二进制日志，MHA可以将最新的二进制日志应用于其他所有的slave服务器上，因此可以保证所有节点的数据一致性。

Mysql MHA的配置过程：

一、在所有节点都要安装MHA node所需要的perl模块，在安装之前先安装epel源：
在server2、server3、server4相同，如下

[root@server2 ~]# rpm -ivh epel-release-6-8.noarch.rpmwarning: epel-release-6-8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEYPreparing...                                                 ########################################### [100%]   1:epel-release                                            ########################################### [100%]yum install -y perl-DBD-MySQL

二、在所有的节点都要安装MHA node（包括manager节点）

1.配置MHA node主机
Server3和server4也要进行相同的操作

[root@server2 ~]# tar xf mha4mysql-node-0.56.tar.gz [root@server2 ~]# cd mha4mysql-node-0.56[root@server2 mha4mysql-node-0.56]# perl Makefile.PL （如果这出现报错，那么根据报错安装所需要的软件包解决依赖性即可。我在做的时候需要的是：yum install perl-devel -y yum install perl-CPAN -y）[root@server2 mha4mysql-node-0.56]# make && make install

2.配置MHA manager主机(在172.25.78.7上)
1）如1配置MHA node

rpm -ivh epel-release-6-8.noarch.rpm --- yum install -y perl-DBD-MySQL --- yum install perl-devel -y, yum install perl-CPAN -y ---- tar xf mha4mysql-node-0.56.tar.gz  ---- cd mha4mysql-node-0.56 --- perl Makefile.PL  --- make && make install

2)安装MHA Manager

tar xf mha4mysql-node-0.56.tar.gz cd mha4mysql-node-0.56perl Makefile.PL make && make install

在上述过程中会出现依赖性问题，需要安装yum install -y perl-Log-Dispatch-2.27-1.el6.noarch.rpm perl-Parallel-ForkManager-0.7.9-1.el6.noarch.rpm
此时安装完成后，在此目录下回产生相应的脚本：

[root@server7 mha4mysql-manager-0.56]# cd /usr/local/bin/[root@server7 bin]# lsapply_diff_relay_logs  masterha_master_monitorfilter_mysqlbinlog     masterha_master_switchmasterha_check_repl    masterha_secondary_checkmasterha_check_ssh     masterha_stopmasterha_check_status  purge_relay_logsmasterha_conf_host     save_binary_logsmasterha_manager

3.配置SSH登录不需要密码验证
1）在server2（172.25.78.2master）上：

[root@server2 ~]#ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa.Your public key has been saved in /root/.ssh/id_rsa.pub.The key fingerprint is:71:d5:bb:58:56:92:d3:03:27:a7:06:e2:68:6c:5c:e0 root@server2The key's randomart image is:+--[ RSA 2048]----+|       ..o ..+.= ||      + + ... X.o||       E...  o =.||      o  o  . +  ||        S    + . ||            . .  ||                 ||                 ||                 |+-----------------+[root@server2 ~]# ssh-copy-id -i /root/.ssh/id_rsa root@172.25.78.3The authenticity of host '172.25.78.3 (172.25.78.3)' can't be established.RSA key fingerprint is cc:15:ee:47:ce:a9:31:60:f7:e2:20:14:6e:26:00:de.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added '172.25.78.3' (RSA) to the list of known hosts.root@172.25.78.3's password: Now try logging into the machine, with "ssh 'root@172.25.78.3'", and check in:  .ssh/authorized_keysto make sure we haven't added extra keys that you weren't expecting.[root@server2 ~]# ssh-copy-id -i /root/.ssh/id_rsa root@172.25.78.4root@172.25.78.4's password: Now try logging into the machine, with "ssh 'root@172.25.78.4'", and check in:  .ssh/authorized_keysto make sure we haven't added extra keys that you weren't expecting.

2）在slave上

A.在server3（172.25.78.3）上：

ssh-keygen -t rsa分发给server2和server4ssh-copy-id -i /root/.ssh/id_rsa.pub root@172.25.78.2ssh-copy-id -i /root/.ssh/id_rsa.pub root@172.25.78.4

B.在server4（172.25.78.4）上：

ssh-keygen -t rsa分发给server2和server3ssh-copy-id  -i  /root/.ssh/id_rsa.pub  root@172.25.78.2ssh-copy-id  -i  /root/.ssh/id_rsa.pub  root@172.25.78.3

3）在monitor上

ssh-keygen -t rsa分发给master和slavessh-copy-id -i /root/.ssh/id_rsa.pub root@172.25.78.2ssh-copy-id  -i /root/.ssh/id_rsa.pub  root@172.25.78.3ssh-copy-id -i /root/.ssh/id_rsa.pub root@172.25.78.4

4.配置MHA
在172.25.78.7（server7）上面
1）创建mkdir -p /etc/mastermha

2）cp /root/mha4mysql-manager-0.56/samples/conf/app1.cnf /etc/mastermha/

vim app1.cnf

[server default]manager_workdir=/var/log/masterha/app1（设置manager的工作目录）manager_log=/var/log/masterha/app1/manager.log（设置manager的日志）master_binlog_dir=/data/mysql（设置master，保存binlog的位置，以便MHA可以找到master的日志）#master_ip_failover_script=/usr/local/bin/master_ip_failover（设置自动failover时候的切换脚本）#master_ip_online_change_script=/usr/local/bin/master_ip_online_changepassword=Westos+007（mysql的密码，就是 server7上的密码）user=root（设置监控用户root）ping_interval=1（设置监控主库，发送ping包的时间间隔，默认三秒，尝试三次没有回应，自动进行railover）remote_workdir=/tmp（设置远端mysql在发送切换是binlog的保存位置）repl_password=Westos+007（设置复制用户的密码）----公用数据库的repl_user=mysql（复制用户）report_script=/usr/local/bin/send_report（设置发送切换后发送的报警脚本）secondary_check_script=/usr/local/bin/masterha_secondary_check -s server3-s server2 --username=root --master_host=server2 --master_ip=172.25.78.2 --master-port=3306（MHA到server2的监控能够出现问题，MHA manager会尝试从server3登录到server2）ssh_user=rootshutdown_script=""[server1]hostname=172.25.78.2port=3306[server2]hostname=172.25.78.3candidate_master=1（设置为候选master，即使不是集群中事件最新的slave）port=3306check_repl_delay=0（默认情况下，如果一个slave落后master100M的话，MHA将不会选择这个为master，设置为0，那么MHA触发切换在选择一个新的master时，会忽略复制延迟）[server3]hostname=172.25.78.4port=3306

5.设置relay log的清除方式（在每个slave节点上）：
在slave 172.25.78.3操作：
mysql−uroot−pWestos+007−e“setglobalrelaylogpurge=0”在slave2172.25.78.4操作： mysql -uroot -pWestos+007 -e “set global relay_log_purge=0”

注意：（网上看的）
MHA在发生切换的过程中，从库的恢复过程中依赖于relay log的相关信息，所以这里要将relay log的自动清除设置为OFF，采用手动清除relay log的方式。在默认情况下，从服务器上的中继日志会在SQL线程执行完毕后被自动删除。但是在MHA环境中，这些中继日志在恢复其他从服务器时可能会被用到，因此需要禁用中继日志的自动删除功能。定期清除中继日志需要考虑到复制延时的问题。在ext3的文件系统下，删除大的文件需要一定的时间，会导致严重的复制延时。为了避免复制延时，需要暂时为中继日志创建硬链接，因为在linux系统中通过硬链接删除大文件速度会很快。（在mysql数据库中，删除大表时，通常也采用建立硬链接的方式）

6.检查ssh配置
[root@server7 mastermha]# masterha_check_ssh –conf=/etc/mastermha/app1.cnf（检查ssh配置）
此时会出现一系列的报错，说明依赖性缺失。需要安装如下依赖性：

yum install -y perl-Log-Dispatch-2.27-1.el6.noarch.rpmperl-Mail-Sender-0.8.16-3.el6.noarch.rpm perl-Mail-Sendmail-0.79-12.el6.noarch.rpm perl-MIME-Lite-3.027-2.el6.noarch.rpm perl-MIME-Lite-HTML-1.23-2.el6.noarch.rpm  perl-Log-Dispatch-2.27-1.el6.noarch.rpmperl-Parallel-ForkManager-0.7.9-1.el6.noarch.rpmperl-Config-Tiny-2.12-7.1.el6.noarch.rpm

[root@server7 ~]# masterha_check_ssh –conf=/etc/mastermha/app1.cnf（检查整个复制环境状况）

Sat Aug  5 14:04:46 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Sat Aug  5 14:04:46 2017 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..Sat Aug  5 14:04:46 2017 - [info] Reading server configuration from /etc/mastermha/app1.cnf..Sat Aug  5 14:04:46 2017 - [info] Starting SSH connection tests..Sat Aug  5 14:04:47 2017 - [debug] Sat Aug  5 14:04:46 2017 - [debug]  Connecting via SSH from root@172.25.78.2(172.25.78.2:22) to root@172.25.78.3(172.25.78.3:22)..Sat Aug  5 14:04:46 2017 - [debug]   ok.Sat Aug  5 14:04:46 2017 - [debug]  Connecting via SSH from root@172.25.78.2(172.25.78.2:22) to root@172.25.78.4(172.25.78.4:22)..Sat Aug  5 14:04:46 2017 - [debug]   ok.Sat Aug  5 14:04:47 2017 - [debug] Sat Aug  5 14:04:46 2017 - [debug]  Connecting via SSH from root@172.25.78.3(172.25.78.3:22) to root@172.25.78.2(172.25.78.2:22)..Sat Aug  5 14:04:47 2017 - [debug]   ok.Sat Aug  5 14:04:47 2017 - [debug]  Connecting via SSH from root@172.25.78.3(172.25.78.3:22) to root@172.25.78.4(172.25.78.4:22)..Sat Aug  5 14:04:47 2017 - [debug]   ok.Sat Aug  5 14:04:47 2017 - [debug] Sat Aug  5 14:04:47 2017 - [debug]  Connecting via SSH from root@172.25.78.4(172.25.78.4:22) to root@172.25.78.2(172.25.78.2:22)..Sat Aug  5 14:04:47 2017 - [debug]   ok.Sat Aug  5 14:04:47 2017 - [debug]  Connecting via SSH from root@172.25.78.4(172.25.78.4:22) to root@172.25.78.3(172.25.78.3:22)..Sat Aug  5 14:04:47 2017 - [debug]   ok.Sat Aug  5 14:04:47 2017 - [info] All SSH connection tests passed successfully.

[root@server7 mastermha]# masterha_check_repl –conf=/etc/mastermha/app1.cnf

Sat Aug  5 15:02:17 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Sat Aug  5 15:02:17 2017 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..Sat Aug  5 15:02:17 2017 - [info] Reading server configuration from /etc/mastermha/app1.cnf..Sat Aug  5 15:02:17 2017 - [info] MHA::MasterMonitor version 0.56.Sat Aug  5 15:02:17 2017 - [info] GTID failover mode = 1Sat Aug  5 15:02:17 2017 - [info] Dead Servers:Sat Aug  5 15:02:17 2017 - [info] Alive Servers:Sat Aug  5 15:02:17 2017 - [info]   172.25.78.2(172.25.78.2:3306)Sat Aug  5 15:02:17 2017 - [info]   172.25.78.3(172.25.78.3:3306)Sat Aug  5 15:02:17 2017 - [info]   172.25.78.4(172.25.78.4:3306)Sat Aug  5 15:02:17 2017 - [info] Alive Slaves:Sat Aug  5 15:02:17 2017 - [info]   172.25.78.3(172.25.78.3:3306)  Version=5.7.17-log (oldest major version between slaves) log-bin:enabledSat Aug  5 15:02:17 2017 - [info]     GTID ONSat Aug  5 15:02:17 2017 - [info]     Replicating from 172.25.78.2(172.25.78.2:3306)Sat Aug  5 15:02:17 2017 - [info]     Primary candidate for the new Master (candidate_master is set)Sat Aug  5 15:02:17 2017 - [info]   172.25.78.4(172.25.78.4:3306)  Version=5.7.17-log (oldest major version between slaves) log-bin:enabledSat Aug  5 15:02:17 2017 - [info]     GTID ONSat Aug  5 15:02:17 2017 - [info]     Replicating from 172.25.78.2(172.25.78.2:3306)Sat Aug  5 15:02:17 2017 - [info] Current Alive Master: 172.25.78.2(172.25.78.2:3306)Sat Aug  5 15:02:17 2017 - [info] Checking slave configurations..Sat Aug  5 15:02:17 2017 - [info] Checking replication filtering settings..Sat Aug  5 15:02:17 2017 - [info]  binlog_do_db= test, binlog_ignore_db= mysqlSat Aug  5 15:02:17 2017 - [info]  Replication filtering check ok.Sat Aug  5 15:02:17 2017 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.Sat Aug  5 15:02:17 2017 - [info] Checking SSH publickey authentication settings on the current master..Sat Aug  5 15:02:17 2017 - [info] HealthCheck: SSH to 172.25.78.2 is reachable.Sat Aug  5 15:02:17 2017 - [info] 172.25.78.2(172.25.78.2:3306) (current master) +--172.25.78.3(172.25.78.3:3306) +--172.25.78.4(172.25.78.4:3306)Sat Aug  5 15:02:17 2017 - [info] Checking replication health on 172.25.78.3..Sat Aug  5 15:02:17 2017 - [info]  ok.Sat Aug  5 15:02:17 2017 - [info] Checking replication health on 172.25.78.4..Sat Aug  5 15:02:17 2017 - [info]  ok.Sat Aug  5 15:02:17 2017 - [warning] master_ip_failover_script is not defined.Sat Aug  5 15:02:17 2017 - [warning] shutdown_script is not defined.Sat Aug  5 15:02:17 2017 - [info] Got exit code 0 (Not master dead).MySQL Replication Health is OK.

出现报错：

[root@server7 mastermha]# masterha_check_repl --conf=/etc/mastermha/app1.cnf Sat Aug  5 15:00:13 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Sat Aug  5 15:00:13 2017 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..Sat Aug  5 15:00:13 2017 - [info] Reading server configuration from /etc/mastermha/app1.cnf..Sat Aug  5 15:00:13 2017 - [info] MHA::MasterMonitor version 0.56.Sat Aug  5 15:00:13 2017 - [error][/usr/local/share/perl5/MHA/Server.pm, ln260] Checking slave status failed on 172.25.78.2(172.25.78.2:3306). err=Got error when executing SHOW SLAVE STATUS. Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operationSat Aug  5 15:00:13 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations.  at /usr/local/share/perl5/MHA/ServerManager.pm line 315Sat Aug  5 15:00:13 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers.Sat Aug  5 15:00:13 2017 - [info] Got exit code 1 (Not master dead).MySQL Replication Health is NOT OK!解决办法：在master和slave端都要进行授权：mysql> grant all on *.* to mysql@'172.25.78.%' identified by 'Westos+007';Query OK, 0 rows affected, 1 warning (0.39 sec)而且在Failover两种方式：一种虚拟IP地址，一种全局配置文件。MHA并没有并未有限定是使用哪一种方式，而是让用户自己选择，虚拟ip地址的方式会牵扯到其他的软件，如keepalive软件，而且还要修改脚本master_ip_failover。所以暂时注释这个选项。

7.检查MHA Manager的状态
[root@server7 mysql]# masterha_check_status –conf=/etc/mastermha/app1.cnf
app1 is stopped(2:NOT_RUNNING).

注：如果正常，会显示“PING_OK”,否则会显示“NOT——RUNNING”，这表示MHA监控没有开启。

开启MHA Manager监控

[root@server7 mysql]# mkdir -p /var/log/masterha/app1/[root@server7 mysql]# nohup masterha_manager --conf=/etc/masrmha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &[1] 1316[root@server7 mysql]# masterha_check_status --conf=/etc/mastermha/app1.cnf app1 (pid:1316) is running(0:PING_OK), master:172.25.78.2显示监控已经开始监控了，而且master主机为172.25.78.2

–remove_dead_master_conf //该参数代表当发生主从切换后，老的主库的ip将会从配置文件中移除。
–manger_log //日志存放位置
–ignore_last_failover //在缺省情况下，如果MHA检测到连续发生宕机，且两次宕机间隔不足8小时的话，则不会进行Failover，之所以这样限制是为了避免ping-pong效应。该参数代表忽略上次MHA触发切换产生的文件，默认情况下，MHA发生切换后会在日志目录，也就是上面我设置的/data产生app1.failover.complete文件，下次再次切换的时候如果发现该目录下存在该文件将不允许触发切换，除非在第一次切换后收到删除该文件，为了方便，这里设置为–ignore_last_failover。

[root@server7 mysql]# tail -n 20 /var/log/masterha/app1/manager.log Sat Aug  5 17:27:33 2017 - [info]     Replicating from 172.25.78.2(172.25.78.2:3306)Sat Aug  5 17:27:33 2017 - [info] Current Alive Master: 172.25.78.2(172.25.78.2:3306)Sat Aug  5 17:27:33 2017 - [info] Checking slave configurations..Sat Aug  5 17:27:33 2017 - [info] Checking replication filtering settings..Sat Aug  5 17:27:33 2017 - [info]  binlog_do_db= test, binlog_ignore_db= mysqlSat Aug  5 17:27:33 2017 - [info]  Replication filtering check ok.Sat Aug  5 17:27:33 2017 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.Sat Aug  5 17:27:33 2017 - [info] Checking SSH publickey authentication settings on the current master..Sat Aug  5 17:27:33 2017 - [info] HealthCheck: SSH to 172.25.78.2 is reachable.Sat Aug  5 17:27:33 2017 - [info] 172.25.78.2(172.25.78.2:3306) (current master) +--172.25.78.3(172.25.78.3:3306) +--172.25.78.4(172.25.78.4:3306)Sat Aug  5 17:27:33 2017 - [warning] master_ip_failover_script is not defined.Sat Aug  5 17:27:33 2017 - [warning] shutdown_script is not defined.Sat Aug  5 17:27:33 2017 - [info] Set master ping interval 1 seconds.Sat Aug  5 17:27:33 2017 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s 172.25.78.3 -s 172.25.78.22 --user=root --master_host=server2 --master_ip=172.25.78.2 --master-port=3306Sat Aug  5 17:27:33 2017 - [info] Starting ping health check on 172.25.78.2(172.25.78.2:3306)..Sat Aug  5 17:27:33 2017 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..其中“Ping（SELECT）succeded,waiting until MYSQL doesn't respond...”说明整个系统开始监控了。

8.手动切换failover（MHA Manager必须没有运行）

首先停掉MHA Manager

[root@server7 app1]# masterha_stop --conf=/etc/mastermha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1

这时手动更换master

[root@server7 app1]# masterha_master_switch --master_state=dead -conf=/etc/mastermha/app1.cnf --dead_master_host=172.25.78.2 --dead_moster_port=3306 --new-master_host=172.25.78.3 --new_master_port=3306 --ignore_last_failover--dead_master_ip=<dead_master_ip> is not set. Using 172.25.78.2.--dead_master_port=<dead_master_port> is not set. Using 3306.Sat Aug  5 21:05:49 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Sat Aug  5 21:05:49 2017 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..Sat Aug  5 21:05:49 2017 - [info] Reading server configuration from /etc/mastermha/app1.cnf..Sat Aug  5 21:05:49 2017 - [info] MHA::MasterFailover version 0.56.Sat Aug  5 21:05:49 2017 - [info] Starting master failover.Sat Aug  5 21:05:49 2017 - [info] Sat Aug  5 21:05:49 2017 - [info] * Phase 1: Configuration Check Phase..Sat Aug  5 21:05:49 2017 - [info] Sat Aug  5 21:05:49 2017 - [info] GTID failover mode = 1Sat Aug  5 21:05:49 2017 - [info] Dead Servers:Sat Aug  5 21:05:49 2017 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln187] None of server is dead. Stop failover.Sat Aug  5 21:05:49 2017 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln177] Got ERROR:  at /usr/local/bin/masterha_master_switch line 53此时报错，原因是MHA Manager检测到没有dead server，报错，并且同时结束了failover。那么必须手动关闭主库，才能正常切换。

关闭主master再手动切换

[root@server2 ~]# /etc/init.d/mysqld stopStopping mysqld:                                           [  OK  ][root@server7 app1]# masterha_master_switch --master_state=dead -conf=/etc/mastermha/app1.cnf --dead_master_host=172.25.78.2 --dead_moster_port=3306 --new-master_host=172.25.78.3 --new_master_port=3306 --ignore_last_failoverapp1: MySQL Master failover 172.25.78.2(172.25.78.2:3306) to 172.25.78.3(172.25.78.3:3306) succeeded（此时显示切换成功）Master 172.25.78.2(172.25.78.2:3306) is down!Check MHA Manager logs at server7 for details.Started manual(interactive) failover.Selected 172.25.78.3(172.25.78.3:3306) as a new master.172.25.78.3(172.25.78.3:3306): OK: Applying all logs succeeded.172.25.78.4(172.25.78.4:3306): OK: Slave started, replicating from 172.25.78.3(172.25.78.3:3306)172.25.78.3(172.25.78.3:3306): Resetting slave info succeeded.Master failover to 172.25.78.3(172.25.78.3:3306) completed successfully.Sat Aug  5 21:12:59 2017 - [info] Sending mail..Unknown option: conf