MHA安装配置

1. 前言

        MHA能够在较短的时间内实现自动故障检测和故障转移，通常在10-30秒以内;在复制框架中，MHA能够很好地解决复制过程中的数据一致性问题，由于不需要在现有的replication中添加额外的服务器，仅需要一个manager节点，而一个Manager能管理多套复制，所以能大大地节约服务器的数量;另外，安装简单，无性能损耗，以及不需要修改现有的复制部署也是它的优势之处。

         MHA还提供在线主库切换的功能，能够安全地切换当前运行的主库到一个新的主库中(通过将从库提升为主库),大概0.5-2秒内即可完成。
MHA提供了上述功能，使得其在适用于对高可用性，数据完整性要求高的场合，还有要求几乎non-stop的主库维护。

         这次主要是讲解MHA的快速部署和简单配置，后续补充高级特性以及相关架构吧。

2. 环境介绍

环境有限，我使用的是vmware虚拟机，搭建三台服务器，1主2从结构。其中manager管理节点放在其中一台从库中。

1）server1（node节点）

OS：CentOS 6.4 X64

IPADDR：172.16.21.12（eth0）；192.168.8.6（eth1）

HOSTNAME：mysql-mha01

DB：Mysql 5.5.43-log MySQL Community Server (GPL)

SOFTWARE：mha4mysql-node-0.56-0.el6.noarch.rpm

防火墙关闭；selinux关闭；

2）server2（node节点）

OS：CentOS 6.4 X64

IPADDR：172.16.21.13（eth2）；192.168.8.7（eth3）

HOSTNAME：mysql-mha02

DB：Mysql 5.5.43-log MySQL Community Server (GPL)

SOFTWARE：mha4mysql-node-0.56-0.el6.noarch.rpm

防火墙关闭；selinux关闭；

3）server3（node节点，manager节点）

OS：CentOS 6.4 X64

IPADDR：172.16.21.15（eth2）；192.168.8.8（eth3）

HOSTNAME：mysql-mha01

DB：Mysql 5.5.43-log MySQL Community Server (GPL)

SOFTWARE：mha4mysql-node-0.56-0.el6.noarch.rpm；mha4mysql-manager-0.56-0.el6.noarch.rpm

防火墙关闭；selinux关闭；

MHA软件下载地址 https://code.google.com/p/mysql-master-ha/wiki/Downloads?tm=2

3. 软件安装

3.1 Node节点安装

在3个节点上安装node软件包，首先安装依赖包

yum install perl-DBD-MySQL

安装node软件包

 rpm -ivh mha4mysql-node-0.56-0.el6.noarch.rpm

3.2 Manager节点安装

安装依赖包

yum install perl-Params-Validate perl-LWP-Simple perl-Config-Tiny perl-Time-HiRes perl-Log-Dispatch perl-Parallel-ForkManager 

我当时配置的163的yum源没有perl-Log-Dispatch和perl-Parallel-ForkManager，所以就自己下载的rpm包安装，大家可以到http://pkgs.repoforge.org/ 下载自己需要的包。

rpm -ivh perl-Log-Dispatch-2.26-1.el6.rf.noarch.rpm perl-Parallel-ForkManager-0.7.5-2.2.el6.rf.noarch.rpm

安装manager软件包

rpm -ivh mha4mysql-manager-0.56-0.el6.noarch.rpm

4. MHA配置

4.1 Mysql复制配置

        此处就不详细说明了，大家可以参考我以前的文章。http://blog.csdn.net/wjf870128/article/details/45170089 和 http://blog.csdn.net/wjf870128/article/details/45149573。

我搭建的是半同步复制的一主双从结构。其中mysql-mha02为master节点，其他为slave节点。复制账户密码repl/slavepass。（PS我们公司环境是5.5的所以这里的虚拟机也是用到的mysql5.5。现在MHA支持5.6的GTID复制，因此大家可以考虑使用5.6的gtid搭建，可以方便master由于崩溃恢复后重新加入集群管理）

4.2 MHA配置

在管理节点mysql-mha02中建立配置文件

mkdir -p /etc/MHAtouch /etc/MHA/MHAmanager.conf

查看manager配置文件

[server default]  # mysql user and password  user=root  password=123456  ssh_user=root  # mysql replication user and password  repl_user=repl  repl_password=slavepass  # working directory on the manager  manager_workdir=/var/log/masterha/manager  # working directory on MySQL servers  remote_workdir=/var/log/masterha/node  #Master Binlog dir  master_binlog_dir=/var/lib/mysql/logs  #secondary route to detect the reachable of master  secondary_check_script= masterha_secondary_check -s mysql-mha03 --user=root --master_host=mysql-mha02 --master_ip=172.16.21.13 --master_port=3306  #define the manager log  manager_log=/var/log/masterha/manager.log    [server1]  hostname=mysql-mha01    [server2]  hostname=mysql-mha02    [server3]  hostname=mysql-mha03

MHA当中详细参数大家可以参考 https://code.google.com/p/mysql-master-ha/wiki/Parameters。

master_binlog_dir标志着master节点binlog的存放位置，如果大家没有使用默认的/var/lib/mysql的话需要单独制定。MHA在故障切换的时候如果master节点可以ssh联通的话会自动到该目录抓取日志文件来防止slave节点日志不完整。

workdir是manager节点和node节点的工作目录，在MHA管理启动时候，如果目录不存在，会通过ssh来进行创建。请保证ssh用户的权限。

4.3 主机信任

MHA环境需要保证manager和node之间的信任。

ssh-keygen -t rsa ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.8.6ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.8.7ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.8.8

验证机器信任

[root@mysql-mha03 ~]# masterha_check_ssh --conf=/etc/MHA/MHAmanager.conf Thu Jul 23 14:33:26 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Thu Jul 23 14:33:26 2015 - [info] Reading application default configuration from /etc/MHA/MHAmanager.conf..Thu Jul 23 14:33:26 2015 - [info] Reading server configuration from /etc/MHA/MHAmanager.conf..Thu Jul 23 14:33:26 2015 - [info] Starting SSH connection tests..Thu Jul 23 14:33:26 2015 - [debug] Thu Jul 23 14:33:26 2015 - [debug]  Connecting via SSH from root@mysql-mha01(192.168.8.6:22) to root@mysql-mha02(192.168.8.7:22)..Thu Jul 23 14:33:26 2015 - [debug]   ok.Thu Jul 23 14:33:26 2015 - [debug]  Connecting via SSH from root@mysql-mha01(192.168.8.6:22) to root@mysql-mha03(192.168.8.8:22)..Thu Jul 23 14:33:26 2015 - [debug]   ok.Thu Jul 23 14:33:27 2015 - [debug] Thu Jul 23 14:33:26 2015 - [debug]  Connecting via SSH from root@mysql-mha02(192.168.8.7:22) to root@mysql-mha01(192.168.8.6:22)..Thu Jul 23 14:33:26 2015 - [debug]   ok.Thu Jul 23 14:33:26 2015 - [debug]  Connecting via SSH from root@mysql-mha02(192.168.8.7:22) to root@mysql-mha03(192.168.8.8:22)..Thu Jul 23 14:33:27 2015 - [debug]   ok.Thu Jul 23 14:33:27 2015 - [debug] Thu Jul 23 14:33:27 2015 - [debug]  Connecting via SSH from root@mysql-mha03(192.168.8.8:22) to root@mysql-mha01(192.168.8.6:22)..Thu Jul 23 14:33:27 2015 - [debug]   ok.Thu Jul 23 14:33:27 2015 - [debug]  Connecting via SSH from root@mysql-mha03(192.168.8.8:22) to root@mysql-mha02(192.168.8.7:22)..Thu Jul 23 14:33:27 2015 - [debug]   ok.Thu Jul 23 14:33:27 2015 - [info] All SSH connection tests passed successfully.

4.4 开启MHA

验证复制环境

[root@mysql-mha03 ~]# masterha_check_repl --conf=/etc/MHA/MHAmanager.conf Thu Jul 23 14:40:27 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Thu Jul 23 14:40:27 2015 - [info] Reading application default configuration from /etc/MHA/MHAmanager.conf..Thu Jul 23 14:40:27 2015 - [info] Reading server configuration from /etc/MHA/MHAmanager.conf..Thu Jul 23 14:40:27 2015 - [info] MHA::MasterMonitor version 0.56.Thu Jul 23 14:40:28 2015 - [info] GTID failover mode = 0Thu Jul 23 14:40:28 2015 - [info] Dead Servers:Thu Jul 23 14:40:28 2015 - [info] Alive Servers:Thu Jul 23 14:40:28 2015 - [info]   mysql-mha01(192.168.8.6:3306)Thu Jul 23 14:40:28 2015 - [info]   mysql-mha02(192.168.8.7:3306)Thu Jul 23 14:40:28 2015 - [info]   mysql-mha03(192.168.8.8:3306)Thu Jul 23 14:40:28 2015 - [info] Alive Slaves:Thu Jul 23 14:40:28 2015 - [info]   mysql-mha01(192.168.8.6:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 14:40:28 2015 - [info]     Replicating from 192.168.8.7(192.168.8.7:3306)Thu Jul 23 14:40:28 2015 - [info]   mysql-mha03(192.168.8.8:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 14:40:28 2015 - [info]     Replicating from 192.168.8.7(192.168.8.7:3306)Thu Jul 23 14:40:28 2015 - [info] Current Alive Master: mysql-mha02(192.168.8.7:3306)Thu Jul 23 14:40:28 2015 - [info] Checking slave configurations..Thu Jul 23 14:40:28 2015 - [info]  read_only=1 is not set on slave mysql-mha01(192.168.8.6:3306).Thu Jul 23 14:40:28 2015 - [info]  read_only=1 is not set on slave mysql-mha03(192.168.8.8:3306).Thu Jul 23 14:40:28 2015 - [info] Checking replication filtering settings..Thu Jul 23 14:40:28 2015 - [info]  binlog_do_db= , binlog_ignore_db= Thu Jul 23 14:40:28 2015 - [info]  Replication filtering check ok.Thu Jul 23 14:40:28 2015 - [info] GTID (with auto-pos) is not supportedThu Jul 23 14:40:28 2015 - [info] Starting SSH connection tests..Thu Jul 23 14:40:29 2015 - [info] All SSH connection tests passed successfully.Thu Jul 23 14:40:29 2015 - [info] Checking MHA Node version..Thu Jul 23 14:40:29 2015 - [info]  Version check ok.Thu Jul 23 14:40:29 2015 - [info] Checking SSH publickey authentication settings on the current master..Thu Jul 23 14:40:30 2015 - [info] HealthCheck: SSH to mysql-mha02 is reachable.Thu Jul 23 14:40:30 2015 - [info] Master MHA Node version is 0.56.Thu Jul 23 14:40:30 2015 - [info] Checking recovery script configurations on mysql-mha02(192.168.8.7:3306)..Thu Jul 23 14:40:30 2015 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql/logs --output_file=/var/log/masterha/node/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000013 Thu Jul 23 14:40:30 2015 - [info]   Connecting to root@192.168.8.7(mysql-mha02:22)..   Creating /var/log/masterha/node if not exists..    ok.  Checking output directory is accessible or not..   ok.  Binlog found at /var/lib/mysql/logs, up to mysql-bin.000013Thu Jul 23 14:40:30 2015 - [info] Binlog setting check done.Thu Jul 23 14:40:30 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..Thu Jul 23 14:40:30 2015 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=mysql-mha01 --slave_ip=192.168.8.6 --slave_port=3306 --workdir=/var/log/masterha/node --target_version=5.5.43-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxxThu Jul 23 14:40:30 2015 - [info]   Connecting to root@192.168.8.6(mysql-mha01:22)..   Checking slave recovery environment settings..    Opening /var/lib/mysql/relay-log.info ... ok.    Relay log found at /var/lib/mysql, up to mysql-mha01-relay-bin.000005    Temporary relay log file is /var/lib/mysql/mysql-mha01-relay-bin.000005    Testing mysql connection and privileges.. done.    Testing mysqlbinlog output.. done.    Cleaning up test file(s).. done.Thu Jul 23 14:40:30 2015 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=mysql-mha03 --slave_ip=192.168.8.8 --slave_port=3306 --workdir=/var/log/masterha/node --target_version=5.5.43-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxxThu Jul 23 14:40:30 2015 - [info]   Connecting to root@192.168.8.8(mysql-mha03:22)..   Checking slave recovery environment settings..    Opening /var/lib/mysql/relay-log.info ... ok.    Relay log found at /var/lib/mysql, up to mysql-mha03-relay-bin.000005    Temporary relay log file is /var/lib/mysql/mysql-mha03-relay-bin.000005    Testing mysql connection and privileges.. done.    Testing mysqlbinlog output.. done.    Cleaning up test file(s).. done.Thu Jul 23 14:40:30 2015 - [info] Slaves settings check done.Thu Jul 23 14:40:30 2015 - [info] mysql-mha02(192.168.8.7:3306) (current master) +--mysql-mha01(192.168.8.6:3306) +--mysql-mha03(192.168.8.8:3306)Thu Jul 23 14:40:30 2015 - [info] Checking replication health on mysql-mha01..Thu Jul 23 14:40:30 2015 - [info]  ok.Thu Jul 23 14:40:30 2015 - [info] Checking replication health on mysql-mha03..Thu Jul 23 14:40:30 2015 - [info]  ok.Thu Jul 23 14:40:30 2015 - [warning] master_ip_failover_script is not defined.Thu Jul 23 14:40:30 2015 - [warning] shutdown_script is not defined.Thu Jul 23 14:40:30 2015 - [info] Got exit code 0 (Not master dead).MySQL Replication Health is OK.

启动mha manager

[root@mysql-mha03 ~]# nohup masterha_manager --conf=/etc/MHA/MHAmanager.conf &[1] 5552

查看启动日志

Thu Jul 23 14:49:01 2015 - [info] MHA::MasterMonitor version 0.56.Thu Jul 23 14:49:01 2015 - [info] GTID failover mode = 0Thu Jul 23 14:49:01 2015 - [info] Dead Servers:Thu Jul 23 14:49:01 2015 - [info] Alive Servers:Thu Jul 23 14:49:01 2015 - [info]   mysql-mha01(192.168.8.6:3306)Thu Jul 23 14:49:01 2015 - [info]   mysql-mha02(192.168.8.7:3306)Thu Jul 23 14:49:01 2015 - [info]   mysql-mha03(192.168.8.8:3306)Thu Jul 23 14:49:01 2015 - [info] Alive Slaves:Thu Jul 23 14:49:01 2015 - [info]   mysql-mha01(192.168.8.6:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 14:49:01 2015 - [info]     Replicating from 192.168.8.7(192.168.8.7:3306)Thu Jul 23 14:49:01 2015 - [info]   mysql-mha03(192.168.8.8:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 14:49:01 2015 - [info]     Replicating from 192.168.8.7(192.168.8.7:3306)Thu Jul 23 14:49:01 2015 - [info] Current Alive Master: mysql-mha02(192.168.8.7:3306)Thu Jul 23 14:49:01 2015 - [info] Checking slave configurations..Thu Jul 23 14:49:01 2015 - [info]  read_only=1 is not set on slave mysql-mha01(192.168.8.6:3306).Thu Jul 23 14:49:01 2015 - [info]  read_only=1 is not set on slave mysql-mha03(192.168.8.8:3306).Thu Jul 23 14:49:01 2015 - [info] Checking replication filtering settings..Thu Jul 23 14:49:01 2015 - [info]  binlog_do_db= , binlog_ignore_db= Thu Jul 23 14:49:01 2015 - [info]  Replication filtering check ok.Thu Jul 23 14:49:01 2015 - [info] GTID (with auto-pos) is not supportedThu Jul 23 14:49:01 2015 - [info] Starting SSH connection tests..Thu Jul 23 14:49:02 2015 - [info] All SSH connection tests passed successfully.Thu Jul 23 14:49:02 2015 - [info] Checking MHA Node version..Thu Jul 23 14:49:03 2015 - [info]  Version check ok.Thu Jul 23 14:49:03 2015 - [info] Checking SSH publickey authentication settings on the current master..Thu Jul 23 14:49:03 2015 - [info] HealthCheck: SSH to mysql-mha02 is reachable.Thu Jul 23 14:49:03 2015 - [info] Master MHA Node version is 0.56.Thu Jul 23 14:49:03 2015 - [info] Checking recovery script configurations on mysql-mha02(192.168.8.7:3306)..Thu Jul 23 14:49:03 2015 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql/logs --output_file=/var/log/masterha/node/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000013 Thu Jul 23 14:49:03 2015 - [info]   Connecting to root@192.168.8.7(mysql-mha02:22)..   Creating /var/log/masterha/node if not exists..    ok.  Checking output directory is accessible or not..   ok.  Binlog found at /var/lib/mysql/logs, up to mysql-bin.000013Thu Jul 23 14:49:03 2015 - [info] Binlog setting check done.Thu Jul 23 14:49:03 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..Thu Jul 23 14:49:03 2015 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=mysql-mha01 --slave_ip=192.168.8.6 --slave_port=3306 --workdir=/var/log/masterha/node --target_version=5.5.43-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxxThu Jul 23 14:49:03 2015 - [info]   Connecting to root@192.168.8.6(mysql-mha01:22)..   Checking slave recovery environment settings..    Opening /var/lib/mysql/relay-log.info ... ok.    Relay log found at /var/lib/mysql, up to mysql-mha01-relay-bin.000005    Temporary relay log file is /var/lib/mysql/mysql-mha01-relay-bin.000005    Testing mysql connection and privileges.. done.    Testing mysqlbinlog output.. done.    Cleaning up test file(s).. done.Thu Jul 23 14:49:03 2015 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=mysql-mha03 --slave_ip=192.168.8.8 --slave_port=3306 --workdir=/var/log/masterha/node --target_version=5.5.43-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxxThu Jul 23 14:49:03 2015 - [info]   Connecting to root@192.168.8.8(mysql-mha03:22)..   Checking slave recovery environment settings..    Opening /var/lib/mysql/relay-log.info ... ok.    Relay log found at /var/lib/mysql, up to mysql-mha03-relay-bin.000005    Temporary relay log file is /var/lib/mysql/mysql-mha03-relay-bin.000005    Testing mysql connection and privileges.. done.    Testing mysqlbinlog output.. done.    Cleaning up test file(s).. done.Thu Jul 23 14:49:04 2015 - [info] Slaves settings check done.Thu Jul 23 14:49:04 2015 - [info] mysql-mha02(192.168.8.7:3306) (current master) +--mysql-mha01(192.168.8.6:3306) +--mysql-mha03(192.168.8.8:3306)Thu Jul 23 14:49:04 2015 - [warning] master_ip_failover_script is not defined.Thu Jul 23 14:49:04 2015 - [warning] shutdown_script is not defined.Thu Jul 23 14:49:04 2015 - [info] Set master ping interval 3 seconds.Thu Jul 23 14:49:04 2015 - [info] Set secondary check script: masterha_secondary_check -s mysql-mha03 --user=root --master_host=mysql-mha02 --master_ip=172.16.21.13 --master_port=3306Thu Jul 23 14:49:04 2015 - [info] Starting ping health check on mysql-mha02(192.168.8.7:3306)..Thu Jul 23 14:49:04 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

5. 验证MHA

5.1 故障切换

我们将master节点关闭

[root@mysql-mha02 ~]# service mysql stopShutting down MySQL...[  OK  ]

查看manager的输出日志

Thu Jul 23 14:59:13 2015 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)Thu Jul 23 14:59:13 2015 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql/logs --output_file=/var/log/masterha/node/save_binary_logs_test --manager_version=0.56 --binlog_prefix=mysql-binThu Jul 23 14:59:13 2015 - [info] Executing secondary network check script: masterha_secondary_check -s mysql-mha03 --user=root --master_host=mysql-mha02 --master_ip=172.16.21.13 --master_port=3306  --user=root  --master_host=mysql-mha02  --master_ip=192.168.8.7  --master_port=3306 --master_user=root --master_password=123456 --ping_type=SELECTThu Jul 23 14:59:14 2015 - [info] HealthCheck: SSH to mysql-mha02 is reachable.Monitoring server mysql-mha03 is reachable, Master is not reachable from mysql-mha03. OK.Thu Jul 23 14:59:14 2015 - [info] Master is not reachable from all other monitoring servers. Failover should start.Thu Jul 23 14:59:16 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)Thu Jul 23 14:59:16 2015 - [warning] Connection failed 2 time(s)..Thu Jul 23 14:59:19 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)Thu Jul 23 14:59:19 2015 - [warning] Connection failed 3 time(s)..Thu Jul 23 14:59:22 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)Thu Jul 23 14:59:22 2015 - [warning] Connection failed 4 time(s)..Thu Jul 23 14:59:22 2015 - [warning] Master is not reachable from health checker!Thu Jul 23 14:59:22 2015 - [warning] Master mysql-mha02(192.168.8.7:3306) is not reachable!Thu Jul 23 14:59:22 2015 - [warning] SSH is reachable.Thu Jul 23 14:59:22 2015 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/MHA/MHAmanager.conf again, and trying to connect to all servers to check server status..Thu Jul 23 14:59:22 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Thu Jul 23 14:59:22 2015 - [info] Reading application default configuration from /etc/MHA/MHAmanager.conf..Thu Jul 23 14:59:22 2015 - [info] Reading server configuration from /etc/MHA/MHAmanager.conf..Thu Jul 23 14:59:22 2015 - [info] GTID failover mode = 0Thu Jul 23 14:59:22 2015 - [info] Dead Servers:Thu Jul 23 14:59:22 2015 - [info]   mysql-mha02(192.168.8.7:3306)Thu Jul 23 14:59:22 2015 - [info] Alive Servers:Thu Jul 23 14:59:22 2015 - [info]   mysql-mha01(192.168.8.6:3306)Thu Jul 23 14:59:22 2015 - [info]   mysql-mha03(192.168.8.8:3306)Thu Jul 23 14:59:22 2015 - [info] Alive Slaves:Thu Jul 23 14:59:22 2015 - [info]   mysql-mha01(192.168.8.6:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 14:59:22 2015 - [info]     Replicating from 192.168.8.7(192.168.8.7:3306)Thu Jul 23 14:59:22 2015 - [info]   mysql-mha03(192.168.8.8:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 14:59:22 2015 - [info]     Replicating from 192.168.8.7(192.168.8.7:3306)Thu Jul 23 14:59:22 2015 - [info] Checking slave configurations..Thu Jul 23 14:59:22 2015 - [info]  read_only=1 is not set on slave mysql-mha01(192.168.8.6:3306).Thu Jul 23 14:59:22 2015 - [info]  read_only=1 is not set on slave mysql-mha03(192.168.8.8:3306).Thu Jul 23 14:59:22 2015 - [info] Checking replication filtering settings..Thu Jul 23 14:59:22 2015 - [info]  Replication filtering check ok.Thu Jul 23 14:59:22 2015 - [info] Master is down!Thu Jul 23 14:59:22 2015 - [info] Terminating monitoring script.Thu Jul 23 14:59:22 2015 - [info] Got exit code 20 (Master dead).Thu Jul 23 14:59:22 2015 - [info] MHA::MasterFailover version 0.56.Thu Jul 23 14:59:22 2015 - [info] Starting master failover.Thu Jul 23 14:59:22 2015 - [info] Thu Jul 23 14:59:22 2015 - [info] * Phase 1: Configuration Check Phase..Thu Jul 23 14:59:22 2015 - [info] Thu Jul 23 14:59:23 2015 - [info] GTID failover mode = 0Thu Jul 23 14:59:23 2015 - [info] Dead Servers:Thu Jul 23 14:59:23 2015 - [info]   mysql-mha02(192.168.8.7:3306)Thu Jul 23 14:59:23 2015 - [info] Checking master reachability via MySQL(double check)...Thu Jul 23 14:59:23 2015 - [info]  ok.Thu Jul 23 14:59:23 2015 - [info] Alive Servers:Thu Jul 23 14:59:23 2015 - [info]   mysql-mha01(192.168.8.6:3306)Thu Jul 23 14:59:23 2015 - [info]   mysql-mha03(192.168.8.8:3306)Thu Jul 23 14:59:23 2015 - [info] Alive Slaves:Thu Jul 23 14:59:23 2015 - [info]   mysql-mha01(192.168.8.6:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 14:59:23 2015 - [info]     Replicating from 192.168.8.7(192.168.8.7:3306)Thu Jul 23 14:59:23 2015 - [info]   mysql-mha03(192.168.8.8:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 14:59:23 2015 - [info]     Replicating from 192.168.8.7(192.168.8.7:3306)Thu Jul 23 14:59:23 2015 - [info] Starting Non-GTID based failover.Thu Jul 23 14:59:23 2015 - [info] Thu Jul 23 14:59:23 2015 - [info] ** Phase 1: Configuration Check Phase completed.Thu Jul 23 14:59:23 2015 - [info] Thu Jul 23 14:59:23 2015 - [info] * Phase 2: Dead Master Shutdown Phase..Thu Jul 23 14:59:23 2015 - [info] Thu Jul 23 14:59:23 2015 - [info] Forcing shutdown so that applications never connect to the current master..Thu Jul 23 14:59:23 2015 - [warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address.Thu Jul 23 14:59:23 2015 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.Thu Jul 23 14:59:23 2015 - [info] * Phase 2: Dead Master Shutdown Phase completed.Thu Jul 23 14:59:23 2015 - [info] Thu Jul 23 14:59:23 2015 - [info] * Phase 3: Master Recovery Phase..Thu Jul 23 14:59:23 2015 - [info] Thu Jul 23 14:59:23 2015 - [info] * Phase 3.1: Getting Latest Slaves Phase..Thu Jul 23 14:59:23 2015 - [info] Thu Jul 23 14:59:23 2015 - [info] The latest binary log file/position on all slaves is mysql-bin.000014:107Thu Jul 23 14:59:23 2015 - [info] Latest slaves (Slaves that received relay log files to the latest):Thu Jul 23 14:59:23 2015 - [info]   mysql-mha01(192.168.8.6:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 14:59:23 2015 - [info]     Replicating from 192.168.8.7(192.168.8.7:3306)Thu Jul 23 14:59:23 2015 - [info]   mysql-mha03(192.168.8.8:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 14:59:23 2015 - [info]     Replicating from 192.168.8.7(192.168.8.7:3306)Thu Jul 23 14:59:23 2015 - [info] The oldest binary log file/position on all slaves is mysql-bin.000014:107Thu Jul 23 14:59:23 2015 - [info] Oldest slaves:Thu Jul 23 14:59:23 2015 - [info]   mysql-mha01(192.168.8.6:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 14:59:23 2015 - [info]     Replicating from 192.168.8.7(192.168.8.7:3306)Thu Jul 23 14:59:23 2015 - [info]   mysql-mha03(192.168.8.8:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 14:59:23 2015 - [info]     Replicating from 192.168.8.7(192.168.8.7:3306)Thu Jul 23 14:59:23 2015 - [info] Thu Jul 23 14:59:23 2015 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..Thu Jul 23 14:59:23 2015 - [info] Thu Jul 23 14:59:23 2015 - [info] Fetching dead master's binary logs..Thu Jul 23 14:59:23 2015 - [info] Executing command on the dead master mysql-mha02(192.168.8.7:3306): save_binary_logs --command=save --start_file=mysql-bin.000014  --start_pos=107 --binlog_dir=/var/lib/mysql/logs --output_file=/var/log/masterha/node/saved_master_binlog_from_mysql-mha02_3306_20150723145922.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56  Creating /var/log/masterha/node if not exists..    ok. Concat binary/relay logs from mysql-bin.000014 pos 107 to mysql-bin.000014 EOF into /var/log/masterha/node/saved_master_binlog_from_mysql-mha02_3306_20150723145922.binlog ..  Dumping binlog format description event, from position 0 to 107.. ok.  Dumping effective binlog data from /var/lib/mysql/logs/mysql-bin.000014 position 107 to tail(126).. ok. Concat succeeded.Thu Jul 23 14:59:23 2015 - [info] scp from root@192.168.8.7:/var/log/masterha/node/saved_master_binlog_from_mysql-mha02_3306_20150723145922.binlog to local:/var/log/masterha/manager/saved_master_binlog_from_mysql-mha02_3306_20150723145922.binlog succeeded.Thu Jul 23 14:59:23 2015 - [info] HealthCheck: SSH to mysql-mha01 is reachable.Thu Jul 23 14:59:24 2015 - [info] HealthCheck: SSH to mysql-mha03 is reachable.Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info] * Phase 3.3: Determining New Master Phase..Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info] Finding the latest slave that has all relay logs for recovering other slaves..Thu Jul 23 14:59:24 2015 - [info] All slaves received relay logs to the same position. No need to resync each other.Thu Jul 23 14:59:24 2015 - [info] Searching new master from slaves..Thu Jul 23 14:59:24 2015 - [info]  Candidate masters from the configuration file:Thu Jul 23 14:59:24 2015 - [info]  Non-candidate masters:Thu Jul 23 14:59:24 2015 - [info] New master is mysql-mha01(192.168.8.6:3306)Thu Jul 23 14:59:24 2015 - [info] Starting master failover..Thu Jul 23 14:59:24 2015 - [info] From:mysql-mha02(192.168.8.7:3306) (current master) +--mysql-mha01(192.168.8.6:3306) +--mysql-mha03(192.168.8.8:3306)To:mysql-mha01(192.168.8.6:3306) (new master) +--mysql-mha03(192.168.8.8:3306)Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info] * Phase 3.3: New Master Diff Log Generation Phase..Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.Thu Jul 23 14:59:24 2015 - [info] Sending binlog..Thu Jul 23 14:59:24 2015 - [info] scp from local:/var/log/masterha/manager/saved_master_binlog_from_mysql-mha02_3306_20150723145922.binlog to root@mysql-mha01:/var/log/masterha/node/saved_master_binlog_from_mysql-mha02_3306_20150723145922.binlog succeeded.Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info] * Phase 3.4: Master Log Apply Phase..Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.Thu Jul 23 14:59:24 2015 - [info] Starting recovery on mysql-mha01(192.168.8.6:3306)..Thu Jul 23 14:59:24 2015 - [info]  Generating diffs succeeded.Thu Jul 23 14:59:24 2015 - [info] Waiting until all relay logs are applied.Thu Jul 23 14:59:24 2015 - [info]  done.Thu Jul 23 14:59:24 2015 - [info] Getting slave status..Thu Jul 23 14:59:24 2015 - [info] This slave(mysql-mha01)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000014:107). No need to recover from Exec_Master_Log_Pos.Thu Jul 23 14:59:24 2015 - [info] Connecting to the target slave host mysql-mha01, running recover script..Thu Jul 23 14:59:24 2015 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='root' --slave_host=mysql-mha01 --slave_ip=192.168.8.6  --slave_port=3306 --apply_files=/var/log/masterha/node/saved_master_binlog_from_mysql-mha02_3306_20150723145922.binlog --workdir=/var/log/masterha/node --target_version=5.5.43-log --timestamp=20150723145922 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxxThu Jul 23 14:59:24 2015 - [info] Applying differential binary/relay log files /var/log/masterha/node/saved_master_binlog_from_mysql-mha02_3306_20150723145922.binlog on mysql-mha01:3306. This may take long time...Applying log files succeeded.Thu Jul 23 14:59:24 2015 - [info]  All relay logs were successfully applied.Thu Jul 23 14:59:24 2015 - [info] Getting new master's binlog name and position..Thu Jul 23 14:59:24 2015 - [info]  mysql-bin.000014:107Thu Jul 23 14:59:24 2015 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='mysql-mha01 or 192.168.8.6', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000014', MASTER_LOG_POS=107, MASTER_USER='repl', MASTER_PASSWORD='xxx';Thu Jul 23 14:59:24 2015 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.Thu Jul 23 14:59:24 2015 - [info] ** Finished master recovery successfully.Thu Jul 23 14:59:24 2015 - [info] * Phase 3: Master Recovery Phase completed.Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info] * Phase 4: Slaves Recovery Phase..Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info] -- Slave diff file generation on host mysql-mha03(192.168.8.8:3306) started, pid: 6359. Check tmp log /var/log/masterha/manager/mysql-mha03_3306_20150723145922.log if it takes time..Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info] Log messages from mysql-mha03 ...Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.Thu Jul 23 14:59:24 2015 - [info] End of log messages from mysql-mha03.Thu Jul 23 14:59:24 2015 - [info] -- mysql-mha03(192.168.8.8:3306) has the latest relay log events.Thu Jul 23 14:59:24 2015 - [info] Generating relay diff files from the latest slave succeeded.Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..Thu Jul 23 14:59:24 2015 - [info] Thu Jul 23 14:59:24 2015 - [info] -- Slave recovery on host mysql-mha03(192.168.8.8:3306) started, pid: 6361. Check tmp log /var/log/masterha/manager/mysql-mha03_3306_20150723145922.log if it takes time..Thu Jul 23 14:59:25 2015 - [info] Thu Jul 23 14:59:25 2015 - [info] Log messages from mysql-mha03 ...Thu Jul 23 14:59:25 2015 - [info] Thu Jul 23 14:59:24 2015 - [info] Sending binlog..Thu Jul 23 14:59:24 2015 - [info] scp from local:/var/log/masterha/manager/saved_master_binlog_from_mysql-mha02_3306_20150723145922.binlog to root@mysql-mha03:/var/log/masterha/node/saved_master_binlog_from_mysql-mha02_3306_20150723145922.binlog succeeded.Thu Jul 23 14:59:24 2015 - [info] Starting recovery on mysql-mha03(192.168.8.8:3306)..Thu Jul 23 14:59:24 2015 - [info]  Generating diffs succeeded.Thu Jul 23 14:59:24 2015 - [info] Waiting until all relay logs are applied.Thu Jul 23 14:59:24 2015 - [info]  done.Thu Jul 23 14:59:24 2015 - [info] Getting slave status..Thu Jul 23 14:59:24 2015 - [info] This slave(mysql-mha03)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000014:107). No need to recover from Exec_Master_Log_Pos.Thu Jul 23 14:59:24 2015 - [info] Connecting to the target slave host mysql-mha03, running recover script..Thu Jul 23 14:59:24 2015 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='root' --slave_host=mysql-mha03 --slave_ip=192.168.8.8  --slave_port=3306 --apply_files=/var/log/masterha/node/saved_master_binlog_from_mysql-mha02_3306_20150723145922.binlog --workdir=/var/log/masterha/node --target_version=5.5.43-log --timestamp=20150723145922 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxxThu Jul 23 14:59:24 2015 - [info] Applying differential binary/relay log files /var/log/masterha/node/saved_master_binlog_from_mysql-mha02_3306_20150723145922.binlog on mysql-mha03:3306. This may take long time...Applying log files succeeded.Thu Jul 23 14:59:24 2015 - [info]  All relay logs were successfully applied.Thu Jul 23 14:59:24 2015 - [info]  Resetting slave mysql-mha03(192.168.8.8:3306) and starting replication from the new master mysql-mha01(192.168.8.6:3306)..Thu Jul 23 14:59:24 2015 - [info]  Executed CHANGE MASTER.Thu Jul 23 14:59:25 2015 - [info]  Slave started.Thu Jul 23 14:59:25 2015 - [info] End of log messages from mysql-mha03.Thu Jul 23 14:59:25 2015 - [info] -- Slave recovery on host mysql-mha03(192.168.8.8:3306) succeeded.Thu Jul 23 14:59:25 2015 - [info] All new slave servers recovered successfully.Thu Jul 23 14:59:25 2015 - [info] Thu Jul 23 14:59:25 2015 - [info] * Phase 5: New master cleanup phase..Thu Jul 23 14:59:25 2015 - [info] Thu Jul 23 14:59:25 2015 - [info] Resetting slave info on the new master..Thu Jul 23 14:59:25 2015 - [info]  mysql-mha01: Resetting slave info succeeded.Thu Jul 23 14:59:25 2015 - [info] Master failover to mysql-mha01(192.168.8.6:3306) completed successfully.Thu Jul 23 14:59:25 2015 - [info] ----- Failover Report -----MHAmanager: MySQL Master failover mysql-mha02(192.168.8.7:3306) to mysql-mha01(192.168.8.6:3306) succeededMaster mysql-mha02(192.168.8.7:3306) is down!Check MHA Manager logs at mysql-mha03:/var/log/masterha/manager.log for details.Started automated(non-interactive) failover.The latest slave mysql-mha01(192.168.8.6:3306) has all relay logs for recovery.Selected mysql-mha01(192.168.8.6:3306) as a new master.mysql-mha01(192.168.8.6:3306): OK: Applying all logs succeeded.mysql-mha03(192.168.8.8:3306): This host has the latest relay log events.Generating relay diff files from the latest slave succeeded.mysql-mha03(192.168.8.8:3306): OK: Applying all logs succeeded. Slave started, replicating from mysql-mha01(192.168.8.6:3306)mysql-mha01(192.168.8.6:3306): Resetting slave info succeeded.Master failover to mysql-mha01(192.168.8.6:3306) completed successfully.

发现MHA已经成功将主节点配置到mysql-mha01上。切换后manager节点自动关闭。

[root@mysql-mha03 ~]# ps -ef|grep -i masterroot       6414   3295  0 15:00 pts/1    00:00:00 grep -i master

登录到mysql-mha03节点查看

mysql> show slave status\G*************************** 1. row ***************************               Slave_IO_State: Waiting for master to send event                  Master_Host: 192.168.8.6                  Master_User: repl                  Master_Port: 3306                Connect_Retry: 20              Master_Log_File: mysql-bin.000014          Read_Master_Log_Pos: 107               Relay_Log_File: mysql-mha03-relay-bin.000002                Relay_Log_Pos: 253        Relay_Master_Log_File: mysql-bin.000014             Slave_IO_Running: Yes            Slave_SQL_Running: Yes              Replicate_Do_DB:           Replicate_Ignore_DB:            Replicate_Do_Table:        Replicate_Ignore_Table:       Replicate_Wild_Do_Table:   Replicate_Wild_Ignore_Table:                    Last_Errno: 0                   Last_Error:                  Skip_Counter: 0          Exec_Master_Log_Pos: 107              Relay_Log_Space: 415              Until_Condition: None               Until_Log_File:                 Until_Log_Pos: 0           Master_SSL_Allowed: No           Master_SSL_CA_File:            Master_SSL_CA_Path:               Master_SSL_Cert:             Master_SSL_Cipher:                Master_SSL_Key:         Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No                Last_IO_Errno: 0                Last_IO_Error:                Last_SQL_Errno: 0               Last_SQL_Error:   Replicate_Ignore_Server_Ids:              Master_Server_Id: 11 row in set (0.00 sec)

host已经改变成功。MHA配置成功。

5.2 主动切换

mha可以实现master的主动切换，可以让我们方便的维护集群环境。如我们可以将master切换到别的节点，维护原master节点。

需要将启动的manager节点关闭掉，否则会报错。

[root@mysql-mha03 ~]# masterha_stop --conf=/etc/MHA/MHAmanager.conf Stopped MHAmanager successfully.[1]+  Exit 1                  nohup masterha_manager --conf=/etc/MHA/MHAmanager.conf

我们主动切换master节点，原master节点成为slave节点

[root@mysql-mha03 ~]# masterha_master_switch --master_state=alive --conf=/etc/MHA/MHAmanager.conf

查看输出

Thu Jul 23 15:29:38 2015 - [info] MHA::MasterRotate version 0.56.Thu Jul 23 15:29:38 2015 - [info] Starting online master switch..Thu Jul 23 15:29:38 2015 - [info] Thu Jul 23 15:29:38 2015 - [info] * Phase 1: Configuration Check Phase..Thu Jul 23 15:29:38 2015 - [info] Thu Jul 23 15:29:38 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Thu Jul 23 15:29:38 2015 - [info] Reading application default configuration from /etc/MHA/MHAmanager.conf..Thu Jul 23 15:29:38 2015 - [info] Reading server configuration from /etc/MHA/MHAmanager.conf..Thu Jul 23 15:29:39 2015 - [info] GTID failover mode = 0Thu Jul 23 15:29:39 2015 - [info] Current Alive Master: mysql-mha01(192.168.8.6:3306)Thu Jul 23 15:29:39 2015 - [info] Alive Slaves:Thu Jul 23 15:29:39 2015 - [info]   mysql-mha02(192.168.8.7:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 15:29:39 2015 - [info]     Replicating from 192.168.8.6(192.168.8.6:3306)Thu Jul 23 15:29:39 2015 - [info]   mysql-mha03(192.168.8.8:3306)  Version=5.5.43-log (oldest major version between slaves) log-bin:enabledThu Jul 23 15:29:39 2015 - [info]     Replicating from 192.168.8.6(192.168.8.6:3306)It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on mysql-mha01(192.168.8.6:3306)? (YES/no): yesThu Jul 23 15:29:41 2015 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..Thu Jul 23 15:29:41 2015 - [info]  ok.Thu Jul 23 15:29:41 2015 - [info] Checking MHA is not monitoring or doing failover..Thu Jul 23 15:29:41 2015 - [info] Checking replication health on mysql-mha02..Thu Jul 23 15:29:41 2015 - [info]  ok.Thu Jul 23 15:29:41 2015 - [info] Checking replication health on mysql-mha03..Thu Jul 23 15:29:41 2015 - [info]  ok.Thu Jul 23 15:29:41 2015 - [info] Searching new master from slaves..Thu Jul 23 15:29:41 2015 - [info]  Candidate masters from the configuration file:Thu Jul 23 15:29:41 2015 - [info]  Non-candidate masters:Thu Jul 23 15:29:41 2015 - [info] From:mysql-mha01(192.168.8.6:3306) (current master) +--mysql-mha02(192.168.8.7:3306) +--mysql-mha03(192.168.8.8:3306)To:mysql-mha02(192.168.8.7:3306) (new master) +--mysql-mha03(192.168.8.8:3306)Starting master switch from mysql-mha01(192.168.8.6:3306) to mysql-mha02(192.168.8.7:3306)? (yes/NO): yesThu Jul 23 15:29:48 2015 - [info] Checking whether mysql-mha02(192.168.8.7:3306) is ok for the new master..Thu Jul 23 15:29:48 2015 - [info]  ok.Thu Jul 23 15:29:48 2015 - [info] ** Phase 1: Configuration Check Phase completed.Thu Jul 23 15:29:48 2015 - [info] Thu Jul 23 15:29:48 2015 - [info] * Phase 2: Rejecting updates Phase..Thu Jul 23 15:29:48 2015 - [info] master_ip_online_change_script is not defined. If you do not disable writes on the current master manually, applications keep writing on the current master. Is it ok to proceed? (yes/NO): yesThu Jul 23 15:29:59 2015 - [info] Locking all tables on the orig master to reject updates from everybody (including root):Thu Jul 23 15:29:59 2015 - [info] Executing FLUSH TABLES WITH READ LOCK..Thu Jul 23 15:29:59 2015 - [info]  ok.Thu Jul 23 15:29:59 2015 - [info] Orig master binlog:pos is mysql-bin.000014:107.Thu Jul 23 15:29:59 2015 - [info]  Waiting to execute all relay logs on mysql-mha02(192.168.8.7:3306)..Thu Jul 23 15:29:59 2015 - [info]  master_pos_wait(mysql-bin.000014:107) completed on mysql-mha02(192.168.8.7:3306). Executed 0 events.Thu Jul 23 15:29:59 2015 - [info]   done.Thu Jul 23 15:29:59 2015 - [info] Getting new master's binlog name and position..Thu Jul 23 15:29:59 2015 - [info]  mysql-bin.000015:107Thu Jul 23 15:29:59 2015 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='mysql-mha02 or 192.168.8.7', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000015', MASTER_LOG_POS=107, MASTER_USER='repl', MASTER_PASSWORD='xxx';Thu Jul 23 15:29:59 2015 - [info] Thu Jul 23 15:29:59 2015 - [info] * Switching slaves in parallel..Thu Jul 23 15:29:59 2015 - [info] Thu Jul 23 15:29:59 2015 - [info] -- Slave switch on host mysql-mha03(192.168.8.8:3306) started, pid: 6819Thu Jul 23 15:29:59 2015 - [info] Thu Jul 23 15:29:59 2015 - [info] Log messages from mysql-mha03 ...Thu Jul 23 15:29:59 2015 - [info] Thu Jul 23 15:29:59 2015 - [info]  Waiting to execute all relay logs on mysql-mha03(192.168.8.8:3306)..Thu Jul 23 15:29:59 2015 - [info]  master_pos_wait(mysql-bin.000014:107) completed on mysql-mha03(192.168.8.8:3306). Executed 0 events.Thu Jul 23 15:29:59 2015 - [info]   done.Thu Jul 23 15:29:59 2015 - [info]  Resetting slave mysql-mha03(192.168.8.8:3306) and starting replication from the new master mysql-mha02(192.168.8.7:3306)..Thu Jul 23 15:29:59 2015 - [info]  Executed CHANGE MASTER.Thu Jul 23 15:29:59 2015 - [info]  Slave started.Thu Jul 23 15:29:59 2015 - [info] End of log messages from mysql-mha03 ...Thu Jul 23 15:29:59 2015 - [info] Thu Jul 23 15:29:59 2015 - [info] -- Slave switch on host mysql-mha03(192.168.8.8:3306) succeeded.Thu Jul 23 15:29:59 2015 - [info] Unlocking all tables on the orig master:Thu Jul 23 15:29:59 2015 - [info] Executing UNLOCK TABLES..Thu Jul 23 15:29:59 2015 - [info]  ok.Thu Jul 23 15:29:59 2015 - [info] All new slave servers switched successfully.Thu Jul 23 15:29:59 2015 - [info] Thu Jul 23 15:29:59 2015 - [info] * Phase 5: New master cleanup phase..Thu Jul 23 15:29:59 2015 - [info] Thu Jul 23 15:29:59 2015 - [info]  mysql-mha02: Resetting slave info succeeded.Thu Jul 23 15:29:59 2015 - [info] Switching master to mysql-mha02(192.168.8.7:3306) completed successfully.

查看mysql-mysql03节点，已经成功切换到mysql-mha02为master

mysql> show slave status\G*************************** 1. row ***************************               Slave_IO_State: Waiting for master to send event                  Master_Host: 192.168.8.7                  Master_User: repl                  Master_Port: 3306                Connect_Retry: 20              Master_Log_File: mysql-bin.000015          Read_Master_Log_Pos: 107               Relay_Log_File: mysql-mha03-relay-bin.000002                Relay_Log_Pos: 253        Relay_Master_Log_File: mysql-bin.000015             Slave_IO_Running: Yes            Slave_SQL_Running: Yes              Replicate_Do_DB:           Replicate_Ignore_DB:            Replicate_Do_Table:        Replicate_Ignore_Table: