Mysql半同步复制+MHA+Keepalived部署
来源：互联网 发布：网络防御 动态防御 编辑：程序博客网 时间：2024/04/29 10:04
目前公司mysql服务器只是部署了简单的mysql主从，并没有自动故障转移，故此部署Mysql半同步复制+MHA+Keepalived进行测试，理论就不过多作说明了，网上资料很多，此次部署也有参考网上资料。
目前MHA主流架构是一主多从，此次部署采用一主二从（一台充当master，一台充当备用master，一台充当从库）
拓扑图：
环境：
4台centos6.8 私有云主机，VIP为192.168.1.251
主机名 IP 职能 部署软件vm01 192.168.1.211 MHA-Manager MHA-Manager,MHA-nodevm02 192.168.1.212 mysql-master mysql5.6.35,MHA-nodevm03 192.168.1.221 mysql-master-backup mysql5.6.35,MHA-node,keepalivedvm04 192.168.1.229 mysql-slave mysql5.6.35,MHA-node,keepalived所有主机/etc/hosts文件添加
192.168.1.211 vm01192.168.1.212 vm02192.168.1.221 vm03192.168.1.229 vm04所有服务器防火墙和SELINUX关闭
一、服务器mysql环境部署
1、mysql源码安装（本次部署使用mysql-5.6.35）
安装编译源码所需要工具及库
[root@vm02 ~]# cd /usr/local/src/[root@vm02 src]# yum install gcc gcc-c++ ncurses-devel perl -y下载并安装cmake(mysql5.5以后源码安装需要使用cmake)
[root@vm02 src]# wget https://cmake.org/files/v3.8/cmake-3.8.1.tar.gz[root@vm02 src]# tar zxf cmake-3.8.1.tar.gz [root@vm02 src]# cd cmake-3.8.1[root@vm02 cmake-3.8.1]# ./configure [root@vm02 cmake-3.8.1]# make -j `cat /proc/cpuinfo |grep processor|wc -l` && make install创建mysql用户及组
[root@vm03 src]# groupadd mysql[root@vm03 src]# useradd -g mysql -s /bin/false -M mysql创建mysql安装目录及数据库存放目录
[root@vm02 src]# mkdir /data/mysql[root@vm02 src]# mkdir /data/mysql/data下载mysql源码(如果嫌麻烦的话也可用二进制包方式安装，便于维护)
[root@vm02 src]# wget https://cdn.mysql.com/archives/mysql-5.6/mysql-5.6.35.tar.gz[root@vm02 src]# cd mysql-5.6.35[root@vm02 mysql-5.6.35]# cmake ./-DCMAKE_INSTALL_PREFIX=/data/mysql \-DMYSQL_DATADIR=/data/mysql/data \-DMYSQL_UNIX_ADDR=/data/mysql/mysql.sock \-DDEFAULT_CHARSET=utf8 \-DDEFAULT_COLLATION=utf8_general_ci \-DWITH_INNOBASE_STORAGE_ENGINE=1 \-DWITH_ARCHIVE_STORAGE_ENGINE=1 \-DWITH_BLACKHOLE_STORAGE_ENGINE=1 \-DMYSQL_TCP_PORT=3306 \-DEXTRA_CHARSETS=all \-DENABLED_LOCAL_INFILE=1在此处，我碰到一个很怪异的情况，cmake生成的CMackCache.txt文件中mysql及数据库目录的安装目录仍是默认的/usr/local/mysql，而并非我指定的目录，手动修改掉
[root@vm02 mysql-5.6.35]# sed -i 's#/usr/local/mysql#/data/mysql#' CMakeCache.txt [root@vm02 mysql-5.6.35]# make -j `cat /proc/cpuinfo|grep processor|wc -l`[root@vm02 mysql-5.6.35]# make install修改mysql目录属主及组
[root@vm02 mysql-5.6.35]# chown -R mysql.mysql /data/mysql/初始化mysql数据库
[root@vm02 mysql-5.6.35]# cd /data/mysql/[root@vm02 mysql]# scripts/mysql_install_db --user=mysql --datadir=/data/mysql/data/将mysql bin及lib目录加入PATH环境变量
[root@vm02 mysql]# echo "export PATH=/data/mysql/bin:/data/mysql/lib:$PATH" >>/etc/profile[root@vm02 mysql]# . /etc/profile复制mysql启动配置文件
[root@vm02 mysql]# cp support-files/my-default.cnf /etc/my.cnf修改为：
[client]port = 3306socket = /data/mysql/mysql.sock[mysql]no-auto-rehash[mysqld]user=mysqlcharacter_set_server=utf8mb4basedir = /data/mysqldatadir = /data/mysql/dataport = 3306# 此次测试部署中 server_id取ip的最后一段，server_id可以随意取，只要互不相同server_id = 212 socket = /data/mysql/mysql.socklower_case_table_names=1#mysql回收空闲连接的时间(7天)wait_timeout=604800symbolic-links=0#二进制日志相关参数log-bin=/data/mysql/data/mysql-binexpire_logs_days=7binlog_cache_size = 1Mbinlog_format=mixedbinlog_checksum=none#slave相关参数skip-slave-startrelay_log_purge=0relay_log=mysql-relay-binrelay_log_index=mysql-relay-bin.indexback_log = 100max_connect_errors = 2000table_open_cache = 1024skip-external-lockingskip-name-resolvekey_buffer_size = 128Mbulk_insert_buffer_size = 32Mmax_allowed_packet = 100Msort_buffer_size = 16Mread_buffer_size = 16Mjoin_buffer_size = 16Mread_rnd_buffer_size = 16Mmyisam_sort_buffer_size = 128Mthread_cache_size = 8query_cache_size = 256Mthread_stack = 192Kmax_connections = 500tmp_table_size= 256Mmax_heap_table_size= 128Mexplicit_defaults_for_timestamp=trueslow_query_loglog_output='Table,File'slow_query_log_file=/data/mysql/slow.loglong_query_time = 2innodb_file_format = Barracudainnodb_file_per_table = 1innodb_large_prefix = oninnodb_data_file_path = ibdata1:10M:autoextendinnodb_log_file_size = 256Minnodb_log_files_in_group = 3innodb_buffer_pool_size = 2Ginnodb_log_buffer_size = 4Minnodb_flush_log_at_trx_commit = 2innodb_file_per_tableinnodb_lock_wait_timeout = 120innodb_thread_concurrency = 8innodb_autoextend_increment = 128M[mysqldump]quickmax_allowed_packet = 100M[mysqld_safe]open-files-limit = 8192log_error =/data/mysql/mysqld.logpid-file=/data/mysql/mysqld.pid[mysqlhotcopy]interactive-timeout复制mysql启动脚本文件并创建为服务
[root@vm02 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld[root@vm02 mysql]# chkconfig --add mysqld手动创建mysql日志文件，否则启动mysql时报错
[root@vm02 mysql]# touch /data/mysql/mysqld.log[root@vm02 mysql]# chown mysql.mysql /data/mysql/mysqld.log启动mysql
[root@vm02 mysql]# service mysqld restart设置mysql root账号初始密码为123456
[root@vm02 mysql]# mysqladmin -uroot password 123456mysql命令行中删除匿名账户
[root@vm02 mysql]# mysql -uroot -pmysql> delete from mysql.user where user="";mysql> update mysql.user set password=password("123456") where user='root'; mysql> flush privileges;vm03、vm04服务器上mysql安装配置方法完全一样，仅/etc/my.cnf中server_id的值不一样，各自取服务器ip最后一段作为自己的server_id
二、配置主从半同步复制
在mysql master(vm02)上创建主从复制账号
[root@vm02 ~]# mysql -uroot -pmysql> grant replication slave on *.* to 'repl'@'192.168.1.%' identified by '654321';备份mysql master上数据库(或者直接物理备份到slave上也行）
其中–master-data=2代表备份时刻记录master的Binlog位置和Position，–single-transaction意思是获取一致性快照，-R意思是备份存储过程和函数，–triggres的意思是备份触发器，-A代表备份所有的库。
[root@vm02 ~]# mysqldump -uroot -p --master-data=2 --single-transaction --default-character-set=utf8mb4 -R --triggers -A >master_data.sql将备份的数据传送到vm03,vm04服务器
[root@vm02 ~]# hosts="vm03 vm04"[root@vm02 ~]# for host in ${hosts[*]};do rsync -avzP master_data.sql $host:/usr/local/src;done;在vm03,vm04上恢复数据
[root@vm03 src]# mysql -uroot -p123456 <master_data.sql[root@vm04 src]# mysql -uroot -p123456 <master_data.sql 在vm03(mysql-master-backup)上配置主从同步，此master备机上也需开启bin-log
[root@vm03 src]# mysql -uroot -p123456mysql> change master to master_host='192.168.1.212',master_port=3306,master_user='repl',master_password='654321',master_log_file='mysql-bin.000006',master_log_pos=324;Query OK, 0 rows affected, 2 warnings (0.04 sec)mysql> start slave;Query OK, 0 rows affected (0.00 sec)mysql> show slave status\G;*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.1.212 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000006 Read_Master_Log_Pos: 324 Relay_Log_File: mysql-relay-bin.000002 Relay_Log_Pos: 279 Relay_Master_Log_File: mysql-bin.000006 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 324 Relay_Log_Space: 448 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 212 Master_UUID: 55cfc4e2-3567-11e7-b46d-123456789012 Master_Info_File: /data/mysql/data/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 01 row in set (0.00 sec)vm04(slave)操作与vm02操作相同，由于这台只做从库，只需要修改server id即可,不需要开启log-bin。
MySQL复制默认是异步复制，Master将事件写入binlog，但并不知道Slave是否或何时已经接收且已处理。在异步复制的机制的情况下，如果Master宕机，事务在Master上已提交，但很可能这些事务没有传到任何的Slave上。假设有Master->Salve故障转移的机制，此时Slave也可能会丢失事务。
半同步复制的功能要在Master，Slave都开启，半同步复制才会起作用；否则，只开启一边，它依然为异步复制。
mysql5.5及更高版本才有半同步复制功能。在MySQL上安装插件需要数据库支持动态载入。检查是否支持，用如下检测：
mysql> show global variables like 'have_dynamic_loading';+----------------------+-------+| Variable_name | Value |+----------------------+-------+| have_dynamic_loading | YES |+----------------------+-------+1 row in set (0.01 sec)半同步复制是基于复制的环境。也就是说配置半同步复制前，已有复制的环境。
在所有mysql节点(主，从)安装半同步插件
mysql> INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';Query OK, 0 rows affected (0.04 sec)mysql> INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';Query OK, 0 rows affected (0.02 sec)如果不清楚Plugin的目录，用如下查找：
mysql> show global variables like 'plugin_dir';+---------------+-------------------------+| Variable_name | Value |+---------------+-------------------------+| plugin_dir | /data/mysql/lib/plugin/ |+---------------+-------------------------+[root@vm02 ~]# ls /data/mysql/lib/plugin/adt_null.so connection_control.so mypluglib.so qa_auth_server.so validate_password.soauth.so daemon_example.ini mysql_no_login.so semisync_master.soauth_socket.so debug qa_auth_client.so semisync_slave.soauth_test_plugin.so libdaemon_example.so qa_auth_interface.so test_udf_services.so检查Plugin是否已正确安装
mysql> show plugins; +----------------------------+----------+--------------------+--------------------+---------+| Name | Status | Type | Library | License |+----------------------------+----------+--------------------+--------------------+---------+| binlog | ACTIVE | STORAGE ENGINE | NULL | GPL || mysql_native_password | ACTIVE | AUTHENTICATION | NULL | GPL || mysql_old_password | ACTIVE | AUTHENTICATION | NULL | GPL || sha256_password | ACTIVE | AUTHENTICATION | NULL | GPL || CSV | ACTIVE | STORAGE ENGINE | NULL | GPL || MEMORY | ACTIVE | STORAGE ENGINE | NULL | GPL || MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL || MRG_MYISAM | ACTIVE | STORAGE ENGINE | NULL | GPL || ARCHIVE | ACTIVE | STORAGE ENGINE | NULL | GPL || BLACKHOLE | ACTIVE | STORAGE ENGINE | NULL | GPL || FEDERATED | DISABLED | STORAGE ENGINE | NULL | GPL || InnoDB | ACTIVE | STORAGE ENGINE | NULL | GPL || INNODB_TRX | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_LOCKS | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_LOCK_WAITS | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_CMP | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_CMP_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_CMPMEM | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_CMPMEM_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_CMP_PER_INDEX | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_CMP_PER_INDEX_RESET | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_BUFFER_PAGE | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_BUFFER_PAGE_LRU | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_BUFFER_POOL_STATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_METRICS | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_FT_DEFAULT_STOPWORD | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_FT_DELETED | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_FT_BEING_DELETED | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_FT_CONFIG | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_FT_INDEX_CACHE | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_FT_INDEX_TABLE | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_SYS_TABLES | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_SYS_TABLESTATS | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_SYS_INDEXES | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_SYS_COLUMNS | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_SYS_FIELDS | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_SYS_FOREIGN | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_SYS_FOREIGN_COLS | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_SYS_TABLESPACES | ACTIVE | INFORMATION SCHEMA | NULL | GPL || INNODB_SYS_DATAFILES | ACTIVE | INFORMATION SCHEMA | NULL | GPL || PERFORMANCE_SCHEMA | ACTIVE | STORAGE ENGINE | NULL | GPL || partition | ACTIVE | STORAGE ENGINE | NULL | GPL || rpl_semi_sync_master | 1.0 | ACTIVE | REPLICATION | 2.0 | semisync_master.so | 1.4 | He Zhenxing | Semi-synchronous replication master | GPL | ON || rpl_semi_sync_slave | 1.0 | ACTIVE | REPLICATION | 2.0 | semisync_slave.so | 1.4 | He Zhenxing | Semi-synchronous replication slave | GPL | ON |+----------------------------+----------+--------------------+--------------------+---------+或者
mysql> select * from information_schema.plugins where PLUGIN_NAME like "rpl_semi_sync%";+----------------------+----------------+---------------+-------------+---------------------+--------------------+------------------------+---------------+-------------------------------------+----------------+-------------+| PLUGIN_NAME | PLUGIN_VERSION | PLUGIN_STATUS | PLUGIN_TYPE | PLUGIN_TYPE_VERSION | PLUGIN_LIBRARY | PLUGIN_LIBRARY_VERSION | PLUGIN_AUTHOR | PLUGIN_DESCRIPTION | PLUGIN_LICENSE | LOAD_OPTION |+----------------------+----------------+---------------+-------------+---------------------+--------------------+------------------------+---------------+-------------------------------------+----------------+-------------+| rpl_semi_sync_master | 1.0 | ACTIVE | REPLICATION | 2.0 | semisync_master.so | 1.4 | He Zhenxing | Semi-synchronous replication master | GPL | ON || rpl_semi_sync_slave | 1.0 | ACTIVE | REPLICATION | 2.0 | semisync_slave.so | 1.4 | He Zhenxing | Semi-synchronous replication slave | GPL | ON |+----------------------+----------------+---------------+-------------+---------------------+--------------------+------------------------+---------------+-------------------------------------+----------------+-------------+2 rows in set (0.00 sec)在Master上执行：
mysql> SET GLOBAL rpl_semi_sync_master_enabled = 1;在所有Slave上执行(包括master备机)： SET GLOBAL rpl_semi_sync_slave_enabled = 1;
mysql> show variables like "rpl_semi_sync_slave_enabled";+-----------------------------+-------+| Variable_name | Value |+-----------------------------+-------+| rpl_semi_sync_slave_enabled | OFF |+-----------------------------+-------+1 row in set (0.00 sec)mysql> SET GLOBAL rpl_semi_sync_slave_enabled = 1;Query OK, 0 rows affected (0.00 sec)mysql> show variables like "rpl_semi_sync_slave_enabled";+-----------------------------+-------+| Variable_name | Value |+-----------------------------+-------+| rpl_semi_sync_slave_enabled | ON |+-----------------------------+-------+1 row in set (0.00 sec)如果在一个正在运行的Slave上开启半同步复制的功能，必须先停止Slave I/O,将其启用半同步后，再开启Slave I/O.
mysql> STOP SLAVE IO_THREAD; START SLAVE IO_THREAD;如果不这样做，Slave还是会以异步的方式进行复制。
正如大家所知，如果不将变量的设置写到配置文件，下次重启数据库，将失效。写入配置文件：
Master上：
[mysqld]rpl_semi_sync_master_enabled=1rpl_semi_sync_master_timeout=10000 # 10秒（默认）所有Slave上：
[mysqld]rpl_semi_sync_slave_enabled=1三、配置ssh免密码登录
配置MHA-Manager(vm01)到所有MHA-node(vm02,vm03,vm04)
[root@vm01 ~]# ssh-keygen -t rsa #一直回车[root@vm01 ~]# hosts="vm02 vm03 vm04"[root@vm01 ~]# for host in ${hosts[*]};do ssh-copy-id -i /root/.ssh/id_rsa.pub $host;done;配置mysql master(vm02)到所有MHA-node(vm01,vm03,vm04)
[root@vm02 ~]# ssh-keygen -t rsa #一直回车[root@vm02 ~]# hosts="vm01 vm03 vm04"[root@vm02 ~]# for host in ${hosts[*]};do ssh-copy-id -i /root/.ssh/id_rsa.pub $host;done;配置mysql slave(vm03,vm04)到所有MHA-node
[root@vm03 ~]# ssh-keygen -t rsa #一直回车[root@vm03 ~]# hosts="vm01 vm02 vm04" [root@vm03 ~]# for host in ${hosts[*]};do ssh-copy-id -i /root/.ssh/id_rsa.pub $host;done;[root@vm04 ~]# ssh-keygen -t rsa #一直回车[root@vm04 ~]# hosts="vm01 vm02 vm03" [root@vm04 ~]# for host in ${hosts[*]};do ssh-copy-id -i /root/.ssh/id_rsa.pub $host;done;四、在mysql master数据库中创建mha管理用户
mysql> grant all privileges on *.* to 'mha'@'192.168.1.%' identified by 'mha@password';Query OK, 0 rows affected (0.01 sec)mysql> flush privileges;Query OK, 0 rows affected (0.00 sec)与此同时，在从库上检察是否同步
mysql> select host,user,password from mysql.user; +------------------+------+-------------------------------------------+| host | user | password |+------------------+------+-------------------------------------------+| localhost | root | *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 || vm02.localdomain | root | *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 || 127.0.0.1 | root | *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 || ::1 | root | *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 || 192.168.1.% | repl | *2A032F7C5BA932872F0F045E0CF6B53CF702F2C5 || 192.168.1.% | mha | *792262F3BDF0EAA8F94DA7C5FA717730EF732632 |+------------------+------+-------------------------------------------+五、安装MHA
MHA特点：
MHA监控复制架构的主服务器，一旦检测到主服务器故障，就会自动进行故障转移。即使有些从服务器没有收到最新的relay log，MHA自动从最新的从服务器上识别差异的relay log并把这些日志应用到其他从服务器上，因此所有的从服务器保持一致性了。MHA通常在几秒内完成故障转移，9-12秒可以检测出主服务器故障，7-10秒内关闭故障的主服务器以避免脑裂，几秒中内应用差异的relay log到新的主服务器上，整个过程可以在10-30s内完成。还可以设置优先级指定其中的一台slave作为master的候选人。由于MHA在slaves之间修复一致性，因此可以将任何slave变成新的master，而不会发生一致性的问题，从而导致复制失败。
mha下载及文档地址: https://code.google.com/p/mysql-master-ha/
国内访问谷歌需要翻墙(如果没有vpn账号的话，可以到http://vpn.disa8.com下载一个免费的vpn)
所有服务器配置epel的yum源，安装相关依赖包
rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpmrpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6sed -i 's/^#baseurl/baseurl/g' /etc/yum.repos.d/epel.reposed -i 's/^mirrorlist/#mirrorlist/g' /etc/yum.repos.d/epel.reposed -i 's/^#baseurl/baseurl/g' /etc/yum.repos.d/epel-testing.reposed -i 's/^mirrorlist/#mirrorlist/g' /etc/yum.repos.d/epel-testing.repo在所有运行MySQL服务的服务器上安装运行MHA-Node，无论是master还是slave。由于MHA-Manager需要MHA-Node，因此在运行MHA-Manager的服务器上也需要安装MHA-Node
安装mha-manager依赖
[root@vm01 src]#yum install perl-Config perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes cpan安装mha-manager(可以使用rpm包安装，更方便)
[root@vm01 src]# tar zxf mha4mysql-manager-0.55.tar.gz[root@vm01 src]# cd mha4mysql-manager-0.55# 检查mha manager所需依赖包是否安装[root@vm01 mha4mysql-manager-0.55]# perl Makefile.PL *** Module::AutoInstall version 1.03*** Checking for Perl dependencies...[Core Features]- DBI ...loaded. (1.609)- DBD::mysql ...loaded. (4.013)- Time::HiRes ...loaded. (1.9721)- Config::Tiny ...missing.- Log::Dispatch ...loaded. (2.26)- Parallel::ForkManager ...loaded. (0.7.9)- MHA::NodeConst ...missing.==> Auto-install the 2 mandatory module(s) from CPAN? [y] y*** Dependencies will be installed the next time you type 'make'.*** Module::AutoInstall configuration finished.Checking if your kit is complete...Looks goodWarning: prerequisite Config::Tiny 0 not found.Warning: prerequisite MHA::NodeConst 0 not found.Writing Makefile for mha4mysql::manager# 安装mha-manager[root@vm01 mha4mysql-manager-0.55]# make[root@vm01 mha4mysql-manager-0.55]# make install安装完成后，会在/usr/local/bin目录下生成以下脚本文件
[root@vm01 ~]# ls /usr/local/bin/masterha_*/usr/local/bin/masterha_check_repl /usr/local/bin/masterha_check_status /usr/local/bin/masterha_manager /usr/local/bin/masterha_master_switch /usr/local/bin/masterha_stop/usr/local/bin/masterha_check_ssh /usr/local/bin/masterha_conf_host /usr/local/bin/masterha_master_monitor /usr/local/bin/masterha_secondary_check此外，mha-manager源码包解压出来，会有一些脚本模板可供参考(因为这些脚本不完整，需要自己修改，这是软件开发着留给我们自己发挥的，如果开启下面的任何一个脚本对应的参数，而对应这里的脚本又没有修改，则会抛错，可能会被坑的很惨)
1、master_ip_failover是自动切换时vip管理的脚本，不是必须，如果我们使用keepalived的，我们可以自己编写脚本完成对vip的管理，比如监控mysql，如果mysql异常，我们停止keepalived就行，这样vip就会自动漂移
[root@vm01 scripts]# ls -l /usr/local/src/mha4mysql-manager-0.55/samples/scripts/master_ip_failover -rwxr-xr-x 1 root root 3648 Dec 13 2012 /usr/local/src/mha4mysql-manager-0.55/samples/scripts/master_ip_failover2、master_ip_online_change是在线切换时vip的管理，不是必须，同样可以可以自行编写简单的shell完成
[root@vm01 scripts]# ls -l /usr/local/src/mha4mysql-manager-0.55/samples/scripts/master_ip_online_change-rwxr-xr-x 1 root root 9568 Dec 13 2012 /usr/local/src/mha4mysql-manager-0.55/samples/scripts/master_ip_online_change3、power_manager故障发生后关闭主机的脚本，不是必须
[root@vm01 scripts]# ls -l /usr/local/src/mha4mysql-manager-0.55/samples/scripts/power_manager-rwxr-xr-x 1 root root 11867 Dec 13 2012 /usr/local/src/mha4mysql-manager-0.55/samples/scripts/power_manager4、send_report因故障切换后发送报警的脚本，不是必须，可自行编写简单的shell完成
[root@vm01 scripts]# ls -l /usr/local/src/mha4mysql-manager-0.55/samples/scripts/send_report-rwxr-xr-x 1 root root 1360 Dec 13 2012 /usr/local/src/mha4mysql-manager-0.55/samples/scripts/send_report安装mha-node依赖
[root@vm01 src]# yum install perl-DBD-MySQL perl-Module-Install cpan安装mha-node（源码或rpm包安装均可）
[root@vm01 src]# rpm -ivh mha4mysql-node-0.54-0.el6.noarch.rpm Preparing... ########################################### [100%] 1:mha4mysql-node ########################################### [100%]或者
[root@vm01 src]# yum localinstall mha4mysql-node-0.54-0.el6.noarch.rpm -y其它服务器(vm02,vm03,vm04)安装mha-node及其依赖和vm01安装完全一样。
六、配置MHA
源码包里有配置文件模板，可参考: mha4mysql-manager-0.55源码压缩包里samples/conf目录下的模板配置文件。
创建mha配置文件目录
[root@vm01 src]# mkdir -p /etc/mha/app1mha配置文件app1.conf
（在软件包解压后的目录里面有样例配置文件）
[root@vm01 app1]# cat /etc/mha/app1/app1.conf [server default]# default 全局配置manager_workdir=/etc/mha/app1 #设置manager的工作目录manager_log=/etc/mha/app1/manager.log #设置manager的日志master_binlog_dir= /data/mysql/data #设置master保存binlog的位置，以便MHA可以找到master的日志，我这里的也就是mysql的数据目录ssh_user=root # 设置ssh的登录用户名user=mha # 设置mha监控用户password=mha@password #mha监控用户的密码repl_user=repl # 主从复制账号repl_password=654321 # 主从复制账号的密码secondary_check_script= /usr/local/bin/masterha_secondary_check -s 192.168.1.221 -s 192.168.1.229 # 一旦MHA到master的监控之间出现问题，MHA Manager将会尝试从192.168.1.221登录到masterping_interval=3 # 设置监控主库，发送ping包的时间间隔，默认是3秒，尝试三次没有回应的时候自动进行failovermaster_ip_failover_script= /etc/mha/app1/master_ip_failover #设置自动failover时候的切换脚本#shutdown_script= /etc/mha/app1/masterha/power_manager#report_script= /etc/mha/app1/send_report#master_ip_online_change_script= /etc/mha/app1/master_ip_online_change #设置手动切换时候的切换脚本[server1]# vm02 masterhostname=192.168.1.212port=3306candidate_master=1 [server2]# vm03 master-backuphostname=192.168.1.221port=3306candidate_master=1 # 设置为候选master，如果设置该参数以后，发生主从切换以后将会将此从库提升为主库，即使这个主库不是集群中事件最新的slavecheck_repl_delay=0 # 默认情况下如果一个slave落后master 100M的relay logs的话，MHA将不会选择该slave作为一个新的master，因为对于这个slave的恢复需要花费很长时间，通过设置check_repl_delay=0,MHA触发切换在选择一个新的master的时候将会忽略复制延时，这个参数对于设置了candidate_master=1的主机非常有用，因为这个候选主在切换的过程中一定是新的master[server3]# vm04 mysql slavehostname=192.168.1.229port=3306no_master=1所有slave服务器上 设置定时任务清理relay log
注意：
MHA在发生切换的过程中，从库的恢复过程中依赖于relay log的相关信息，所以这里要将relay log的自动清除设置为OFF（已将relay_log_purge=0写入到了my.cnf配置文件），采用手动清除relay log的方式。在默认情况下，从服务器上的中继日志会在SQL线程执行完毕后被自动删除。但是在MHA环境中，这些中继日志在恢复其他从服务器时可能会被用到，因此需要禁用中继日志的自动删除功能。定期清除中继日志需要考虑到复制延时的问题。在ext3的文件系统下，删除大的文件需要一定的时间，会导致严重的复制延时。为了避免复制延时，需要暂时为中继日志创建硬链接，因为在linux系统中通过硬链接删除大文件速度会很快。（在mysql数据库中，删除大表时，通常也采用建立硬链接的方式）
[root@vm03 ~]# cat /etc/cron.d/purge_relay_logs0 7 * * * /usr/bin/purge_relay_logs --user=root --password=123456 --disable_relay_log_purge --port=3306 --workdir=/data/mysql/data/ >>/etc/mha/app1/purge_relay_logs.log 2>&1参数说明：
--user=root # mysql用户名--password=123456 # mysql用户密码--port=3306 # mysql端口号--workdir=/data/mysql/data # 指定创建relay log的硬链接的位置，默认是/var/tmp，由于系统不同分区创建硬链接文件会失败，故需要执行硬链接具体位置，成功执行脚本后，硬链接的中继日志文件被删除--disable_relay_log_purge # 默认情况下，如果relay_log_purge=1，脚本会什么都不清理，自动退出，通过设定这个参数，当relay_log_purge=1的情况下会将relay_log_purge设置为0。清理relay log之后，最后将参数设置为OFF。故障转移脚本，虚拟ip(vip）配置为自己的：192.168.1.251
[root@vm01 app1]# cat /etc/mha/app1/master_ip_failover #!/usr/bin/env perluse strict;use warnings FATAL => 'all';use Getopt::Long;my ( $command, $ssh_user, $orig_master_host, $orig_master_ip, $orig_master_port, $new_master_host, $new_master_ip, $new_master_port);my $vip = '192.168.1.251/24'; # Virtual IPmy $key = "1";my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip";my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down";$ssh_user = "root";GetOptions( 'command=s' => \$command, 'ssh_user=s' => \$ssh_user, 'orig_master_host=s' => \$orig_master_host, 'orig_master_ip=s' => \$orig_master_ip, 'orig_master_port=i' => \$orig_master_port, 'new_master_host=s' => \$new_master_host, 'new_master_ip=s' => \$new_master_ip, 'new_master_port=i' => \$new_master_port,);exit &main();sub main { print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n"; if ( $command eq "stop" || $command eq "stopssh" ) { # $orig_master_host, $orig_master_ip, $orig_master_port are passed. # If you manage master ip address at global catalog database, # invalidate orig_master_ip here. my $exit_code = 1; #eval { # print "Disabling the VIP on old master: $orig_master_host \n"; # &stop_vip(); # $exit_code = 0; #}; eval { print "Disabling the VIP on old master: $orig_master_host \n"; #my $ping=`ping -c 1 10.0.0.13 | grep "packet loss" | awk -F',' '{print $3}' | awk '{print $1}'`; #if ( $ping le "90.0%" && $ping gt "0.0%" ){ #$exit_code = 0; #} #else { &stop_vip(); # updating global catalog, etc $exit_code = 0; #} }; if ($@) { warn "Got Error: $@\n"; exit $exit_code; } exit $exit_code; } elsif ( $command eq "start" ) { # all arguments are passed. # If you manage master ip address at global catalog database, # activate new_master_ip here. # You can also grant write access (create user, set read_only=0, etc) here. my $exit_code = 10; eval { print "Enabling the VIP - $vip on the new master - $new_master_host \n"; &start_vip(); $exit_code = 0; }; if ($@) { warn $@; exit $exit_code; } exit $exit_code; } elsif ( $command eq "status" ) { print "Checking the Status of the script.. OK \n"; `ssh $ssh_user\@$orig_master_ip \" $ssh_start_vip \"`; exit 0; } else { &usage(); exit 1; }}# A simple system call that enable the VIP on the new mastersub start_vip() { `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;}# A simple system call that disable the VIP on the old_mastersub stop_vip() { `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;}sub usage { print "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";}# the end.backup master & slave 设置read_only防止被误写
master备机和slave都要设置read_only=1，如果master自动切换后，会由mha设置原master备机为read_only=0
从库对外提供读服务，之所以没有写进配置文件，是因为随时slave会提升为master
mysql> select @@read_only;+-------------+| @@read_only |+-------------+| 0 |+-------------+1 row in set (0.00 sec)mysql> set global read_only=1;Query OK, 0 rows affected (0.00 sec)mysql> select @@read_only;+-------------+| @@read_only |+-------------+| 1 |+-------------+1 row in set (0.00 sec)MHA维护
创建软连接(在所有mysql上都执行)
ln -s /data/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlogln -s /data/mysql/bin/mysql /usr/bin/mysql检查主从复制情况 masterha_check_repl –conf=/etc/mha/app1/app1.conf
[root@vm01 app1]# masterha_check_repl --conf=/etc/mha/app1/app1.conf Thu May 11 22:11:42 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Thu May 11 22:11:42 2017 - [info] Reading application default configurations from /etc/mha/app1/app1.conf..Thu May 11 22:11:42 2017 - [info] Reading server configurations from /etc/mha/app1/app1.conf..Thu May 11 22:11:42 2017 - [info] MHA::MasterMonitor version 0.55.Thu May 11 22:11:42 2017 - [info] Dead Servers:Thu May 11 22:11:42 2017 - [info] Alive Servers:Thu May 11 22:11:42 2017 - [info] 192.168.1.212(192.168.1.212:3306)Thu May 11 22:11:42 2017 - [info] 192.168.1.221(192.168.1.221:3306)Thu May 11 22:11:42 2017 - [info] 192.168.1.229(192.168.1.229:3306)Thu May 11 22:11:42 2017 - [info] Alive Slaves:Thu May 11 22:11:42 2017 - [info] 192.168.1.221(192.168.1.221:3306) Version=5.6.35-log (oldest major version between slaves) log-bin:enabledThu May 11 22:11:42 2017 - [info] Replicating from 192.168.1.212(192.168.1.212:3306)Thu May 11 22:11:42 2017 - [info] Primary candidate for the new Master (candidate_master is set)Thu May 11 22:11:42 2017 - [info] 192.168.1.229(192.168.1.229:3306) Version=5.6.35-log (oldest major version between slaves) log-bin:disabledThu May 11 22:11:42 2017 - [info] Replicating from 192.168.1.212(192.168.1.212:3306)Thu May 11 22:11:42 2017 - [info] Not candidate for the new Master (no_master is set)Thu May 11 22:11:42 2017 - [info] Current Alive Master: 192.168.1.212(192.168.1.212:3306)Thu May 11 22:11:42 2017 - [info] Checking slave configurations..Thu May 11 22:11:42 2017 - [warning] log-bin is not set on slave 192.168.1.229(192.168.1.229:3306). This host can not be a master.Thu May 11 22:11:42 2017 - [info] Checking replication filtering settings..Thu May 11 22:11:42 2017 - [info] binlog_do_db= , binlog_ignore_db= Thu May 11 22:11:42 2017 - [info] Replication filtering check ok.Thu May 11 22:11:42 2017 - [info] Starting SSH connection tests..Thu May 11 22:11:43 2017 - [info] All SSH connection tests passed successfully.Thu May 11 22:11:43 2017 - [info] Checking MHA Node version..Thu May 11 22:11:44 2017 - [info] Version check ok.Thu May 11 22:11:44 2017 - [info] Checking SSH publickey authentication settings on the current master..Thu May 11 22:11:44 2017 - [info] HealthCheck: SSH to 192.168.1.212 is reachable.Thu May 11 22:11:44 2017 - [info] Master MHA Node version is 0.54.Thu May 11 22:11:44 2017 - [info] Checking recovery script configurations on the current master..Thu May 11 22:11:44 2017 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/data --output_file=/var/tmp/save_binary_logs_test --manager_version=0.55 --start_file=mysql-bin.000008 Thu May 11 22:11:44 2017 - [info] Connecting to root@192.168.1.212(192.168.1.212).. Creating /var/tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /data/mysql/data, up to mysql-bin.000008Thu May 11 22:11:44 2017 - [info] Master setting check done.Thu May 11 22:11:44 2017 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..Thu May 11 22:11:44 2017 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha' --slave_host=192.168.1.221 --slave_ip=192.168.1.221 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.35-log --manager_version=0.55 --relay_log_info=/data/mysql/data/relay-log.info --relay_dir=/data/mysql/data/ --slave_pass=xxxThu May 11 22:11:44 2017 - [info] Connecting to root@192.168.1.221(192.168.1.221:22).. Checking slave recovery environment settings.. Opening /data/mysql/data/relay-log.info ... ok. Relay log found at /data/mysql/data, up to mysql-relay-bin.000011 Temporary relay log file is /data/mysql/data/mysql-relay-bin.000011 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done.Thu May 11 22:11:44 2017 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha' --slave_host=192.168.1.229 --slave_ip=192.168.1.229 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.35-log --manager_version=0.55 --relay_log_info=/data/mysql/data/relay-log.info --relay_dir=/data/mysql/data/ --slave_pass=xxxThu May 11 22:11:44 2017 - [info] Connecting to root@192.168.1.229(192.168.1.229:22).. Checking slave recovery environment settings.. Opening /data/mysql/data/relay-log.info ... ok. Relay log found at /data/mysql/data, up to mysql-relay-bin.000011 Temporary relay log file is /data/mysql/data/mysql-relay-bin.000011 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done.Thu May 11 22:11:45 2017 - [info] Slaves settings check done.Thu May 11 22:11:45 2017 - [info] 192.168.1.212 (current master) +--192.168.1.221 +--192.168.1.229Thu May 11 22:11:45 2017 - [info] Checking replication health on 192.168.1.221..Thu May 11 22:11:45 2017 - [info] ok.Thu May 11 22:11:45 2017 - [info] Checking replication health on 192.168.1.229..Thu May 11 22:11:45 2017 - [info] ok.Thu May 11 22:11:45 2017 - [info] Checking master_ip_failover_script status:Thu May 11 22:11:45 2017 - [info] /etc/mha/app1/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.212 --orig_master_ip=192.168.1.212 --orig_master_port=3306 IN SCRIPT TEST====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.1.251/24===Checking the Status of the script.. OK Thu May 11 22:11:45 2017 - [info] OK.Thu May 11 22:11:45 2017 - [warning] shutdown_script is not defined.Thu May 11 22:11:45 2017 - [info] Got exit code 0 (Not master dead).MySQL Replication Health is OK.检查ssh连接情况 masterha_check_ssh –conf=/etc/mha/app1/app1.conf
[root@vm01 app1]# masterha_check_ssh --conf=/etc/mha/app1/app1.conf Thu May 11 22:15:23 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.Thu May 11 22:15:23 2017 - [info] Reading application default configurations from /etc/mha/app1/app1.conf..Thu May 11 22:15:23 2017 - [info] Reading server configurations from /etc/mha/app1/app1.conf..Thu May 11 22:15:23 2017 - [info] Starting SSH connection tests..Thu May 11 22:15:24 2017 - [debug] Thu May 11 22:15:23 2017 - [debug] Connecting via SSH from root@192.168.1.212(192.168.1.212:22) to root@192.168.1.221(192.168.1.221:22)..Thu May 11 22:15:24 2017 - [debug] ok.Thu May 11 22:15:24 2017 - [debug] Connecting via SSH from root@192.168.1.212(192.168.1.212:22) to root@192.168.1.229(192.168.1.229:22)..Thu May 11 22:15:24 2017 - [debug] ok.Thu May 11 22:15:24 2017 - [debug] Thu May 11 22:15:24 2017 - [debug] Connecting via SSH from root@192.168.1.221(192.168.1.221:22) to root@192.168.1.212(192.168.1.212:22)..Thu May 11 22:15:24 2017 - [debug] ok.Thu May 11 22:15:24 2017 - [debug] Connecting via SSH from root@192.168.1.221(192.168.1.221:22) to root@192.168.1.229(192.168.1.229:22)..Thu May 11 22:15:24 2017 - [debug] ok.Thu May 11 22:15:25 2017 - [debug] Thu May 11 22:15:24 2017 - [debug] Connecting via SSH from root@192.168.1.229(192.168.1.229:22) to root@192.168.1.212(192.168.1.212:22)..Thu May 11 22:15:25 2017 - [debug] ok.Thu May 11 22:15:25 2017 - [debug] Connecting via SSH from root@192.168.1.229(192.168.1.229:22) to root@192.168.1.221(192.168.1.221:22)..Thu May 11 22:15:25 2017 - [debug] ok.Thu May 11 22:15:25 2017 - [info] All SSH connection tests passed successfully.七、启动mha
vm01服务器上启动mha
[root@vm01 app1]# nohup /usr/local/bin/masterha_manager --conf=/etc/mha/app1/app1.conf --ignore_fail_on_start > /etc/mha/app1/mha_manager.log < /dev/null 2>&1 &[1] 10980[root@vm01 app1]# ps -ef|grep masterha_managerroot 10980 10886 0 22:20 pts/2 00:00:00 perl /usr/local/bin/masterha_manager --conf=/etc/mha/app1/app1.conf --ignore_fail_on_startroot 11082 10886 0 22:21 pts/2 00:00:00 grep masterha_manager此时，检查mysql master服务器上（vm02）的虚拟ip（vip）
[root@vm02 ~]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 12:34:56:78:90:12 brd ff:ff:ff:ff:ff:ff inet 192.168.1.212/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.251/24 brd 192.168.1.255 scope global secondary eth0:1 inet6 fe80::1034:56ff:fe78:9012/64 scope link valid_lft forever preferred_lft forever在mysql master数据库(vm02)上创建应用账号app，密码为app@password
mysql> grant alter,create,select,update,insert,delete on *.* to 'app'@'192.168.1.%' identified by 'app@password'; Query OK, 0 rows affected (0.00 sec)mysql> flush privileges;Query OK, 0 rows affected (0.00 sec)在vm01服务器上模拟应用程序远程使用vip连接数据库操作
[root@vm01 app1]# mysql -uapp -p -h192.168.1.251Enter password: Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 46Server version: 5.6.35-log Source distributionCopyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> show databases;+--------------------+| Database |+--------------------+| information_schema || mysql || performance_schema || test |+--------------------+4 rows in set (0.00 sec)mysql> create database zabbix;Query OK, 1 row affected (0.01 sec)mysql> use zabbixDatabase changedmysql> create table test_table(id int,name varchar(20));Query OK, 0 rows affected (0.05 sec)mysql> insert into test_table values(1,'zhangjian');Query OK, 1 row affected (0.00 sec)模拟master宕机故障，mha自动故障转移，切换master
[root@vm02 ~]# ps -ef|grep mysqlroot 15125 1 0 17:32 ? 00:00:00 /bin/sh /data/mysql/bin/mysqld_safe --datadir=/data/mysql/data --pid-file=/data/mysql/data/vm02.localdomain.pidmysql 15951 15125 0 17:32 ? 00:00:11 /data/mysql/bin/mysqld --basedir=/data/mysql --datadir=/data/mysql/data --plugin-dir=/data/mysql/lib/plugin --user=mysql --log-error=/data/mysql/mysqld.log --open-files-limit=8192 --pid-file=/data/mysql/data/vm02.localdomain.pid --socket=/data/mysql/mysql.sock --port=3306root 16669 16368 0 22:55 pts/4 00:00:00 grep mysql[root@vm02 ~]# pkill mysqld[root@vm02 ~]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 12:34:56:78:90:12 brd ff:ff:ff:ff:ff:ff inet 192.168.1.212/24 brd 192.168.1.255 scope global eth0 inet6 fe80::1034:56ff:fe78:9012/64 scope link valid_lft forever preferred_lft forever[root@vm02 ~]# !psps -ef|grep mysqlroot 16723 16708 0 22:59 pts/1 00:00:00 grep mysql此时，mysql-backup（vm03)服务器上
[root@vm03 ~]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether fa:9a:21:e9:4a:00 brd ff:ff:ff:ff:ff:ff inet 192.168.1.221/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.251/24 brd 192.168.1.255 scope global secondary eth0:1 inet6 fe80::f89a:21ff:fee9:4a00/64 scope link valid_lft forever preferred_lft forever[root@vm03 ~]# mysql -uroot -p123456Warning: Using a password on the command line interface can be insecure.Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 17Server version: 5.6.35-log Source distributionCopyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> show master status\G*************************** 1. row *************************** File: mysql-bin.000005 Position: 344 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec)mysql> show slave status\GEmpty set (0.00 sec)此时要注意：
因为前面我们是将半同步复制的配置写死在my.cnf配置文件中了，此时master切换后，需要将新master和原master中关于半同步复制的内容对调，同时在mysql命令行中执行set global操作，如下：
新Master:
[mysqld] rpl_semi_sync_master_enabled=1 rpl_semi_sync_master_timeout=10000 # 默认10秒mysql> set global rpl_semi_sync_master_enabled=1;Query OK, 0 rows affected (0.00 sec)mysql> set global rpl_semi_sync_slave_enabled=0;Query OK, 0 rows affected (0.00 sec)原Master:
[mysqld] rpl_semi_sync_slave_enabled=1 mysql> set global rpl_semi_sync_master_enabled=0;Query OK, 0 rows affected (0.00 sec)mysql> set global rpl_semi_sync_slave_enabled=0; Query OK, 0 rows affected (0.00 sec)所有slave上重启io_thread：
mysql> stop slave io_thread;Query OK, 0 rows affected (0.00 sec)mysql> start slave io_thread;Query OK, 0 rows affected (0.00 sec)发现有一个问题，在原master上重启io_thread后，Rpl_semi_sync_slave_status状态仍是OFF.检查半同步复制配置及状态：
show variables like ‘%rpl%’; show status like ‘%rpl%’;此时，vm04（mysql slave）服务器上
[root@vm04 ~]# mysql -uroot -p123456Warning: Using a password on the command line interface can be insecure.Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 44Server version: 5.6.35-log Source distributionCopyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> show master status;Empty set (0.00 sec)mysql> show slave status\G*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.1.221 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000005 Read_Master_Log_Pos: 344 Relay_Log_File: mysql-relay-bin.000002 Relay_Log_Pos: 503 Relay_Master_Log_File: mysql-bin.000005 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 344 Relay_Log_Space: 672 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 221 Master_UUID: c520243d-356a-11e7-b483-fa9a21e94a00 Master_Info_File: /data/mysql/data/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 01 row in set (0.00 sec)此时，从vm01服务器模拟应用程序继续远程使用vip连接数据库操作
mysql> insert into test_table values(1,'zhangjian');ERROR 2006 (HY000): MySQL server has gone awayNo connection. Trying to reconnect...Connection id: 16Current database: zabbixQuery OK, 1 row affected (0.01 sec)八、模拟原宕机master恢复
发现，原宕机master恢复后，没有自动变为新master的从库，但vip也没有重新设置回来。新的master仍保持其master地位不变。
[root@vm02 ~]# ps -ef|grep mysqlroot 16764 16708 0 00:02 pts/1 00:00:00 grep mysql[root@vm02 ~]# service mysqld restart ERROR! MySQL server PID file could not be found!Starting MySQL.. SUCCESS! [root@vm02 ~]# !psps -ef|grep mysqlroot 16805 1 0 00:02 pts/1 00:00:00 /bin/sh /data/mysql/bin/mysqld_safe --datadir=/data/mysql/data --pid-file=/data/mysql/data/vm02.localdomain.pidmysql 17631 16805 10 00:02 pts/1 00:00:00 /data/mysql/bin/mysqld --basedir=/data/mysql --datadir=/data/mysql/data --plugin-dir=/data/mysql/lib/plugin --user=mysql --log-error=/data/mysql/mysqld.log --open-files-limit=8192 --pid-file=/data/mysql/data/vm02.localdomain.pid --socket=/data/mysql/mysql.sock --port=3306root 17658 16708 0 00:02 pts/1 00:00:00 grep mysql[root@vm02 ~]# mysql -uroot -p123456Warning: Using a password on the command line interface can be insecure.Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 1Server version: 5.6.35-log Source distributionCopyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> show master status\G*************************** 1. row *************************** File: mysql-bin.000009 Position: 120 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec)mysql> show slave status\G Empty set (0.01 sec)原宕机的master开启后，开启mha会报错，原因是检测到非slave服务器(即两个master,尽管只有一个生效)导致启动失败
mha是一次性的，主库发生切换后，切换完成过，mha进程就自动退了。
[root@vm01 scripts]# nohup /usr/local/bin/masterha_manager --conf=/etc/mha/app1/app1.conf --ignore_fail_on_start > /etc/mha/app1/mha_manager.log < /dev/null 2>&1 &[root@vm01 scripts]# nohup /usr/local/bin/masterha_manager --conf=/etc/mha/app1/app1.conf --ignore_fail_on_start > /etc/mha/app1/mha_manager.log < /dev/null 2>&1 &[1] 14621[root@vm01 scripts]# ps -ef|grep managerroot 14627 29444 0 15:12 pts/0 00:00:00 grep manager[1]+ Exit 1 nohup /usr/local/bin/masterha_manager --conf=/etc/mha/app1/app1.conf --ignore_fail_on_start > /etc/mha/app1/mha_manager.log < /dev/null 2>&1[root@vm01 ~]# cat /etc/mha/app1/manager.logFri May 12 15:06:21 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln604] There are 2 non-slave servers! MHA manages at most one non-slave server. Check configurations.Fri May 12 15:06:21 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln386] Error happend on checking configurations. at /usr/local/share/perl5/MHA/MasterMonitor.pm line 300Fri May 12 15:06:21 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln482] Error happened on monitoring servers.Fri May 12 15:06:21 2017 - [info] Got exit code 1 (Not master dead).此时查看MHA Manager监控是不正常的
[root@vm01 ~]# /usr/local/bin/masterha_check_status --conf=/etc/mha/app1/app1.conf app1 is stopped(2:NOT_RUNNING).此时，需要修改mha manager配置文件(app1.conf)，将原宕机后恢复的master重做为新master的slave。
在新master上全库备份，用于宕机后恢复的原master用来做新master的slave主从同步用，以下得到的备份文件中会有用于主从同步的change master的相关信息。
[root@vm03 ~]# mysqldump -uroot -p123456 --master-data=2 --single-transaction --default-character-set=utf8mb4 -R --triggers -A >new_master_data.sql将新master上的全库备份文件复制到原宕机Master(vm02)上，恢复数据库，并从数据库备份文件中得到change master所需要的信息。
[root@vm03 ~]# rsync -avzP new_master_data.sql vm02:/root# 恢复新主库执行全备时完整数据[root@vm02 ~]# mysql -uroot -p123456 <new_master_data.sql # 从备份文件中得到change master所需要信息[root@vm02 ~]# grep -i "change master to" new_master_data.sql |head -n1-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000005', MASTER_LOG_POS=344;原宕机master（vm02)变为新master（vm03）的从库，同时在vm02上设置 只读set global read_only=1;
在执行change master前我在新master上插入了几条测试数据用于验证同步
# vm02服务器上mysql> CHANGE MASTER TO master_host='192.168.1.221' , master_user='repl' , master_password='654321' , master_port=3306 , MASTER_LOG_FILE='mysql-bin.000005', MASTER_LOG_POS=344; Query OK, 0 rows affected, 2 warnings (0.02 sec)mysql> start slave;Query OK, 0 rows affected (0.00 sec)mysql> set global read_only=1;Query OK, 0 rows affected (0.00 sec)mysql> show slave status\G;*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.1.221 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000005 Read_Master_Log_Pos: 594 Relay_Log_File: mysql-relay-bin.000002 Relay_Log_Pos: 529 Relay_Master_Log_File: mysql-bin.000005 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 594 Relay_Log_Space: 698 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 221 Master_UUID: c520243d-356a-11e7-b483-fa9a21e94a00 Master_Info_File: /data/mysql/data/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 01 row in set (0.00 sec)ERROR: No query specifiedmysql> select * from zabbix.test_table;+------+-----------+| id | name |+------+-----------+| 1 | zhangjian || 1 | zhangjian || 2 | yy || 3 | fangfang || 8 | hm || 6 | mm |+------+-----------+6 rows in set (0.00 sec)此时，再在MHA manager（vm01）管理节点上开启MHA，正常
注意：
在开启MHA前，要修改MHA manager配置文件secondary_check_script那部分
[root@vm01 ~]# vim /etc/mha/app1/app1.confsecondary_check_script= /usr/local/bin/masterha_secondary_check -s 192.168.1.212 -s 192.168.1.229开启mha
[root@vm01 ~]# nohup /usr/local/bin/masterha_manager --conf=/etc/mha/app1/app1.conf --ignore_fail_on_start > /etc/mha/app1/mha_manager.log < /dev/null 2>&1 &[1] 26635[root@vm01 ~]# ps -ef|grep managerroot 26635 14062 2 17:56 pts/1 00:00:00 perl /usr/local/bin/masterha_manager --conf=/etc/mha/app1/app1.conf --ignore_fail_on_startroot 26678 14062 0 17:56 pts/1 00:00:00 grep manager[root@vm01 ~]# ps -ef|grep managerroot 26635 14062 1 17:56 pts/1 00:00:00 perl /usr/local/bin/masterha_manager --conf=/etc/mha/app1/app1.conf --ignore_fail_on_startroot 26682 14062 0 17:56 pts/1 00:00:00 grep manager[root@vm01 ~]# ps -ef|grep managerroot 26635 14062 1 17:56 pts/1 00:00:00 perl /usr/local/bin/masterha_manager --conf=/etc/mha/app1/app1.conf --ignore_fail_on_startroot 26685 14062 0 17:56 pts/1 00:00:00 grep manager[root@vm01 ~]# ps -ef|grep managerroot 26635 14062 1 17:56 pts/1 00:00:00 perl /usr/local/bin/masterha_manager --conf=/etc/mha/app1/app1.conf --ignore_fail_on_startroot 26692 14062 0 17:56 pts/1 00:00:00 grep manager检查 MHA状态正常
[root@vm01 ~]# /usr/local/bin/masterha_check_status --conf=/etc/mha/app1/app1.conf app1 (pid:26635) is running(0:PING_OK), master:192.168.1.221查看mha启动日志
[root@vm01 ~]# cat /etc/mha/app1/manager.log Fri May 12 17:56:18 2017 - [info] MHA::MasterMonitor version 0.55.Fri May 12 17:56:18 2017 - [info] Dead Servers:Fri May 12 17:56:18 2017 - [info] Alive Servers:Fri May 12 17:56:18 2017 - [info] 192.168.1.212(192.168.1.212:3306)Fri May 12 17:56:18 2017 - [info] 192.168.1.221(192.168.1.221:3306)Fri May 12 17:56:18 2017 - [info] 192.168.1.229(192.168.1.229:3306)Fri May 12 17:56:18 2017 - [info] Alive Slaves:Fri May 12 17:56:18 2017 - [info] 192.168.1.212(192.168.1.212:3306) Version=5.6.35-log (oldest major version between slaves) log-bin:enabledFri May 12 17:56:18 2017 - [info] Replicating from 192.168.1.221(192.168.1.221:3306)Fri May 12 17:56:18 2017 - [info] Primary candidate for the new Master (candidate_master is set)Fri May 12 17:56:18 2017 - [info] 192.168.1.229(192.168.1.229:3306) Version=5.6.35-log (oldest major version between slaves) log-bin:disabledFri May 12 17:56:18 2017 - [info] Replicating from 192.168.1.221(192.168.1.221:3306)Fri May 12 17:56:18 2017 - [info] Not candidate for the new Master (no_master is set)Fri May 12 17:56:18 2017 - [info] Current Alive Master: 192.168.1.221(192.168.1.221:3306)Fri May 12 17:56:18 2017 - [info] Checking slave configurations..Fri May 12 17:56:18 2017 - [warning] log-bin is not set on slave 192.168.1.229(192.168.1.229:3306). This host can not be a master.Fri May 12 17:56:18 2017 - [info] Checking replication filtering settings..Fri May 12 17:56:18 2017 - [info] binlog_do_db= , binlog_ignore_db= Fri May 12 17:56:18 2017 - [info] Replication filtering check ok.Fri May 12 17:56:18 2017 - [info] Starting SSH connection tests..Fri May 12 17:56:20 2017 - [info] All SSH connection tests passed successfully.Fri May 12 17:56:20 2017 - [info] Checking MHA Node version..Fri May 12 17:56:20 2017 - [info] Version check ok.Fri May 12 17:56:20 2017 - [info] Checking SSH publickey authentication settings on the current master..Fri May 12 17:56:20 2017 - [info] HealthCheck: SSH to 192.168.1.221 is reachable.Fri May 12 17:56:20 2017 - [info] Master MHA Node version is 0.54.Fri May 12 17:56:20 2017 - [info] Checking recovery script configurations on the current master..Fri May 12 17:56:20 2017 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/data --output_file=/var/tmp/save_binary_logs_test --manager_version=0.55 --start_file=mysql-bin.000005 Fri May 12 17:56:20 2017 - [info] Connecting to root@192.168.1.221(192.168.1.221).. Creating /var/tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /data/mysql/data, up to mysql-bin.000005Fri May 12 17:56:20 2017 - [info] Master setting check done.Fri May 12 17:56:20 2017 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..Fri May 12 17:56:20 2017 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha' --slave_host=192.168.1.212 --slave_ip=192.168.1.212 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.35-log --manager_version=0.55 --relay_log_info=/data/mysql/data/relay-log.info --relay_dir=/data/mysql/data/ --slave_pass=xxxFri May 12 17:56:20 2017 - [info] Connecting to root@192.168.1.212(192.168.1.212:22).. Checking slave recovery environment settings.. Opening /data/mysql/data/relay-log.info ... ok. Relay log found at /data/mysql/data, up to mysql-relay-bin.000002 Temporary relay log file is /data/mysql/data/mysql-relay-bin.000002 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done.Fri May 12 17:56:21 2017 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha' --slave_host=192.168.1.229 --slave_ip=192.168.1.229 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.35-log --manager_version=0.55 --relay_log_info=/data/mysql/data/relay-log.info --relay_dir=/data/mysql/data/ --slave_pass=xxxFri May 12 17:56:21 2017 - [info] Connecting to root@192.168.1.229(192.168.1.229:22).. Checking slave recovery environment settings.. Opening /data/mysql/data/relay-log.info ... ok. Relay log found at /data/mysql/data, up to mysql-relay-bin.000002 Temporary relay log file is /data/mysql/data/mysql-relay-bin.000002 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done.Fri May 12 17:56:21 2017 - [info] Slaves settings check done.Fri May 12 17:56:21 2017 - [info] 192.168.1.221 (current master) +--192.168.1.212 +--192.168.1.229Fri May 12 17:56:21 2017 - [info] Checking master_ip_failover_script status:Fri May 12 17:56:21 2017 - [info] /etc/mha/app1/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.221 --orig_master_ip=192.168.1.221 --orig_master_port=3306 IN SCRIPT TEST====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.1.251/24===Checking the Status of the script.. OK Fri May 12 17:56:21 2017 - [info] OK.Fri May 12 17:56:21 2017 - [warning] shutdown_script is not defined.Fri May 12 17:56:21 2017 - [info] Set master ping interval 3 seconds.Fri May 12 17:56:21 2017 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s 192.168.1.212 -s 192.168.1.229Fri May 12 17:56:21 2017 - [info] Starting ping health check on 192.168.1.221(192.168.1.221:3306)..Fri May 12 17:56:21 2017 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..其中”[info] Ping(SELECT) succeeded, waiting until MySQL doesn’t respond..”说明整个系统已经开始监控了。
关闭MHA Manage监控 masterha_stop –conf=/etc/mha/app1/app1.conf
[root@vm01 scripts]# /usr/local/bin/masterha_stop --conf=/etc/mha/app1/app1.conf Stopped app1 successfully.[root@vm01 scripts]# ps -ef|grep managerroot 27826 29444 0 18:09 pts/0 00:00:00 grep manager# 再次开启，让mha时刻监控mysql集群状态[root@vm01 scripts]# nohup /usr/local/bin/masterha_manager --conf=/etc/mha/app1/app1.conf --ignore_fail_on_start > /etc/mha/app1/mha_manager.log < /dev/null 2>&1 &[1] 28041[root@vm01 scripts]# ps -ef|grep managerroot 28041 29444 2 18:11 pts/0 00:00:00 perl /usr/local/bin/masterha_manager --conf=/etc/mha/app1/app1.conf --ignore_fail_on_start十、配置VIP
vip配置可以采用两种方式，一种通过keepalived的方式管理虚拟ip的浮动；另外一种通过脚本方式启动虚拟ip的方式（即不需要keepalived或者heartbeat类似的软件）。
前面测试是通过脚本方式启动的虚拟ip。
初步测试完毕，暂时先写这么多了，lvs+keepalived部署后续再写。
1 0
- Mysql半同步复制+MHA+Keepalived部署
- MySQL半同步+MHA+Keepalived部署指导
- mysql5.6+mha+keepalived+半同步
- MySQL MHA+Keepalived
- mysql mha+keepalived
- mysql+mha+keepalived实现MHA架构
- atlas（keepalived）+mysql（MHA）
- MySQL MHA+lvs+keepalived线上生产环境
- MHA+Keepalived
- MySQL半同步复制
- mysql半同步复制
- MySQL半同步复制
- MySQL半同步复制
- MySql-半同步复制
- MySQL半同步复制
- MySQL半同步复制
- MySQL半同步复制
- mysql半同步复制
- WCF入门教程（一）简介
- python多线程模块--threading三部曲（一）
- sql server 加密解密
- xcode无法打印崩溃信息以及 乱打无效日志
- (0045) iOS 开发之MBProgressHUD 源码学习
- Mysql半同步复制+MHA+Keepalived部署
- Android Canvas 方法总结
- iframe 自适应
- Nginx基本功能极速入门
- 位运算简单举例
- JAVA解析JSON数据
- linux下zip命令打包与解包
- 我的第一篇文章
- 当推荐系统遇上深度学习
