通过Keepalived实现Redis Failover自动故障切换功能

设计思路：
redis 主和 redis从  正常同步；
redis主keepalived 启动，执行redis?_master.sh脚本，vip在自己身上；
redis 从keepalived 启动，执行redis?_slave.sh脚本；
redis主服务关闭，则redis主执行redis?_fault.sh脚本， vip落在redis 从上，redis从执行redis-_master.sh脚本，变成主；
redis主服务开启，则redis主执行redis?_slave.sh脚本变成从，vip还在redis 从上，还是主；
redis从服务关闭，则redis从执行redis?_fault.sh脚本， vip落在redis主上，redis主执行redis-_master.sh脚本，变成主；
redis从服务开启，则redis从执行redis?_slave.sh脚本变成从， vip还在redis主上，redis主继续做主，此时为redis初试主从状态；

环境：
192.168.1.235     redis
192.168.1.236     redis-slave
192.168.1.237     vip





一:安装keepalived+LVS
(redis和redis-slave两台服务器都安装)

命令：ipvsadm  查看是否安装了LVS

安装LVS：yum -y install ipvsadm

wget http://www.keepalived.org/software/keepalived-1.2.2.tar.gz

tar -zxvf keepalived-1.2.2.tar.gz
cd keepalived-1.2.2
./configure
configure时若出现以下结果：


不推荐使用yum install -y kernel-devel安装 安装完成后建立一下链接在配置：
ln -s /usr/src/kernels/2.6.18-194.el5-i686/ /usr/src/linux/

上述过程完毕后，在进行configure：


make
make install


cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/
chmod 755 /etc/init.d/keepalived
cp /usr/local/sbin/keepalived /usr/sbin/
cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/

#运行 由于keepalived配置文件不是在启动时一次性加载并读取完成的，所以必须是完整路径，可以根据log判断是否加载了正确的配置文件
keepalived –f /usr/local/etc/keepalived/keepalived.conf
运行过程可以查看log：
tail -f /var/log/message

二:安装redis
(redis和redis-slave两台服务器都安装)

tar   -zxvf redis-2.4.17.tar.gz
cd redis-2.4.17
./configure
make
make install

redis-slve只需注意一点
[root@redis bin]# pwd
/usr/local/bin
[root@redis bin]# vim redis.conf
116  slaveof 192.168.1.235 6379

三：通过Keepalived实现Redis Failover自动故障切换

修改Master和Slave的/etc/hosts文件
[root@redis rc.d]# cat /etc/hosts
192.168.1.235   redis
192.168.1.236   redis-slave

Master上是
[root@redis rc.d]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=redis

Slave上是
[root@redis-slave linux]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=redis-slave
修改完主机名重启机器生效


默认安装完成keepalived之后是没有配置文件的，因此我们需要手动创建：
首先，在Master上创建如下配置文件：
[root@redis  linux]# vim /etc/keepalived/keepalived.conf
global_defs {
   notification_email {
     [e-mail]
   }
   notification_email_from keepalived@redis
   router_id redis
}

vrrp_script chk_redis {
    script "/etc/keepalived/scripts/redis_check.sh"
    interval 1
}

vrrp_instance mes_Redis {
    state MASTER
    interface eth0            #设置实例绑定的网卡
    garp_master_delay 10      #在切换到master状态后，延迟进行gratuitous ARP请求
    smtp_alert                #使用global_defs中提供的邮件地址和smtp服务器发送邮件通知
    virtual_router_id 3       #VPID标记
    priority 100              #优先级，高优先级竞选为master
    nopreempt                 #设置为不抢占 注：这个配置只能设置在backup主机上，而且这个主机优先级要比另外一台高
    advert_int 1              #检查间隔，默认1秒
    authentication {          #设置认证
        auth_type PASS        #认证方式
        auth_pass 1111        #认证密码
    }
    virtual_ipaddress {       #设置vip
        192.168.1.237
    }
    track_script {
        chk_redis
    }
    notify_master /etc/keepalived/scripts/redis_master.sh
    notify_backup /etc/keepalived/scripts/redis_slave.sh
    notify_fault  /etc/keepalived/scripts/redis_fault.sh
    notify_stop   /etc/keepalived/scripts/redis_stop.sh
}

然后，在Slave上创建如下配置文件：
[root@redis-slave linux]# vim /etc/keepalived/keepalived.conf
global_defs {
   notification_email {
     [e-mail]
   }
   notification_email_from keepalived@redis-slave
   router_id redis-salve
}

vrrp_script chk_redis {
    script "/etc/keepalived/scripts/redis_check.sh"
    interval 1
}

vrrp_instance mes_Redis {
    state BACKUP
    interface eth0
    garp_master_delay 10
    smtp_alert
    virtual_router_id 3
    priority 90
    nopreempt
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.237
    }
    track_script {
        chk_redis
    }
    notify_master /etc/keepalived/scripts/redis_master.sh
    notify_backup /etc/keepalived/scripts/redis_slave.sh
    notify_fault  /etc/keepalived/scripts/redis_fault.sh
    notify_stop   /etc/keepalived/scripts/redis_stop.sh
}


在Master和Slave上创建监控Redis的脚本
mkdir /etc/keepalived/scripts
vim /etc/keepalived/scripts/redis_check.sh
#!/bin/bash
ALIVE=`/usr/local/bin/redis-cli PING`
if [ "$ALIVE" == "PONG" ]; then
  echo $ALIVE
  exit 0
else
  echo $ALIVE
  exit 1
fi

编写以下负责运作的关键脚本：
notify_master /etc/keepalived/scripts/redis_master.sh
notify_backup /etc/keepalived/scripts/redis_slave.sh
notify_fault /etc/keepalived/scripts/redis_fault.sh
notify_stop /etc/keepalived/scripts/redis_stop.sh

因为Keepalived在转换状态时会依照状态来呼叫：
当进入Master状态时会呼叫notify_master
当进入Backup状态时会呼叫notify_backup
当发现异常情况时进入Fault状态呼叫notify_fault
当Keepalived程序终止时则呼叫notify_stop

首先，在Redis Master上创建notity_master与notify_backup脚本：
vim /etc/keepalived/scripts/redis_master.sh
#!/bin/bash

REDISCLI="/usr/local/bin/redis-cli"
LOGFILE="/var/log/keepalived-redis-state.log"

echo "[master]" >> $LOGFILE
date >> $LOGFILE
echo "Being master...." >> $LOGFILE 2>&1

echo "Run SLAVEOF cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF 192.168.1.236 6379 >> $LOGFILE  2>&1
sleep 10 #延迟10秒以后待数据同步完成后再取消同步状态

echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1

vim /etc/keepalived/scripts/redis_slave.sh
#!/bin/bash

REDISCLI="/usr/local/bin/redis-cli"
LOGFILE="/var/log/keepalived-redis-state.log"

echo "[backup]" >> $LOGFILE
date >> $LOGFILE
echo "Being slave...." >> $LOGFILE 2>&1

sleep 15 #延迟15秒待数据被对方同步完成之后再切换主从角色
echo "Run SLAVEOF cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF 192.168.1.236 6379 >> $LOGFILE  2>&1


接着，在Redis Slave上创建notity_master与notify_backup脚本：

vim /etc/keepalived/scripts/redis_master.sh
#!/bin/bash

REDISCLI="/usr/local/bin/redis-cli"
LOGFILE="/var/log/keepalived-redis-state.log"

echo "[master]" >> $LOGFILE
date >> $LOGFILE
echo "Being master...." >> $LOGFILE 2>&1

echo "Run SLAVEOF cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF 192.168.1.235 6379 >> $LOGFILE  2>&1
sleep 10 #延迟10秒以后待数据同步完成后再取消同步状态

echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1

vim /etc/keepalived/scripts/redis_slave.sh
#!/bin/bash

REDISCLI="/usr/local/bin/redis-cli"
LOGFILE="/var/log/keepalived-redis-state.log"

echo "[backup]" >> $LOGFILE
date >> $LOGFILE
echo "Being slave...." >> $LOGFILE 2>&1

sleep 15 #延迟15秒待数据被对方同步完成之后再切换主从角色
echo "Run SLAVEOF cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF 192.168.1.235 6379 >> $LOGFILE  2>&1

然后在Master与Slave创建如下相同的脚本：
vim /etc/keepalived/scripts/redis_fault.sh
#!/bin/bash

LOGFILE=/var/log/keepalived-redis-state.log

echo "[fault]" >> $LOGFILE
date >> $LOGFILE

vim /etc/keepalived/scripts/redis_stop.sh
#!/bin/bash

LOGFILE=/var/log/keepalived-redis-state.log

echo "[stop]" >> $LOGFILE
date >> $LOGFILE


在Master与Slave给脚本都加上可执行权限：
chmod +x /etc/keepalived/scripts/*.sh

脚本创建完成以后，我们开始按照如下流程进行测试：
1.启动Master上的Redis
[root@redis bin]# pwd
/usr/local/bin
[root@redis bin]# ./redis-server redis.conf

2.启动Slave上的Redis
[root@redis bin]# pwd
/usr/local/bin
[root@redis bin]# ./redis-server redis.conf

3.启动Master上的Keepalived
/etc/init.d/keepalived start

4.启动Slave上的Keepalived
/etc/init.d/keepalived start

5.尝试通过VIP连接Redis:
[root@redis bin]#pwd
/usr/local/bin
[root@redis bin]# ./redis-cli -h 192.168.1.237 info
role:master
slave0:192.168.1.236,6379,online
连接成功，Slave也连接上来了

6.尝试插入一些数据：
[root@redis bin]# ./redis-cli -h 192.168.1.237 SET Hello Redis
从VIP读取数据
[root@redis bin]# ./redis-cli -h 192.168.1.237 GET Hello
"Redis"


从Master读取数据
[root@redis bin]# ./redis-cli -h 192.168.1.235  GET Hello
"Redis"

从Slave读取数据
[root@redis-slave bin]# ./redis-cli -h 192.168.1.235  GET Hello
"Redis"

下面，模拟故障产生：
将Master上的Redis进程杀死：
[root@redis bin]# ./redis-cli shutdown

查看Master上的Keepalived日志
[root@redis scripts]# tail /var/log/keepalived-redis-state.log
[fault]
Thu Sep 27 08:29:01 CST 2012

同时Slave上的日志显示：
[root@redis-slave scripts]# tail /var/log/keepalived-redis-state.log
[master]
Thu Nov 15 12:06:04 CST 2012
Being master....
Run SLAVEOF cmd ...
OK
Run SLAVEOF NO ONE cmd ...
OK

然后我们可以发现，Slave已经接管服务，并且担任Master的角色了。
./redis-cli -h 192.168.1.237 info
./redis-cli -h 192.168.1.236 info
role:master

然后我们恢复Master的Redis进程
主变成slave


然后把236redis停掉
235恢复主的角色，在把236redis开启

恢复235是主，236是备

自动切换成功！