基于keepalived、redis sentinel的多实例redis集群【1】---路由以及哨兵配置

来源：互联网发布：第一届大数据教育论坛编辑：程序博客网时间：2024/04/30 11:05

硬件

机器名IP作用master192.168.0.2redis的master服务器,两个主实例slave1192.168.0.3redis的slave服务器，两个从实例slave2192.168.0.4redis的slave服务器，两个从实例route1192.168.0.5【虚拟IP：192.168.0.7】keepalived和redis sentinel服务器，承载写redis的VIP【虚拟ip】，做写的双机热备的主master指定，redis哨兵的安装节点1route2192.168.0.6【虚拟IP：192.168.0.8】keepalived和redis sentinel服务器，承载读redis的VIP，做读的负载均衡和写的双机热备的master备份路由指定，redis哨兵的安装节点2

route1

1.安装redis在route1上，安装路径/usr/local/redis/

2.在redis安装路径下创建scripts目录，将需要的脚本复制到此处：

1RunCmd.py基础功能模块，提供redis服务超时检查2master_config_set.py将master的save参数配置为空3redischeck.py检查master的redis服务状态4slave_config_set.py将slave的save参数配置为特定值5weightchange.py调整读的redis服务在keepalived的权重

详细的keepalived配置，

! Configuration File for keepalivedglobal_defs {   notification_email {     接收邮箱      }   notification_email_from 发送邮箱   smtp_server 邮件服务器   smtp_connect_timeout 30   router_id LVS_DEVEL}vrrp_instance VI_1 {    state MASTER    interface eth1    lvs_sync_daemon_inteface eth1    virtual_router_id 100    priority 160    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }        virtual_ipaddress {        192.168.0.7    }         }vrrp_instance VI_2 {    state BACKUP    interface eth1    lvs_sync_daemon_inteface eth1    virtual_router_id 101    priority 100    advert_int 1    authentication {auth_type PASSauth_pass 1111    }    virtual_ipaddress {192.168.0.    }}virtual_server 192.168.0.7 6379 {delay_loop 3lb_algo rrlb_kind DR#nat_mask 255.255.255.0persistence_timeout 15protocol TCPreal_server 192.168.0.2 6379 {weight 8notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.2 6379"MISC_CHECK { misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.2 6379"misc_timeout 5misc_dynamic}}real_server 192.168.0.3 6379 {weight 3notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.3 6379"MISC_CHECK {misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.3 6379"misc_timeout 5misc_dynamic} }real_server 192.168.0.4 6379 {weight 3notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.4 6379"MISC_CHECK {misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.4 6379"misc_timeout 5msic_dynamic}}}virtual_server 192.168.0.7 6380 {delay_loop 3lb_algo rrlb_kind DR#nat_mask 255.255.255.0persistence_timeout 15protocol TCPreal_server 192.168.0.2 6380 {weight 8notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.2 6380"MISC_CHECK { misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.2 6380"misc_timeout 5misc_dynamic}}real_server 192.168.0.3 6380 {weight 3notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.3 6380"MISC_CHECK {misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.3 6380"misc_timeout 5misc_dynamic} }real_server 192.168.0.4 6380 {weight 3notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.4 6380"MISC_CHECK {misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.4 6380"misc_timeout 5msic_dynamic}}}virtual_server 192.168.0.8 6379 {delay_loop 3lb_algo wrrlb_kind DRpersistence_timeout 30protocol TCPreal_server 192.168.0.2 6379 {weight 6notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.2 6379"MISC_CHECK {misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.2 6379"misc_timeout 5misc_dynamic}} real_server 192.168.0.3 6379 {weight 2notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.3 6379"MISC_CHECK {misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.3 6379"misc_timeout 5misc_dynamic}}real_server 192.168.0.4 6379 {weight 2notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.4 6379"MISC_CHECK {misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.4 6379"misc_timeout 5misc_dynamic}}}virtual_server 192.168.0.8 6380 {delay_loop 3lb_algo wrrlb_kind DRpersistence_timeout 30protocol TCPreal_server 192.168.0.2 6380 {weight 6notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.2 6380"MISC_CHECK {misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.2 6380"misc_timeout 5misc_dynamic}} real_server 192.168.0.3 6380 {weight 2notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.3 6380"MISC_CHECK {misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.3 6380"misc_timeout 5misc_dynamic}}real_server 192.168.0.4 6380 {weight 2notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.4 6380"MISC_CHECK {misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.4 6380"misc_timeout 5misc_dynamic}}}

route2的keepalived配置文件

! Configuration File for keepalivedglobal_defs {   notification_email {     接受邮箱   }   notification_email_from 发送邮箱   smtp_server 邮件服务器   smtp_connect_timeout 30   router_id LVS_DEVEL}vrrp_instance VI_1 {    state BACKUP    interface eth1    lvs_sync_daemon_inteface eth1    virtual_router_id 100    priority 100    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }        virtual_ipaddress {        192.168.0.7    }     }vrrp_instance VI_2 {    state MASTER    interface eth1    lvs_sync_daemon_inteface eth1    virtual_router_id 101    priority 151    advert_int 1    authentication {auth_type PASSauth_pass 1111    }    virtual_ipaddress { 192.168.0.8    }}virtual_server 192.168.0.7 6379 {delay_loop 3lb_algo rrlb_kind DRpersistence_timeout 15protocol TCPreal_server 192.168.0.2 6379 {weight 8notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.2 6379"MISC_CHECK {misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.2 6379"misc_timeout 5misc_dynamic}}real_server 192.168.0.3 6379 {weight 3notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.3 6379"MISC_CHECK {misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.3 6379"misc_timeout 5misc_dynamic}}real_server 192.168.0.4 6379 {weight 3notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.4 6379"MISC_CHECK {misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.4 6379"misc_timeout 5misc_dynamic}}}virtual_server 192.168.0.7 6380 {delay_loop 3lb_algo rrlb_kind DRpersistence_timeout 15protocol TCPreal_server 192.168.0.2 6380 {weight 8notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.2 6380"MISC_CHECK {misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.2 6380"misc_timeout 5misc_dynamic}}real_server 192.168.0.3 6380 {weight 3notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.3 6380"MISC_CHECK {misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.3 6380"misc_timeout 5misc_dynamic}}real_server 192.168.0.4 6380 {weight 3notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.4 6380"MISC_CHECK {misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.4 6380"misc_timeout 5misc_dynamic}}}virtual_server 192.168.0.8 6379 {delay_loop 3lb_algo wrrlb_kind DRpersistence_timeout 30protocol TCPreal_server 192.168.0.2 6379 {weight 1notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.2 6379"MISC_CHECK {misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.2 6379"misc_timeout 5misc_dynamic}}real_server 192.168.0.3 6379 {weight 2notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.3 6379"MISC_CHECK {misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.3 6379"misc_timeout 5misc_dynamic}}real_server 192.168.0.4 6379 {weight 2notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.4 6379"MISC_CHECK {misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.4 6379"misc_timeout 5misc_dynamic}}}virtual_server 192.168.0.8 6380 {delay_loop 3lb_algo wrrlb_kind DRpersistence_timeout 30protocol TCPreal_server 192.168.0.2 6380 {weight 1notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.2 6380"MISC_CHECK {misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.2 6380"misc_timeout 5misc_dynamic}}real_server 192.168.0.3 6380 {weight 2notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.3 6380"MISC_CHECK {misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.3 6380"misc_timeout 5misc_dynamic}}real_server 192.168.0.4 6380 {weight 2notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.4 6380"MISC_CHECK {misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.4 6380"misc_timeout 5misc_dynamic}}}

在keepalived使用的脚本

RunCmd.py

#!/usr/bin/pythonimport os;  import sys;  import time;  import fcntl;  import select;  import signal;  import commands;  import subprocess; class RunCmd:      def __init__(self):          pass;           def Run(self,ip,port, nTimeOut = 0, nIntervalTime = 0.1):          lsCmd=['/usr/local/redis/bin/redis-cli','-h',ip,'-p',port,'ping']        oProc = subprocess.Popen(lsCmd, stdout =subprocess.PIPE, stderr = subprocess.PIPE)          istimeout=False        nStartTime = time.time()                  while True:              time.sleep(nIntervalTime)               print("1:")             print(oProc.poll())            if None != oProc.poll():                  break;                        if (nTimeOut > 0) and (time.time() - nStartTime) > nTimeOut:                istimeout=True                                 break;         print("2:")         print(istimeout)         if istimeout:             print(oProc.poll())             if None == oProc.poll():                  self.KillAll(oProc.pid)          print("3:")         print(istimeout)        return istimeout      def KillAll(self, nKillPid, nKillSignal = signal.SIGKILL):          nRet, strOutput = commands.getstatusoutput("kill "+str(nKillPid));#as root run                 return (True, strOutput)

master_config_set.py脚本

#!/usr/bin/pythonfrom RunCmd import RunCmdimport sys,commandsoCmd = RunCmd();  istimeout = oCmd.Run(sys.argv[1],sys.argv[2], 0.1)if not istimeout:    cmd="/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" info"    str=commands.getoutput(cmd)    ismaster=str.count("role:master")    zero=0    if ismaster>zero:        t=commands.getoutput("/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" config set save \"\"")        print t

slave_config_set.py

#!/usr/bin/pythonfrom RunCmd import RunCmdimport sys,commandsoCmd = RunCmd();  istimeout = oCmd.Run(sys.argv[1],sys.argv[2], 0.1)if not istimeout:    cmd="/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" info"    str=commands.getoutput(cmd)    isslave=str.count("role:slave")    zero=0    if isslave>zero:        t=commands.getoutput("/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" config set save \"90 1 300 10 60 1000\"")        print t

redischeck.py

#!/usr/bin/pythonfrom RunCmd import RunCmdimport sys,commandsoCmd = RunCmd();  istimeout = oCmd.Run(sys.argv[1],sys.argv[2], 0.1)if not istimeout:cmd="/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" info"str=commands.getoutput(cmd)ismaster=str.count("role:master")zero=0if ismaster>zero:  sys.exit(0)else:  sys.exit(1)else:sys.exit(1)

weightchange.py

#!/usr/bin/pythonfrom RunCmd import RunCmdimport sys,commandsoCmd = RunCmd();  istimeout = oCmd.Run(sys.argv[1],sys.argv[2], 0.1)if not istimeout:result=1cmd="/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" ping"strping=commands.getoutput(cmd)zero=0ispong=-100ispong=strping.count("PONG")if ispong>zero:result=0if result>zero:sys.exit(1)else:cmdmaster="/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" info"str=commands.getoutput(cmdmaster)ismaster=-100ismaster=str.count("role:master")if ismaster>zero:  sys.exit(3)else:  sys.exit(10)else:sys.exit(1)

redis的哨兵的配置文件sentinel.conf

# Example sentinel.conf# port <sentinel-port># The port that this sentinel instance will run onport 26379# sentinel monitor <master-name> <ip> <redis-port> <quorum>## Tells Sentinel to monitor this slave, and to consider it in O_DOWN# (Objectively Down) state only if at least <quorum> sentinels agree.## Note: master name should not include special characters or spaces.# The valid charset is A-z 0-9 and the three characters ".-_".sentinel monitor mymaster 192.168.0.2 6379 2sentinel monitor mymaster6380 192.168.0.2 6380 2# sentinel auth-pass <master-name> <password>## Set the password to use to authenticate with the master and slaves.# Useful if there is a password set in the Redis instances to monitor.## Note that the master password is also used for slaves, so it is not# possible to set a different password in masters and slaves instances# if you want to be able to monitor these instances with Sentinel.## However you can have Redis instances without the authentication enabled# mixed with Redis instances requiring the authentication (as long as the# password set is the same for all the instances requiring the password) as# the AUTH command will have no effect in Redis instances with authentication# switched off.## Example:## sentinel auth-pass mymaster MySUPER--secret-0123passw0rd# sentinel down-after-milliseconds <master-name> <milliseconds>## Number of milliseconds the master (or any attached slave or sentinel) should# be unreachable (as in, not acceptable reply to PING, continuously, for the# specified period) in order to consider it in S_DOWN state (Subjectively# Down).## Default is 30 seconds.sentinel down-after-milliseconds mymaster 3800sentinel down-after-milliseconds mymaster6380 3800# sentinel can-failover <master-name> <yes|no>## Specify if this Sentinel can start the failover for this master.sentinel can-failover mymaster yessentinel can-failover mymaster6380 yes# sentinel parallel-syncs <master-name> <numslaves>## How many slaves we can reconfigure to point to the new slave simultaneously# during the failover. Use a low number if you use the slaves to serve query# to avoid that all the slaves will be unreachable at about the same# time while performing the synchronization with the master.sentinel parallel-syncs mymaster 1sentinel parallel-syncs mymaster6380 1# sentinel failover-timeout <master-name> <milliseconds>## Specifies the failover timeout in milliseconds. When this time has elapsed# without any progress in the failover process, it is considered concluded by# the sentinel even if not all the attached slaves were correctly configured# to replicate with the new master (however a "best effort" SLAVEOF command# is sent to all the slaves before).## Also when 25% of this time has elapsed without any advancement, and there# is a leader switch (the sentinel did not started the failover but is now# elected as leader), the sentinel will continue the failover doing a# "takeover".## Default is 15 minutes.sentinel failover-timeout mymaster 90000sentinel failover-timeout mymaster6380 90000# SCRIPTS EXECUTION## sentinel notification-script and sentinel reconfig-script are used in order# to configure scripts that are called to notify the system administrator# or to reconfigure clients after a failover. The scripts are executed# with the following rules for error handling:## If script exists with "1" the execution is retried later (up to a maximum# number of times currently set to 10).## If script exists with "2" (or an higher value) the script execution is# not retried.## If script terminates because it receives a signal the behavior is the same# as exit code 1.## A script has a maximum running time of 60 seconds. After this limit is# reached the script is terminated with a SIGKILL and the execution retried.# NOTIFICATION SCRIPT## sentinel notification-script <master-name> <script-path># # Call the specified notification script for any sentienl event that is# generated in the WARNING level (for instance -sdown, -odown, and so forth).# This script should notify the system administrator via email, SMS, or any# other messaging system, that there is something wrong with the monitored# Redis systems.## The script is called with just two arguments: the first is the event type# and the second the event description.## The script must exist and be executable in order for sentinel to start if# this option is provided.## Example:## sentinel notification-script mymaster /var/redis/notify.sh# CLIENTS RECONFIGURATION SCRIPT## sentinel client-reconfig-script <master-name> <script-path>## When the failover starts, ends, or is aborted, a script can be called in# order to perform application-specific tasks to notify the clients that the# configuration has changed and the master is at a different address.# # The script is called in the following cases:## Failover started (a slave is already promoted)# Failover finished (all the additional slaves already reconfigured)# Failover aborted (in that case the script was previously called when the#                   failover started, and now gets called again with swapped#                   addresses).## The following arguments are passed to the script:## <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>## <state> is "start", "end" or "abort"# <role> is either "leader" or "observer"# # The arguments from-ip, from-port, to-ip, to-port are used to communicate# the old address of the master and the new address of the elected slave# (now a master) in the case state is "start" or "end".## For abort instead the "from" is the address of the promoted slave and# "to" is the address of the original master address, since the failover# was aborted.## This script should be resistant to multiple invocations.## Example:## sentinel client-reconfig-script mymaster /var/redis/reconfig.sh

在两个route上修改/etc/sysctl.conf文件

net.ipv4.ip_forward=1#转发开启

执行sysctl -p让文件起效

有防火墙需要设置防火墙转发