翼支付门户架构之redis高可用部署及监控

一、Redis Sentinel简介

Redis Sentinel是redis自带的集群管理工具，主要功能有

1、监控（Monitoring）：Redis Sentinel实时监控主服务器和从服务器运行状态；

2、提醒（Notification）：当被监控的某个Redis服务器出现问题时，Redis Sentinal可以向系统管理员发送通知，也可以通过API向其他程序发送通知；

3、自动故障转移（Automatic failover）：当一个主服务器不能正常运行时，Redis Sentinel可以将一个从服务器升级为主服务器，并对其他从服务器进行配置，让他们使用新的主服务器。当应用程序连接到Redis服务器时，Redis Sentinel会告之新的主服务器地址和端口；

Redis Sentinel是一个分布式系统，你可以在架构中运行多个Sentinel进程，这些进程通过相互通信来判断一个主服务器是否断线，以及是否应该执行故障转移。

在配置Redis Sentinel时，至少需要有一个1个Master和1个Slave。当Master失效后，Redis Sentinel会报出失效警告，并通过故障转移将Slave提升为Master，并提供读写服务。当失效的Master恢复后，Redis Sentinel会自动识别，将Master自动转换为Slave并完成数据同步。

通过Redis Sentinel可以实现Redis零手工干预并且短时间内进行M-S切换，减少业务影响时间。

二、硬件需求

未来预防单节点故障，需要配置至少两台服务器，配置要求一致。

CPU：>2 CORES；

内存：>16 GB；

磁盘：>100 GB；

三、拓扑结构

       在两个服务器中分别都部署Redis和Redis Sentinel。当Master中的Redis出现故障时（Redis进程终止、服务器僵死、服务器断电等），由Redis Sentinel将Master权限切换至Slave Redis中，并将只读模式更改为可读可写模式，应用程序通过Redis Sentinel确定当前Master Redis位置，进行重新连接。

根据业务模式，可以制定两种拓扑结构：单M-S结构和双M-S结构。如果有足够多的服务器，可以配置多M-S结构。

       1）、单M-S结构

        单M-S结构特点是在Master服务器中配置MasterRedis（Redis-1M）和MasterSentinel（Sentinel-1M）。Slave服务器中配置SlaveRedis（Redis-1S）和SlaveSentinel（Sentinel-1S）。其中 MasterRedis可以提供读写服务，但是SlaveRedis只能提供只读服务。因此，在业务压力比较大的情况下，可以选择将只读业务放在SlaveRedis中进行。

       

    2）、双M-S结构

          双M-S结构的特点是在每台服务器上配置一个MasterRedis，同时部署一个Slave Redis。由两个RedisSentinel同时对4个Redis进行监控。两个MasterRedis可以同时对应用程序提供读写服务，即便其中一个服务器出现故障，另一个服务器也可以同时运行两个MasterRedis提供读写服务。缺点是两个Masterredis之间无法实现数据共享，不适合存在大量用户数据关联的应用使用。

         

         3）、优劣对比

            两个结构各有优缺点，分别适用于不同的应用场景：

            单M-S结构适用于不同用户数据存在关联，但应用可以实现读写分离的业务模式。Master主要提供写操作，Slave主要提供读操作，充分利用硬件资源。

             双（多）M-S结构适用于用户间不存在或者存在较少的数据关联的业务模式，读写效率是单M-S的两（多）倍，但要求故障时单台服务器能够承担两个MaterRedis的资源需求。

四、配置部署

       单M-S结构和双M-S结构配置相差无几，

      下面以单M-S结构配置为例

      1）、在Server-1M上配置Redis-1M:

# viredis-1M.conf

## masterredis-1M

##daemonize默认为no，修改为yes，启用后台运行

daemonizeyes

 

# Redis默认pid文件位置redis.pid

#当运行多个redis服务时，需要指定不同的 pid文件和端口

pidfileredis-1M.pid

 

##端口号

port 6379

 

##验证口令   

requirepass************* 

masterauth*************

 

#绑定可连接Redis的IP地址,不设置将处理所有请求

# bind 127.0.0.1

 

#客户端连接的超时时间,单位为秒,超时后会关闭连接（0为不设置）

timeout 0

 

#日志记录等级

loglevel notice

 

#设置数据库的个数

databases16

#日志刷新策略（Master禁用）

#save 9001

#save 30010

#save 6010000

 

#是否使用压缩镜像备份

rdbcompression yes

 

#镜像备份文件的文件名

dbfilename redis-1M_dump.rdb

 

#镜像备份路径，默认值为./

dir /redis/backup

 

#设置该数据库为其他数据库的从数据库,主库无需设置

#slaveof

#slaveof

 

#指定与主数据库连接时需要的密码验证，主库无需设置

#masterauth

#masterauth

 

#如果slave-serve-stale-data设置成 'no'，slave会返回"SYNCwith master in #progress"错误信息，但INFO和SLAVEOF命令除外。

slave-serve-stale-data yes

 

#客户端连接访问口令

# requirepassfoobared

 

#限制同时连接的客户数量，防止过多的client导致内存耗尽。如果有足够内存可以不进行#设置

#maxclients 10000

 

#设置redis能够使用的最大内存。

#maxmemory

 

##启用增量（Master禁用） 

appendonly no

 

#增量日志文件名，默认值为appendonly.aof

appendfilenameappendonly.aof

 

#设置对appendonly.aof文件进行同步的频率

#always表示每次有写操作都进行同步,everysec表示对写操作进行累积,每秒同步一次。

#no表示等操作系统进行数据缓存同步到磁盘，都进行同步,everysec表示对写操作进行累#积,每秒同步一次

appendfsynceverysec

 

#是否重置Hash表

#设置成yes后redis将每100毫秒使用1毫秒CPU时间来对redis的hash表重新hash，##可降低内存的使用。当使用场景有较为严格的实时性需求,不能接受Redis时不时的对请##求有2毫秒的延迟的话，把这项配置为no。如果没有这么严格的实时性要求,可以设置为#yes,能够尽可能快的释放内存。

activerehashingyes

 

##Slave开启只读模式

slave-read-only yes

     2）、在Server-1S上配置Redis-1S：

# Redis默认pid文件位置redis.pid

#当运行多个redis服务时，需要指定不同的 pid文件和端口

pidfileredis-1S.pid

 

##端口号

port 6380

 

#镜像备份文件的文件名

dbfilenameredis-1S_dump.rdb

 

#设置该数据库为其他数据库的从数据库,主库无需设置

Slaveof server-1M 6379

 

##启用增量（Master禁用） 

appendonly yes

 

#-----------------其他参数与redis-1M保持一致-----------------

daemonize yes

#……

     3）、Redis Sentinel配置

     在Server-1M配置sentinel-1M：

     sentinel.conf的配置信息如下：

    

# Example sentinel.conf# port <sentinel-port># The port that this sentinel instance will run onport 26379daemonize yes# sentinel announce-ip <ip># sentinel announce-port <port>## The above two configuration directives are useful in environments where,# because of NAT, Sentinel is reachable from outside via a non-local address.## When announce-ip is provided, the Sentinel will claim the specified IP address# in HELLO messages used to gossip its presence, instead of auto-detecting the# local address as it usually does.## Similarly when announce-port is provided and is valid and non-zero, Sentinel# will announce the specified TCP port.## The two options don't need to be used together, if only announce-ip is# provided, the Sentinel will announce the specified IP and the server port# as specified by the "port" option. If only announce-port is provided, the# Sentinel will announce the auto-detected local IP and the specified port.## Example:## sentinel announce-ip 1.2.3.4# dir <working-directory># Every long running process should have a well-defined working directory.# For Redis Sentinel to chdir to /tmp at startup is the simplest thing# for the process to don't interfere with administrative tasks such as# unmounting filesystems.dir "/tmp"# sentinel monitor <master-name> <ip> <redis-port> <quorum>## Tells Sentinel to monitor this master, and to consider it in O_DOWN# (Objectively Down) state only if at least <quorum> sentinels agree.## Note that whatever is the ODOWN quorum, a Sentinel will require to# be elected by the majority of the known Sentinels in order to# start a failover, so no failover can be performed in minority.## Slaves are auto-discovered, so you don't need to specify slaves in# any way. Sentinel itself will rewrite this configuration file adding# the slaves using additional configuration options.# Also note that the configuration file is rewritten when a# slave is promoted to master.## Note: master name should not include special characters or spaces.# The valid charset is A-z 0-9 and the three characters ".-_".sentinel monitor server-1M 127.0.0.1 6380 1sentinel config-epoch server-1M 4# sentinel auth-pass <master-name> <password>## Set the password to use to authenticate with the master and slaves.# Useful if there is a password set in the Redis instances to monitor.## Note that the master password is also used for slaves, so it is not# possible to set a different password in masters and slaves instances# if you want to be able to monitor these instances with Sentinel.## However you can have Redis instances without the authentication enabled# mixed with Redis instances requiring the authentication (as long as the# password set is the same for all the instances requiring the password) as# the AUTH command will have no effect in Redis instances with authentication# switched off.## Example:## sentinel auth-pass mymaster MySUPER--secret-0123passw0rd# sentinel down-after-milliseconds <master-name> <milliseconds>## Number of milliseconds the master (or any attached slave or sentinel) should# be unreachable (as in, not acceptable reply to PING, continuously, for the# specified period) in order to consider it in S_DOWN state (Subjectively# Down).## Default is 30 seconds.sentinel leader-epoch server-1M 4sentinel known-slave server-1M 127.0.0.1 6379# sentinel parallel-syncs <master-name> <numslaves>## How many slaves we can reconfigure to point to the new slave simultaneously# during the failover. Use a low number if you use the slaves to serve query# to avoid that all the slaves will be unreachable at about the same# time while performing the synchronization with the master.sentinel monitor server-1S 127.0.0.1 6380 1sentinel config-epoch server-1S 0# sentinel failover-timeout <master-name> <milliseconds>## Specifies the failover timeout in milliseconds. It is used in many ways:## - The time needed to re-start a failover after a previous failover was#   already tried against the same master by a given Sentinel, is two#   times the failover timeout.## - The time needed for a slave replicating to a wrong master according#   to a Sentinel current configuration, to be forced to replicate#   with the right master, is exactly the failover timeout (counting since#   the moment a Sentinel detected the misconfiguration).## - The time needed to cancel a failover that is already in progress but#   did not produced any configuration change (SLAVEOF NO ONE yet not#   acknowledged by the promoted slave).## - The maximum time a failover in progress waits for all the slaves to be#   reconfigured as slaves of the new master. However even after this time#   the slaves will be reconfigured by the Sentinels anyway, but not with#   the exact parallel-syncs progression as specified.## Default is 3 minutes.sentinel leader-epoch server-1S 3sentinel current-epoch 4# SCRIPTS EXECUTION## sentinel notification-script and sentinel reconfig-script are used in order# to configure scripts that are called to notify the system administrator# or to reconfigure clients after a failover. The scripts are executed# with the following rules for error handling:## If script exits with "1" the execution is retried later (up to a maximum# number of times currently set to 10).## If script exits with "2" (or an higher value) the script execution is# not retried.## If script terminates because it receives a signal the behavior is the same# as exit code 1.## A script has a maximum running time of 60 seconds. After this limit is# reached the script is terminated with a SIGKILL and the execution retried.# NOTIFICATION SCRIPT## sentinel notification-script <master-name> <script-path>## Call the specified notification script for any sentinel event that is# generated in the WARNING level (for instance -sdown, -odown, and so forth).# This script should notify the system administrator via email, SMS, or any# other messaging system, that there is something wrong with the monitored# Redis systems.## The script is called with just two arguments: the first is the event type# and the second the event description.## The script must exist and be executable in order for sentinel to start if# this option is provided.## Example:## sentinel notification-script mymaster /var/redis/notify.sh# CLIENTS RECONFIGURATION SCRIPT## sentinel client-reconfig-script <master-name> <script-path>## When the master changed because of a failover a script can be called in# order to perform application-specific tasks to notify the clients that the# configuration has changed and the master is at a different address.## The following arguments are passed to the script:## <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>## <state> is currently always "failover"# <role> is either "leader" or "observer"## The arguments from-ip, from-port, to-ip, to-port are used to communicate# the old address of the master and the new address of the elected slave# (now a master).## This script should be resistant to multiple invocations.## Example:## sentinel client-reconfig-script mymaster /var/redis/reconfig.sh

    4）检测redis sentinel的配置是否成功

    [root@esxi3v03 redis-2.8.19]# pwd
/home/bppf_tools/redis/instance-2/redis-2.8.19
[root@esxi3v03 redis-2.8.19]# src/redis-cli -p 6380
127.0.0.1:6380> set name bianji
OK
127.0.0.1:6380> get name
"bianji"
127.0.0.1:6380>

[root@esxi3v03 redis-2.8.19]# pwd
/home/bppf_tools/redis/instance-1/redis-2.8.19
[root@esxi3v03 redis-2.8.19]# src/redis-cli -p 6379
127.0.0.1:6379> set name wenz
(error) READONLY You can't write against a read only slave.
127.0.0.1:6379> get name
"bianji"
127.0.0.1:6379>

[root@esxi3v03 redis-2.8.19]# ps -ef | grep redis
root     49162     1  0 Dec26 ?        00:00:00 src/redis-server *:6380    
root     54809     1  0 10:58 ?        00:00:00 src/redis-server *:26379                 
root     54865     1  0 11:04 ?        00:00:00 src/redis-server *:6379    
root     54929 54623  0 11:17 pts/0    00:00:00 grep redis
[root@esxi3v03 redis-2.8.19]# kill -9 49162

[root@esxi3v03 redis-2.8.19]# pwd
/home/bppf_tools/redis/instance-1/redis-2.8.19
[root@esxi3v03 redis-2.8.19]# src/redis-cli -p 6379
127.0.0.1:6379> set name yuyu
OK
127.0.0.1:6379> get name
"yuyu"
127.0.0.1:6379> quit
[root@esxi3v03 redis-2.8.19]#

[root@esxi3v03 redis-2.8.19]# src/redis-cli -p 6380
127.0.0.1:6380> set name kehao
(error) READONLY You can't write against a read only slave.
127.0.0.1:6380> get name
"yuyu"
127.0.0.1:6380> quit
[root@esxi3v03 redis-2.8.19]#
   可以看出redis-sentinel已经配置成功了！