KeepAlived的高可用实验

Nginx作为七层负载均衡器的高可用实验

为了减少服务器数量，在每个Web Server 各上部署两个基于不同端口的虚拟机，每个Web Server 分别模拟为两台 Web server 。

绘制实验拓扑图

IP分配：

Nginx 1 hostname ：node1

Nginx 2 hostname ：node2

Nginx 1 Address ：172.16.50.1(面向公网)、192.168.50.11（面向后端服务器）

Nginx 2 Address ：172.16.50.2(面向公网)、192.168.50.12（面向后端服务器

Web Server 1 Address ：192.168.50.13

Web Server 2 Address ：192.168.50.14

各节点间的关系如图所示。

准备工作：

安装nginx

同步时间

先配置两台Web Server

• 创建默认页面

mkidr -pv /web/nginx/html/vhost{1,2}echo 'Web Server 1 --> 192.168.50.13:80 ' > /web/nginx/html/vhost1/index.htmlecho 'Web Server 1 --> 192.168.50.13:8080 ' > /web/nginx/html/vhost2/index.html

• 编辑配置文件，创建虚拟机

另一台Web Server与下面的配置相同，仅仅将IP地址修改为192.168.50.14即可

vim /etc/nginx/nginx.conf# 删除原配置文件中的server配置段，自定义两个server配置段如下：server {    listen       80;    server_name  192.168.50.13;    root /web/nginx/html/vhost1;}server {    listen      8080;    server_name 192.168.50.13;    root    /web/nginx/html/vhost2;}

• 启动两台Web Server 的Nginx服务，测试访问所有虚拟机是否可以正常提供服务。

配置Nginx服务器的的反代和负载均衡功能

另一台配置相同，仅需将upstream配置段中的server改为192.168.50.14，将server配置段中的server_name改为172.16.50.2即可。

upstream websrvs {    server 192.168.50.13:80 weight=1;    server 192.168.50.13:8080 weight=1;}server {    listen       80;    server_name  172.16.50.1;    root         /usr/share/nginx/html;    location / {        proxy_pass http://websrvs;    }}

配置高可用

- 注意事项：

HA Cluster的配置前提：

(1) 各节点时间必须同步；

ntp, chrony

(2) 确保iptables及selinux不会成为阻碍；

(3) 各节点之间可通过主机名互相通信（对KA并非必须）；

建议使用/etc/hosts文件实现； 

(4) 确保各节点的用于集群服务的接口支持multicast通信；

D类地址：224~239，注意239开头的网段不可用开启组播的命令：ip link set nulticast on dev ens33

(5) 各节点之间的root用户可以基于密钥认证的ssh服务完成互相通信；（并非必须）

(6) keepalived程序修改配置后如果要重新加载配置，必须先停止keepalived程序再启动，不应使用restart方式重启。

• 安装keepalived程序包

单主模型配置

• 修改配置文件，假设流动VIP为172.16.50.50/16

• 主服务器配置

! Configuration File for keepalivedglobal_defs {   notification_email {                root@localhost     }   notification_email_from keepalived@localhost    smtp_server 127.0.0.1    smtp_connect_timeout 30   router_id node1    vrrp_mcast_group4 224.10.10.10}vrrp_instance VI_1 {    state MASTER    interface ens33    virtual_router_id 10    priority 100    advert_int 1    authentication {        auth_type PASS        auth_pass 70924d6fa    }    virtual_ipaddress {                172.16.50.50/16 dev ens33 label ens33:0    notify_master "/etc/keepalived/mail_from_keepalived.sh master"    notify_backup "/etc/keepalived/mail_from_keepalived.sh backup"    notify_fault "/etc/keepalived/mail_from_keepalived.sh fault"}

• 备服务器配置

! Configuration File for keepalivedglobal_defs {   notification_email {        root@localhost    }   notification_email_from keepalived@localhost   smtp_server 127.0.0.1    smtp_connect_timeout 30   router_id node2   vrrp_mcast_group4 224.10.10.10}vrrp_instance VI_1 {    state BACKUP     interface ens33    virtual_router_id 10    priority 93    advert_int 1    authentication {        auth_type PASS        auth_pass 70924d6fa    }    virtual_ipaddress {        172.16.50.50/16 dev ens33 label ens33:0    notify_master "/etc/keepalived/mail_from_keepalived.sh master"    notify_backup "/etc/keepalived/mail_from_keepalived.sh backup"    notify_fault "/etc/keepalived/mail_from_keepalived.sh fault"}

• 编写通知脚本: mail_from_keepalived.sh

vim mail_from_keepalived.sh#!/bin/bash#contact='root@localhost'notify(){    local mailsubject="$(hostname) to be $1, Virtual_IP floating"    local mailbody="$(date +'%F %T'):Vrrp transition,$(hostname) changed to be '$1'"    echo "$mailbody" | mail -s "$mailsubject" $contact}case $1 inmaster)    notify master    ;;backup)    notify backup    ;;fault)    notify fault    ;;*)    echo "Usage : $(basename $0) {master|bakcup|fault}"    exit 1    ;;esac

• 测试效果：

for ((i=1;i<=10;i++)); do curl 172.16.50.50;done

双主模型

• 修改配置文件，假设流动VIP为172.16.50.99/16

服务器Nginx 1 对50为主，对99为备的配置，在最后附加以下内容：

vrrp_instance VI_2 {    state BACKUP    interface ens33    virtual_router_id 20    priority 92    advert_int 1    authentication {        auth_type PASS        auth_pass 7b9a4d6ft    }    virtual_ipaddress {                172.16.50.99/16 dev ens33 label ens33:1           }}

服务器Nginx 2 对50为备，对99为主的配置，在配置文件最后附加以下内容：

vrrp_instance VI_2 {    state MASTER     interface ens33    virtual_router_id 20    priority 99    advert_int 1    authentication {        auth_type PASS        auth_pass 7b9a4d6ft    }    virtual_ipaddress {        172.16.50.99/16 dev ens33 label ens33:1    }}

• 测试结果：

for ((i=1;i<=6;i++)); do curl 172.16.50.50;done;echo;for ((i=1;i<=6;i++)); do curl 172.16.50.99;done

对nginx进程监控并根据监控结果执行指定动作

• 需要使用killall命令，该命令由程序包 psmisc 提供，CentOS最小化版没有此程序包。

修改通知脚本，增加启动Nginx的功能，详细配置如下：

#!/bin/bash#contact='root@localhost'notify(){        local mailsubject="$(hostname) to be $1, Virtual_IP floating"        local mailbody="$(date +'%F %T'):Vrrp transition,$(hostname) changed to be '$1'"        echo "$mailbody" | mail -s "$mailsubject" $contact}case $1 inmaster)        systemctl start nginx.service        notify master        ;;backup)        systemctl start nginx.service        notify backup        ;;fault)        notify fault        ;;*)        echo "Usage : $(basename $0) {master|bakcup|fault}"        exit 1        ;;esac

Nginx 1 主机的配置文件：

! Configuration File for keepalivedglobal_defs {   notification_email {                root@localhost     }   notification_email_from keepalived@localhost    smtp_server 127.0.0.1    smtp_connect_timeout 30   router_id node1    vrrp_mcast_group4 224.10.10.10}vrrp_script chk_down10 {        script "[[ -f /etc/keepalived/down10 ]] && exit 1 || exit 0"        weight -9        interval 1        fall 1        rise 1}vrrp_script chk_down20 {        script "[[ -f /etc/keepalived/down20 ]] && exit 1 || exit 0"        weight -9        interval 1        fall 1        rise 1}vrrp_script chk_nginx {        script "killall -0 nginx && exit 0 || exit 1"        weight -10        interval 1        fall 2        rise 1}vrrp_instance VI_1 {    state MASTER    interface ens33    virtual_router_id 10    priority 100    advert_int 1    authentication {    auth_type PASS    auth_pass 70924d6fa    }    virtual_ipaddress {                172.16.50.50/16 dev ens33 label ens33:0            }        track_script {                chk_down10        }        track_script {                chk_nginx        }    notify_master "/etc/keepalived/mail_from_keepalived.sh master"    notify_backup "/etc/keepalived/mail_from_keepalived.sh backup"    notify_fault "/etc/keepalived/mail_from_keepalived.sh fault"}vrrp_instance VI_2 {    state BACKUP    interface ens33    virtual_router_id 20    priority 92    advert_int 1    authentication {        auth_type PASS        auth_pass 7b9a4d6ft    }    virtual_ipaddress {                172.16.50.99/16 dev ens33 label ens33:1           }        track_script {                chk_down20        }        track_script {                chk_nginx        }    notify_master "/etc/keepalived/mail_from_keepalived.sh master"    notify_backup "/etc/keepalived/mail_from_keepalived.sh backup"    notify_fault "/etc/keepalived/mail_from_keepalived.sh fault"}

Nginx 2 主机配置文件：

! Configuration File for keepalivedglobal_defs {   notification_email {        root@localhost    }   notification_email_from keepalived@localhost   smtp_server 127.0.0.1    smtp_connect_timeout 30   router_id node2   vrrp_mcast_group4 224.10.10.10}vrrp_script chk_down10 {        script "[[ -f /etc/keepalived/down10 ]] && exit 1 || exit 0"        weight -9        interval 1        fall 1        rise 1}vrrp_script chk_down20 {        script "[[ -f /etc/keepalived/down20 ]] && exit 1 || exit 0"        weight -9        interval 1        fall 1        rise 1}vrrp_script chk_nginx {        script "killall -0 nginx && exit 0 || exit 1"        interval 1        fall 2        rise 1}vrrp_instance VI_1 {    state BACKUP     interface ens33    virtual_router_id 10    priority 93    advert_int 1    authentication {    auth_type PASS    auth_pass 70924d6fa    }    virtual_ipaddress {        172.16.50.50/16 dev ens33 label ens33:0       }        track_script {                chk_down10        }        track_script {                chk_nginx        }    notify_master "/etc/keepalived/mail_from_keepalived.sh master"    notify_backup "/etc/keepalived/mail_from_keepalived.sh backup"    notify_fault "/etc/keepalived/mail_from_keepalived.sh fault"}vrrp_instance VI_2 {    state MASTER     interface ens33    virtual_router_id 20    priority 99    advert_int 1    authentication {    auth_type PASS    auth_pass 7b9a4d6ft    }    virtual_ipaddress {        172.16.50.99/16 dev ens33 label ens33:1    }        track_script {                chk_down20        }        track_script {                chk_nginx        }    notify_master "/etc/keepalived/mail_from_keepalived.sh master"    notify_backup "/etc/keepalived/mail_from_keepalived.sh backup"    notify_fault "/etc/keepalived/mail_from_keepalived.sh fault"}

• 测试：

通过在Nginx 1 主机和Nginx 2 主机 在对应目录下创建down10或down20 文件，操控VIP的优先级、转换主备，配合停止/启动Nginx，会有很多玩法，慢慢尝试吧。

提示：如果手动停止Nginx，此时主备发生转换，Nginx不能自动启动，还原需要手动启动Nginx。

for ((i=1;i<=4;i++)); do curl 172.16.50.50;done;echo;for ((i=1;i<=4;i++)); do curl 172.16.50.99;done