keepalived+lvs

###############keepalived+lvs#############

一.知识梳理
LVS+keepalived基于完成开源软件的构架实现负载均衡高可用
1.LVS
LVS是Linux Virtual Server的简写，即Linux虚拟服务器，是一个虚拟的服务集群系统。有三种负载均衡技术（VS/NAT、VS/TUN、VS/DR），八种调度算法。

2.keeplived
keepalived在这里主要做realserver健康检查以及loadbalance主机和backup主机之间failover的实现。
keepalived是一个基于VRRP协议来实现WEB服务高可用方案，可以用来避免单点故障，一个web服务器至少有两台服务器运行keepalived，一台主服务器（MASTER），一台备用服务器（BACKUP），但是对外只表现一个vip，主服务器会发送特定消息给备用服务器，当备用服务器收到消息时，备用服务器就会接管ipswich，继续提供服务，从而保证了高可用性。
3.kepalive工作原理
Layer3:keepalived使用layer3的方式工作时，keepalived会定期向服务器群中发送一个ICMP的数据包（即我们平时用的ping程序），如果发现某台服务器的IP地址没有激活，keepalived便会报告这台服务器是小，并将他从服务器群中剔除。Layer3的方式是以服务器的IP第孩子是否有效作为服务器工作正常与否的标准。

Layer4:主要以TCP端口的状态来决定服务器工作正常与否。如web sercer的服务端口一般是80.如果keepalived检测到80端口没有启动，则keepalived将这台服务器从服务群中删除。

Layer5:layer5就是工作载具体的应用层，比layer3,4要复杂一点，载网络上占用的宽带也要打一些。Keepalived将根据用户的设定检查服务器的运行是否正常。如果设定不相符，则keepalived将把服务器从群中踢除。



二.实现
1.lvs配置
server1
[root@server1 ha.d]# yum install -y ipvsadm        
[root@server1 ha.d]# ipvsadm -A -t 172.25.38.100:80 -s rr    
##添加策略 指定vip 172.25.66.100:80，-t指tcp，-s rr 指定轮询
[root@server1 ha.d]# ip addr add 172.25.66.100/24 dev eth0    ##添加虚拟ip
[root@server1 ha.d]# ipvsadm -L                    ##列出ipvsadm策略
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.25.66.100:http rr

[root@server1 ha.d]# ipvsadm -a -t 172.25.66.100:80 -r 172.25.66.2:80 -g    
[root@server1 ha.d]# ipvsadm -a -t 172.25.66.100:80 -r 172.25.66.3:80 -g
##将虚拟ip映射到真实的ip，-r指定realserver -t指定tcp协议 -g指dr模式
[root@server1 ha.d]# ipvsadm -L                    ##查看策略已加入
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.25.66.100:http rr
  -> server2:http                 Route   1      0          0         
  -> server3:http                 Route   1      0          0   
[root@server1 ha.d]# /etc/init.d/ipvsadm save            ##保存策略，不保存的话关机策略即失效
ipvsadm: Saving IPVS table to /etc/sysconfig/ipvsadm:      [  OK  ]
[root@server1 ha.d]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.25.66.100:80 rr
  -> 172.25.66.2:80               Route   1      0          0         
  -> 172.25.66.3:80               Route   1      0          0         


server2
[root@server2 ~]# ip addr add 172.25.66.100/32 dev eth0        ##在realserver上加一个和vip相同的ip地址
[root@server2 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:64:ed:04 brd ff:ff:ff:ff:ff:ff
    inet 172.25.66.2/24 brd 172.25.66.255 scope global eth0
    inet 172.25.66.100/32 scope global eth0
    inet6 fe80::5054:ff:fe64:ed04/64 scope link
       valid_lft forever preferred_lft forever
[root@server2 ~]#  yum install -y arptables_jf

为防止客户端在访问vip时直接访问到realserver，在realserver设置一定的策略
[root@server2 ~]# arptables -A IN -d 172.25.66.100 -j DROP
##访问realserver的100ip时的请求直接丢弃
[root@server2 ~]# arptables -A OUT -s 172.25.66.100 -j mangle --mangle-ip-s 172.25.66.2
##realserver发出的信息全部转为他的真实ip
[root@server2 ~]# /etc/init.d/arptables_jf save            ##保存策略
Saving current rules to /etc/sysconfig/arptables:          [  OK  ]
[root@server2 ~]# /etc/init.d/httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 172.25.6.2 for ServerName
                                                           [  OK  ]


server3（两个realserver做同样的配置）
[root@server3 ~]# ip addr add 172.25.66.100/32 dev eth0
[root@server3 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:c7:a3:48 brd ff:ff:ff:ff:ff:ff
    inet 172.25.66.3/24 brd 172.25.66.255 scope global eth0
    inet 172.25.66.100/32 scope global eth0
    inet6 fe80::5054:ff:fec7:a348/64 scope link
       valid_lft forever preferred_lft forever
[root@server3 ~]# yum install -y arptables_jf
[root@server3 ~]# arptables -A IN -d 172.25.66.100 -j DROP
[root@server3 ~]# arptables -A OUT -s 172.25.66.100 -j mangle --mangle-ip-s 172.25.66.3
[root@server3 ~]# /etc/init.d/arptables_jf save
Saving current rules to /etc/sysconfig/arptables:          [  OK  ]
[root@server3 ~]# /etc/init.d/httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 172.25.6.3 for ServerName
                                                          [  OK  ]

2.keeplived配置
[root@server1 new]# tar zxf keepalived-1.2.20.tar.gz
[root@server1 new]# cd keepalived-1.2.20
[root@server1 keepalived-1.2.20]# yum install -y openssl-devel             ##解决依赖性
[root@server1 keepalived-1.2.20]# ./configure --prefix=/usr/local/keepalived    ##源码编译安装

[root@server1 keepalived-1.2.20]# make && make instal

l

[root@server1 keepalived]# ln -s /usr/local/keepalived/sbin/keepalived  /sbin
[root@server1 keepalived]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived  /etc/sysconfig/
[root@server1 keepalived]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived  /etc/init.d/
[root@server1 keepalived]# ln -s /usr/local/keepalived/etc/keepalived/ /etc/
[root@server1 keepalived]# cd /etc/keepalived/
[root@server1 keepalived]# ls
keepalived.conf  samples
[root@server1 keepalived]# ll /etc/init.d/keepalived
lrwxrwxrwx 1 root root 48 Jul 26 13:30 /etc/init.d/keepalived -> /usr/local/keepalived/etc/rc.d/init.d/keepalived
[root@server1 keepalived]# vim keepalived.conf                     ##编辑主配置文件
  1 ! Configuration File for keepalived
  2
  3 global_defs {
  4    notification_email {
  5      root@localhost                ##设置报警邮件地址，如果要开启邮件报警，要开启本机的sendmail服务
  6         }
  7    notification_email_from keepalived@server1##设置邮件的发送地址
  8    smtp_server 127.0.0.1            ##设置smpt server邮件服务端的地址
  9    smtp_connect_timeout 30             ##设置连接smptserver的超时时间
 10    router_id LVS_DEVEL                
 11    vrrp_skip_check_adv_addr
 12    vrrp_strict
 13 }
 14
 15 vrrp_instance VI_1 {
 16     state MASTER            ##指定keepalived的角色，MASTER表示此主机为主服务器，若为BACKUP则表示备用服务器
 17     interface eth0                ##指定HA检测网络的接口
 18     virtual_router_id 51            ##虚拟路由标识，MASTER和BACKUP的必须相同
 19     priority 100            ##定义优先级，数字越大优先级越高，因此MASTER的必须年比BACKUP的数字大，否则就会有冲突
 20     advert_int 1                ##设定MASTER和BACKUP负载均衡之间同步检查的时间间隔，单位为秒
 21     authentication {            ##设定验证类型和密码
 22         auth_type PASS
 23         auth_pass 1111
 24     }
 25     virtual_ipaddress {            ##指定虚拟ip
 26         172.25.66.100
 27         }
 28 }
 29 ##虚拟服务器定义部分
 30 virtual_server 172.25.66.100 80 {        ##设定虚拟服务器ip地址及端口，两者以空格隔开
 31     delay_loop 6                ##设置运行情况检查时间，单位为秒
 32     lb_algo rr                ##设置负载均衡算法，这里设置为rr算法，即轮询算法
 33     lb_kind DR                ##设置lvs实现负载均衡机制，有NAT，TUN，DR三种模式
 34     #persistence_timeout 50            ##会话保留时间，单位为秒
 35     protocol TCP                ##指定转发协议类型
 36
 37
 38
 39      real_server 172.25.66.2 80 {        ##配置realserver，需要指定真实ip与其端口，用空格隔开
 40          weight 1                ##配置服务节点的权重值
 41          TCP_CHECK {            ##realserver检查状态设置部分
 42              connect_timeout 3        ##3s无响应时超时
 43              nb_get_retry 3            ##重复次数
 44              delay_before_retry 3        ##重复间隔
 45                  }
 46          }
 47      real_server 172.25.66.3 80 {
 48          weight 1
 49          TCP_CHECK {
 50              connect_timeout 3
 51              nb_get_retry 3
 52              delay_before_retry 3
 53                  }
 54   }
 55 }



test2
[root@server1 local]# scp -r keepalived/ test2:/usr/local/
[root@test2 local]# ls
bin  etc  games  include  keepalived  lib  lib64  libexec  sbin  share  src
[root@test2 local]# cd keepalived/
[root@test2 keepalived]# ls
bin  etc  sbin  share
[root@test2 keepalived]# ln -s /usr/local/keepalived/sbin/keepalived /sbin/
[root@test2 keepalived]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived  /etc/init.d/
[root@test2 keepalived]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived  /etc/sysconfig/
[root@test2 ~]# mkdir /etc/keepalived
[root@server1 keepalived]# scp keepalived.conf 172.25.66.11:/etc/keepalived/
[root@test2 keepalived]# cd /etc/keepalived/
[root@test2 keepalived]# ls
keepalived.conf
[root@test2 keepalived]# vim keepalived.conf
1 ! Configuration File for keepalived
  2
  3 global_defs {
  4    notification_email {
  5      root@localhost
  6         }
  7    notification_email_from keepalived@test2
  8    smtp_server 127.0.0.1
  9    smtp_connect_timeout 30
 10    router_id LVS_DEVEL
 11    vrrp_skip_check_adv_addr
 12    vrrp_strict
 13 }
 14
 15 vrrp_instance VI_1 {
 16     state BACKUP                ##备用服务器
 17     interface eth0
 18     virtual_router_id 51
 19     priority 50                ##优先级必须低于主服务器
 20     advert_int 1
 21     authentication {
 22         auth_type PASS
 23         auth_pass 1111
 24     }
 25     virtual_ipaddress {
 26         172.25.66.100
 27         }
 28 }
 29
 30 virtual_server 172.25.66.100 80 {
 31     delay_loop 6
 32     lb_algo rr
 33     lb_kind DR
 34     #persistence_timeout 50
 35     protocol TCP
 36
 37
 38
 39      real_server 172.25.66.2 80 {
 40          weight 1
 41          TCP_CHECK {
 42              connect_timeout 3
 43              nb_get_retry 3
 44              delay_before_retry 3
 45                  }
 46          }
 47      real_server 172.25.66.3 80 {
 48          weight 1
 49          TCP_CHECK {
 50              connect_timeout 3
 51              nb_get_retry 3
 52              delay_before_retry 3
 53                  }
 54   }
 55 }


[root@server1 keepalived]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]
[root@server1 keepalived]# tail -f /var/log/messages
Jul 26 13:46:25 server1 Keepalived_healthcheckers[24222]: Removing service [172.25.66.3]:80 from VS [172.25.66.100]:80
Jul 26 13:46:25 server1 Keepalived_healthcheckers[24222]: Remote SMTP server [127.0.0.1]:25 connected.
Jul 26 13:46:25 server1 Keepalived_healthcheckers[24222]: SMTP alert successfully sent.
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: TCP connection to [172.25.66.2]:80 failed.
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: Check on service [172.25.66.2]:80 failed after 1 retry.
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: Removing service [172.25.66.2]:80 from VS [172.25.66.100]:80
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: Lost quorum 1-0=1 > 0 for VS [172.25.66.100]:80
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: Remote SMTP server [127.0.0.1]:25 connected.
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: SMTP alert successfully sent.
Jul 26 13:46:28 server1 Keepalived_vrrp[24223]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.25.66.100
[root@server1 keepalived]# ip addr                    ##接管vip
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:c1:37:57 brd ff:ff:ff:ff:ff:ff
    inet 172.25.66.1/24 brd 172.25.66.255 scope global eth0
    inet 172.25.66.100/32 scope global eth0
    inet6 fe80::5054:ff:fec1:3757/64 scope link
       valid_lft forever preferred_lft forever
[root@server1 keepalived]# ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.25.66.100:http rr
  -> server2:http                 Route   1      0          3         

  -> server3:http                 Route   1      0          4

    

[root@test2 keepalived]# /etc/init.d/keepalived startStarting keepalived:                                       [  OK  ]



测试：
server2和server3实现lvs
server1和test2实现keepalived
1）当server1和test2的keepalived服务都正常时,server1的keepalived工作（默认master工作，这里server1被设置为master）

[kiosk@foundation6 Desktop]$ curl 172.25.66.100
<h1>server2-www.westos.org</h1>
[kiosk@foundation6 Desktop]$ curl 172.25.66.100
<h1>server3-www.westos.org</h1>
[kiosk@foundation6 Desktop]$ arp -an | grep 100
? (172.25.66.100) at 52:54:00:c1:37:57 [ether] on br0
[root@server1 local]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:c1:37:57 brd ff:ff:ff:ff:ff:ff
    inet 172.25.66.1/24 brd 172.25.66.255 scope global eth0
    inet 172.25.66.100/32 scope global eth0
    inet6 fe80::5054:ff:fec1:3757/64 scope link
       valid_lft forever preferred_lft forever

2）当master即server1挂掉时，test2的keepalived工作

[root@server1 local]# /etc/init.d/keepalived stop
Stopping keepalived:                                       [  OK  ]
[kiosk@foundation6 Desktop]$ curl 172.25.66.100
<h1>server3-www.westos.org</h1>
[kiosk@foundation6 Desktop]$ arp -an | grep 100
? (172.25.66.100) at 52:54:00:7a:98:49 [ether] on br0
[root@test2 keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:7a:98:49 brd ff:ff:ff:ff:ff:ff
    inet 172.25.66.11/24 brd 172.25.66.255 scope global eth0
    inet 172.25.66.100/32 scope global eth0
    inet6 fe80::5054:ff:fe7a:9849/64 scope link
       valid_lft forever preferred_lft forever





keepalived+ftp
[root@server1 ~]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]
[root@server1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:c1:37:57 brd ff:ff:ff:ff:ff:ff
    inet 172.25.66.1/24 brd 172.25.66.255 scope global eth0
    inet 172.25.66.100/32 scope global eth0
    inet6 fe80::5054:ff:fec1:3757/64 scope link
       valid_lft forever preferred_lft forever
[root@server1 ~]#  ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.25.66.100:http rr
  -> server2:http                 Route   1      0          1         
  -> server3:http                 Route   1      0          1         

[root@server1 ~]# vim /etc/keepalived/keepalived.conf
 57 virtual_server 172.25.66.100 21 {
 58     delay_loop 6
 59     lb_algo rr
 60     lb_kind DR
 61     persistence_timeout 50
 62     protocol TCP
 63
 64
 65
 66      real_server 172.25.66.2 21 {
 67          weight 1
 68          TCP_CHECK {
 69              connect_timeout 3
 70              nb_get_retry 3
 71              delay_before_retry 3
 72                  }
 73          }
 74      real_server 172.25.66.3 21 {
 75          weight 1
 76          TCP_CHECK {
 77              connect_timeout 3
 78              nb_get_retry 3
 79              delay_before_retry 3
 80                  }
 81   }
 82 }
##这里我们主要强调persistence_timeout 50这个参数，persistence_timeout会话保持时间，单位是秒。这个选项对动态网站很有用处：当用户从远程用帐号进行登陆网站时，有了这个会话保持功能，就能把用户的请求转发给同一个应用服务器。在这里，我们来做一个假设，假定现在有一个lvs 环境，使用DR转发模式，真实服务器有3个，负载均衡器不启用会话保持功能。当用户第一次访问的时候，他的访问请求被负载均衡器转给某个真实服务器，这样他看到一个登陆页面，第一次访问完毕；接着他在登陆框填写用户名和密码，然后提交；这时候，问题就可能出现了---登陆不能成功。因为没有会话保持，负载均衡器可能会把第2次的请求转发到其他的服务器，因此在配置有些需要保持绘画的服务时，这里必须做相应的时常配置，否则会影响业务或数据处理，
[root@server1 ~]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.25.66.100:ftp rr persistent 50
  -> server2:ftp                  Route   1      0          0         
  -> server3:ftp                  Route   1      0          0         
TCP  172.25.66.100:http rr
  -> server2:http                 Route   1      0          0         
  -> server3:http                 Route   1      0          0         


[root@server2 ~]# yum install y vsftpd
[root@server2 ~]# /etc/init.d/vsftpd start
Starting vsftpd for vsftpd:   
[root@server2 ~]# touch  /var/ftp/serve2

[root@server3 ~]# yum install -y vsftpd
[root@server3 ~]# /etc/init.d/vsftpd start
Starting vsftpd for vsftpd:                                [  OK  ]
[root@server3 ~]# touch   /var/ftp/server3


停止任意一台真机服务，不会影响虚拟FTP服务，虚拟FTP服务会自动连上可以提供服务的FTP服务器，如果所有的FTP真机服务都宕机了，则虚拟FTP也将无法访问。当真机FTP服务恢复正常后，虚拟FTP将自动添加真机服务至虚拟队列中，无人值守。