ubuntu+haproxy+keepalived主从负载

来源：互联网发布：怎么找网络水军编辑：程序博客网时间：2024/05/01 20:06

一.测试环境：

系统：ubuntu server 12.04

haproxy版本：1.4.24

keepalived版本：keepalived-1.2.7

haporxy01：eth0：172.16.1.36 eth1：192.168.100.36

haporxy02：eth0：172.16.1.37 eth1：192.168.100.37

vip1：172.16.1.30

vip2：172.16.1.31

nignx1 website显示：nginx1

nginx2 website显示：ningx2

nginx3 website显示：nginx3

nginx4 website显示：nginx4

二.网络结构

user

(vip1) | (vip2)

haproxy01-------keepalived-------haproxy02

/ \ / \

--------------------------------------------------------

| nginx1 nginx2 nginx3 nginx4 |

--------------------------------------------------------

三.安装

1.安装keepalvied

主机haproxy01：

wget http://www.keepalived.org/software/keepalived-1.2.7.tar.gz

tar xf keepalived-1.2.7.tar.gz

cd keepalived-1.2.7

./configure --prefix=/usr/local/

提示：

checking for openssl/ssl.h... no

configure: error:

!!! OpenSSL is not properly installed on your system. !!!

!!! Can not include OpenSSL headers files. !!!

安装libssl.dev

apt-get install libssl.dev

继续：

./configure --prefix=/usr/local/

提示：

checking for poptGetContext in -lpopt... no

configure: error: Popt libraries is required

安装libpopt-dev

apt-get install libpopt-dev

继续：

./configure --prefix=/usr/local/

make

make install

2.编辑keepalived.conf

mkdir /etc/keepalived/

vi /etc/keepalived/keepalived.conf

######## keepalived.conf ########

global_defs {

router_id LVS_DEVEL

}

vrrp_instance VI_1 {

state MASTER # haproxy02:BACKUP

interface eth0

virtual_router_id 51

priority 91 # 比haproxy02大

advert_int 1

authentication {

auth_type PASS

auth_pass 123456789

}

virtual_ipaddress {

172.16.1.30

}

vrrp_instance VI_2 {

state BACKUP # haproxy02:MASTER

interface eth0

virtual_router_id 52

priority 90 # 比haproxy02小

advert_int 1

authentication {

auth_type PASS

auth_pass 123456789

}

virtual_ipaddress {

172.16.1.31

}

######## keepalived.conf ########

3.启动keepalived

/usr/local/sbin/keepalvied -f /etc/keepalived/keepalived.conf

4.查看vip

ip addr

显示：

eth0：

inet 172.16.1.30/32 scope global eth0

inet 172.16.1.31/32 scope global eth0 #（haproxy02的keepalived未启动，vip2在haproxy01中，haproxy02中keepalived启动后，vip2自动跳转到haproxy02中eth0上）

主机haproxy02：

keepalived安装同上，keepalived.conf如下：

######## keepalived.conf ########

global_defs {

router_id LVS_DEVEL

}

vrrp_instance VI_1 {

state BACKUP # haproxy01:MASTER

interface eth0

virtual_router_id 51

priority 90 # 比haproxy01小

advert_int 1

authentication {

auth_type PASS

auth_pass 123456789

}

virtual_ipaddress {

172.16.1.30

}

vrrp_instance VI_2 {

state MASTER # haproxy01:BACKUP

interface eth0

virtual_router_id 52

priority 91 # 比haproxy01大

advert_int 1

authentication {

auth_type PASS

auth_pass 123456789

}

virtual_ipaddress {

172.16.1.31

}

######## keepalived.conf ########

启动keepalived

/usr/local/sbin/keepalvied -f /etc/keepalived/keepalived.conf

查看vip：

ip addr

显示：

eth0：

inet 172.16.1.31/32 scope global eth0 # (haproxy中eth0上vip2消失，只有vip1)

停止任意一台keepalived服务，另外一台均可自动生成vip1和vip2，确保2台haproxy高可用性

2.安装haproxy

wget http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.24.tar.gz

tar xf haproxy-1.4.24.tar.gz

cd haproxy-1.4.24

查看ubuntu版本信息：

uname -a

显示：

Linux ubuntu37 3.2.0-51-generic #77-Ubuntu SMP Wed Jul 24 20:18:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

make TARGET=37-ubuntu

make PREFIX=/usr/local/haproxy install

建立haproxy用户

useradd haproxy

2.编辑haproxy.cfg

mkdir /etc/haproxy

vi /etc/haproxy/haproxy.cfg

######## haproxy.cfg ########

global

log 127.0.0.1 local0

maxconn 51200

user haproxy

group haproxy

daemon

defaults

log 127.0.0.1 local3

mode http

option dontlognull

balance roundrobin

retries 2

option redispatch

option abortonclose

maxconn 51200

contimeout 5000

clitimeout 50000

srvtimeout 50000

listen haproxy01

bind 172.16.1.30:80

mode http

option httpclose

option forceclose

option forwardfor

option originalto

option redispatch

balance leastconn

timeout check 5s

stats uri /stats # haproxy负载监控页面例：http://172.16.1.30/stats

stats refresh 15s # 监控页面刷新时间

stats realm baison-test-Haproxy

stats auth admin:admin # 监控页面账号密码

stats hide-version # 隐藏haproxy版本信息

cookie SESSION_COOKIE insert indirect nocache

maxconn 40960

server nginx1 192.168.100.11:80 weight 1 cookie nginx1 check inter 2000 rise 2 fall 3

server nginx2 192.168.100.12:80 weight 1 cookie nginx2 check inter 2000 rise 2 fall 3

listen haproxy02

bind 172.16.1.31:80

mode http

option httpclose

option forceclose

option forwardfor

option originalto

option redispatch

balance leastconn

cookie SESSION_COOKIE insert indirect nocache

maxconn 40960

server nginx3 192.168.100.13:80 weight 1 cookie nginx3 check inter 2000 rise 2 fall 3

server nginx4 192.168.100.14:80 weight 1 cookie nginx4 check inter 2000 rise 2 fall 3

######## haproxy.cfg ########

启动haproxy：

/usr/local/haproxy/sbin/haproxy -f /etc/haproxy/haproxy.cfg

主机haproxy02：

安装同上，haproxy.cfg同上

启动haproxy：

/usr/local/haproxy/sbin/haproxy -f /etc/haproxy/haproxy.cfg

四：测试

通过浏览器访问：

http://172.16.1.30

页面显示：nginx1、nginx2轮询切换

http://172.16.1.31

页面显示：nginx3、nginx4轮询切换

可通过监控页面查看负载情况：

http://172.16.1.30/stats

停止任意一台haproxy服务，站点均不受影响

五：haproxy日志

1.在/etc/rsyslog.conf中：

添加haproxy日志路径:

local0.* /var/log/haproxy.log

local3.* /var/log/haproxy.log

取消注释:

#$ModLoad imudp ==》$ModLoad imudp

#$UDPServerRun 514 ==》$UDPServerRun 514

重启rsyslog服务

service rsyslog restart

2.核实services文件

grep 514 /etc/services

显示：

syslog 514/udp 为ok!!!

如不存在，手动添加：

vi /etc/default/rsyslog

RSYSLOGD_OPTIONS="-r -c 5"

重启syslog服务，

service rsyslog restart

六：优化haproxy服务器

1.内核结果如下：

sysctl -p

net.ipv4.ip_forward = 1

net.ipv4.ip_nonlocal_bind = 1

net.ipv4.tcp_syncookies = 1

net.ipv4.tcp_tw_reuse = 1

net.ipv4.ip_local_port_range = 1024 65535

net.nf_conntrack_max = 1310720

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_fin_timeout = 15

net.core.netdev_max_backlog = 4096

net.core.rmem_max = 16777216

net.core.somaxconn = 1310720

net.core.wmem_max = 16777216

net.ipv4.tcp_max_syn_backlog = 20480

net.ipv4.tcp_max_tw_buckets = 400000

net.ipv4.tcp_no_metrics_save = 1

net.ipv4.tcp_rmem = 4096 87380 16777216

net.ipv4.tcp_syn_retries = 2

net.ipv4.tcp_synack_retries = 2

net.ipv4.tcp_wmem = 4096 65536 16777216

vm.min_free_kbytes = 65536

net.ipv4.tcp_sack = 1

net.ipv4.tcp_timestamps = 1

net.ipv4.tcp_tw_recycle = 1

拷贝至sysctl.conf中

vi /etc/sysctl.conf

2.ulimit数值永久化

执行：

ulimit -SHn

显示：

1024

永久化：

a.在/etc/pam.d/login中，添加：

pam_limits.so (有时候系统默认添加)

查找文件位置：

find / -name pam_limits.so

显示：

/lib/x86_64-linux-gnu/security/pam_limits.so

b.在/etc/security/limits.conf中，添加：

root soft nofile 10240 #实际值

root hard nofile 10240 #实际值

c.修改 /etc/rc.local 添加：

echo 8061540 > /proc/sys/fs/file-max

执行：

echo 8061540 > /proc/sys/fs/file-max

3.nginx服务器内核参数优化

sysctl -p

net.ipv4.tcp_fin_timeout = 30

net.ipv4.tcp_keepalive_intvl = 2

net.ipv4.tcp_keepalive_probes = 2

net.ipv4.tcp_keepalive_time = 120

net.ipv4.tcp_syn_retries = 10

net.ipv4.tcp_sack = 1

net.ipv4.tcp_timestamps = 1

0 0