LVS环境搭建及图形化管理(Pacemaker管理+heartbeat心跳+ipvs(DR)+ldirector健康检查及转发)
来源:互联网 发布:绘制平面图的软件 编辑:程序博客网 时间:2024/05/14 22:12
LVS环境搭建及图形化管理(Pacemaker管理+heartbeat心跳+ipvs(DR)ldirector健康检查及转发)
很久没有正儿八经的写技术博客了,希望各位看官海涵。接下来,直接进入主题。
为了使用一个名叫LCMC(好像叫Linux Cluster Managerment Console)的lvs图形化管理界面,我这次lvs的主角确定为heartbeat(基本停止开发,只进行维护)。之前在公司的生产环境进行试验,所以LCMC可以直接连接公网进行wget和yum安装,所以一切都显得非常便捷,基本一个主备模式的lvs只需要几分钟就轻松搞定。为了防止试验环境对生产网的干扰,只得把主机放到测试环境中去。由于测试环境无法直接访问公网,所以花了一天把所需的安装包都收集好。好了,开始安装。
1.系统安装
操作系统:RHEL5.8 X86_64 版本
内核版本: 2.6.18-308.el5
虚拟IP(VIP) : eth0:192.168.29.100 eth1:xxx.xxx.xxx.45/46
主LVS(ds1.lvs.stefentang): eth0:192.168.29.45 eth1:xxx.xxx.xxx.45
备LVS(ds2.lvs.stefentang): eth0:192.168.29.46 eth1:xxx.xxx.xxx.46
实机1(rs1.lvs.stefentang): eth0:192.168.29.47(45) eth0:无
实机2(rs2.lvs.stefentang): eth0:192.168.29.48(46) eth0:无
3.软件安装
为了便于日后集成到操作系统的安装包中去,都采用rpm包安装(所需安装包稍后上传)
在ds1和ds2上分别安装:
配置实机环路地址 lo:0(/etc/init.d/lvs_rs)
并且修改heartbeat的以下三个参数后,就缩短到17秒,估计也不能改得太小,造成误判。
很久没有正儿八经的写技术博客了,希望各位看官海涵。接下来,直接进入主题。
为了使用一个名叫LCMC(好像叫Linux Cluster Managerment Console)的lvs图形化管理界面,我这次lvs的主角确定为heartbeat(基本停止开发,只进行维护)。之前在公司的生产环境进行试验,所以LCMC可以直接连接公网进行wget和yum安装,所以一切都显得非常便捷,基本一个主备模式的lvs只需要几分钟就轻松搞定。为了防止试验环境对生产网的干扰,只得把主机放到测试环境中去。由于测试环境无法直接访问公网,所以花了一天把所需的安装包都收集好。好了,开始安装。
1.系统安装
操作系统:RHEL5.8 X86_64 版本
内核版本: 2.6.18-308.el5
2.配置
借图一用
虚拟IP(VIP) : eth0:192.168.29.100 eth1:xxx.xxx.xxx.45/46
主LVS(ds1.lvs.stefentang): eth0:192.168.29.45 eth1:xxx.xxx.xxx.45
备LVS(ds2.lvs.stefentang): eth0:192.168.29.46 eth1:xxx.xxx.xxx.46
实机1(rs1.lvs.stefentang): eth0:192.168.29.47(45) eth0:无
实机2(rs2.lvs.stefentang): eth0:192.168.29.48(46) eth0:无
3.软件安装
为了便于日后集成到操作系统的安装包中去,都采用rpm包安装(所需安装包稍后上传)
在ds1和ds2上分别安装:
[root@ds1 ~]# rpm -ivh --force *.rpm
[root@ds2 ~]# rpm -ivh --force *.rpm执行结果:
Preparing... ########################################### [100%] 1:libtool-ltdl ########################################### [ 2%] 2:net-snmp-libs ########################################### [ 4%] 3:libesmtp ########################################### [ 5%] 4:openhpi-libs ########################################### [ 7%] 5:OpenIPMI-libs ########################################### [ 9%] 6:lm_sensors ########################################### [ 11%] 7:libqb ########################################### [ 13%] 8:imake ########################################### [ 14%] 9:drbd83 ########################################### [ 16%] 10:bzip2 ########################################### [ 18%] 11:pacemaker-debuginfo ########################################### [ 20%] 12:openais-debuginfo ########################################### [ 21%] 13:heartbeat-debuginfo ########################################### [ 23%] 14:glib2-devel ########################################### [ 25%] 15:cluster-glue-debuginfo ########################################### [ 27%] 16:autoconf ########################################### [ 29%] 17:zlib-devel ########################################### [ 30%] 18:libxml2-devel ########################################### [ 32%] 19:automake ########################################### [ 34%] 20:libtool-ltdl-devel ########################################### [ 36%] 21:perl-TimeDate ########################################### [ 38%] 22:openib ########################################### [ 39%] 23:libsepol-devel ########################################### [ 41%] 24:libselinux-devel ########################################### [ 43%] 25:libgpg-error-devel ########################################### [ 45%] 26:libgcrypt-devel ########################################### [ 46%] 27:keyutils-libs-devel ########################################### [ 48%] 28:e2fsprogs-devel ########################################### [ 50%] 29:krb5-devel ########################################### [ 52%] 30:openssl-devel ########################################### [ 54%] 31:libesmtp-devel ########################################### [ 55%] 32:libxslt-devel ########################################### [ 57%] 33:libtool ########################################### [ 59%] 34:bzip2-devel ########################################### [ 61%] 35:kmod-drbd83 ########################################### [ 63%] 36:libqb-devel ########################################### [ 64%] 37:resource-agents-debugin########################################### [ 66%] 38:cluster-glue-libs ########################################### [ 68%] 39:cluster-glue ########################################### [ 70%] 40:libibverbs ########################################### [ 71%] 41:resource-agents ########################################### [ 73%] 42:corosync ########################################### [ 75%] 43:librdmacm ########################################### [ 77%] 44:corosynclib ########################################### [ 79%] 45:cluster-glue-libs-devel########################################### [ 80%] 46:libnes ########################################### [ 82%] 47:corosynclib-devel ########################################### [ 84%] 48:heartbeat-libs ########################################### [ 86%] 49:pacemaker-libs ########################################### [ 88%] 50:openaislib ########################################### [ 89%] 51:heartbeat ########################################### [ 91%] 52:openais ########################################### [ 93%] 53:pacemaker ########################################### [ 95%] 54:heartbeat-devel ########################################### [ 96%] 55:openaislib-devel ########################################### [ 98%] 56:pacemaker-libs-devel ########################################### [100%]4.配置HB(/etc/ha.d/ha.cf)可以由LCMC来完成,也可以手动编辑。
#集群中的节点不会自动加入autojoin none#heartbeat会记录debug日志,如果启用use_logd,则此选项会被忽略debugfile /var/log/ha-debug#记录所有non-debug消息,如果启用use_logd,则此选项会被忽略logfile /var/log/ha-log#告诉heartbeat记录那些sysloglogfacility local0#指定两个心跳检测包的时间间隔keepalive 1#多久以后心跳检测决定集群中的node已经挂掉deadtime 30#心跳包检测的延时事件,如果延时,只是往日志中记录warning日志,并不切换服务warntime 10#在heartbeat启动后,在多长时间内宣布node是dead状态,因为有时候系统启动后,网络还需要一段时间才能启动initdead 120#如果udpport指令在bcast ucast指令的前面,则使用哪个端口进行广播,否则使用默认端口udpport 694#设置使用哪个网络接口发送UDP广播包,可以设置多个网络接口bcast eth1 eth1#设置在哪个网络接口进行多播心跳检测mcast eth0 239.192.0.0 694 1 0#设置使用哪个网络接口进行UDP单播心跳检测,在ds2上为10.0.0.45#ucast eth1 10.0.0.46#在主节点的服务恢复后,是否把从节点的服务切换回来auto_failback off#告诉集群中有哪些节点,node名称必须是uname -n显示出来的名称,可以在一个node中设置多个节点,也可以多次设置node,每一个在集群中的node都必须被列出来node ds1.lvs.stefentang ds2.lvs.stefentang#设置ping节点,ping节点用来检测网络连接#ping 10.0.0.254#开启Pacemaker cluster manager,因为历史原因,次选项默认是off,但是应该保持该选项值为respawn。在设置为respawn默认自动使用以下配置#pacemaker respawncrm respawncompression bz2compression_threshold 20traditional_compression on#默认配置文件中下面还有很多选项,由于暂时用不到所以暂时忽略
5.配置/etc/ha.d/authkeys(LCMC完成,authkeys文件属性必须是600,否则日志会报错。)
## generated by drbd-guiauth 11 sha1 9AO5FmCl9RwBZZLIepNERwSbMMsmgeiF
6.配置ldirectord(/etc/ha.d/ldirectord.cf)可以手动编辑,也可以拷贝ldirectord的share目录下的文件
## Sample ldirectord configuration file to configure various virtual services.## Ldirectord will connect to each real server once per second and request# /index.html. If the data returned by the server does not contain the# string "Test Message" then the test fails and the real server will be# taken out of the available pool. The real server will be added back into# the pool once the test succeeds. If all real servers are removed from the# pool then localhost:80 is added to the pool as a fallback measure.# Global Directiveschecktimeout=3checkinterval=1#fallback=127.0.0.1:80autoreload=yeslogfile="/var/log/ldirectord.log"#logfile="local0"emailalert="tanghuifeng@besttone.com.cn"emailalertfreq=3600emailalertstatus=allquiescent=yes# Sample for an http virtual servicevirtual=192.168.29.100:80 real=192.168.29.47(45):80 gate real=192.168.29.48(46):80 gate# real=192.168.6.6:80 gate# fallback=127.0.0.1:80 gate service=http request="health/health.html" receive="running status: available"# virtualhost=some.domain.com.au scheduler=rr #persistent=600 #netmask=255.255.255.255 protocol=tcp checktype=negotiate checkport=80 request="health/health.html" receive="running status: available" virtualhost=ds1/ds2.lvs.stefentang
测试心跳是否正常: tcpdump -i eth1 -n -p udp port 694
7.安装http服务器
lvs配置完后,配置真实服务器[root@rs1 ~]# rpm -ivh /mnt/Server/apr-1.2.7-11.el5_6.5.x86_64.rpm[root@rs1 ~]# rpm -ivh /mnt/Server/postgresql-libs-8.1.23-1.el5_7.3.x86_64.rpm[root@rs1 ~]# rpm -ivh /mnt/Server/apr-util-1.2.7-11.el5_5.2.x86_64.rpm[root@rs1 ~]# rpm -ivh /mnt/Server/mailcap-2.1.23-1.fc6.noarch.rpm[root@rs1 ~]# rpm -ivh /mnt/Server/httpd-2.2.3-63.el5.x86_64.rpm[root@rs1 ~]# service httpd start
http服务80端口根目录添加一个health/health.html,内容为
[47(45)/48(46).DS1/2]running status: available
然后浏览器输入http://RIP/health/health.html ,能正确看到内容。配置实机环路地址 lo:0(/etc/init.d/lvs_rs)
#!/bin/bash#chkconfig: 2345 80 90#description:lvs real server set lo:0 VIP addressVIP=192.168.29.100 #virtual ip case "$1" in start) echo " start LVS of REALServer" /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce ;; stop) /sbin/ifconfig lo:0 down echo "close LVS Directorserver" echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce ;; *) echo "Usage: $0 {start|stop}" exit 1 esac然后
chmod u+x lvs_rs
/etc/inid.d/lvs_rs start
验证ldirectord是否做健康检查可以尝试查看httpd的access日志文件如(访问源从29.45变成29.46期间LVS做了一次切换):
192.168.29.45 - - [08/Aug/2013:07:32:16 +0800] "GET /health/health.html HTTP/1.1" 200 372 "-" "libwww-perl/5.805"192.168.29.45 - - [08/Aug/2013:07:32:17 +0800] "GET /health/health.html HTTP/1.1" 200 372 "-" "libwww-perl/5.805"192.168.29.46 - - [08/Aug/2013:07:33:12 +0800] "GET /health/health.html HTTP/1.1" 200 372 "-" "libwww-perl/5.805"192.168.29.46 - - [08/Aug/2013:07:33:13 +0800] "GET /health/health.html HTTP/1.1" 200 372 "-" "libwww-perl/5.805"
开始怎么也想不通,为啥切换动作差不多花了整整55秒,后来发现DNS配置有问题,干脆把DNS解析注释掉(修改/etc/resolv.conf文件):
192.168.29.45 - - [17/Aug/2013:02:46:37 +0800] "GET /health/health.html HTTP/1.1" 200 364 "-" "libwww-perl/5.805"192.168.29.45 - - [17/Aug/2013:02:46:38 +0800] "GET /health/health.html HTTP/1.1" 200 364 "-" "libwww-perl/5.805"192.168.29.46 - - [17/Aug/2013:02:46:55 +0800] "GET /health/health.html HTTP/1.1" 200 364 "-" "libwww-perl/5.805"192.168.29.46 - - [17/Aug/2013:02:46:59 +0800] "GET /health/health.html HTTP/1.1" 200 364 "-" "libwww-perl/5.805"
并且修改heartbeat的以下三个参数后,就缩短到17秒,估计也不能改得太小,造成误判。
warntime 5deadtime 10initdead 30
接下来都可以在LCMC端操作了,稍后配图。
8.查看配置结果
[root@ds1 ~]# crm_monAttempting connection to the cluster...============Last updated: Thu Aug 8 06:59:53 2013Stack: HeartbeatCurrent DC: ds1.lvs.stefentang (a4f7c50b-823d-4434-bb1d-850aa18918b2) - partition with quorumVersion: 1.0.12-unknown2 Nodes configured, unknown expected votes2 Resources configured.============Online: [ ds1.lvs.stefentang ds2.lvs.stefentang ]res_IPaddr2_VIP (ocf::heartbeat:IPaddr2): Started ds1.lvs.stefentangres_ldirectord_1 (ocf::heartbeat:ldirectord): Started ds1.lvs.stefentang
[root@ds2 ~]# crm_monAttempting connection to the cluster...============Last updated: Thu Aug 8 07:01:15 2013Stack: HeartbeatCurrent DC: ds1.lvs.stefentang (a4f7c50b-823d-4434-bb1d-850aa18918b2) - partition with quorumVersion: 1.0.12-unknown2 Nodes configured, unknown expected votes2 Resources configured.============Online: [ ds1.lvs.stefentang ds2.lvs.stefentang ]res_IPaddr2_VIP (ocf::heartbeat:IPaddr2): Started ds1.lvs.stefentangres_ldirectord_1 (ocf::heartbeat:ldirectord): Started ds1.lvs.stefentang
最后用ab并发1000,10000连接,压了一下,查看了分发情况如下:
IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.29.100:http rr -> 192.168.29.45:http Local 1 3 5071 -> 192.168.29.46:http Route 1 4 4930
Server Software: Apache/2.2.3Server Hostname: 192.168.29.100Server Port: 80Document Path: /is/checkNumber.doDocument Length: 1336 bytesConcurrency Level: 1000Time taken for tests: 68.166 secondsComplete requests: 10000Failed requests: 9883 (Connect: 0, Receive: 0, Length: 9883, Exceptions: 0)Write errors: 0Total transferred: 16318053 bytesHTML transferred: 13448053 bytesRequests per second: 146.70 [#/sec] (mean)Time per request: 6816.590 [ms] (mean)Time per request: 6.817 [ms] (mean, across all concurrent requests)Transfer rate: 233.78 [Kbytes/sec] receivedConnection Times (ms) min mean[+/-sd] median maxConnect: 1 7 42.7 5 3014Processing: 81 6228 1490.0 6244 11462Waiting: 33 3396 2218.5 3159 10526Total: 86 6235 1490.7 6250 11467
心血来潮,测了一下ab的极限,呵呵:
==================================10000=====================================InActConn连接数上不去了,保持在2600左右IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.29.100:http rr -> 192.168.29.45:http Local 1 4996 2563 -> 192.168.29.46:http Route 1 5003 2557==================================20000=====================================ActiveConn连接数上不去了,保持在7500左右,ab压力机报错:apr_socket_connect(): 由于系统缓冲区空间不足或队列已满,不能执行套接字上的操作。(730055)IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.29.100:http rr -> 192.168.29.45:http Local 1 7624 2188 -> 192.168.29.46:http Route 1 7571 2186
- LVS环境搭建及图形化管理(Pacemaker管理+heartbeat心跳+ipvs(DR)+ldirector健康检查及转发)
- LVS环境搭建及图形化管理(Pacemaker管理+heartbeat心跳+ipvs(DR)+ldirector健康检查及转发)
- LVS(一)LVS简介及LVS-DR模式配置搭建
- LVS-DR模式原理及搭建过程
- Mysql cluster+heartbeat+ldirector+lvs 集群解决方案
- dr.elephant 环境搭建及使用详解
- 运维笔记30 (lvs负载均衡的(nat,tun,dr三种模式,ldirectord健康检测,ldirectord结合heartbeat)keepalived+lvs介绍)
- RabbitMQ概念及环境搭建(二)RabbitMQ Broker管理
- LVS+Heartbeat 实施案例及经验分享
- Heartbeat,LVS ,Keepalived,Ldirectord功能及配置
- IPVS-DR+heartbeat+ldirectord实现高可用负载均衡集
- IPVS-DR+heartbeat+ldirectord实现高可用负载均衡集
- Linux集群--简单实现LVS-DNAT及LVS-DR
- lvs-dr实验搭建
- heartbeat(高可用)+LVS(负载均衡+健康检查)服务配置(apache测试)
- 高可用负载均衡集群Heartbeat+Ldirectord+lvs-DR的搭建
- heartbeat集群心跳配置及mgmt调用修改方法
- Centos6.5上安装xen及图形化界面管理
- Struts2基础学习(3)-结果集
- struts1和struts2安全性比较
- README.TXT
- C可变参数函数 实现
- 如何使图片在DIV 中水平垂直居中
- LVS环境搭建及图形化管理(Pacemaker管理+heartbeat心跳+ipvs(DR)+ldirector健康检查及转发)
- hdu-4615 Partition
- 模式识别绪论1:基本概念
- sqlplus命令行工具格式化命令
- 错误argument of type "char *" is incompatible with parameter of type "LPCWSTR"的解决方法
- c# 线程池中的所有线程结束之后执行的事件
- 在treeWidget中加入控件
- ORACLE DATABASE LINK
- 二叉树的探索 poj 2255