RHEL6平台keepalived+lvs+iscsi+gfs搭建高可用负载均衡web集群

没啥新技术，基本上是把以前发过的帖子串了一下，练练手。
主要实现的是一个高可用负载均衡web服务器集群，适合lamp架构。
前端使用两台服务器做lvs+keepalived负载调度器，中间可以用N台做apache+php应用服务器，后面使用两台做mysql高可用双机，最后面用了一台虚拟机做文件服务器。
一共开了7个虚拟机。
系统环境：RHEL6.0 kvm虚拟机
lvs调度服务器：192.168.0.1   192.168.0.2
apache服务器:192.168.0.3    192.168.0.4
mysql服务器:192.168.0.7   192.168.0.8
文件服务器:192.168.0.10
-----------------------------------------------------------------------------------------------------------
一、lvs调度服务器
-----------------------------------------------------------------------------------------------------------
Keepalived+lvs
服务器环境:
系统均是:RHEL6.0 内核:2.6.32-71.el6.i686)
Virtual IP:192.168.0.50
Load Balancer:192.168.0.1
Backup:192.168.0.2
Real Server 1:192.168.0.3
Real Server 2:192.168.0.4

主备机上的软件包安装与配置
yum install ipvsadm kernel­-devel -y
wget http://www.keepalived.org/software/keepalived-1.1.20.tar.gz
tar zxf keepalived-1.1.20.tar.gz
cd keepalived­1.1.20
./configure ­­prefix=/usr/local/keepalived ­­with­kernel­dir=/usr/src/kernels/2.6.32-71.el6.i686/
86/

Keepalived configuration
­­­­­­­­­­­­­­­­­­­­­­Keepalived version  : 1.1.17
Compiler: gcc
Compiler flags: ­g ­O2
Extra Lib: ­lpopt ­lssl ­lcrypto
Use IPVS Framework: Yes;注意编译时一定要支持 lvs
IPVS sync daemon support: Yes
Use VRRP Framework: Yes
Use LinkWatch: No
Use Debug flags: No
make
make install

mkdir /etc/keepalived
ln ­-s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
ln ­-s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
ln ­-s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
ln -s /usr/local/keepalived/bin/* /bin/
ln -s /usr/local/keepalived/sbin/* /sbin/

vi /etc/rc.local
modprobe ip_vs

vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
   notification_email {
        [e-mail]#接收警报的 email 地址,可以添加多个
   }
   notification_email_from root@localhost
   smtp_server 127.0.0.1#使用本机转发 email
   smtp_connect_timeout 30
   router_id LVS_DEVEL#load balancer 的标识 ID,用于 email 警报
}
vrrp_instance VI_1 {
    state MASTER#备机改为 BACKUP,此状态是由 priority 的值来决定的,当前
priority 的值小于备机的值,那么将会失去 MASTER 状态
    interface eth0#HA 监测网络接口
    virtual_router_id 50#主、备机的 virtual_router_id 必须相同
    priority 150#主机的优先级,备份机改为 50
    advert_int 1#主备之间的通告间隔秒数
    authentication {
        auth_type PASS#主备切换时的验证
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.0.111#HA 虚拟 ip,可加多个
    }
}
     
     
     
  
virtual_server 192.168.0.111 80 {
    delay_loop 6#每隔 6 秒查询 realserver状态
    lb_algo rr#lvs 调度算法,这里使用轮叫
    lb_kind DR#lvs 负载均衡机制,这里使用直连路由
#    persistence_timeout 50  #同一 IP 的连接 60 秒内被分配到同一台 realserver
    protocol TCP  #用 TCP 协议检查 realserver 状态
    real_server 192.168.0.1 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3  #故障重试秒数
            nb_get_retry 3  #重试延迟
            delay_before_retry 3
        }
    }
    real_server 192.168.0.2 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}


分别在 realserver 上执行以下命令
vi /etc/sysctl.conf
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
sysctl -p
ifconfig eth0:0 192.168.0.200 netmask 255.255.255.255 up
route add -host 192.168.0.200 dev eth0:0

vi /etc/rc.local
ifconfig eth0:0 192.168.0.200 netmask 255.255.255.255 up
route add -host 192.168.0.200 dev eth0:0


echo `hostname` > /var/www/html/index.html
service httpd start
测试:
访问 http://192.168.0.50,看到页面在两个 realserver 上切换表示成功!
你也可以通过 ipvsadm -Lnc 查看详细连接情况!
-------------------------------------------------------------------------------------------------------------------
二、mysql服务器
-------------------------------------------------------------------------------------------------------------------
mysql双主高可用
-------------------------------------
系统环境：
RHEL6.0_I386
VIP 192.168.0.51
real server1 192.167.0.7
real server2 192.168.0.8
-------------------------------------
1.server1和server2安装好mysql并修改配置文件：
yum install mysql-server
vi /etc/my.cnf:
[mysqld]
log-bin=MySQL-bin
server-id=1 ##在server2上配置为server-id=2

-------------------------------------
2.server1和server2相互设置为主从同步，（双主）.
server1:
mysql> grant replication slave on *.* to 'cluster'@'%' identified by 'cluster';
mysql> show master status;
-------------------------
MySQL-bin.000001   236
-------------------------
------------------------------------
server2:
mysql> change master to
    -> master_host='192.168.0.7',
    -> master_user='cluster',
    -> master_password='cluster',
    -> master_log_file='MySQL-bin.000001',
    -> master_log_pos=236;

mysql> start slave;
mysql> show slave status;
在server2和server1上执行相反的操作，使其互为主从.
--------------------------------------
3.安装软件包
yum install gcc popt-devel kernel-devel openssl-devel ipvsadm make
tar xf keepalived-***.tar.gz
cd keepalived-*

./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-71.el6.i686
make && make install

modprobe ip_vs #此处如果系统没有自动加载此模块将导致keepalived启动后无法找到lvs负载均衡协议
mkdir /etc/keepalived/
ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
ln -s /usr/local/keepalived/bin/* /bin/
ln -s /usr/local/keepalived/sbin/* /sbin/
---------------------------------------------------
4.修改server1/server2配置文件
server1:
vi /etc/keepalived/keepalived.conf:


! Configuration File for keepalived

global_defs {
   notification_email {
     [e-mail]
   }
   notification_email_from root@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id MYSQL-HA ##确保和server2相同
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51 ##确保和server2相同，同网内不同集群此项必须不同,否则发生冲突
    priority 100   ##此处server2上设置为50
    advert_int 1
    nopreempt  ##不抢占，只在priority高的server1上设置，server2上此项注释掉
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.0.51
    }
}

virtual_server 192.168.0.51 3306 {
    delay_loop 2
    lb_algo wrr
    lb_kind DR
    persistence_timeout 60
    protocol TCP

    real_server 192.168.0.7 3306 {
##server2上此处改为192.168.0.8(即server2本机ip)
        weight 3
        notify_down /usr/local/keepalived/bin/mysql.sh
            TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
            connect_port 3306
        }
    }
}
--------------------------------
server1和server2上都添加此检测脚本，作用是当mysql停止工作时自动关闭本机的keepalived
从而实现将故障机器踢出（因每台机器上keepalived只添加了本机为realserver）.
vi /usr/local/keepalived/bin/mysql.sh:
#!/bin/sh
pkill keepalived
--------------------------------
vi /etc/rc.local:
modprobe ip_vs ##此模块如果无法自动加载则需手动加载
--------------------------------
server1和server2启动keepalived守护进程.
/etc/init.d/keepalived start

------------------------------------------------------------------------------------------------------------
三、文件服务器
------------------------------------------------------------------------------------------------------------
data server:192.168.0.10
data client1:192.168.0.3
data client2:192.168.0.4
-----------------------------

data server:
yum install luci -y

/etc/init.d/luci start
访问https://192.168.0.10/:8084/ 使用系统帐号密码登录进入创建一个集群并将client1/2加为节点
然后添加一个虚拟fence设备并加入client1 client2


yum install scsi-target-utils -y
chkconfig tgtd on
/etc/init.d/tgtd start
tgtadm --lld iscsi --op new --mode target --tid 1 -T webdata
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/sda
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL

tgtadm --lld iscsi --op show --mode target #验证是否ok
vi /etc/rc.local 将上面的配置命令写入rc.local
tgtadm --lld iscsi --op new --mode target --tid 1 -T webdata
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/sda
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL


data client:
yum install ricci -y
/etc/init.d/ricci start
lvmconf --enable-cluster

yum install iscsi-initiator-utils -y
iscsiadm -m discovery -t sendtargets -p 192.168.0.10
iscsiadm -m node -T webdata -p 192.168.0.10 -l
信息自动保存至配置文件/var/lib/iscsi/nodes/webdata/192.168.0.10,3260,1/default
fdisk -l 查看可看到多了一块/dev/sda硬盘

pvcreate /dev/sda
vgcreate datavg /dev/sda
lvcreate -L 1020M -n lv1 datavg
cman_tool status|grep Name #查看cluster name
Cluster Name: web_cluster
创建gfs文件系统
mkfs.gfs2 -p lock_dlm -t web_cluster:gfs -j 2 /dev/datavg/lv1
#此处-j 2 提供两台client主机连接

mount -t gfs2 /dev/datavg/lv1 /mnt
vi /etc/fstab
/dev/datavg/lv1         /mnt                    gfs2    defaults        0 0
/etc/init.d/gfs2 start  #文件系统自动挂在到/mnt
chkconfig cman on
chkconfig rgmanager on
chkconfig ricci on
chkconfig modclusterd on
chkconfig clvmd on
chkconfig gfs2 on

如果gfs开机时无法识别到而lvscan命令又遇到lv状态为inactive：
lvchange -ay /dev/gfsvg/gfs
-------------------------------------------------------------------------------------
就这样就搞定了。。。。