6.1 Keepalived+LVS集群介绍
-
Keepalived和LVS共同构建了一个高效的负载均衡和高可用性解决方案:LVS作为负载均衡器,负责在集群中的多个服务器间分配流量,以其高性能和可扩展性确保应用程序能够处理大量的并发请求;而Keepalived则作为高可用性保证,通过VRRP协议监控LVS服务的状态并在主服务器发生故障时自动进行故障转移,确保服务的持续可用性和无感知切换。这种结合利用了两者的优势,为关键的网络服务提供了一个稳定、可靠且高效的运行环境。
机器名称 | IP地址 | 子网掩码 | 说明 |
---|---|---|---|
Keepalived-01 | 192.168.110.31 | 255.255.255.0 | 负载均衡+高可用 |
Keepalived-02 | 192.168.110.32 | 255.255.255.0 | 负载均衡+高可用 |
RS1 | 192.168.110.33 | 255.255.255.0 | 真实服务器1 |
RS2 | 192.168.110.34 | 255.255.255.0 | 真实服务器2 |
Client | 192.168.110.35 | 255.255.255.0 | 客户端 |
6.2 后端RS配置
-
配置系统服务脚本
[root@LVS-RS1 ~]# vim /etc/init.d/LVS_RS #!/bin/bash # Startup script to handle the initialisation of LVS # chkconfig: - 28 72 # description: Initialise the Linux Virtual Server for DR # Provides: ipvsadm # Required-Start: $local_fs $network $named # Required-Stop: $local_fs $remote_fs $network # Short-Description: Initialise the Linux Virtual Server # Description: The Linux Virtual Server is a highly scalable and highly # available server built on a cluster of real servers, with the load # balancer running on Linux. LOCK=/var/lock/ipvsadm.lock VIP=192.168.110.10 . /etc/rc.d/init.d/functions start() {PID=`ifconfig | grep lo:20 | wc -l`if [ $PID -ne 0 ]; thenecho "The LVS-DR-RIP Server is already running !"else/sbin/ifconfig lo:20 $VIP netmask 255.255.255.255 broadcast $VIP up/sbin/route add -host $VIP dev lo:20echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignoreecho "2" >/proc/sys/net/ipv4/conf/lo/arp_announceecho "1" >/proc/sys/net/ipv4/conf/all/arp_ignoreecho "2" >/proc/sys/net/ipv4/conf/all/arp_announce/bin/touch $LOCKecho "starting LVS-DR-RIP server is ok !"fi } stop() {/sbin/route del -host $VIP dev lo:20/sbin/ifconfig lo:20 down >/dev/nullecho "0" >/proc/sys/net/ipv4/conf/lo/arp_ignoreecho "0" >/proc/sys/net/ipv4/conf/lo/arp_announceecho "0" >/proc/sys/net/ipv4/conf/all/arp_ignoreecho "0" >/proc/sys/net/ipv4/conf/all/arp_announcerm -rf $LOCKecho "stopping LVS-DR-RIP server is ok !" } status() {if [ -e $LOCK ]; thenecho "The LVS-DR-RIP Server is already running !"elseecho "The LVS-DR-RIP Server is not running !"fi } case "$1" instart)start;;stop)stop;;restart)stopstart;;status)status;;*)echo "Usage: $1 {start|stop|restart|status}"exit 1 esac exit 0 [root@LVS-RS1 ~]# chmod +x /etc/init.d/LVS_RS [root@LVS-RS1 ~]# chkconfig --add LVS_RS [root@LVS-RS1 ~]# systemctl start LVS_RS.service [root@LVS-RS1 ~]# scp /etc/init.d/LVS_RS 192.168.110.34:/etc/init.d/ #发给另一台RS [root@LVS-RS2 ~]# chmod +x /etc/init.d/LVS_RS [root@LVS-RS2 ~]# chkconfig --add LVS_RS [root@LVS-RS2 ~]# systemctl start LVS_RS.service
6.3 Keepalived配置
6.3.1 基于TCP的健康检测
6.3.1.1 Keepalived-01配置
[root@Keepalived-01 ~]# cp /etc/keepalived/keepalived.conf{,.bak} [root@Keepalived-01 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs {router_id LVS_node1 } vrrp_instance LVS {state MASTERinterface ens160virtual_router_id 51priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.110.10} } virtual_server 192.168.110.10 80 {delay_loop 6 lb_algo rrlb_kind DRprotocol TCP real_server 192.168.110.33 80 {weight 1TCP_CHECK {connect_timeout 3retry 3delay_before_retry 3connect_port 80}}real_server 192.168.110.34 80 {weight 1TCP_CHECK {connect_timeout 3retry 3delay_before_retry 3connect_port 80}} } [root@Keepalived-01 ~]# systemctl start keepalived.service [root@Keepalived-01 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr-> 192.168.110.33:80 Route 1 0 0 -> 192.168.110.34:80 Route 1 0 0
6.3.1.2 Keepalived-02配置
[root@Keepalived-02 ~]# cp /etc/keepalived/keepalived.conf{,.bak} [root@Keepalived-02 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs {router_id LVS_node2 } vrrp_instance LVS {state MASTERinterface ens160virtual_router_id 51priority 80advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.110.10} } virtual_server 192.168.110.10 80 {delay_loop 6 lb_algo rrlb_kind DRprotocol TCP real_server 192.168.110.33 80 {weight 1TCP_CHECK {connect_timeout 3retry 3delay_before_retry 3connect_port 80}}real_server 192.168.110.34 80 {weight 1TCP_CHECK {connect_timeout 3retry 3delay_before_retry 3connect_port 80}} } [root@Keepalived-02 ~]# systemctl start keepalived.service [root@Keepalived-02 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr-> 192.168.110.33:80 Route 1 0 0 -> 192.168.110.34:80 Route 1 0 0
6.3.1.3 访问测试
[root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
6.3.1.4 模拟故障转移
[root@Keepalived-01 ~]# systemctl stop keepalived.service [root@Keepalived-02 ~]# ip address show ens160 #VIP漂移到Keepalived-02 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:b0:0d:30 brd ff:ff:ff:ff:ff:ffaltname enp3s0inet 192.168.110.32/24 brd 192.168.110.255 scope global noprefixroute ens160valid_lft forever preferred_lft foreverinet 192.168.110.10/32 scope global ens160valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:feb0:d30/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done #访问正常 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 [root@Keepalived-01 ~]# systemctl start keepalived.service [root@Keepalived-01 ~]# ip address show ens160 #恢复后回来 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:d1:a9:eb brd ff:ff:ff:ff:ff:ffaltname enp3s0inet 192.168.110.31/24 brd 192.168.110.255 scope global noprefixroute ens160valid_lft forever preferred_lft foreverinet 192.168.110.10/32 scope global ens160valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:fed1:a9eb/64 scope link noprefixroute valid_lft forever preferred_lft forever
6.3.1.5 后端服务器故障
[root@LVS-RS1 ~]# systemctl stop nginx.service [root@Keepalived-02 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr-> 192.168.110.34:80 Route 1 0 9 [root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done #访问正常 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2
6.3.2 基于HTTP的健康检测
6.3.2.1 生成远程网页的MD5哈希值
-
genhash 是一个命令行工具,用于生成远程网页的MD5哈希值。它可以用于监控HTTP和HTTPS服务,特别是在Keepalived配置中,用于健康检查。genhash 可以通过HTTP或HTTPS连接到网页,并生成页面数据的MD5哈希值,该哈希值可以在Keepalived配置文件中使用。
[root@Keepalived-01 ~]# genhash -s 192.168.110.33 -p 80 -u /index.html MD5SUM = fd0508d1ccc6c66c14977e54ffc7faef [root@Keepalived-01 ~]# genhash -s 192.168.110.34 -p 80 -u /index.html MD5SUM = 0632aaa5fb77608b1a4736d47aacb62c [root@Keepalived-02 ~]# genhash -s 192.168.110.33 -p 80 -u /index.html MD5SUM = fd0508d1ccc6c66c14977e54ffc7faef [root@Keepalived-02 ~]# genhash -s 192.168.110.34 -p 80 -u /index.html MD5SUM = 0632aaa5fb77608b1a4736d47aacb62c
6.3.2.2 Keepalived-01配置
[root@Keepalived-01 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs {router_id LVS_node1 } vrrp_instance LVS {state MASTERinterface ens160virtual_router_id 51priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.110.10} } virtual_server 192.168.110.10 80 {delay_loop 6 lb_algo rrlb_kind DRprotocol TCP real_server 192.168.110.33 80 {weight 1 HTTP_GET {url {path /index.htmldigest fd0508d1ccc6c66c14977e54ffc7faef}connect_timeout 3nb_get_retry 3delay_before_retry 3}}real_server 192.168.110.34 80 {weight 1HTTP_GET {url {path /index.htmldigest 0632aaa5fb77608b1a4736d47aacb62c }connect_timeout 3nb_get_retry 3delay_before_retry 3}} } [root@Keepalived-01 ~]# systemctl start keepalived.service [root@Keepalived-01 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr-> 192.168.110.33:80 Route 1 0 0 -> 192.168.110.34:80 Route 1 0 0
6.3.2.3 Keepalived-02配置
[root@Keepalived-02 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs {router_id LVS_node2 } vrrp_instance LVS {state MASTERinterface ens160virtual_router_id 51priority 80advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.110.10} } virtual_server 192.168.110.10 80 {delay_loop 6 lb_algo rrlb_kind DRprotocol TCP real_server 192.168.110.34 80 {weight 1 HTTP_GET {url {path /index.htmldigest fd0508d1ccc6c66c14977e54ffc7faef}connect_timeout 3nb_get_retry 3delay_before_retry 3}}real_server 192.168.110.34 80 {weight 1HTTP_GET {url {path /index.htmldigest 0632aaa5fb77608b1a4736d47aacb62c }connect_timeout 3nb_get_retry 3delay_before_retry 3}} } [root@Keepalived-02 ~]# systemctl start keepalived.service [root@Keepalived-02 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr-> 192.168.110.33:80 Route 1 0 0 -> 192.168.110.34:80 Route 1 0 0
6.3.2.4 访问测试
[root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
6.3.2.5 模拟故障转移
[root@Keepalived-01 ~]# systemctl stop keepalived.service [root@Keepalived-02 ~]# ip address show ens160 #VIP漂移到Keepalived-02 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:b0:0d:30 brd ff:ff:ff:ff:ff:ffaltname enp3s0inet 192.168.110.32/24 brd 192.168.110.255 scope global noprefixroute ens160valid_lft forever preferred_lft foreverinet 192.168.110.10/32 scope global ens160valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:feb0:d30/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done #访问正常 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 [root@Keepalived-01 ~]# systemctl start keepalived.service [root@Keepalived-01 ~]# ip address show ens160 #恢复后回来 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:d1:a9:eb brd ff:ff:ff:ff:ff:ffaltname enp3s0inet 192.168.110.31/24 brd 192.168.110.255 scope global noprefixroute ens160valid_lft forever preferred_lft foreverinet 192.168.110.10/32 scope global ens160valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:fed1:a9eb/64 scope link noprefixroute valid_lft forever preferred_lft forever
6.3.2.6 后端服务器故障
[root@LVS-RS1 ~]# systemctl stop nginx.service [root@Keepalived-01 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr-> 192.168.110.34:80 Route 1 0 6 [root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2
6.3.3 Keepalived健康检查(MISC方式)
6.3.3.1 编写健康脚本
[root@Keepalived-01 ~]# vim /etc/keepalived/chk_web.sh #!/bin/bash# 检查传入的参数数量是否正确 if [ $# -ne 2 ]; thenecho "Error, Parameter error"exit 1 else# 使用 nmap 进行端口扫描,获取端口状态n=$(nmap -Pn -p$2 -sS -vv $1 | grep "^$2" | awk '{print $2}') # 判断端口是否开放if [ "$n" = "open" ]; thenexit 0elseexit 1fi fi [root@Keepalived-01 ~]# chmod +x /etc/keepalived/chk_web.sh [root@Keepalived-01 ~]# scp /etc/keepalived/chk_web.sh 192.168.110.32:/etc/keepalived/
6.3.3.2 keepalived-01配置
[root@Keepalived-01 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs {router_id LVS_node1 } vrrp_instance LVS {state MASTERinterface ens160virtual_router_id 51priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.110.10} } virtual_server 192.168.110.10 80 {delay_loop 6lb_algo rrlb_kind DRprotocol TCP real_server 192.168.110.33 80 {weight 1MISC_CHECK {misc_path "/etc/keepalived/chk_web.sh 192.168.110.33 80"misc_timeout 3}} real_server 192.168.110.34 80 {weight 1MISC_CHECK {misc_path "/etc/keepalived/chk_web.sh 192.168.110.34 80"misc_timeout 3}} } [root@Keepalived-01 ~]# systemctl start keepalived.service [root@Keepalived-01 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr-> 192.168.110.33:80 Route 1 0 0 -> 192.168.110.34:80 Route 1 0 0
6.3.3.3 keepalived-02配置
[root@Keepalived-02 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs {router_id LVS_node2 } vrrp_instance LVS {state MASTERinterface ens160virtual_router_id 51priority 80advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.110.10} } virtual_server 192.168.110.10 80 {delay_loop 6lb_algo rrlb_kind DRprotocol TCP real_server 192.168.110.33 80 {weight 1MISC_CHECK {misc_path "/etc/keepalived/chk_web.sh 192.168.110.33 80"misc_timeout 3}} real_server 192.168.110.34 80 {weight 1MISC_CHECK {misc_path "/etc/keepalived/chk_web.sh 192.168.110.34 80"misc_timeout 3}} } [root@Keepalived-02 ~]# systemctl start keepalived.service
6.3.3.4 访问测试
[root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
6.3.3.5 模拟故障转移
[root@Keepalived-01 ~]# systemctl stop keepalived.service [root@Keepalived-02 ~]# ip address show ens160 #VIP漂移到Keepalived-02 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:b0:0d:30 brd ff:ff:ff:ff:ff:ffaltname enp3s0inet 192.168.110.32/24 brd 192.168.110.255 scope global noprefixroute ens160valid_lft forever preferred_lft foreverinet 192.168.110.10/32 scope global ens160valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:feb0:d30/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done #访问正常 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 [root@Keepalived-01 ~]# systemctl start keepalived.service [root@Keepalived-01 ~]# ip address show ens160 #恢复后回来 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:d1:a9:eb brd ff:ff:ff:ff:ff:ffaltname enp3s0inet 192.168.110.31/24 brd 192.168.110.255 scope global noprefixroute ens160valid_lft forever preferred_lft foreverinet 192.168.110.10/32 scope global ens160valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:fed1:a9eb/64 scope link noprefixroute valid_lft forever preferred_lft forever
6.3.3.6 后端服务器故障
[root@LVS-RS1 ~]# systemctl stop nginx.service [root@Keepalived-01 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr-> 192.168.110.34:80 Route 1 0 6 [root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2