使用LVS的 NAT 模式实现 3 台RS的轮询访问
- 1.配置 RS(NAT模式)
- 2. 配置 LVS 主机(仅主机、NAT模式)
- 2.1 配置仅主机网卡(192.168.66.150/24 VIP )
- 2.2 配置 NAT 网卡(192.168.88.6/24 DIP)
- 2.3 LVS 规则匹配
- 2.4 配置内核转发参数
- 3. 配置客户端IP(仅主机模式:192.168.66.100/24)
- 报错纠错
- curl报错
- ipvs配置规则报错
使用LVS的 NAT 模式实现 3 台RS的轮询访问
节点规划:
主机 | 角色 | 系统 | 网络 | IP |
---|---|---|---|---|
client | client | redhat 9.5 | 仅主机 | 192.168.66.100/24 |
lvs | lvs | redhat 9.5 | 仅主机 NAT | 192.168.66.150/24 VIP 192.168.88.6/24 DIP |
nginx | rs1 | redhat 9.5 | NAT | 192.168.88.7/24 |
nginx | rs2 | redhat 9.5 | NAT | 192.168.88.17/24 |
nginx | rs3 | redhat 9.5 | NAT | 192.168.88.27/24 |
仅主机网段:192.168.66.0/24
NAT网段:192.168.88.0/24
1.配置 RS(NAT模式)
- 挂载安装nginx
- echo首页内容
echo $(hostname -I) > /usr/share/nginx/html/index.html
- 关闭防火墙、selinux
- 开启 nginx 服务
- curl本机 IP 查看首页内容
- RS1、RS2、RS3同样操作:
[root@rs1 ~]# mount /dev/sr0 /mnt
mount: /mnt: WARNING: source write-protected, mounted read-only.
[root@rs1 ~]# dnf install nginx -y
Updating Subscription Management repositories.
.....
Complete![root@rs1 ~]# echo $(hostname -I) > /usr/share/nginx/html/index.html
[root@rs1 ~]# systemctl stop firewalld
[root@rs1 ~]# setenforce 0
[root@rs1 ~]# systemctl start nginx
[root@rs1 ~]# curl 192.168.88.7
192.168.88.7
2. 配置 LVS 主机(仅主机、NAT模式)
开启主机之前,先编辑虚拟机,设置两张网卡:
保险起见:防止不能上外网,开启虚拟机后,先把 ipvsamd 软件安装上
[root@lvs ~]# mount /dev/sr0 /mnt/
mount: /mnt: WARNING: source write-protected, mounted read-only.
[root@lvs ~]# dnf install ipvsadm -y
- ipvsadm软件包安装目录
[root@lvs ~]# rpm -ql ipvsadm
/etc/sysconfig/ipvsadm-config #配置文件
/usr/lib/.build-id
/usr/lib/.build-id/0b
/usr/lib/.build-id/0b/d10d85dc0121855898c34f27a7730b50772fcc
/usr/lib/systemd/system/ipvsadm.service #服务启动文件
/usr/sbin/ipvsadm #主程序
/usr/sbin/ipvsadm-restore #规则重载工具
/usr/sbin/ipvsadm-save #规则保存工具
/usr/share/doc/ipvsadm
/usr/share/doc/ipvsadm/MAINTAINERS
/usr/share/doc/ipvsadm/README
/usr/share/man/man8/ipvsadm-restore.8.gz
/usr/share/man/man8/ipvsadm-save.8.gz
/usr/share/man/man8/ipvsadm.8.gz
- 命令使用
#管理集群服务规则:ipvsadm -A|E virtual-service [-s scheduler] [-p [timeout]] [-M netmask] [--pe persistence_engine] [-b sched-flags]ipvsadm -D virtual-service #删除ipvsadm -C #清空ipvsadm -R #重载,相当于ipvsadm-restoreipvsadm -S [-n] #保存,相当于ipvsadm-save
#管理集群中的RSipvsadm -a|e virtual-service -r server-address [options]ipvsadm -d virtual-service -r server-addressipvsadm -L|l [virtual-service] [options] #查看ipvsadm -Z [virtual-service] #清空计数器ipvsadm --set tcp tcpfin udpipvsadm --start-daemon {master|backup} [daemon-options]ipvsadm --stop-daemon {master|backup}ipvsadm -h
[root@lvs ~]# nmcli c show
NAME UUID TYPE DEVICE
Wired connection 1 90b171af-f007-36a1-aec7-477b813d9ccd ethernet ens224
ens160 d622d6da-1540-371d-8def-acd3db9bd38d ethernet ens160
lo b5d28f8d-076d-4e67-9fba-afa12cc2e571 loopback lo
# 修改连接名称
[root@lvs ~]# nmcli c modify 'Wired connection 1' connection.id ens224
[root@lvs ~]# nmcli c show
NAME UUID TYPE DEVICE
ens224 90b171af-f007-36a1-aec7-477b813d9ccd ethernet ens224
ens160 d622d6da-1540-371d-8def-acd3db9bd38d ethernet ens160
lo b5d28f8d-076d-4e67-9fba-afa12cc2e571 loopback lo [root@lvs ~]# ip ad
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
#仅主机网卡
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:0c:29:26:e1:1b brd ff:ff:ff:ff:ff:ffaltname enp3s0inet 192.168.66.128/24 brd 192.168.66.255 scope global dynamic noprefixroute ens160valid_lft 1673sec preferred_lft 1673secinet6 fe80::20c:29ff:fe26:e11b/64 scope link noprefixroute valid_lft forever preferred_lft forever
#NAT网卡
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:26:e1:25 brd ff:ff:ff:ff:ff:ffaltname enp19s0inet 192.168.88.149/24 brd 192.168.88.255 scope global dynamic noprefixroute ens224valid_lft 1673sec preferred_lft 1673secinet6 fe80::54c2:3ed3:5085:89a0/64 scope link noprefixroute valid_lft forever preferred_lft forever
2.1 配置仅主机网卡(192.168.66.150/24 VIP )
[root@lvs ~]# nmcli c modify ens160 ipv4.addresses 192.168.66.150/24 ipv4.gateway 192.168.66.2 ipv4.method manual connection.autoconnect yes
[root@lvs ~]# nmcli c up ens160
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)
[root@lvs ~]# ip ad
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:26:e1:1b brd ff:ff:ff:ff:ff:ffaltname enp3s0inet 192.168.66.150/24 brd 192.168.66.255 scope global noprefixroute ens160valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:fe26:e11b/64 scope link noprefixroute valid_lft forever preferred_lft forever
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:26:e1:25 brd ff:ff:ff:ff:ff:ffaltname enp19s0inet 192.168.88.149/24 brd 192.168.88.255 scope global dynamic noprefixroute ens224valid_lft 1272sec preferred_lft 1272secinet6 fe80::54c2:3ed3:5085:89a0/64 scope link noprefixroute valid_lft forever preferred_lft forever
2.2 配置 NAT 网卡(192.168.88.6/24 DIP)
[root@lvs ~]# nmcli c modify ens224 ipv4.addresses 192.168.88.6/24 ipv4.gateway 192.168.88.2 ipv4.method manual connection.autoconnect yes
[root@lvs ~]# nmcli c up ens224
[root@lvs ~]# ip ad
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:26:e1:1b brd ff:ff:ff:ff:ff:ffaltname enp3s0inet 192.168.66.150/24 brd 192.168.66.255 scope global noprefixroute ens160valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:fe26:e11b/64 scope link noprefixroute valid_lft forever preferred_lft forever
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:26:e1:25 brd ff:ff:ff:ff:ff:ffaltname enp19s0inet 192.168.88.6/24 brd 192.168.88.255 scope global noprefixroute ens224valid_lft forever preferred_lft foreverinet6 fe80::54c2:3ed3:5085:89a0/64 scope link noprefixroute valid_lft forever preferred_lft forever
2.3 LVS 规则匹配
- 安装 ipvsadm(在配置网卡之前)
保险起见:防止不能上外网,先把 ipvsamd 软件安装上 - 启动 ipvsadm(lvs 服务器中已可以访问RS主机,客户端暂不能通过访问 VIP 访问到后端的 RS 服务器)
ipvsadm-save > /etc/sysconfig/ipvsadm
systemctl start ipvsadm
- LVS规则匹配(LVS主机的仅主机VIP-IP,对应RS主机NAT-IP)
ipvsadm -A -t 192.168.66.150:80 -s rr
ipvsadm -a -t 192.168.66.150:80 -r 192.168.88.7:80 -m -w 2
ipvsadm -a -t 192.168.66.150:80 -r 192.168.88.17:80 -m -w 2
ipvsadm -a -t 192.168.66.150:80 -r 192.168.88.27:80 -m -w 2
- 重启ipvsadm服务
systemctl restart ipvsadm
# 启动 ipvsadm
[root@lvs ~]# ipvsadm-save > /etc/sysconfig/ipvsadm
[root@lvs ~]# systemctl start ipvsadm
#lvs 服务器中已可以访问RS主机,客户端暂不能通过访问 VIP 访问到后端的 RS 服务器(原因是没有做 LVS 规则匹配)
[root@lvs ~]# curl 192.168.88.7
192.168.88.7
[root@lvs ~]# curl 192.168.88.17
192.168.88.17
[root@lvs ~]# curl 192.168.88.27
192.168.88.27# LVS规则匹配
#配置一条规则
[root@lvs ~]# ipvsadm -A -t 192.168.66.150:80 -s rr
-A:指定规则
-t:指定虚拟的主机的地址
-s:算法 rr#为规则增加RS
[root@lvs ~]# ipvsadm -a -t 192.168.66.150:80 -r 192.168.88.7:80 -m -w 2
[root@lvs ~]# ipvsadm -a -t 192.168.66.150:80 -r 192.168.88.17:80 -m -w 2
[root@lvs ~]# ipvsadm -a -t 192.168.66.150:80 -r 192.168.88.27:80 -m -w 2
-a:虚拟的服务器地址
-r:真实的服务器地址
-m:NAT模式-Masq(不指定即为DR模式-route)
-W:权重root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.66.150:80 rr-> 192.168.88.7:80 Masq 2 0 0 -> 192.168.88.17:80 Masq 2 0 0 -> 192.168.88.27:80 Masq 2 0 0# 重启ipvsadm
[root@lvs ~]# systemctl restart ipvsadm
重启好后,可通过客户端来测试访问 VIP 能否访问到后端的 RS 服务器,此时不报错,但是没有数据返回。原因是需要配置内核转发参数
2.4 配置内核转发参数
- 配置内核转发参数
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
或者
vim /etc/sysctl.conf
net.ipv4.ip_forward=1
- 执行命令来生效。
sysctl -p
- 重启ipvsadm服务
systemctl restart ipvsadm
# 配置内核转发参数
[root@lvs ~]# echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf# 执行命令来生效
[root@lvs ~]# sysctl -p
net.ipv4.ip_forward = 1# 重启ipvsadm服务
[root@lvs ~]# systemctl restart ipvsadm
3. 配置客户端IP(仅主机模式:192.168.66.100/24)
[root@client ~]# nmcli c modify ens160 ipv4.addresses 192.168.66.100/24 ipv4.gateway 192.168.66.2 connection.autoconnect yes
[root@client ~]# nmcli c up ens160
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/3)
[root@client ~]# ip ad
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:d8:fd:b7 brd ff:ff:ff:ff:ff:ffaltname enp3s0inet 192.168.66.100/24 brd 192.168.66.255 scope global noprefixroute ens160valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:fed8:fdb7/64 scope link noprefixroute valid_lft forever preferred_lft forever
客户端通过 VIP 成功 经过 LVS 主机访问到 RS 主机
[root@client ~]# curl 192.168.66.150
192.168.88.7
[root@client ~]# curl 192.168.66.150
192.168.88.17
[root@client ~]# curl 192.168.66.150
192.168.88.27
报错纠错
curl报错
- RS主机配置nginx后访问本机IP报错
- nignx服务未启动
systemctl start nginx - 防火墙/selinux
setenforce 0
systemctl stop firewalld
- nignx服务未启动
[root@rs1 ~]# curl 192.168.88.7
curl: (7) Failed to connect to 192.168.88.7 port 80: Connection refused
- 客户端主机curl访问IP报错
[root@client ~]# curl 192.168.66.150
curl: (7) Failed to connect to 192.168.66.150 port 80: No route to host
#lvs防火墙未关
[root@lvs ~]# systemctl stop firewalld
新的报错:
[root@client ~]# curl 192.168.66.150
curl: (7) Failed to connect to 192.168.66.150 port 80: Connection refused
ipvs配置规则报错
- 在网络配置中指定了一个无效的虚拟服务器地址和端口
[root@lvs ~]# ipvsadm -a -t 192.168.66.150:80 -r 1923168.88.7:80 -m -w 2
illegal real server address[:port] specified[root@lvs ~]# ipvsadm -a -t 192.168.66.150:80 -r 192.168.88.7:80 -m -w 2
[root@lvs ~]# ipvsadm -a -t 192.168.66.150:80 -r 192.168.88.17:80 -m -w 2
[root@lvs ~]# ipvsadm -a -t 192.168.66.150:80 -r 192.168.88.27:80 -m -w 2