【Docker】Swarm的ingress网络

Docker Swarm Ingress网络是Docker集群中的一种网络模式，它允许在Swarm集群中运行的服务通过一个公共的入口点进行访问。Ingress网络将外部流量路由到Swarm集群中的适当服务，并提供负载均衡和服务发现功能。

在Docker Swarm中，Ingress网络使用了一种称为"Routing Mesh"的技术。Routing Mesh通过在Swarm集群的每个节点上创建一组代理来实现负载均衡和服务发现。这些代理将外部流量路由到适当的服务，并自动处理服务的扩展和缩减。

ingress routing mesh是docker swarm网络里最复杂的一部分内容，包括多方面的内容：

iptables的Destination NAT流量转发
Linux bridge, network namespace
使用IPVS技术做负载均衡
包括容器间的通信（overlay）和入方向流量的端口转发

主机规划：

node1：172.19.177.14，角色为Leader
node2：172.19.188.123，角色为Worker

$ sudo docker node ls
ID                            HOSTNAME   STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
u36el9nkyamqgihcwj5yk4cwe *   node1      Ready     Active         Leader           24.0.2
0e8q2v9glvt56jbk9mpbkhc9l     node2      Ready     Active                          24.0.2

service创建

创建一个service，名为web，通过-p把端口映射出来：

$ sudo docker service create --name web -p 8080:80 --replicas 2 containous/whoami
j2bzt3mi7yedm4um6g5o96ndd
overall progress: 2 out of 2 tasks
1/2: running   [==================================================>]
2/2: running   [==================================================>]

我们使用的镜像containous/whoami是一个简单的web服务，能返回服务器的hostname，和基本的网络信息，比如IP地址。

查询service：

$ sudo docker service ps web
ID             NAME      IMAGE                      NODE      DESIRED STATE   CURRENT STATE            ERROR     PORTS
q9s4ggh2oaq2   web.1     containous/whoami:latest   node2     Running         Running 14 seconds ago
1vomhe5jo3hq   web.2     containous/whoami:latest   node1     Running         Running 14 seconds ago

service的访问

8080这个端口到底映射到哪里了？尝试访问两个swarm节点的IP加端口8080：

$ curl 172.19.177.14:8080
Hostname: 3a0d0ab2c13b
IP: 127.0.0.1
IP: 10.0.0.5
IP: 172.20.0.3
RemoteAddr: 10.0.0.2:51986
GET / HTTP/1.1
Host: 172.19.177.14:8080
User-Agent: curl/7.58.0
Accept: */*$ curl 172.19.188.123:8080
Hostname: 1356383a5cdc
IP: 127.0.0.1
IP: 10.0.0.6
IP: 172.18.0.3
RemoteAddr: 10.0.0.3:58008
GET / HTTP/1.1
Host: 172.19.188.123:8080
User-Agent: curl/7.58.0
Accept: */*

可以看到两个节点的8080端口都可以访问，并且回应的容器是不同的（hostname），也就是有负载均衡的效果。

ingress数据包的走向

以node1节点为例，来看看数据到底是如何达到service中的container：

$ sudo iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 2583 packets, 256K bytes)pkts bytes target     prot opt in     out     source               destination7447  447K DOCKER-INGRESS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
17212 1036K DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCALChain INPUT (policy ACCEPT 1830 packets, 114K bytes)pkts bytes target     prot opt in     out     source               destinationChain OUTPUT (policy ACCEPT 1816 packets, 110K bytes)pkts bytes target     prot opt in     out     source               destination147 10427 DOCKER-INGRESS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCALChain POSTROUTING (policy ACCEPT 1816 packets, 110K bytes)pkts bytes target     prot opt in     out     source               destination7   420 MASQUERADE  all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match src-type LOCAL0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/08   672 MASQUERADE  all  --  *      !docker_gwbridge  172.18.0.0/16        0.0.0.0/0Chain DOCKER (2 references)pkts bytes target     prot opt in     out     source               destination0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/00     0 RETURN     all  --  docker_gwbridge *       0.0.0.0/0            0.0.0.0/0Chain DOCKER-INGRESS (2 references)pkts bytes target     prot opt in     out     source               destination2   120 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8080 to:172.18.0.2:80807587  457K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

外部访问的流量首先进入nat表的PREROUTING链，PREROUTING链的第一条规则会将所有的流量转发至DOCKER-INGRESS自定义链，在DOCKER-INGRESS链中可以看到一条DNAT的规则，所有访问本地8080端口的流量都被转发到 172.18.0.2:8080。

那么这个172.18.0.2是谁的ip呢？

查看docker_gwbridge网桥详情

首先172.18.0.0/16这个网段是网桥docker_gwbridge的，所以这个地址肯定是连在了docker_gwbridge上。

$ sudo docker network inspect docker_gwbridge
[{"Name": "docker_gwbridge","Id": "06c86fd2ac810906cc53669d4a1f01c0036fb3b7a35863f23a898a7a7faa5dfd","Created": "2023-11-17T09:07:00.959710462Z","Scope": "local","Driver": "bridge","EnableIPv6": false,"IPAM": {"Driver": "default","Options": null,"Config": [{"Subnet": "172.18.0.0/16","Gateway": "172.18.0.1"}]},"Internal": false,"Attachable": false,"Ingress": false,"ConfigFrom": {"Network": ""},"ConfigOnly": false,"Containers": {"1356383a5cdcd023acb9fc5090d983be4717161ef6e8416b38070f84e2d38b72": {"Name": "gateway_7e04894d7701","EndpointID": "9cda42b1d59f9643905169f92fd2d24c62c8d451096c4c002c8c5c0733d3c50e","MacAddress": "02:42:ac:12:00:03","IPv4Address": "172.18.0.3/16","IPv6Address": ""},"ingress-sbox": {"Name": "gateway_ingress-sbox","EndpointID": "5011de467d80ba0f1bc9b97b6625f2dc6104cc4913495fe84af9e27661df0730","MacAddress": "02:42:ac:12:00:02","IPv4Address": "172.18.0.2/16","IPv6Address": ""}},"Options": {"com.docker.network.bridge.enable_icc": "false","com.docker.network.bridge.enable_ip_masquerade": "true","com.docker.network.bridge.name": "docker_gwbridge"},"Labels": {}}
]

可以看到除了容器web.2（gateway_7e04894d7701）连接到docker_gwbridge这个网桥上，还有一个叫ingress-sbox的容器也连接了docker_gwbridge，它的IP地址是172.18.0.2/16。

ingress-sbox网络命名空间

这个ingress-sbox其实并不是一个容器，而是一个网络命名空间(network namespace),　我们可以通过下面的方式进入到这个命名空间：

$ sudo ls /run/docker/netns/
1-m4np8jn3j5  7e04894d7701  ingress_sbox$ sudo nsenter --net="/run/docker/netns/ingress_sbox" ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft forever
119: eth0@if120: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group defaultlink/ether 02:42:0a:00:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 10.0.0.2/24 brd 10.0.0.255 scope global eth0valid_lft forever preferred_lft foreverinet 10.0.0.4/32 scope global eth0valid_lft forever preferred_lft forever
121: eth1@if122: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group defaultlink/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 1inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1valid_lft forever preferred_lft forever

通过查看IP地址，发现这个命名空间连接了两个网络，一个eth1是连接了docker_gwbridge，另外一个eth0连接了ingress这个网络。

ingress-sbox对流量的处理

前面我们已经看到访问宿主机的8080端口，宿主机会将流量转发到ingress-sbox（172.18.0.2）的8080端口，接下来看下ingress-sbox对流量是怎么处理的？

$ sudo nsenter --net="/run/docker/netns/ingress_sbox" iptables -nvL -t mangle
Chain PREROUTING (policy ACCEPT 20 packets, 1820 bytes)pkts bytes target     prot opt in     out     source               destination12   804 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8080 MARK set 0x10fChain INPUT (policy ACCEPT 12 packets, 804 bytes)pkts bytes target     prot opt in     out     source               destination0     0 MARK       all  --  *      *       0.0.0.0/0            10.0.0.4             MARK set 0x10fChain FORWARD (policy ACCEPT 8 packets, 1016 bytes)pkts bytes target     prot opt in     out     source               destinationChain OUTPUT (policy ACCEPT 12 packets, 804 bytes)pkts bytes target     prot opt in     out     source               destinationChain POSTROUTING (policy ACCEPT 20 packets, 1820 bytes)pkts bytes target     prot opt in     out     source               destination

MARK是iptables的一个目标，用于给数据包打上指定mark，我们可以看到ingress-sbox对8080端口的流量打上MARK为0x10f，对应的十进制为271。

由于ingress_sbox会通过ipvs负载转发数据包到某个容器的虚拟ip上（即Routing Mesh路由转发），故需要通过ipvsadm指令查看对应的路由结果。

此时，我们查看ipvs负载路由，通过命令ipvsadm可以发现标记位271会将数据包轮询(rr)发送到10.0.0.5和10.0.0.6这两个IP地址。

$ sudo nsenter --net="/run/docker/netns/ingress_sbox" ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  271 rr-> 10.0.0.5:0                   Masq    1      0          0-> 10.0.0.6:0                   Masq    1      0          0