目录

1、Kubernetes网络组件

1.1、Flannel网络组件

1.2、Calico 网络插件

2、环境准备

2.1、主机初始化配置

2.2、部署docker环境

3、部署kubernetes集群

3.1、组件介绍

3.2、配置阿里云yum源

3.3、安装kubelet kubeadm kubectl

3.4、配置init-config.yaml

3.5、安装master节点

3.6、安装node节点

3.7、部署Calico网络插件

4、Calico网络策略基础

4.1、创建服务

4.2、启用网络隔离

4.3、测试网络隔离

4.4、允许通过网络策略进行访问

5、Calico网络策略进阶

5.1、创建服务

5.2、拒绝所有入口流量

5.3、允许进入Nginx的流量

5.4、拒绝所有出口流量

5.5、允许DNS出口流量

5.6、允许出口流量到Nginx

---

1、Kubernetes网络组件

        随着 Docker 容器化兴起，云计算面对的挑战越来越大，例如：网络管理、存储管理等。一个数据中心中基本上都有成百上千个容器，这么多的容器需要运维人员集中管理。而在云计算的世界中，计算是最基础的，存储是最重要的，网络则是最复杂的。Kubernetes 网络的实现不是集群内部自己实现，而是依赖于第三方网络插件。本次主要介绍 Kubernetes网络组件中的一个重要成员——Calico。

1.1、Flannel网络组件

        Flannel 是 CoreOS 团队针对 Kubernetes 设计的一个网络规划服务。它的功能是让集群中的不同节点主机创建的 Docker 容器都具有全集群唯一的虚拟 IP 地址。在默认的 Docker 配置中，每个节点上的 Docker 服务会分别负责所在节点容器的 IP 分配。这样导致的问题是，不同节点上容器可能获得相同的内网 IP 地址。

        Flannel 的设计目的就是为集群中的所有节点重新规划 IP 地址的使用规则，从而使得不同节点上的容器能够获得“同属一个内网”且”不重复的”IP 地址，并让属于不同节点上的容器能够直接通过内网 IP 地址通信。

        Flannel 实质上是一种“覆盖网络(overlay network)”，也就是将 TCP 数据包装在另一种网络包里面进行路由转发和通信，目前已经支持 UDP、VxLAN、AWS VPC 和 GCE 路由等数据转发方式，默认的节点间数据通信方式是 UDP 转发。

 

        数据从源容器中发出后，经由所在主机的 docker0 虚拟网卡转发到 flannel0 虚拟网卡，

这是个 P2P 的虚拟网卡，flanneld 服务监听在网卡的另外一端。

        Flannel 通过 Etcd 服务维护了一张节点间的路由表。源主机的 flanneld 服务将原本的数据内容 UDP 封装后根据自己的路由表投递给目的节点的 flanneld 服务，数据到达以后被解包，然后直接进入目的节点的 flannel0 虚拟网卡，之后被转发到目的主机的 docker0 虚拟网卡，最后就像本机容器通信一下的有 docker0 路由到达目标容器。

1.2、Calico 网络插件

        Calico 是一种基于 BGP 的、纯三层的、容器间互通的网络方案。与 OpenStack、Kubenetes、AWS、GCE 等云平台都能够良好的集成。在虚拟化平台中，如 OpenStack、Docker 等都需要实现 workloads 之间互连，但同时也需要对容器做隔离控制，就像在Internet 中的服务仅开放 80 端口、公有云的多租户一样，提供隔离和管控机制。

        而在多数的虚拟化平台实现中，通常使用二层隔离技术来实现容器的网络，这些二层技术有一些弊端，比如需要依赖 VLAN、bridge 和隧道等技术。其中 bridge 带来了复杂性，vlan 隔离和 tunnel 隧道则消耗更多的资源并对物理环境有要求，随着网络规模的增大，整体会变得越加复杂。我们尝试把 Host 当作 Internet 中的路由器，同样使用 BGP 同步路由，并使用 Iptables 来做安全访问策略，最终设计出了 Calico 方案。

 

（1）Calico 网络模型工作组件

• Felix：运行在每一台 Host 的 agent 进程，主要负责网络接口管理和监听、路由、ARP管理、ACL 管理和同步、状态上报等。
• etcd：分布式键值存储，主要负责网络元数据一致性，确保 Calico 网络状态的准确性，可以与 kubernetes 共用；
• BGP Client（BIRD）：Calico 为每一台 Host 部署一个 BGP Client，使用 BIRD 实现。BIRD 是一个单独的持续发展的项目，实现了众多动态路由协议比如 BGP、OSPF、RIP 等。在 Calico 的角色是监听 Host 上由 Felix 注入的路由信息，然后通过 BGP 协议广播告诉剩余 Host 节点，从而实现网络互通。
• BGP Route Reflector：在大型网络规模中，如果仅仅使用 BGP client 形成 mesh 全网互联的方案就会导致规模限制，因为所有节点之间俩俩互联，需要 N^2 个连接，为了解决这个规模问题，可以采用 BGP 的 Router Reflector 的方法，使所有 BGP Client仅与特定 RR 节点互联并做路由同步，从而大大减少连接数。
• CalicoCtl：Calico 命令行管理工具。

（2）Calico 网络 Node 之间两种网络

• IPIP
• 从字面来理解，就是把一个 IP 数据包又套在一个 IP 包里，即把 IP 层封装到 IP 层的一个 tunnel。它的作用相当于一个基于 IP 层的网桥。一般来说，普通的网桥是基于 mac 层的，根本不需 IP，而这个 ipip 则是通过两端的路由做一个 tunnel，把两个本来不通的网络通过点对点连接起来。
• BGP
• 边界网关协议（Border Gateway Protocol, BGP）是互联网上一个核心的去中心化自治路由协议。它通过维护 IP 路由表或‘前缀’表来实现自治系统（AS）之间的可达性，属于矢量路由协议。BGP 不使用传统的内部网关协议（IGP）的指标，而使用基于路径、网络策略或规则集来决定路由。因此，它更适合被称为矢量性协议，而不是路由协议。BGP通俗的讲就是讲接入到机房的多条线路（如电信、联通、移动等）融合为一体，实现多线单 IP，BGP 机房的优点：服务器只需要设置一个 IP 地址，最佳访问路由是由网络上的骨干路由器根据路由跳数与其它技术指标来确定的，不会占用服务器的任何系统。

• 2、环境准备

  操作系统	IP地址	主机名	组件
  CentOS7.5	192.168.2.116	k8s-master	kubeadm、kubelet、kubectl、docker-ce
  CentOS7.5	192.168.2.117	k8s-node1	kubeadm、kubelet、kubectl、docker-ce
  CentOS7.5	192.168.2.118	k8s-node2	kubeadm、kubelet、kubectl、docker-ce

  注意：所有主机配置推荐CPU：2C+  Memory：2G+

 

2.1、主机初始化配置

所有主机配置禁用防火墙和selinux 

[root@localhost ~]# setenforce 0
[root@localhost ~]# iptables -F
[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# systemctl disable firewalld
[root@localhost ~]# systemctl stop NetworkManager
[root@localhost ~]# systemctl disable NetworkManager
[root@localhost ~]# sed -i '/^SELINUX=/s/enforcing/disabled/' /etc/selinux/config

配置主机名并绑定hosts，不同主机名称不同

[root@localhost ~]# hostname k8s-master
[root@localhost ~]# bash
[root@k8s-master ~]# cat << EOF >> /etc/hosts
192.168.2.116 k8s-master
192.168.2.117 k8s-node1
192.168.2.118 k8s-node2
EOF[root@k8s-master ~]# scp /etc/hosts 192.168.2.117:/etc/
[root@k8s-master ~]# scp /etc/hosts 192.168.2.118:/etc/[root@localhost ~]# hostname k8s-node1
[root@localhost ~]# bash
[root@k8s-node1 ~]#[root@localhost ~]# hostname k8s-node2
[root@localhost ~]# bash
[root@k8s-node2 ~]#

主机配置初始化

[root@k8s-master ~]# yum -y install vim wget net-tools lrzsz[root@k8s-master ~]# swapoff -a
[root@k8s-master ~]# sed -i '/swap/s/^/#/' /etc/fstab[root@k8s-node01 ~]# cat << EOF >> /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
[root@k8s-node01 ~]# modprobe br_netfilter
[root@k8s-node01 ~]# sysctl -p

2.2、部署docker环境

三台主机上分别部署 Docker 环境，因为 Kubernetes 对容器的编排需要 Docker 的支持。

[root@k8s-master ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo[root@k8s-master ~]# yum install -y yum-utils device-mapper-persistent-data lvm2

使用 YUM 方式安装 Docker 时，推荐使用阿里的 YUM 源。

[root@k8s-master ~]# yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo[root@k8s-master ~]# yum clean all && yum makecache fast [root@k8s-master ~]# yum -y install docker-ce
[root@k8s-master ~]# systemctl start docker
[root@k8s-master ~]# systemctl enable docker

镜像加速器（所有主机配置）

[root@k8s-master ~]# cat << END > /etc/docker/daemon.json
{"registry-mirrors":[ "https://nyakyfun.mirror.aliyuncs.com" ]
}
END
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl restart docker

3、部署kubernetes集群

3.1、组件介绍

三个节点都需要安装下面三个组件

• kubeadm：安装工具，使所有的组件都会以容器的方式运行
• kubectl：客户端连接K8S API工具
• kubelet：运行在node节点，用来启动容器的工具

3.2、配置阿里云yum源

        使用 YUM 方式安装 Kubernetes时，推荐使用阿里的 YUM 源。

        准备好基础环境和 Docker 环境，下面就开始通过 Kubeadm 来部署 Kubernetes 集群。首先，安装 Kubelet、Kubeadm 和 Kubectl。

[root@k8s-master ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo 
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpghttps://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF[root@k8s-master ~]# ls /etc/yum.repos.d/backup  Centos-7.repo  CentOS-Media.repo  CentOS-x86_64-kernel.repo  docker-ce.repo  kubernetes.repo

3.3、安装kubelet kubeadm kubectl

所有主机配置

[root@k8s-master ~]# yum install -y kubelet-1.20.0 kubeadm-1.20.0 kubectl-1.20.0
[root@k8s-master ~]# systemctl enable kubelet

        kubelet 刚安装完成后，通过 systemctl start kubelet 方式是无法启动的，需要加入节点或初始化为 master 后才可启动成功。

3.4、配置init-config.yaml

        Kubeadm 提供了很多配置项，Kubeadm 配置在 Kubernetes 集群中是存储在ConfigMap 中的，也可将这些配置写入配置文件，方便管理复杂的配置项。Kubeadm 配内容是通过 kubeadm config 命令写入配置文件的。

        在master节点安装，master 定于为192.168.2.116，通过如下指令创建默认的init-config.yaml文件：

[root@k8s-master ~]# kubeadm config print init-defaults > init-config.yaml

init-config.yaml配置

[root@k8s-master ~]# cat init-config.yaml 
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:- system:bootstrappers:kubeadm:default-node-tokentoken: abcdef.0123456789abcdefttl: 24h0m0susages:- signing- authentication
kind: InitConfiguration
localAPIEndpoint:advertiseAddress: 192.168.2.116		//master节点IP地址bindPort: 6443
nodeRegistration:criSocket: /var/run/dockershim.sockname: k8s-master		//如果使用域名保证可以解析，或直接使用 IP 地址taints:- effect: NoSchedulekey: node-role.kubernetes.io/master
---
apiServer:timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:type: CoreDNS
etcd:local:dataDir: /var/lib/etcd		//etcd 容器挂载到本地的目录
imageRepository: registry.aliyuncs.com/google_containers	//修改为国内地址
kind: ClusterConfiguration
kubernetesVersion: v1.19.0
networking:dnsDomain: cluster.localserviceSubnet: 10.96.0.0/12podSubnet: 10.244.0.0/16 	//podSubnet 地址不能与主机物理地址设置为同一网段
scheduler: {}

3.5、安装master节点

master节点镜像提取链接：https://pan.baidu.com/s/1BDLqfw_S08XDBAyvAtB1ug?pwd=gply 
提取码：gply

拉取所需镜像

[root@k8s-master ~]# kubeadm config images list --config init-config.yamlregistry.aliyuncs.com/google_containers/kube-apiserver:v1.20.0
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.20.0
registry.aliyuncs.com/google_containers/kube-scheduler:v1.20.0
registry.aliyuncs.com/google_containers/kube-proxy:v1.20.0
registry.aliyuncs.com/google_containers/pause:3.2
registry.aliyuncs.com/google_containers/etcd:3.4.13-0
registry.aliyuncs.com/google_containers/coredns:1.7.0[root@k8s-master ~]# mkdir master [root@k8s-master ~]# cd master[root@k8s-master master]# rz -E
rz waiting to receive.[root@k8s-master master]# ls | while read line
> do
> docker load < $line
> done

安装matser节点

[root@k8s-master master]# cd [root@k8s-master ~]# kubeadm init --config=init-config.yaml[init] Using Kubernetes version: v1.20.0
[preflight] Running pre-flight checks[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.5. Latest validated version: 19.03
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.2.116]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.2.116 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.2.116 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 13.002529 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxyYour Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:https://kubernetes.io/docs/concepts/cluster-administration/addons/Then you can join any number of worker nodes by running the following on each as root:kubeadm join 192.168.2.116:6443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:ad01378122a860c8be7ace78f3abe732304e3334a15f0a72cf05f7ebb93ca313 

根据提示操作

        kubectl 默认会在执行的用户家目录下面的.kube 目录下寻找config 文件。这里是将在初始化时[kubeconfig]步骤生成的admin.conf 拷贝到.kube/config

[root@k8s-master ~]#   mkdir -p $HOME/.kube
[root@k8s-master ~]#   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]#   sudo chown $(id -u):$(id -g) $HOME/.kube/config

        Kubeadm 通过初始化安装是不包括网络插件的，也就是说初始化之后是不具备相关网络功能的，比如 k8s-master 节点上查看节点信息都是“Not Ready”状态、Pod 的 CoreDNS无法提供服务等。

3.6、安装node节点

根据master安装时的提示信息

[root@k8s-node1 ~]# kubeadm join 192.168.2.116:6443 --token abcdef.0123456789abcdef \
>     --discovery-token-ca-cert-hash sha256:ad01378122a860c8be7ace78f3abe732304e3334a15f0a72cf05f7ebb93ca313 
[preflight] Running pre-flight checks[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.5. Latest validated version: 19.03
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.[root@k8s-master ~]# kubectl get nodes
NAME         STATUS     ROLES                  AGE     VERSION
k8s-master   NotReady   control-plane,master   8m44s   v1.20.0
k8s-node1    NotReady   <none>                 31s     v1.20.0
k8s-node2    NotReady   <none>                 31s     v1.20.0

        前面已经提到，在初始化 k8s-master 时并没有网络相关配置，所以无法跟 node 节点通信，因此状态都是“NotReady”。但是通过 kubeadm join 加入的 node 节点已经在k8s-master 上可以看到。

3.7、部署Calico网络插件

calico.yaml文件提取链接：https://pan.baidu.com/s/1BQ4B6tDNL38ce3y_1k04oA?pwd=rb2u 
提取码：rb2u

安装 Calico 网络插件。

[root@k8s-master ~]# rz -E            #上传calico.yaml文件
rz waiting to receive.[root@k8s-master ~]#  kubectl apply -f calico.yaml
configmap/calico-config unchanged
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org configured
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers configured
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrole.rbac.authorization.k8s.io/calico-node configured
clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged
daemonset.apps/calico-node configured
serviceaccount/calico-node unchanged
deployment.apps/calico-kube-controllers configured
serviceaccount/calico-kube-controllers unchanged
poddisruptionbudget.policy/calico-kube-controllers created

查看node节点状态。

[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
k8s-master   Ready    control-plane,master   26m   v1.20.0
k8s-node1    Ready    <none>                 18m   v1.20.0
k8s-node2    Ready    <none>                 18m   v1.20.0

查看所有pod状态

[root@k8s-master ~]#  kubectl get pod --all-namespacesNAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-744cfdf676-xjz5f   1/1     Running   0          17m
kube-system   calico-node-92974                          1/1     Running   0          18m
kube-system   calico-node-fdwmf                          1/1     Running   0          18m
kube-system   calico-node-tprqg                          1/1     Running   0          18m
kube-system   coredns-7f89b7bc75-jkn49                   1/1     Running   0          40m
kube-system   coredns-7f89b7bc75-nlz64                   1/1     Running   0          40m
kube-system   etcd-k8s-master                            1/1     Running   0          40m
kube-system   kube-apiserver-k8s-master                  1/1     Running   0          40m
kube-system   kube-controller-manager-k8s-master         1/1     Running   0          40m
kube-system   kube-proxy-jcvgc                           1/1     Running   0          40m
kube-system   kube-proxy-nmhwm                           1/1     Running   0          32m
kube-system   kube-proxy-tg47v                           1/1     Running   0          32m
kube-system   kube-scheduler-k8s-master                  1/1     Running   0          40m

4、Calico网络策略基础

4.1、创建服务

创建命名空间。

[root@k8s-master ~]# kubectl create ns policy-demo
namespace/policy-demo created

在 policy-demo 命名空间中创建两个副本的 Nginx Pod。

[root@k8s-master ~]# vim nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: nginxnamespace: policy-demolabels:app: nginx
spec:replicas: 2selector:matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:containers:- name: nginximage: nginxports:- containerPort: 80[root@k8s-master ~]# kubectl apply -f nginx-deployment.yamldeployment.apps/nginx created[root@k8s-master ~]# kubectl get pod -n policy-demo
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7848d4b86f-7bw9n   1/1     Running   0          84s
nginx-7848d4b86f-pcnsn   1/1     Running   0          84s

通过服务暴露 Nginx 的 80 端口。

[root@k8s-master ~]# kubectl expose --namespace=policy-demo deployment nginx --port=80service/nginx exposed[root@k8s-master ~]#  kubectl get all -n policy-demoNAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-7848d4b86f-7bw9n   1/1     Running   0          2m10s
pod/nginx-7848d4b86f-pcnsn   1/1     Running   0          2m10sNAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/nginx   ClusterIP   10.107.124.161   <none>        80/TCP    13sNAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   2/2     2            2           2m10sNAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-7848d4b86f   2         2         2       2m10s

通过 busybox 的 Pod 去访问 Nginx 服务。

[root@k8s-master ~]# kubectl run --namespace=policy-demo access --rm -ti --image busyboxIf you don't see a command prompt, try pressing enter.
/bin/sh
/ # If you don't see a command prompt, try pressing enter.
> / # wget -q nginx -O -
> <!DOCTYPE html>
> <html>
> <head>
> <title>Welcome to nginx!</title>
> <style>
>     body {
>         width: 35em;
>         margin: 0 auto;
>         font-family: Tahoma, Verdana, Arial, sans-serif;
>     }
> </style>
> </head>
> <body>
> <h1>Welcome to nginx!</h1>
> <p>If you see this page, the nginx web server is successfully installed and
> working. Further configuration is required.</p>
> 
> <p>For online documentation and support please refer to
> <a href="http://nginx.org/">nginx.org</a>.<br/>
> Commercial support is available at
> <a href="http://nginx.com/">nginx.com</a>.</p>
> 
> <p><em>Thank you for using nginx.</em></p>
> </body>
> </html>ctrl+pq退出

4.2、启用网络隔离

        在 policy-demo 命名空间中打开隔离。然后 Calico 将阻止连接到该命名空间中的 Pod。执行以下命令将创建一个 NetworkPolicy，该策略将对 policy-demo 名称空间中的所有 Pod实现默认的拒绝行为。

[root@k8s-master ~]# kubectl create -f - <<EOF
> kind: NetworkPolicy
> apiVersion: networking.k8s.io/v1
> metadata:
>   name: default-deny
>   namespace: policy-demo
> spec:
>   podSelector:
>     matchLabels: {}
> EOF
networkpolicy.networking.k8s.io/default-deny created

4.3、测试网络隔离

        启用网络隔离后，所有对 Nginx 服务的访问都将阻止。执行以下命令，尝试再次访问Nginx 服务，查看网络隔离的效果。

[root@k8s-master ~]# kubectl run --namespace=policy-demo access --rm -ti --image busybox
If you don't see a command prompt, try pressing enter.
/ # wget -q --timeout=5 nginx -O -
wget: download timed out            #连接超时

4.4、允许通过网络策略进行访问

        使用 NetworkPolicy 启用对 Nginx 服务的访问。设置允许从 accessPod 传入的连接，但不能从其他任何地方传入。创建 access-nginx 的网络策略具体内容如下所示。

[root@k8s-master ~]# kubectl create -f - <<EOF
> kind: NetworkPolicy
> apiVersion: networking.k8s.io/v1
> metadata:
>   name: access-nginx
>   namespace: policy-demo
> spec:
>   podSelector:
>     matchLabels:
>       app: nginx
>   ingress:
>     - from:
>       - podSelector:
>           matchLabels:
>             run: access
> EOF
networkpolicy.networking.k8s.io/access-nginx created

 从 accessPod 访问该服务。

[root@k8s-master ~]# kubectl run --namespace=policy-demo access --rm -ti --image busybox
If you don't see a command prompt, try pressing enter./ # wget -q --timeout=5 nginx -O -
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>

5、Calico网络策略进阶

5.1、创建服务

删除命令空间 policy-demo，创建新的命名空间 advanced-policy-demo。

[root@k8s-master ~]# kubectl delete ns policy-demonamespace "policy-demo" deleted[root@k8s-master ~]# kubectl create ns advanced-policy-demonamespace/advanced-policy-demo created

使用 YAML 文件创建 Nginx 服务。

[root@k8s-master ~]# vim nginx-deployment.yamlapiVersion: apps/v1
kind: Deployment
metadata:name: nginxnamespace: advanced-policy-demo            #修改为advanced-policy-demolabels:app: nginx
spec:replicas: 2selector:matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:containers:- name: nginximage: nginxports:- containerPort: 80[root@k8s-master ~]# kubectl apply -f nginx-deployment.yamldeployment.apps/nginx created[root@k8s-master ~]# kubectl expose --namespace=advanced-policy-demo deployment nginx --port=80service/nginx exposed

验证访问权限并访问百度测试外网连通性。

[root@k8s-master ~]# kubectl run --namespace=advanced-policy-demo access --rm -ti --image busybox /bin/shIf you don't see a command prompt, try pressing enter./ # wget -q --timeout=5 nginx -O -<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>/ # wget -q --timeout=5 www.baidu.com -O -<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>&copy;2017&nbsp;Baidu&nbsp;<a href=http://www.baidu.com/duty/>使用百度前必读</a>&nbsp; <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a>&nbsp;京ICP证030173号&nbsp; <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>

5.2、拒绝所有入口流量

设置网络策略，要求 Nginx 服务拒绝所有入口流量。然后进行访问权限的验证。

[root@localhost ~]# kubectl create -f - <<EOF
> apiVersion: networking.k8s.io/v1
> kind: NetworkPolicy
> metadata:
>   name: default-deny-ingress
>   namespace: advanced-policy-demo
> spec:
>   podSelector:
>     matchLabels: {}
>   policyTypes:
>   - Ingress 
> EOF
networkpolicy.networking.k8s.io/default-deny-ingress created[root@k8s-master ~]# kubectl run --namespace=advanced-policy-demo access --rm -ti --image busybox /bin/sh/ # wget -q --timeout=5 nginx -O -wget: download timed out/ # wget -q --timeout=5 www.baidu.com -O -<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>&copy;2017&nbsp;Baidu&nbsp;<a href=http://www.baidu.com/duty/>使用百度前必读</a>&nbsp; <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a>&nbsp;京ICP证030173号&nbsp; <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>

        从上述命令执行结果中可以看出，对 Nginx 服务的入口访问被拒绝，而仍然允许对出站 Internet 的出口访问。

5.3、允许进入Nginx的流量

        执行以下命令，创建一个 NetworkPolicy，设置允许流量从 advanced-policy-demo 命名空间中的任何 Pod 到 Nginx Pod。创建策略成功后，就可以访问 Nginx 服务了。

[root@localhost ~]# kubectl create -f - <<EOF
> apiVersion: networking.k8s.io/v1
> kind: NetworkPolicy
> metadata:
>   name: access-nginx
>   namespace: advanced-policy-demo
> spec:
>   podSelector:
>     matchLabels:
>       app: nginx
>   ingress:
>     - from:
>       - podSelector:
>           matchLabels: {}
> EOF
networkpolicy.networking.k8s.io/access-nginx created[root@k8s-master ~]# kubectl run --namespace=advanced-policy-demo access --rm -ti --image busybox /bin/shIf you don't see a command prompt, try pressing enter./ # wget -q --timeout=5 nginx -O -<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>

 

5.4、拒绝所有出口流量

        设置拒绝所有出口流量的网络策略，该策略设置成功后，任何策略未明确允许的入站或出站流量都将被拒绝。

[root@localhost ~]#  kubectl create -f - <<EOF
> apiVersion: networking.k8s.io/v1
> kind: NetworkPolicy
> metadata:
>   name: default-deny-egress
>   namespace: advanced-policy-demo
> spec:
>   podSelector:
>     matchLabels: {}
>   policyTypes:
>   - Egress
> EOF
networkpolicy.networking.k8s.io/default-deny-egress created[root@k8s-master ~]# kubectl run --namespace=advanced-policy-demo access --rm -ti --image busybox /bin/shIf you don't see a command prompt, try pressing enter./ # nslookup nginx
;; connection timed out; no servers could be reached/ # wget -q --timeout=5 www.baidu.com -O -
wget: bad address 'www.baidu.com'

5.5、允许DNS出口流量

        执行以下命令，在 kube-system 名称空间上创建一个标签。该标签的 NetworkPolicy允许 DNS 从 advanced-policy-demo 名称空间中的任何 Pod 到名称空间 kube-system 的出站流量。

[root@localhost ~]# kubectl label namespace kube-system name=kube-systemnamespace/kube-system labeled[root@localhost ~]# kubectl create -f - <<EOF
> apiVersion: networking.k8s.io/v1
> kind: NetworkPolicy
> metadata:
>   name: allow-dns-access
>   namespace: advanced-policy-demo
> spec:
>   podSelector:
>     matchLabels: {}
>   policyTypes:
>   - Egress
>   egress:
>   - to:
>     - namespaceSelector:
>         matchLabels:
>           name: kube-system
>     ports:
>     - protocol: UDP
>       port: 53
> EOF
networkpolicy.networking.k8s.io/allow-dns-access created[root@k8s-master ~]# kubectl run --namespace=advanced-policy-demo access --rm -ti --image busybox /bin/sh
If you don't see a command prompt, try pressing enter.
/ # nslookup nginx
Server:		10.96.0.10
Address:	10.96.0.10:53*** Can't find nginx.svc.cluster.local: No answer
*** Can't find nginx.cluster.local: No answer
*** Can't find nginx.localdomain: No answer
*** Can't find nginx.advanced-policy-demo.svc.cluster.local: No answer
*** Can't find nginx.svc.cluster.local: No answer
*** Can't find nginx.cluster.local: No answer
*** Can't find nginx.localdomain: No answer/ # nslookup www.baidu.com
Server:		10.96.0.10
Address:	10.96.0.10:53Non-authoritative answer:
www.baidu.com	canonical name = www.a.shifen.com*** Can't find www.baidu.com: No answer

        即使 DNS 出口流量被允许，但来自 Advanced-policy-demo 命名空间中所有 Pod 的所有其他出口流量仍被阻止。因此，来自 wget 调用的 HTTP 出口流量仍将失败。

/ # wget -q --timeout=5 nginx -O -
wget: download timed out

5.6、允许出口流量到Nginx

        执行以下命令，创建一个 NetworkPolicy，允许从 advanced-policy-demo 名称空间中的任何 Pod到具有 app: nginx 相同名称空间中标签匹配的 Pod 的出站流量。

[root@localhost ~]# kubectl create -f - <<EOF
> apiVersion: networking.k8s.io/v1
> kind: NetworkPolicy
> metadata:
>   name: allow-egress-to-advance-policy-ns
>   namespace: advanced-policy-demo
> spec:
>   podSelector:
>     matchLabels: {}
>   policyTypes:
>   - Egress
>   egress:
>   - to:
>     - podSelector:
>         matchLabels:
>           app: nginx
> EOF
networkpolicy.networking.k8s.io/allow-egress-to-advance-policy-ns created[root@k8s-master ~]# kubectl run --namespace=advanced-policy-demo access --rm -ti --image busybox /bin/shIf you don't see a command prompt, try pressing enter./ # wget -q --timeout=5 nginx -O -
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>/ # wget -q --timeout=5 www.baidu.com -O -
wget: download timed out

 

访问百度超时，是因为它可以解决 DNS 匹配标签以外的其他任何出口访问 app: nginx

的 advanced-policy-demo 命名空间。