往期回顾:
云端技术驾驭DAY01——云计算底层技术奥秘、云服务器磁盘技术、虚拟化管理、公有云概述
云端技术驾驭DAY02——华为云管理、云主机管理、跳板机配置、制作私有镜像模板
云端技术驾驭DAY03——云主机网站部署、web集群部署、Elasticsearch安装
云端技术驾驭DAY04——Logstash安装部署及插件模块
云端技术驾驭DAY06——容器技术概述、镜像与容器管理、定制简单镜像、容器内安装部署服务
云端技术驾驭DAY07——Dockerfile详解、容器镜像制作、私有仓库
云端技术驾驭DAY08——部署容器服务、Compose微服务管理、harbor仓库部署及管理
云端技术驾驭DAY09——k8s集群安装部署、calico插件部署、计算节点配置管理
云端技术驾驭DAY10——kubectl命令详解、Pod创建过程、Pod的生命周期、定制Pod、资源对象文件
云端技术驾驭DAY11——资源对象文件、Pod自定义命令、多容器Pod、资源监控工具
云端技术驾驭DAY12——Pod调度策略、Pod标签管理、Pod资源配额与限额、全局资源配额与限额策略
云端技术驾驭DAY13
- Pod调度策略管理
- 污点容忍策略
- 污点概述
- 污点策略
- 容忍策略
- 抢占与优先级
- 优先级概述
- 非抢占优先级
- 抢占优先级
- Pod安全
- 特权容器
- 特权容器概述
- 特权容器
- Pod安全策略
- 安全概述
- 限制特权容器
Pod调度策略管理
污点容忍策略
污点概述
- 什么是污点?
- 污点(Taint)是使节点与Pod产生排斥的一类规则
- 污点策略是如何实现
- 污点策略通过嵌合在键值对上的污点标签进行声明
- 污点标签
- 尽量不调度:
PreferNoSchedule
- 不会被调度:
NoSchedule
- 驱逐节点:
NoExecute
- 尽量不调度:
- 管理污点标签
- 污点标签必须绑定在键值对上,格式为:
key=value:污点标签
- 查看污点标签:
kubectl describe nodes [节点名字]
- 设置污点标签:
kubectl taint node [节点名字] key=value:污点标签
- 删除污点标签:
kubectl taint node [节点名字] key=value:污点标签-
- 污点标签必须绑定在键值对上,格式为:
污点策略
- 污点验证
- 为node-0001设置PreferNoschedule污点标签
- 为node-0002设置NoSchedule污点标签
[root@master ~]# kubectl taint node node-0001 k=v1:PreferNoSchedule
node/node-0001 tainted
[root@master ~]# kubectl taint node node-0002 k=v2:NoSchedule
node/node-0002 tainted
[root@master ~]# kubectl describe nodes | grep Taints
Taints: node-role.kubernetes.io/control-plane:NoSchedule
Taints: k=v1:PreferNoSchedule
Taints: k=v2:NoSchedule
Taints: <none>
Taints: <none>
Taints: <none>
- Pod资源文件
[root@master ~]# vim myphp.yaml
---
kind: Pod
apiVersion: v1
metadata:name: myphp
spec:containers:- name: phpimage: myos:php-fpmresources:requests:cpu: 1500m
- 验证测试
[root@master ~]# sed "s,myphp,php1," myphp.yaml | kubectl apply -f -
pod/php1 created
[root@master ~]# sed "s,myphp,php2," myphp.yaml | kubectl apply -f -
pod/php2 created
[root@master ~]# sed "s,myphp,php3," myphp.yaml | kubectl apply -f - // 创建三个Pod
pod/php3 created
[root@master ~]# kubectl get pods -o wide // 发现会优先使用没有污点的节点
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
php1 1/1 Running 0 35s 10.244.240.134 node-0004 <none> <none>
php2 1/1 Running 0 30s 10.244.243.209 node-0003 <none> <none>
php3 1/1 Running 0 26s 10.244.153.139 node-0005 <none> <none>
- 在没有其他节点可用的时候,使用PreferNoSchedule污点所在的节点运行Pod
[root@master ~]# sed "s,myphp,php4," myphp.yaml | kubectl apply -f -
pod/php4 created
[root@master ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
php1 1/1 Running 0 4m19s 10.244.240.134 node-0004 <none> <none>
php2 1/1 Running 0 4m14s 10.244.243.209 node-0003 <none> <none>
php3 1/1 Running 0 4m10s 10.244.153.139 node-0005 <none> <none>
php4 1/1 Running 0 2s 10.244.21.132 node-0001 <none> <none>
- 继续创建Pod
- 即使Pod创建失败,也不会使用NoSchedule所在节点
[root@master ~]# sed "s,myphp,php5," myphp.yaml | kubectl apply -f -
pod/php5 created
[root@master ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
php1 1/1 Running 0 5m37s 10.244.240.134 node-0004 <none> <none>
php2 1/1 Running 0 5m32s 10.244.243.209 node-0003 <none> <none>
php3 1/1 Running 0 5m28s 10.244.153.139 node-0005 <none> <none>
php4 1/1 Running 0 80s 10.244.21.132 node-0001 <none> <none>
php5 0/1 Pending 0 2s <none> <none> <none> <none>
- NoSchedule策略
- 设置NoSchedule污点标签只对新建Pod有效,对于已经创建完成的Pod不会产生影响
[root@master ~]# kubectl taint node node-0003 k=v3:NoSchedule
node/node-0003 tainted
[root@master ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
php1 1/1 Running 0 8m38s 10.244.240.134 node-0004 <none> <none>
php2 1/1 Running 0 8m33s 10.244.243.209 node-0003 <none> <none>
php3 1/1 Running 0 8m29s 10.244.153.139 node-0005 <none> <none>
php4 1/1 Running 0 4m21s 10.244.21.132 node-0001 <none> <none>
php5 0/1 Pending 0 3m3s <none> <none> <none> <none>
- 驱逐策略
- 驱逐策略会删除该节点上的所有Pod
[root@master ~]# kubectl taint node node-0004 k=v4:NoExecute // 为node-0004设置NoExecute策略
node/node-0004 tainted
[root@master ~]# kubectl describe nodes | grep Taints
Taints: node-role.kubernetes.io/control-plane:NoSchedule
Taints: k=v1:PreferNoSchedule
Taints: k=v2:NoSchedule
Taints: k=v3:NoSchedule
Taints: k=v4:NoExecute
Taints: <none>
[root@master ~]# kubectl get pods -o wide // 查看Pod情况,NoExecute污点所在节点的Pod已经被删除了
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
php2 1/1 Running 0 15m 10.244.243.209 node-0003 <none> <none>
php3 1/1 Running 0 15m 10.244.153.139 node-0005 <none> <none>
php4 1/1 Running 0 11m 10.244.21.132 node-0001 <none> <none>
php5 0/1 Pending 0 9m57s <none> <none> <none> <none>
- 删除实验所有污点策略
[root@master ~]# kubectl taint node node-000{1..4} k-
node/node-0001 untainted
node/node-0002 untainted
node/node-0003 untainted
node/node-0004 untainted
容忍策略
- 容忍策略是什么?
- 容忍策略与污点策略相反,某些时候我们需要在有污点的节点上运行Pod,这种无视污点标签的调度方式称为容忍策略
- 容忍策略实验
- 节点node-000{1…2}设置污点标签
k=v1:NoSchedule
- 节点node-000{3…4}设置污点标签
k=v2:NoSchedule
- 节点node-0005设置污点标签
k=v1:NoExecute
- 节点node-000{1…2}设置污点标签
- 精确匹配策略
[root@master ~]# kubectl taint node node-000{1..2} k=v1:NoSchedule
node/node-0001 tainted
node/node-0002 tainted
[root@master ~]# kubectl taint node node-000{3..4} k=v2:NoSchedule
node/node-0003 tainted
node/node-0004 tainted
[root@master ~]# kubectl taint node node-0005 k=v1:NoExecute
node/node-0005 tainted
[root@master ~]# vim myphp.yaml // 容忍k=v1:NoSchedule污点
---
kind: Pod
apiVersion: v1
metadata:name: myphp
spec:tolerations:- operator: Equal // 完全匹配键值对key: k // 键value: v1 // 值effect: NoSchedule // 污点标签containers:- name: phpimage: myos:php-fpmresources:requests:cpu: 1500m
[root@master ~]# for i in php{1..3};do sed "s,myphp,${i}," myphp.yaml ;done|kubectl apply -f -
pod/php1 created
pod/php2 created
pod/php3 created
[root@master ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
php1 1/1 Running 0 6s 10.244.21.133 node-0001 <none> <none>
php2 1/1 Running 0 6s 10.244.147.9 node-0002 <none> <none>
php3 0/1 Pending 0 6s <none> <none> <none> <none>
- 模糊匹配策略
... ...
spec:tolerations:- operator: Exists // 部分匹配key: keffect: NoSchedule
... ...
- 容忍所有node污点
... ...
spec:tolerations:- operator: Exists // 模糊匹配key: k // 键effect: "" // 设置空或删除,代表所有污点标签
... ...
抢占与优先级
优先级概述
- 优先级是什么
- 优先级表示一个Pod相对于其他Pod的重要性
- 优先级有什么用?
- 优先级可以保证重要的Pod被调度运行
- 如何使用优先级和抢占
- 配置优先级类PriorityClass
- 创建Pod时为其设置对应的优先级
- PriorityClasss
- PriorityClasss是一个全局资源对象,它定义了从优先级类名称到优先级整数值的映射
- 优先级在value字段中指定,可以设置小于10亿的整数值,值最大,优先级最高
- PriorityClasss还有两个可选字段:
- globalDefault用于设置默认优先级状态,如果没有任何优先级设置,Pod的优先级为零
- description用来配置描述性信息,告诉用户优先级的用途
- 优先级策略
- 非抢占优先:
- 指的是在调度阶段优先进行调度分配,一旦容器调度完成就不可以抢占,资源不足时,只能等待
- 抢占优先:
- 强制调度一个Pod,如果资源不足无法被调度,调度程序会抢占较低优先级的Pod的资源,来保证高优先级Pod的运行
- 非抢占优先:
非抢占优先级
- 定义优先级资源对象
- 创建一个value为1000的优先级对象
- 创建一个value为500的优先级对象
- 设置非抢占策略Never
[root@master ~]# vim mypriority.yaml
---
kind: PriorityClass
apiVersion: scheduling.k8s.io/v1
metadata:name: high-non // 指定优先级名字
preemptionPolicy: Never // 策略:非抢占
value: 1000 // 优先级---
kind: PriorityClass
apiVersion: scheduling.k8s.io/v1
metadata:name: low-non
preemptionPolicy: Never
value: 500
[root@master ~]# kubectl apply -f mypriority.yaml
priorityclass.scheduling.k8s.io/high-non created
priorityclass.scheduling.k8s.io/low-non created
[root@master ~]# kubectl get priorityclasses.scheduling.k8s.io
NAME VALUE GLOBAL-DEFAULT AGE
high-non 1000 false 15s
low-non 500 false 15s
system-cluster-critical 2000000000 false 5d21h
system-node-critical 2000001000 false 5d21h
- 验证Pod优先级调度策略
- 所有Pod创建在node-0002上
- 创建php1,该Pod使用默认优先级,cpu=1500m
- 创建php1,该Pod使用低优先级,cpu=1500m
- 创建php3,该Pod使用高优先级,cpu=1500m
root@master ~]# vim php1.yaml
---
kind: Pod
apiVersion: v1
metadata:name: php1
spec:nodeSelector:kubernetes.io/hostname: node-0004containers:- name: phpimage: myos:php-fpmresources:requests:cpu: "1500m"
[root@master ~]# vim php2.yaml
---
kind: Pod
apiVersion: v1
metadata:name: php2
spec:nodeSelector:kubernetes.io/hostname: node-0004priorityClassName: low-noncontainers:- name: phpimage: myos:php-fpmresources:requests:cpu: "1500m"
[root@master ~]# vim php3.yaml
---
kind: Pod
apiVersion: v1
metadata:name: php3
spec:nodeSelector:kubernetes.io/hostname: node-0004priorityClassName: high-non containers:- name: phpimage: myos:php-fpmresources:requests:cpu: "1500m"
[root@master ~]# kubectl apply -f php1.yaml
pod/php1 created
[root@master ~]# kubectl apply -f php2.yaml
pod/php2 created
[root@master ~]# kubectl apply -f php3.yaml
pod/php3 created
[root@master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
php1 1/1 Running 0 19s
php2 0/1 Pending 0 16s
php3 0/1 Pending 0 14s
[root@master ~]# kubectl delete pod php1
pod "php1" deleted
[root@master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
php2 0/1 Pending 0 70s
php3 1/1 Running 0 68s
抢占优先级
- 定义优先级资源对象
[root@master ~]# vim mypriority.yaml
---
kind: PriorityClass
apiVersion: scheduling.k8s.io/v1
metadata:name: high-non
preemptionPolicy: Never
value: 1000---
kind: PriorityClass
apiVersion: scheduling.k8s.io/v1
metadata:name: low-non
preemptionPolicy: Never
value: 500---
kind: PriorityClass
apiVersion: scheduling.k8s.io/v1
metadata:name: high
preemptionPolicy: PreemptLowerPriority // 策略:抢占优先级
value: 1000---
kind: PriorityClass
apiVersion: scheduling.k8s.io/v1
metadata:name: low
preemptionPolicy: PreemptLowerPriority
value: 500
[root@master ~]# kubectl apply -f mypriority.yaml
priorityclass.scheduling.k8s.io/high created
priorityclass.scheduling.k8s.io/low created
[root@master ~]# kubectl get priorityclasses.scheduling.k8s.io
NAME VALUE GLOBAL-DEFAULT AGE
high 1000 false 4s
high-non 1000 false 2h
low 500 false 4s
low-non 500 false 2h
system-cluster-critical 2000000000 false 21d
system-node-critical 2000001000 false 21d
- 验证抢占优先级
// 替换优先级策略
[root@master ~]# sed 's,-non,,' -i php?.yaml// 默认优先级 Pod
[root@master ~]# kubectl apply -f php1.yaml
pod/php1 created
[root@master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
php1 1/1 Running 0 6s// 高优先级 Pod
[root@master ~]# kubectl apply -f php3.yaml
pod/php3 created
[root@master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
php3 1/1 Running 0 9s// 低优先级 Pod
[root@master ~]# kubectl apply -f php2.yaml
pod/php2 created
[root@master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
php2 0/1 Pending 0 3s
php3 1/1 Running 0 9s
Pod安全
特权容器
特权容器概述
- 什么是特权容器?
- 容器是通过名称空间技术隔离的,有时候我们执行一些应用服务,需要使用或修改敏感的系统信息,这时容器需要突破隔离限制,获取更高的权限,这类容器统称特权容器
- 运行特权容器会有一些安全风险,这种模式下运行容器对宿主机拥有root权限,可以突破隔离直接控制宿主机的资源配置
特权容器
- 制作特权容器(修改主机名和hosts文件)
[root@master ~]# vim root.yaml
---
kind: Pod
apiVersion: v1
metadata:name: root
spec:hostname: myhost // 修改容器主机名hostAliases: // 修改 /etc/hosts- ip: 192.168.1.30 // IP 地址hostnames: // 名称键值对- harbor // 主机名containers:- name: apacheimage: myos:httpd
[root@master ~]# kubectl apply -f root.yaml
pod/root created
[root@master ~]# kubectl exec -it root -- /bin/bash
[root@myhost html]# hostname
myhost
[root@myhost html]# cat /etc/hosts
... ...
# Entries added by HostAliases.
192.168.1.30 harbor
- 制作root特权容器
[root@master ~]# vim root.yaml
---
kind: Pod
apiVersion: v1
metadata:name: root
spec:hostPID: true // 特权,共享系统进程hostNetwork: true // 特权,共享主机网络containers:- name: apacheimage: myos:httpdsecurityContext: // 安全上下文值privileged: true // root特权容器
[root@master ~]# kubectl apply -f root.yaml
pod/root created
[root@master ~]# kubectl exec -it root -- /bin/bash
[root@node-0005 html]# pstree -p // 系统进程特权
systemd(1)-+-NetworkManager(595)-+-{NetworkManager}(617)| `-{NetworkManager}(618)|-agetty(1129)... ...
[root@node-0005 html]# ifconfig eth0 // 网络特权
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500inet 192.168.1.55 netmask 255.255.255.0 broadcast 192.168.1.255inet6 fe80::f816:3eff:fe74:cd03 prefixlen 64 scopeid 0x20<link>ether fa:16:3e:74:cd:03 txqueuelen 1000 (Ethernet)
[root@node-0005 ~]# mkdir /sysroot // root用户特权
[root@node-0005 ~]# mount /dev/sda1 /sysroot
[root@node-0005 ~]# chroot /sysroot
sh-4.4# // 此处已经是node节点上的root用户了
Pod安全策略
安全概述
- 什么是Pod安全策略?
- Pod安全策略是集群级别的资源,它能够控制Pod运行的行为,以及它具有访问什么的能力
- 如何使用Pod安全策略?
- Kubernetes服务器版本必须不低于v1.22
- 确保PodSecurity特性门控被启用
- Pod安全策略(LEVEL)
privieged
:不受限制的策略,提供最大可能范围的特权许可,此策略允许特权提升baseline
:弱限制性的策略,禁止已知的策略提升权限,允许使用默认的Pod配置restricted
:非常严格的限制性策略,遵循当前的保护Pod的最佳实践
- Pod准入控制标签(MODE)
- Kubernetes定义了一组标签,你可以设置这些标签来定义某个名字空间上Pod安全性标准级别。所选择的标签定义了检测到潜在违利时所要采取的动作
enforce
:策略违利会导致Pod被拒绝audit
:策略违利会触发审计日志,但是Pod仍可被接受warn
:策略违利会触发用户可见的警告信息,但是Pod仍是被接受的
- Kubernetes定义了一组标签,你可以设置这些标签来定义某个名字空间上Pod安全性标准级别。所选择的标签定义了检测到潜在违利时所要采取的动作
- 语法格式:
pod-security.kubernetes.io/<MODE>:<LEVEL>
限制特权容器
- 设置严格的准入控制,拒绝特权容器
[root@master ~]# kubectl create namespace myprod
namespace/myprod created
[root@master ~]# kubectl label namespaces myprod pod-security.kubernetes.io/enforce=restricted
namespace/myprod labeled
- 在创建特权容器时发出警告提示
[root@master ~]# kubectl create namespace mytest
namespace/mytest created
[root@master ~]# kubectl label namespaces mytest pod-security.kubernetes.io/warn=baseline
namespace/mytest labeled
- 创建特权容器
[root@master ~]# kubectl -n myprod apply -f root.yaml
Error from server (Failure): ... ... // 创建失败
[root@master ~]# kubectl -n mytest apply -f root.yaml
Warning: would violate "latest" version of "baseline" PodSecurity profile: host namespaces (hostNetwork=true, hostPID=true), privileged (container "linux" must not set securityContext.privileged=true)
pod/root created
- 创建安全的Pod
[root@master ~]# vim nonroot.yaml
---
kind: Pod
apiVersion: v1
metadata:name: nonroot
spec:restartPolicy: Alwayscontainers:- name: phpimage: myos:php-fpmsecurityContext:allowPrivilegeEscalation: falserunAsNonRoot: truerunAsUser: 65534seccompProfile:type: "RuntimeDefault"capabilities:drop: ["ALL"]
[root@master ~]# kubectl -n myprod apply -f nonroot.yaml
pod/nonroot created
[root@master ~]# kubectl -n myprod get pods
NAME READY STATUS RESTARTS AGE
nonroot 1/1 Running 0 6s
[root@master ~]# kubectl -n myprod exec -it nonroot -- id // 没有root用户
uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)