问题场景:
今天部署了K8S主节点,Master节点安装了Calico网络插件,在运行node列表结果是NotReady:
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady master 22h v1.19.4
获取kube-system pods 列表,dns为Pending
[root@master ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6d56c8448f-57njl 0/1 Pending 0 22h
coredns-6d56c8448f-k9nm2 0/1 Pending 0 22h
etcd-master 1/1 Running 1 22h
kube-apiserver-master 1/1 Running 1 22h
kube-controller-manager-master 1/1 Running 1 22h
kube-proxy-9dljn 1/1 Running 1 22h
kube-scheduler-master 1/1 Running 1 22h
查看pod coredns-6d56c8448f-57njl 详情日志没有数据,获取pod详情有错误信息:
[root@master ~]# kubectl logs coredns-6d56c8448f-57njl -n kube-system
[root@master ~]#
[root@master ~]# kubectl describe pods coredns-6d56c8448f-57njl -n kube-system
...其他信息
Events:Type Reason Age From Message---- ------ ---- ---- -------Warning FailedScheduling 21h (x114 over 22h) default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.Warning FailedScheduling 51s (x25 over 5m2s) default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
[root@master ~]#
问题解决:
首先看到的是污点,取消污点没有作用,这个是因为长时间node为NotReady,他会吧这个节点加入到污点,不在调度此node,但这恰恰是master节点。
先使用[journalctl]命令查看系统日志关于K8S的:
[root@master ~]# journalctl -f -u kubelet
-- Logs begin at 二 2024-04-30 21:01:33 CST. --
4月 30 21:11:05 master kubelet[962]: W0430 21:11:05.628506 962 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
4月 30 21:11:06 master kubelet[962]: E0430 21:11:06.818698 962 kubelet.go:2103] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
4月 30 21:11:10 master kubelet[962]: W0430 21:11:10.628945 962 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
4月 30 21:11:11 master kubelet[962]: E0430 21:11:11.849882 962 kubelet.go:2103] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Unable to update cni config: no networks found in /etc/cni/net.d
是说没有找到cni配置,查看了一下都没有/etc/cni/这个文件夹,证明没有安装cni网络插件。
运行k8s软件状态也能有一部分日志:
[root@master ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node AgentLoaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)Drop-In: /usr/lib/systemd/system/kubelet.service.d└─10-kubeadm.confActive: active (running) since 二 2024-04-30 21:01:44 CST; 11min agoDocs: https://kubernetes.io/docs/Main PID: 962 (kubelet)Tasks: 15Memory: 116.0MCGroup: /system.slice/kubelet.service└─962 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod...4月 30 21:12:45 master kubelet[962]: W0430 21:12:45.637104 962 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
4月 30 21:12:47 master kubelet[962]: E0430 21:12:47.223164 962 kubelet.go:2103] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: ne... uninitialized
4月 30 21:12:50 master kubelet[962]: W0430 21:12:50.637477 962 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
4月 30 21:12:52 master kubelet[962]: E0430 21:12:52.250303 962 kubelet.go:2103] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: ne... uninitialized
4月 30 21:12:55 master kubelet[962]: W0430 21:12:55.637834 962 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
4月 30 21:12:57 master kubelet[962]: E0430 21:12:57.269078 962 kubelet.go:2103] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: ne... uninitialized
4月 30 21:13:00 master kubelet[962]: W0430 21:13:00.638500 962 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
4月 30 21:13:02 master kubelet[962]: E0430 21:13:02.299863 962 kubelet.go:2103] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: ne... uninitialized
4月 30 21:13:05 master kubelet[962]: W0430 21:13:05.638862 962 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
4月 30 21:13:07 master kubelet[962]: E0430 21:13:07.328870 962 kubelet.go:2103] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: ne... uninitialized
Hint: Some lines were ellipsized, use -l to show in full.
可以清晰地看到同样的错误。