云原生Kubernetes: K8S 1.29版本部署ingress-nginx

一、实验

1.环境

2. K8S 1.29版本部署ingress-nginx

二、问题

1.kubectl 如何强制删除 Pod、Namespace 资源

2.创建pod失败

3.pod报错ImagePullBackOff

4.docker如何将镜像上传到官方仓库

5.创建ingress报错

一、实验

1.环境

（1）主机

表1 主机

主机	架构	版本	IP	备注
master	K8S master节点	1.29.0	192.168.204.8
node1	K8S node节点	1.29.0	192.168.204.9
node2	K8S node节点	1.29.0	192.168.204.10	已部署Kuboard

（2）master节点查看集群

1）查看node
kubectl get node2)查看node详细信息
kubectl get node -o wide

（3）查看pod

[root@master ~]# kubectl get pod -A

(4) 访问Kuboard

http://192.168.204.10:30080/kuboard/cluster

查看节点

2. K8S 1.29版本部署ingress-nginx

（1）查阅

https://github.com/kubernetes/ingress-nginx

（2）版本支持图

（3）查看K8S版本

[root@master ~]#  kubectl version

（4）选择ingress-nginx版本

版本为1.29.0需使用ingress-nginx v1.10.0版本

下载

https://github.com/kubernetes/ingress-nginx/blob/controller-v1.10.0/deploy/static/provider/cloud/deploy.yaml

（5）查看配置文件

[root@master ~]# vim deploy.yaml

  1 apiVersion: v12 kind: Namespace3 metadata:4   labels:5     app.kubernetes.io/instance: ingress-nginx6     app.kubernetes.io/name: ingress-nginx7   name: ingress-nginx8 ---9 apiVersion: v110 automountServiceAccountToken: true11 kind: ServiceAccount12 metadata:13   labels:14     app.kubernetes.io/component: controller15     app.kubernetes.io/instance: ingress-nginx16     app.kubernetes.io/name: ingress-nginx17     app.kubernetes.io/part-of: ingress-nginx18     app.kubernetes.io/version: 1.10.019   name: ingress-nginx20   namespace: ingress-nginx21 ---22 apiVersion: v123 kind: ServiceAccount24 metadata:25   labels:26     app.kubernetes.io/component: admission-webhook27     app.kubernetes.io/instance: ingress-nginx28     app.kubernetes.io/name: ingress-nginx29     app.kubernetes.io/part-of: ingress-nginx30     app.kubernetes.io/version: 1.10.031   name: ingress-nginx-admission32   namespace: ingress-nginx33 ---34 apiVersion: rbac.authorization.k8s.io/v135 kind: Role36 metadata:37   labels:38     app.kubernetes.io/component: controller39     app.kubernetes.io/instance: ingress-nginx40     app.kubernetes.io/name: ingress-nginx41     app.kubernetes.io/part-of: ingress-nginx42     app.kubernetes.io/version: 1.10.043   name: ingress-nginx44   namespace: ingress-nginx45 rules:46 - apiGroups:47   - ""48   resources:49   - namespaces50   verbs:51   - get52 - apiGroups:53   - ""54   resources:55   - configmaps56   - pods57   - secrets58   - endpoints59   verbs:60   - get61   - list62   - watch63 - apiGroups:64   - ""65   resources:66   - services67   verbs:68   - get69   - list70   - watch71 - apiGroups:72   - networking.k8s.io73   resources:74   - ingresses75   verbs:76   - get77   - list78   - watch79 - apiGroups:80   - networking.k8s.io81   resources:82   - ingresses/status83   verbs:84   - update85 - apiGroups:86   - networking.k8s.io87   resources:88   - ingressclasses89   verbs:90   - get91   - list92   - watch93 - apiGroups:94   - coordination.k8s.io95   resourceNames:96   - ingress-nginx-leader97   resources:98   - leases99   verbs:
100   - get
101   - update
102 - apiGroups:
103   - coordination.k8s.io
104   resources:
105   - leases
106   verbs:
107   - create
108 - apiGroups:
109   - ""
110   resources:
111   - events
112   verbs:
113   - create
114   - patch
115 - apiGroups:
116   - discovery.k8s.io
117   resources:
118   - endpointslices
119   verbs:
120   - list
121   - watch
122   - get
123 ---
124 apiVersion: rbac.authorization.k8s.io/v1
125 kind: Role
126 metadata:
127   labels:
128     app.kubernetes.io/component: admission-webhook
129     app.kubernetes.io/instance: ingress-nginx
130     app.kubernetes.io/name: ingress-nginx
131     app.kubernetes.io/part-of: ingress-nginx
132     app.kubernetes.io/version: 1.10.0
133   name: ingress-nginx-admission
134   namespace: ingress-nginx
135 rules:
136 - apiGroups:
137   - ""
138   resources:
139   - secrets
140   verbs:
141   - get
142   - create
143 ---
144 apiVersion: rbac.authorization.k8s.io/v1
145 kind: ClusterRole
146 metadata:
147   labels:
148     app.kubernetes.io/instance: ingress-nginx
149     app.kubernetes.io/name: ingress-nginx
150     app.kubernetes.io/part-of: ingress-nginx
151     app.kubernetes.io/version: 1.10.0
152   name: ingress-nginx
153 rules:
154 - apiGroups:
155   - ""
156   resources:
157   - configmaps
158   - endpoints
159   - nodes
160   - pods
161   - secrets
162   - namespaces
163   verbs:
164   - list
165   - watch
166 - apiGroups:
167   - coordination.k8s.io
168   resources:
169   - leases
170   verbs:
171   - list
172   - watch
173 - apiGroups:
174   - ""
175   resources:
176   - nodes
177   verbs:
178   - get
179 - apiGroups:
180   - ""
181   resources:
182   - services
183   verbs:
184   - get
185   - list
186   - watch
187 - apiGroups:
188   - networking.k8s.io
189   resources:
190   - ingresses
191   verbs:
192   - get
193   - list
194   - watch
195 - apiGroups:
196   - ""
197   resources:
198   - events
199   verbs:
200   - create
201   - patch
202 - apiGroups:
203   - networking.k8s.io
204   resources:
205   - ingresses/status
206   verbs:
207   - update
208 - apiGroups:
209   - networking.k8s.io
210   resources:
211   - ingressclasses
212   verbs:
213   - get
214   - list
215   - watch
216 - apiGroups:
217   - discovery.k8s.io
218   resources:
219   - endpointslices
220   verbs:
221   - list
222   - watch
223   - get
224 ---
225 apiVersion: rbac.authorization.k8s.io/v1
226 kind: ClusterRole
227 metadata:
228   labels:
229     app.kubernetes.io/component: admission-webhook
230     app.kubernetes.io/instance: ingress-nginx
231     app.kubernetes.io/name: ingress-nginx
232     app.kubernetes.io/part-of: ingress-nginx
233     app.kubernetes.io/version: 1.10.0
234   name: ingress-nginx-admission
235 rules:
236 - apiGroups:
237   - admissionregistration.k8s.io
238   resources:
239   - validatingwebhookconfigurations
240   verbs:
241   - get
242   - update
243 ---
244 apiVersion: rbac.authorization.k8s.io/v1
245 kind: RoleBinding
246 metadata:
247   labels:
248     app.kubernetes.io/component: controller
249     app.kubernetes.io/instance: ingress-nginx
250     app.kubernetes.io/name: ingress-nginx
251     app.kubernetes.io/part-of: ingress-nginx
252     app.kubernetes.io/version: 1.10.0
253   name: ingress-nginx
254   namespace: ingress-nginx
255 roleRef:
256   apiGroup: rbac.authorization.k8s.io
257   kind: Role
258   name: ingress-nginx
259 subjects:
260 - kind: ServiceAccount
261   name: ingress-nginx
262   namespace: ingress-nginx
263 ---
264 apiVersion: rbac.authorization.k8s.io/v1
265 kind: RoleBinding
266 metadata:
267   labels:
268     app.kubernetes.io/component: admission-webhook
269     app.kubernetes.io/instance: ingress-nginx
270     app.kubernetes.io/name: ingress-nginx
271     app.kubernetes.io/part-of: ingress-nginx
272     app.kubernetes.io/version: 1.10.0
273   name: ingress-nginx-admission
274   namespace: ingress-nginx
275 roleRef:
276   apiGroup: rbac.authorization.k8s.io
277   kind: Role
278   name: ingress-nginx-admission
279 subjects:
280 - kind: ServiceAccount
281   name: ingress-nginx-admission
282   namespace: ingress-nginx
283 ---
284 apiVersion: rbac.authorization.k8s.io/v1
285 kind: ClusterRoleBinding
286 metadata:
287   labels:
288     app.kubernetes.io/instance: ingress-nginx
289     app.kubernetes.io/name: ingress-nginx
290     app.kubernetes.io/part-of: ingress-nginx
291     app.kubernetes.io/version: 1.10.0
292   name: ingress-nginx
293 roleRef:
294   apiGroup: rbac.authorization.k8s.io
295   kind: ClusterRole
296   name: ingress-nginx
297 subjects:
298 - kind: ServiceAccount
299   name: ingress-nginx
300   namespace: ingress-nginx
301 ---
302 apiVersion: rbac.authorization.k8s.io/v1
303 kind: ClusterRoleBinding
304 metadata:
305   labels:
306     app.kubernetes.io/component: admission-webhook
307     app.kubernetes.io/instance: ingress-nginx
308     app.kubernetes.io/name: ingress-nginx
309     app.kubernetes.io/part-of: ingress-nginx
310     app.kubernetes.io/version: 1.10.0
311   name: ingress-nginx-admission
312 roleRef:
313   apiGroup: rbac.authorization.k8s.io
314   kind: ClusterRole
315   name: ingress-nginx-admission
316 subjects:
317 - kind: ServiceAccount
318   name: ingress-nginx-admission
319   namespace: ingress-nginx
320 ---
321 apiVersion: v1
322 data:
323   allow-snippet-annotations: "false"
324 kind: ConfigMap
325 metadata:
326   labels:
327     app.kubernetes.io/component: controller
328     app.kubernetes.io/instance: ingress-nginx
329     app.kubernetes.io/name: ingress-nginx
330     app.kubernetes.io/part-of: ingress-nginx
331     app.kubernetes.io/version: 1.10.0
332   name: ingress-nginx-controller
333   namespace: ingress-nginx
334 ---
335 apiVersion: v1
336 kind: Service
337 metadata:
338   labels:
339     app.kubernetes.io/component: controller
340     app.kubernetes.io/instance: ingress-nginx
341     app.kubernetes.io/name: ingress-nginx
342     app.kubernetes.io/part-of: ingress-nginx
343     app.kubernetes.io/version: 1.10.0
344   name: ingress-nginx-controller
345   namespace: ingress-nginx
346 spec:
347   externalTrafficPolicy: Local
348   ipFamilies:
349   - IPv4
350   ipFamilyPolicy: SingleStack
351   ports:
352   - appProtocol: http
353     name: http
354     port: 80
355     protocol: TCP
356     targetPort: http
357   - appProtocol: https
358     name: https
359     port: 443
360     protocol: TCP
361     targetPort: https
362   selector:
363     app.kubernetes.io/component: controller
364     app.kubernetes.io/instance: ingress-nginx
365     app.kubernetes.io/name: ingress-nginx
366   type: LoadBalancer
367 ---
368 apiVersion: v1
369 kind: Service
370 metadata:
371   labels:
372     app.kubernetes.io/component: controller
373     app.kubernetes.io/instance: ingress-nginx
374     app.kubernetes.io/name: ingress-nginx
375     app.kubernetes.io/part-of: ingress-nginx
376     app.kubernetes.io/version: 1.10.0
377   name: ingress-nginx-controller-admission
378   namespace: ingress-nginx
379 spec:
380   ports:
381   - appProtocol: https
382     name: https-webhook
383     port: 443
384     targetPort: webhook
385   selector:
386     app.kubernetes.io/component: controller
387     app.kubernetes.io/instance: ingress-nginx
388     app.kubernetes.io/name: ingress-nginx
389   type: ClusterIP
390 ---
391 apiVersion: apps/v1
392 kind: Deployment
393 metadata:
394   labels:
395     app.kubernetes.io/component: controller
396     app.kubernetes.io/instance: ingress-nginx
397     app.kubernetes.io/name: ingress-nginx
398     app.kubernetes.io/part-of: ingress-nginx
399     app.kubernetes.io/version: 1.10.0
400   name: ingress-nginx-controller
401   namespace: ingress-nginx
402 spec:
403   minReadySeconds: 0
404   revisionHistoryLimit: 10
405   selector:
406     matchLabels:
407       app.kubernetes.io/component: controller
408       app.kubernetes.io/instance: ingress-nginx
409       app.kubernetes.io/name: ingress-nginx
410   strategy:
411     rollingUpdate:
412       maxUnavailable: 1
413     type: RollingUpdate
414   template:
415     metadata:
416       labels:
417         app.kubernetes.io/component: controller
418         app.kubernetes.io/instance: ingress-nginx
419         app.kubernetes.io/name: ingress-nginx
420         app.kubernetes.io/part-of: ingress-nginx
421         app.kubernetes.io/version: 1.10.0
422     spec:
423       containers:
424       - args:
425         - /nginx-ingress-controller
426         - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
427         - --election-id=ingress-nginx-leader
428         - --controller-class=k8s.io/ingress-nginx
429         - --ingress-class=nginx
430         - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
431         - --validating-webhook=:8443
432         - --validating-webhook-certificate=/usr/local/certificates/cert
433         - --validating-webhook-key=/usr/local/certificates/key
434         - --enable-metrics=false
435         env:
436         - name: POD_NAME
437           valueFrom:
438             fieldRef:
439               fieldPath: metadata.name
440         - name: POD_NAMESPACE
441           valueFrom:
442             fieldRef:
443               fieldPath: metadata.namespace
444         - name: LD_PRELOAD
445           value: /usr/local/lib/libmimalloc.so
446         image: registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
447         imagePullPolicy: IfNotPresent
448         lifecycle:
449           preStop:
450             exec:
451               command:
452               - /wait-shutdown
453         livenessProbe:
454           failureThreshold: 5
455           httpGet:
456             path: /healthz
457             port: 10254
458             scheme: HTTP
459           initialDelaySeconds: 10
460           periodSeconds: 10
461           successThreshold: 1
462           timeoutSeconds: 1
463         name: controller
464         ports:
465         - containerPort: 80
466           name: http
467           protocol: TCP
468         - containerPort: 443
469           name: https
470           protocol: TCP
471         - containerPort: 8443
472           name: webhook
473           protocol: TCP
474         readinessProbe:
475           failureThreshold: 3
476           httpGet:
477             path: /healthz
478             port: 10254
479             scheme: HTTP
480           initialDelaySeconds: 10
481           periodSeconds: 10
482           successThreshold: 1
483           timeoutSeconds: 1
484         resources:
485           requests:
486             cpu: 100m
487             memory: 90Mi
488         securityContext:
489           allowPrivilegeEscalation: false
490           capabilities:
491             add:
492             - NET_BIND_SERVICE
493             drop:
494             - ALL
495           readOnlyRootFilesystem: false
496           runAsNonRoot: true
497           runAsUser: 101
498           seccompProfile:
499             type: RuntimeDefault
500         volumeMounts:
501         - mountPath: /usr/local/certificates/
502           name: webhook-cert
503           readOnly: true
504       dnsPolicy: ClusterFirst
505       nodeSelector:
506         kubernetes.io/os: linux
507       serviceAccountName: ingress-nginx
508       terminationGracePeriodSeconds: 300
509       volumes:
510       - name: webhook-cert
511         secret:
512           secretName: ingress-nginx-admission
513 ---
514 apiVersion: batch/v1
515 kind: Job
516 metadata:
517   labels:
518     app.kubernetes.io/component: admission-webhook
519     app.kubernetes.io/instance: ingress-nginx
520     app.kubernetes.io/name: ingress-nginx
521     app.kubernetes.io/part-of: ingress-nginx
522     app.kubernetes.io/version: 1.10.0
523   name: ingress-nginx-admission-create
524   namespace: ingress-nginx
525 spec:
526   template:
527     metadata:
528       labels:
529         app.kubernetes.io/component: admission-webhook
530         app.kubernetes.io/instance: ingress-nginx
531         app.kubernetes.io/name: ingress-nginx
532         app.kubernetes.io/part-of: ingress-nginx
533         app.kubernetes.io/version: 1.10.0
534       name: ingress-nginx-admission-create
535     spec:
536       containers:
537       - args:
538         - create
539         - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
540         - --namespace=$(POD_NAMESPACE)
541         - --secret-name=ingress-nginx-admission
542         env:
543         - name: POD_NAMESPACE
544           valueFrom:
545             fieldRef:
546               fieldPath: metadata.namespace
547         image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.0@sha256:44d1d0e9f19c63f58b380c5fddaca7cf22c7cee564adeff365225a5df5ef3334
548         imagePullPolicy: IfNotPresent
549         name: create
550         securityContext:
551           allowPrivilegeEscalation: false
552           capabilities:
553             drop:
554             - ALL
555           readOnlyRootFilesystem: true
556           runAsNonRoot: true
557           runAsUser: 65532
558           seccompProfile:
559             type: RuntimeDefault
560       nodeSelector:
561         kubernetes.io/os: linux
562       restartPolicy: OnFailure
563       serviceAccountName: ingress-nginx-admission
564 ---
565 apiVersion: batch/v1
566 kind: Job
567 metadata:
568   labels:
569     app.kubernetes.io/component: admission-webhook
570     app.kubernetes.io/instance: ingress-nginx
571     app.kubernetes.io/name: ingress-nginx
572     app.kubernetes.io/part-of: ingress-nginx
573     app.kubernetes.io/version: 1.10.0
574   name: ingress-nginx-admission-patch
575   namespace: ingress-nginx
576 spec:
577   template:
578     metadata:
579       labels:
580         app.kubernetes.io/component: admission-webhook
581         app.kubernetes.io/instance: ingress-nginx
582         app.kubernetes.io/name: ingress-nginx
583         app.kubernetes.io/part-of: ingress-nginx
584         app.kubernetes.io/version: 1.10.0
585       name: ingress-nginx-admission-patch
586     spec:
587       containers:
588       - args:
589         - patch
590         - --webhook-name=ingress-nginx-admission
591         - --namespace=$(POD_NAMESPACE)
592         - --patch-mutating=false
593         - --secret-name=ingress-nginx-admission
594         - --patch-failure-policy=Fail
595         env:
596         - name: POD_NAMESPACE
597           valueFrom:
598             fieldRef:
599               fieldPath: metadata.namespace
600         image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.0@sha256:44d1d0e9f19c63f58b380c5fddaca7cf22c7cee564adeff365225a5df5ef3334
601         imagePullPolicy: IfNotPresent
602         name: patch
603         securityContext:
604           allowPrivilegeEscalation: false
605           capabilities:
606             drop:
607             - ALL
608           readOnlyRootFilesystem: true
609           runAsNonRoot: true
610           runAsUser: 65532
611           seccompProfile:
612             type: RuntimeDefault
613       nodeSelector:
614         kubernetes.io/os: linux
615       restartPolicy: OnFailure
616       serviceAccountName: ingress-nginx-admission
617 ---
618 apiVersion: networking.k8s.io/v1
619 kind: IngressClass
620 metadata:
621   labels:
622     app.kubernetes.io/component: controller
623     app.kubernetes.io/instance: ingress-nginx
624     app.kubernetes.io/name: ingress-nginx
625     app.kubernetes.io/part-of: ingress-nginx
626     app.kubernetes.io/version: 1.10.0
627   name: nginx
628 spec:
629   controller: k8s.io/ingress-nginx
630 ---
631 apiVersion: admissionregistration.k8s.io/v1
632 kind: ValidatingWebhookConfiguration
633 metadata:
634   labels:
635     app.kubernetes.io/component: admission-webhook
636     app.kubernetes.io/instance: ingress-nginx
637     app.kubernetes.io/name: ingress-nginx
638     app.kubernetes.io/part-of: ingress-nginx
639     app.kubernetes.io/version: 1.10.0
640   name: ingress-nginx-admission
641 webhooks:
642 - admissionReviewVersions:
643   - v1
644   clientConfig:
645     service:
646       name: ingress-nginx-controller-admission
647       namespace: ingress-nginx
648       path: /networking/v1/ingresses
649   failurePolicy: Fail
650   matchPolicy: Equivalent
651   name: validate.nginx.ingress.kubernetes.io
652   rules:
653   - apiGroups:
654     - networking.k8s.io
655     apiVersions:
656     - v1
657     operations:
658     - CREATE
659     - UPDATE
660     resources:
661     - ingresses
662   sideEffects: None

（5）替换镜像源

注意：k8s.io 的image需要修改为阿里云的镜像或其他指定镜像源

参考其他镜像：

https://hub.docker.com/u/anjia0532

更换镜像源

docker pull anjia0532/google-containers.ingress-nginx.controller:v1.10.0

docker pull anjia0532/google-containers.ingress-nginx.kube-webhook-certgen:v1.4.0

这里已重新打标签上传个人docker hub

（6）查看已拉取镜像

[root@node1 ~]# docker images | grep ingress-nginx

（7）生成资源创建ingress-nginx

[root@master ~]# kubectl apply -f deploy.yaml

(8)查看启动的pod和service

[root@master ~]# kubectl get pod,svc -n ingress-nginx  -o wide
NAME                                            READY   STATUS      RESTARTS   AGE    IP               NODE    NOMINATED NODE   READINESS GATES
pod/ingress-nginx-admission-create-dgzgd        0/1     Completed   0          2m2s   10.244.166.139   node1   <none>           <none>
pod/ingress-nginx-admission-patch-c4vgh         0/1     Completed   1          2m2s   10.244.166.138   node1   <none>           <none>
pod/ingress-nginx-controller-5dc4b769bd-mmgc6   1/1     Running     0          2m2s   10.244.166.140   node1   <none>           <none>NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE    SELECTOR
service/ingress-nginx-controller             LoadBalancer   10.101.23.182   <pending>     80:31820/TCP,443:32442/TCP   2m2s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission   ClusterIP      10.103.254.63   <none>        443/TCP                      2m2s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

#主要看ingress-nginx-controller是否启动成功，是通过这个控制器把生成的nginx配置写入/etc/nginx.conf文件中。ingress-nginx-admission-create和ingress-nginx-admission-patch不管，看pod状态为Completed，他们有可能是一次性执行任务，已经运行完成了

（9）Kuboard查看

工作负载

容器组

服务

（10）确认nginx版本

K8S版本为1.29.0需使用nginx版本1.25.3

（11）拉取镜像

docker hub查看

node1节点拉取

[root@node1 ~]# docker pull nginx:1.25.3

node2节点拉取

[root@node2 ~]# docker pull nginx:1.25.3

（12）编写测试的yaml

[root@master ~]# vim nginx-test.yaml

apiVersion: apps/v1
kind: Deployment
metadata:name: my-nginxnamespace: test
spec:replicas: 2selector:matchLabels:app: my-nginxtemplate:metadata:labels:app: my-nginxspec:containers:- name: my-nginximage: nginx:1.25.3imagePullPolicy: IfNotPresentports:- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:name: nginx-svcnamespace: test
spec:ports:- port: 80targetPort: 80protocol: TCP selector:app: my-nginx

（13）生成资源

[root@master ~]# kubectl create ns test[root@master ~]# kubectl apply -f nginx-test.yaml

（14）查看pod与service

[root@master ~]# kubectl get pod,svc -n test

（15）Kuboard查看

工作负载

容器组

服务

（16）创建对应的ingress

[root@master ~]# vim ingress-http.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: ingress-httpnamespace: test
spec:ingressClassName: "nginx"rules:- host: devops.sitehttp:paths:- path: /pathType: Prefixbackend:service:name: nginx-svcport:number: 80

（17）生成资源

[root@master ~]# kubectl apply -f ingress-http.yaml

(18) 查看ingress

[root@master ~]# kubectl get ingress -n test

详细查看

[root@master ~]#  kubectl describe  ingress ingress-http  -n test

（19）Kuboard查看

应用路由

（20）node1节点修改hosts

[root@node1 ~]# vim /etc/hosts

ingress-nginx-controller部署到node1节点的IP为10.244.166.140

10.244.166.140 devops.site

（21）node1节点访问

[root@node1 ~]# curl devops.site

（22）查看nginx 的pod

[root@master ~]# kubectl get pod -n test
NAME                        READY   STATUS    RESTARTS   AGE
my-nginx-7bbcf4d985-2sg9h   1/1     Running   0          48m
my-nginx-7bbcf4d985-ztvrb   1/1     Running   0          48m

修改第一个nginx容器内html文件

[root@master ~]# kubectl exec -it my-nginx-7bbcf4d985-2sg9h -n test /bin/bash
……
# cd /usr/share/nginx/html# ls# cat index.html # echo "my-nginx-7bbcf4d985-2sg9h" > index.html# cat index.html # exit

修改第二个nginx容器内html文件

[root@master ~]# kubectl exec -it my-nginx-7bbcf4d985-ztvrb -n test /bin/bash
……
# cd /usr/share/nginx/html# ls# echo "my-nginx-7bbcf4d985-ztvrb" > index.html# cat index.html # exit

（23）node节点访问

node1节点访问

[root@node1 ~]# curl devops.site

node2节点访问（目前为拒绝连接）

[root@node2 ~]# curl devops.site

（24）ingress扩容

完成：

容器组

（25）查看pod与service

ingress-nginx-controller部署到node2节点的IP为10.244.104.13

[root@master ~]# kubectl get pod,svc -n ingress-nginx  -o wide

（26） node2节点修改hosts

[root@node2 ~]# vim /etc/hosts

ingress-nginx-controller部署到node2节点的IP为10.244.104.13

10.244.104.13 devops.site

（27）node2节点访问

[root@node2 ~]# curl devops.site

（28）最后再次查看容器组

kube-system名称空间

ingress-nginx名称空间

test名称空间

二、问题

1.kubectl 如何强制删除 Pod、Namespace 资源

（1）报错

devops名称空间的jenkins-bc7986c64-rhcr5一直为Terminating状态

（2）原因分析

资源未成功删除。

（3）解决方法

可以通过 kubectl delete 命令中的 “–force --grace-period=0” 来强制删除资源。

# 删除 Pod
kubectl delete pod ${podname} --force --grace-period=0# 删除 Namespace
kubectl delete namespace ${namespace_name} --force --grace-period=0

删除pod:

[root@master ~]# kubectl delete pods jenkins-bc7986c64-rhcr5 -n devops --force --grace-period=0

成功：

2.创建pod失败

（1）报错

节点创建Pod会一直卡在ContainerCreating的状态无法顺利创建并且就绪，READY状态一直为0/1

Kuboard显示

（2）原因分析

①查看pod

[root@master ~]# kubectl describe pod ingress-nginx-admission-create-2m2hs -n ingress-nginx

②node1节点继续查看cni的日志

sudo journalctl -xe | grep cni

③CNI的配置文件默认在/etc/cni/net.d/目录，进入目录查看

[root@node1 ~]# cd /etc/cni/net.d/
[root@node1 net.d]# ls

nodename为node1，正确的

[root@node1 net.d]# vim 10-calico.conflist

④ 查看kubelet日志

[root@node1 ~]# journalctl --since="2024-04-21 9:50:00" --until="2024-04-21 10:14:00" -fu kubelet

显示Failed to stop sandbox

4月 21 10:13:53 node1 kubelet[1083]: E0421 10:13:53.733547    1083 kuberuntime_manager.go:1381] "Failed to stop sandbox" podSandboxID={"Type":"docker","ID":"f0c0260d8f529498d31a198543cc021365e87eb03729d9ef11b0e55c69d0c8b6"}

⑤ 查看节点cri-docker 并重启服务

systemctl status cri-dockersystemctl restart cri-docker

⑥ 综上分析

原因是node1节点的cni容器出现了异常无法为pod分配ip导致的卡在ContainerCreating的状态。

（3）解决方法

删除异常节点的calico-node容器，让它拉起重新同步数据即可修复。

① 删除 calico-node-7wqzs

②已重新拉活

3.pod报错ImagePullBackOff

（1）原因

pod状态为ImagePullBackOff

（2）原因分析

官方给出的yaml文件中拉取的镜像不在docker hub中，在k8s.gcr.io中，所以在国内我们拉取就会报错：ErrImagePull

相关问题的issue:（相关镜像没法上传到docker hub）

https://github.com/kubernetes/ingress-nginx/issues/6335

（3）解决方法

参考项目：

1）GitHub
https://github.com/anjia0532/gcr.io_mirror2)docker hub地址
https://hub.docker.com/u/anjia0532

更换镜像源

[root@node1 ~]#  docker pull anjia0532/google-containers.ingress-nginx.controller:v1.10.0

[root@node1 ~]# docker pull anjia0532/google-containers.ingress-nginx.kube-webhook-certgen:v1.4.0

4.docker如何将镜像上传到官方仓库

（1）ingress-nginx.controller

①添加新的标签，在镜像名称前加上仓库名，jiajianwei为仓库名称

[root@node1 ~]# docker tag anjia0532/google-containers.ingress-nginx.controller:v1.10.0 jiajianwei/google-containers.ingress-nginx.controller:v1.10.0

查看镜像：

[root@node1 ~]# docker images

② 登录公共仓库

 docker login            #登录公共仓库Username：  #账号password：  #密码

③ 上传镜像

[root@node1 ~]# docker push jiajianwei/google-containers.ingress-nginx.controller:v1.10.0

成功：

④在自己的仓库中可以看到上传的镜像，默认上传到公共仓库中

私有仓库需要付费

⑤登出公共仓库

[root@node1 ~]# docker logout

（2）kube-webhook-certgen

①添加新的标签，在镜像名称前加上仓库名，jiajianwei为仓库名称

[root@node1 ~]# docker tag anjia0532/google-containers.ingress-nginx.kube-webhook-certgen:v1.4.0 jiajianwei/google-containers.ingress-nginx.kube-webhook-certgen:v1.4.0

查看镜像：

[root@node1 ~]# docker images

② 登录公共仓库

 docker login            #登录公共仓库Username：  #账号password：  #密码

③ 上传镜像

[root@node1 ~]# docker push jiajianwei/google-containers.ingress-nginx.kube-webhook-certgen:v1.4.0

成功：

④在自己的仓库中可以看到上传的镜像，默认上传到公共仓库中

私有仓库需要付费

⑤登出公共仓库

[root@node1 ~]# docker logout

（3）删除本地已有镜像

[root@node1 ~]# docker rmi -f anjia0532/google-containers.ingress-nginx.controller:v1.10.0 anjia0532/google-containers.ingress-nginx.kube-webhook-certgen:v1.4.0

[root@node1 ~]# docker rmi -f jiajianwei/google-containers.ingress-nginx.controller:v1.10.0 jiajianwei/google-containers.ingress-nginx.kube-webhook-certgen:v1.4.0

(4) 拉取jiajianwei仓库中的镜像

node1节点

[root@node1 ~]# docker pull jiajianwei/google-containers.ingress-nginx.controller:v1.10.0

[root@node1 ~]# docker pull jiajianwei/google-containers.ingress-nginx.kube-webhook-certgen:v1.4.0

node2节点

[root@node2 ~]# docker pull jiajianwei/google-containers.ingress-nginx.controller:v1.10.0

[root@node2 ~]# docker pull jiajianwei/google-containers.ingress-nginx.kube-webhook-certgen:v1.4.0

5.创建ingress报错

（1）报错

Error from server (InternalError): error when creating "ingress-http.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": context deadline exceeded

（2）原因分析

ValidatingWebhookConfiguration未删掉。

（3）解决方法

查看

[root@master ~]# kubectl get ValidatingWebhookConfiguration

删除

[root@master ~]# kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

成功创建ingress: