K8S Helm 安装ingress-nginx/ingress-nginx

安装ingress-nginx/ingress-nginx
- 参考：
- https://www.cnblogs.com/syushin/p/15271304.html

添加helm仓库

- [root@k8s-master ~]# helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
- [root@k8s-master ~]# helm repo update

下载安装包
- helm pull ingress-nginx/ingress-nginx
  - 解压
    - tar -zxvf ingress-nginx-4.0.1.tgz
备份并修改 values.yaml 文件
- 修改controller的镜像地址

controller:name: /enableAnnotationValidations: falseimage:## Keep false as default for now!chroot: falseregistry: docker.ioimage: willdockerhub/ingress-nginx-controller## for backwards compatibility consider setting the full image url via the repository value below## use *either* current default registry/image or repository format or installing chart by providing the values.yaml will fail## repository:tag: "v1.0.0"digest: sha256:0851b34f69f69352bf168e6ccf30e1e20714a264ab1ecd1933e4d8c0fc3215c6pullPolicy: IfNotPresent# www-data -> uid 101runAsUser: 101allowPrivilegeEscalation: true

- 修改 hostNetwork 的值为 true：

# is merged hostNetwork: true ## Use host ports 80 and 443

- dnsPolicy的值改为: ClusterFirstWithHostNet

# to keep resolving names inside the k8s network, use ClusterFirstWithHostNet. dnsPolicy: ClusterFirstWithHostNet

- nodeSelector 添加标签: ingress: "true"，用于部署 ingress-controller 到指定节点

 # -- Node labels for controller pod assignment## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/##nodeSelector:kubernetes.io/os: linuxingress: "true"

- kind类型更改为：DaemonSet

 # -- Use a `DaemonSet` or `Deployment`kind: DaemonSet# -- Annotations to be added to the controller Deployment or DaemonSet

- kube-webhook-certgen的镜像地址改为国内仓库地址

patch:enabled: trueimage:registry: registry.cn-hangzhou.aliyuncs.comimage: google_containers/kube-webhook-certgen## for backwards compatibility consider setting the full image url via the repository value below## use *either* current default registry/image or repository format or installing chart by providing the values.yaml will fail## repository:tag: "v20231011-8b53cabe0"#digest: sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80digest: sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9pullPolicy: IfNotPresent

安装
- 使用当前目录中的values.yaml文件
  - helm install ingress-nginx -n ingress-nginx .
- 指定values.yaml文件，并使用debug参数，方便调试异常
  - helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --values values.yaml --debug
    - upgrade --install 会导致重新下载包
      - 如果不想重新下载，直接使用install
- 安装成功，并给了一个创建ingress的例子

NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'An example Ingress that makes use of the controller:apiVersion: networking.k8s.io/v1kind: Ingressmetadata:name: examplenamespace: foospec:ingressClassName: nginxrules:- host: www.example.comhttp:paths:- pathType: Prefixbackend:service:name: exampleServiceport:number: 80path: /# This section is only required if TLS is to be enabled for the Ingresstls:- hosts:- www.example.comsecretName: example-tlsIf TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:apiVersion: v1kind: Secretmetadata:name: example-tlsnamespace: foodata:tls.crt: <base64 encoded cert>tls.key: <base64 encoded key>type: kubernetes.io/tls

处于pending状态的ingress-nginx-controller

[root@k8s-master01 ingress-nginx]# kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.99.163.44    <pending>     80:31268/TCP,443:31052/TCP   4m54s
ingress-nginx-controller-admission   ClusterIP      10.100.131.12   <none>        443/TCP                      4m54s[root@k8s-master01 ingress-nginx]# kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
ingress-nginx-controller   LoadBalancer   10.99.163.44   <pending>     80:31268/TCP,443:31052/TCP   94s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

给节点打标签ingress=true
- 需要给节点打上刚刚设置的标签ingress=true，让 Pod 调度到指定的节点，比如调度到 master 节点

# 给master节点打上标签 ingress=ture
[root@k8s-master ingress-nginx]# kubectl label node master1 ingress=true
node/master1 labeled# k8s默认集群中，出于安全考虑，默认配置下Kubernetes不会将Pod调度到Master节点。测试环境无所谓，所以执行下面命令去除master的污点：
[root@k8s-master ingress-nginx]# kubectl taint node master1 node-role.kubernetes.io/master-

参考：

How do I set up a pod to allow scheduling on the master · Issue #1814 · k3s-io/k3s · GitHub How do I set up a pod to allow scheduling on the master

K8S 将 pod 调度到指定 nodes 上运行 - 知乎 K8S 将 pod 调度到指定 nodes 上运行

[root@master01 rocketmq]# kubectl describe nodes | grep Taints
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Taints:             <none>
Taints:             <none>
[root@master01 rocketmq]# kubectl taint nodes --all node-role.kubernetes.io/control-plane-
node/master01 untainted
taint "node-role.kubernetes.io/control-plane" not found
taint "node-role.kubernetes.io/control-plane" not found
[root@master01 rocketmq]# kubectl describe nodes | grep Taints
Taints:             <none>
Taints:             <none>
Taints:             <none>
[root@master01 rocketmq]# kubectl get nodes
NAME       STATUS   ROLES           AGE   VERSION
master01   Ready    control-plane   44d   v1.28.2
node02     Ready    <none>          44d   v1.28.2
node03     Ready    <none>          44d   v1.28.2
[root@master01 rocketmq]# kubectl describe nodes | grep Taints
Taints:             <none>
Taints:             <none>
Taints:             <none>
[root@master01 rocketmq]# kubectl get pods -A -owide |grep ingress
ingress-nginx            ingress-nginx-controller-bc96v                           1/1     Running            0                  41s     yourip     master01   <none>           <none>

k8s命令（pod相关（驱逐、强制删除）、让Master当Node用、修改nodeport端口范围、修改k8sDNS、运行nslookup容器）_kubectl drain-CSDN博客

#将 Master 也当作 Node 使用
kubectl taint node nodename node-role.kubernetes.io/master-[root@app01 home]# kubectl taint node app01 node-role.kubernetes.io/master-
node/app01 untainted
[root@app01 home]# #将 Master 恢复成 Master Only 状态
kubectl taint node nodename node-role.kubernetes.io/master="":NoSchedule[root@app01 rabbitmq]# kubectl taint node app01  node-role.kubernetes.io/master="":NoSchedule
node/app01 tainted
[root@app01 rabbitmq]#

- 本次直接将标签打到了node02上
  - 所以配置的域名要指向node02节点的ip地址
  - ingress-nginx-controller对应的pod会自动部署到node02节点上
配置访问hosts
- 192.168.221.132 harbor.david.org
- 测试
  - 用浏览器访问没问，curl缺少header中的一些参数，所以报308错误

[root@k8s-master01 harbor]# curl harbor.david.org
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>

configmap权限异常处理

查看ingress-nginx-controller日志

- [root@k8s-master01 harbor]# kubectl logs -n ingress-nginx ingress-nginx-controller-4nl8l

- - E1122 05:49:57.933496 7 leaderelection.go:329] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:ingress-nginx:ingress-nginx" cannot create resource "configmaps" in API group "" in the namespace "ingress-nginx"
    - 需要create和update权限，修改role即可
      - [root@k8s-master01 nginx]# kubectl edit clusterrole -n ingress-nginx ingress-nginx
        添加
        - create
        - update
        保存并退出，再查看ingress-nginx-controller日志
        [root@k8s-master01 harbor]# kubectl logs -n ingress-nginx ingress-nginx-controller-4nl8l
      - configmaps （用于配置文件注入）

rules:
- apiGroups:- ""resources:- configmaps- endpoints- nodes- pods- secrets- namespacesverbs:- list- watch- create- update

异常调试
- 安装加上debug参数展示详细安装过程，方便排查错误
- client.go:779: [debug] ingress-nginx-admission-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
- 提示job有异常，先找到这个job

[root@k8s-master01 ingress-nginx]# kubectl get job -A
NAMESPACE       NAME                             COMPLETIONS   DURATION   AGE
ingress-nginx   ingress-nginx-admission-create   0/1           8s         8s
ingress-nginx   ingress-nginx-admission-patch    0/1           69m        69m[root@k8s-master01 ingress-nginx]# kubectl get pods -A
ingress-nginx          ingress-nginx-admission-create-r7bps

有个job对应的pod有异常


[root@k8s-master01 ingress-nginx]# kubectl describe pods -n ingress-nginx          ingress-nginx-admission-create-r7bpsNormal   BackOff    5s (x2 over 28s)   kubelet            Back-off pulling image "registry.aliyuncs.com/google_containers/kube-webhook-certgen:v1.5.1@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"

提示镜像有异常

Failed to pull image "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": rpc error: code = NotFound desc = failed to pull and unpack image "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": failed to resolve reference "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80: not found

修改了镜像还不管用，还需要修改sha256

使用ctr下载该镜像

[root@k8s-master01 ingress-nginx]# ctr -n=k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0

找到该镜像

[root@k8s-master01 ingress-nginx]# ctr -n=k8s.io images list

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0 application/vnd.docker.distribution.manifest.list.v2+json sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9 22.1 MiB linux/amd64,linux/arm/v7,linux/arm64,linux/s390x io.cri-containerd.image=managed