Ingress-controller高可用

Ingress Controller是集群流量的接入层，对它做高可用非常重要，可以基于keepalive实现nginx-ingress-controller高可用，具体实现如下：

• Ingress-controller根据Deployment+ nodeSeletor+pod反亲和性方式部署在k8s指定的两个work节点，nginx-ingress-controller这个pod共享宿主机ip，然后通过keepalive+nginx实现nginx-ingress-controller高可用

参考：https://github.com/kubernetes/ingress-nginx
https://github.com/kubernetes/ingress-nginx/tree/main/deploy/static/provider/baremetal

特别注意：

• 因为我们现在安装的k8s版本是1.25，那就需要按照文档步骤ctr -n=k8s.io images import导出镜像，如果k8s版本是1.24之前的，可以用docker load -i解压，现在安装更新到了k8s1.25，所以导出镜像需要用ctr -n=k8s.io images import

将ingress-nginx-controllerv1.1.0.tar.gz 上传至节点
链接：https://pan.baidu.com/s/1BP1Vr0IUxkMYqrP-MDs8vQ?pwd=0n68
提取码：0n68

ctr -n=k8s.io images import ingress-nginx-controllerv1.1.0.tar.gz
docker load -i ingress-nginx-controllerv1.1.0.tar.gz
ctr -n=k8s.io images import kube-webhook-certgen-v1.1.0.tar.gz
docker load -i kube-webhook-certgen-v1.1.0.tar.gz

##如果k8s版本是1.24之前的用docker load -i解压
#docker load -i ingress-nginx-controllerv1.1.0.tar.gz

ingress-deploy.yaml
链接：https://pan.baidu.com/s/1nNx9ZDHCu1stqtV7qo6bJA?pwd=x9pg
提取码：x9pg

kubectl apply -f ingress-deploy.yaml

通过keepalived+nginx实现ingress-nginx-controller高可用

安装nginx主备：

在k8snode1和k8snode2上做nginx主备安装

yum install  epel-release  nginx keepalived nginx-mod-stream  -y

修改nginx配置文件。主备一样

cat /etc/nginx/nginx.conf

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;include /usr/share/nginx/modules/*.conf;events {worker_connections 1024;
}
# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';access_log  /var/log/nginx/k8s-access.log  main;upstream k8s-ingress-controller {server 192.168.40.12:80 weight=5 max_fails=3 fail_timeout=30s;   # k8snode1 IP:PORTserver 192.168.40.13:80 weight=5 max_fails=3 fail_timeout=30s;  # k8snode2 IP:PORT}server {listen 30080; proxy_pass k8s-ingress-controller;}
}http {log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';access_log  /var/log/nginx/access.log  main;sendfile            on;tcp_nopush          on;tcp_nodelay         on;keepalive_timeout   65;types_hash_max_size 2048;include             /etc/nginx/mime.types;default_type        application/octet-stream;}#备注：
#server 反向服务地址和端口
#weight 权重
#max_fails 失败多少次，认为主机已挂掉，则踢出
#fail_timeout 踢出后重新探测时间 

注意：nginx监听端口变成大于30000的端口，比方说30080,这样访问域名:30080就可以了，必须是满足大于30000以上，才能代理ingress-controller

keepalive配置

主keepalived

cat /etc/keepalived/keepalived.conf 

global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc  smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id NGINX_MASTER
} vrrp_script check_nginx {script "/etc/keepalived/check_nginx.sh"
}vrrp_instance VI_1 { state MASTER interface ens33  # 修改为实际网卡名virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 priority 100    # 优先级，备服务器设置 90 advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒 authentication { auth_type PASS      auth_pass 1111 }  # 虚拟IPvirtual_ipaddress { 192.168.40.199/24} track_script {check_nginx} 
}#vrrp_script：指定检查nginx工作状态脚本（根据nginx状态判断是否故障转移）
#virtual_ipaddress：虚拟IP（VIP）

cat /etc/keepalived/check_nginx.sh 

#!/bin/bash
#1、判断Nginx是否存活
counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )
if [ $counter -eq 0 ]; then#2、如果不存活则尝试启动Nginxservice nginx startsleep 2#3、等待2秒后再次获取一次Nginx状态counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )#4、再次进行判断，如Nginx还不存活则停止Keepalived，让地址进行漂移if [ $counter -eq 0 ]; thenservice  keepalived stopfi
fi

chmod +x  /etc/keepalived/check_nginx.sh

备keepalive

cat /etc/keepalived/keepalived.conf 

global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc  smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id NGINX_BACKUP
} vrrp_script check_nginx {script "/etc/keepalived/check_nginx.sh"
}vrrp_instance VI_1 { state BACKUP interface ens33virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 priority 90advert_int 1authentication { auth_type PASS      auth_pass 1111 }  virtual_ipaddress { 192.168.1.199/24} track_script {check_nginx} 
}

cat /etc/keepalived/check_nginx.sh

#!/bin/bash
#1、判断Nginx是否存活
counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )
if [ $counter -eq 0 ]; then#2、如果不存活则尝试启动Nginxservice nginx startsleep 2#3、等待2秒后再次获取一次Nginx状态counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )#4、再次进行判断，如Nginx还不存活则停止Keepalived，让地址进行漂移if [ $counter -eq 0 ]; thenservice  keepalived stopfi
fi

chmod +x /etc/keepalived/check_nginx.sh

注：keepalived根据脚本返回状态码（0为工作正常，非0不正常）判断是否故障转移。

k8snode1和k8snode2启动服务

systemctl daemon-reload
systemctl enable nginx keepalived
systemctl start nginx
systemctl start keepalived

测试vip是否绑定成功

在k8snode1执行

ip addr

测试keepalived：

停掉k8snode1上的keepalived。Vip会漂移到k8snode2

service  keepalived stop

在k8snode1启动keepalived ，Vip又会漂移到k8snode1

systemctl start keepalived

测试Ingress HTTP代理k8s内部pod

部署后端tomcat服务

tomcat:8.5.34-jre8-alpine
链接：https://pan.baidu.com/s/18AU47-mvEMeqFS9IGUQURw?pwd=abyo
提取码：abyo

cat ingress-demo.yaml

apiVersion: v1
kind: Service
metadata:name: tomcatnamespace: default
spec:selector:app: tomcatrelease: canaryports:- name: httptargetPort: 8080port: 8080- name: ajptargetPort: 8009port: 8009
---
apiVersion: apps/v1
kind: Deployment
metadata:name: tomcat-deploynamespace: default
spec:replicas: 2selector:matchLabels:app: tomcatrelease: canarytemplate:metadata:labels:app: tomcatrelease: canaryspec:containers:- name: tomcatimage: tomcat:8.5.34-jre8-alpine imagePullPolicy: IfNotPresent  ports:- name: httpcontainerPort: 8080name: ajpcontainerPort: 8009

kubectl apply -f ingress-demo.yaml

kubectl get pods -l app=tomcat

编写ingress规则

ingress-myapp.yaml
链接：https://pan.baidu.com/s/1UE81sqHEELqPQhB6lx5xEQ?pwd=hxmc
提取码：hxmc

kubectl apply -f ingress-myapp.yaml

编写ingress的配置清单

vim ingress-myapp.yaml 

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: ingress-myappnamespace: default
spec:ingressClassName: nginxrules:- host: tomcat.lucky.comhttp:paths:- backend:service:name: tomcatport:number: 8080path: /pathType: Prefix

kubectl apply -f ingress-myapp.yaml

如果有报错

报错如下：

解决办法：

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
kubectl apply -f ingress-myapp.yaml
kubectl get pods -n ingress-nginx -o wide

修改电脑本地的host文件，增加如下一行，IP为VIP

本地hosts文件位置

C:\Windows\System32\drivers\etc

宿主机浏览器访问：tomcat.lucky.com

代理流程：

tomcat.lucky.com:30080->192.168.40.199:30080->192.168.40.181:80,192.168.40.182:80->svc: tomcat:8080->tomcat-deploy-

测试Ingress HTTPS代理k8s内部pod

构建TLS站点

准备证书，在k8smaster1节点操作

cd /root/
openssl genrsa -out tls.key 2048
openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.lucky.com

生成secret，在k8smaster1节点操作

kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key

查看secret

kubectl get secret

查看tomcat-ingress-secret详细信息

kubectl describe secret tomcat-ingress-secret

创建Ingress

Ingress规则可以参考官方：
https://kubernetes.io/zh/docs/concepts/services-networking/ingress/

cat ingress-tomcat-tls.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: ingress-tomcat-tlsnamespace: default
spec:ingressClassName: nginxtls:- hosts:-  tomcat.lucky.comsecretName: tomcat-ingress-secretrules:- host: tomcat.lucky.comhttp:paths:- path: /pathType:  Prefixbackend:service:name: tomcatport:number: 8080

更新yaml文件

kubectl delete -f ingress-myapp.yaml
kubectl apply -f ingress-tomcat-tls.yaml

再次访问 tomcat.lucky.com

同一个k8s搭建多套Ingress-controller

原理以及注意事项

• ingress可以简单理解为service的service，他通过独立的ingress对象来制定请求转发的规则，把请求路由到一个或多个service中。这样就把服务与请求规则解耦了，可以从业务维度统一考虑业务的暴露，而不用为每个service单独考虑。
• 在同一个k8s集群里，部署两个ingress nginx。一个deploy部署给A的API网关项目用。另一个daemonset部署给其它项目作域名访问用。这两个项目的更新频率和用法不一致，暂时不用合成一个。
• 为了满足多租户场景，需要在k8s集群部署多个ingress-controller，给不同用户不同环境使用。

主要参数设置：

containers:- name: nginx-ingress-controllerimage: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.0args:- /nginx-ingress-controller- --ingress-class=ngx-ds

注意：–ingress-class设置该Ingress Controller可监听的目标Ingress Class标识；
注意：同一个集群中不同套Ingress Controller监听的Ingress Class标识必须唯一，且不能设置为nginx关键字（其是集群默认Ingress Controller的监听标识）；

创建Ingress规则：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: ingress-myappnamespace: default
spec:ingressClassName: nginxrules:- host: tomcat.lucky.comhttp:paths:- path: /pathType:  Prefixbackend:service:name: tomcatport:number: 8080

注意：这里要设置为你前面配置的controller.ingressClass唯一标识

ingressClassName: nginx-ds