准备环境
先把 google 的vm 跑起来…
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-user$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane,master 124d v1.23.6
k8s-node0 Ready <none> 124d v1.23.6
k8s-node1 NotReady <none> 124d v1.23.6
k8s-node3 Ready <none> 104d v1.23.6
当前的环境是干净的
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-user$ kubectl get all -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 81d <none>
准备1个 micro-service 和 1个clusterIP
大概框架如上图：
组件
micro service
我会先构建1个 微服务 bq-api-service 并部署到集群， 设成4个pods
clusterIP
我会构建1个 ClusterIP service: clusterip-bq-api-service 作为上面微服务的loadbanlance 和 revserse proxy
测试pod dns-test
因为没有nodePort 和 ingress, 只能用测试POD 去尝试通过 clusterip service 链接 bq-api-serivce
部署bq-api-service
deployment-bq-api-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:labels: # label of this deploymentapp: bq-api-service # custom definedauthor: nvd11name: deployment-bq-api-service # name of this deploymentnamespace: default
spec:replicas: 4 # desired replica count, Please note that the replica Pods in a Deployment are typically distributed across multiple nodes.revisionHistoryLimit: 10 # The number of old ReplicaSets to retain to allow rollbackselector: # label of the Pod that the Deployment is managing,, it's mandatory, without it , we will get this error # error: error validating data: ValidationError(Deployment.spec.selector): missing required field "matchLabels" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector ..matchLabels:app: bq-api-servicestrategy: # Strategy of upodatetype: RollingUpdate # RollingUpdate or RecreaterollingUpdate:maxSurge: 25% # The maximum number of Pods that can be created over the desired number of Pods during the updatemaxUnavailable: 25% # The maximum number of Pods that can be unavailable during the updatetemplate: # Pod templatemetadata:labels:app: bq-api-service # label of the Pod that the Deployment is managing. must match the selector, otherwise, will get the error Invalid value: map[string]string{"app":"bq-api-xxx"}: `selector` does not match template `labels`spec:containers:- image: europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.7 # image of the containerimagePullPolicy: IfNotPresentname: bq-api-service-containerenv: # set env varaibles- name: APP_ENVIRONMENTvalue: prodrestartPolicy: Always # Restart policy for all containers within the PodterminationGracePeriodSeconds: 10 # The period of time in seconds given to the Pod to terminate gracefully---
apiVersion: v1
kind: Service
metadata:name: clusterip-bq-api-service
spec:selector:app: bq-api-service # for the pods that have the label app: bq-api-serviceports:- protocol: TCPport: 8080targetPort: 8080type: ClusterIP
执行命令：
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/test$ kubectl apply -f bq-api-service.yml
deployment.apps/deployment-bq-api-service created
service/clusterip-bq-api-service unchanged
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/test$ kubectl get ep -o wide
NAME ENDPOINTS AGE
clusterip-bq-api-service 10.244.1.70:8080,10.244.1.71:8080,10.244.2.141:8080 + 1 more... 36s
kubernetes 192.168.0.3:6443 83d
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/test$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deployment-bq-api-service-6f6ffc7866-2mn5n 1/1 Running 0 27s 10.244.2.141 k8s-node0 <none> <none>
deployment-bq-api-service-6f6ffc7866-74bcq 1/1 Running 0 27s 10.244.1.70 k8s-node1 <none> <none>
deployment-bq-api-service-6f6ffc7866-l2xzb 1/1 Running 0 27s 10.244.1.71 k8s-node1 <none> <none>
deployment-bq-api-service-6f6ffc7866-lwxt7 1/1 Running 0 27s 10.244.3.82 k8s-node3 <none> <none>
看起来没什么问题
测试cluster ip 的连接
先进入dns-test
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/test$ kubectl run dns-test --image=odise/busybox-curl --restart=Never -- /bin/sh -c "while true; do echo hello docker; sleep 1; done"
pod/dns-test created
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/test$ kubectl exec -it dns-test -- /bin/sh
/ # 测试调用clusterip service
/ # curl clusterip-bq-api-service:8080/actuator/info
{"app":"Sales API","version":"1.1.7","hostname":"deployment-bq-api-service-6f6ffc7866-2mn5n","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}/ #
/ # curl clusterip-bq-api-service:8080/actuator/info
/ # curl clusterip-bq-api-service:8080/actuator/info
{"app":"Sales API","version":"1.1.7","hostname":"deployment-bq-api-service-6f6ffc7866-l2xzb","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}/ #
看来 reverse proxy 和 loadbalance 都生效～
安装helm
下载：
https://github.com/helm/helm/releases
配置~/.bashrc 就好
export PATH=$PATH:/home/gateman/devtools/helm
测试version
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/test$ helm version
version.BuildInfo{Version:"v3.15.2", GitCommit:"1a500d5625419a524fdae4b33de351cc4f58ec35", GitTreeState:"clean", GoVersion:"go1.22.4"}
通过helm 安装 ingress-nginx controller
把 ingress-ngnix 的repo 添加到helm
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
查看repo list
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/test$ helm repo list
NAME URL
ingress-nginx https://kubernetes.github.io/ingress-nginx
helm 下载 ingress-nginx package
helm pull <package-name>
helm pull ingress-nginx/ingress-nginx
这时， 会下载1个tgz 文件 ingress-nginx-4.10.1.tgz
解压：
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/ingress-nginx/helm/ingress-nginx$ ls -l
total 132
drwxrwxr-x 2 gateman gateman 4096 6月 25 01:26 changelog
-rw-r--r-- 1 gateman gateman 752 4月 26 22:02 Chart.yaml
drwxrwxr-x 2 gateman gateman 4096 6月 25 01:26 ci
-rw-r--r-- 1 gateman gateman 177 4月 26 22:02 OWNERS
-rw-r--r-- 1 gateman gateman 49134 4月 26 22:02 README.md
-rw-r--r-- 1 gateman gateman 11358 4月 26 22:02 README.md.gotmpl
drwxrwxr-x 3 gateman gateman 4096 6月 25 01:26 templates
drwxrwxr-x 2 gateman gateman 4096 6月 25 01:26 tests
-rw-r--r-- 1 gateman gateman 45136 6月 25 02:02 values.yaml
其中values.yaml 就是最终要的配置文件
修改values.yaml
有些教程让我们修改 image repo 的 address 指向阿里云加速， 由于我的server在google cloud 就忽略这一步了。
-
controller.kind -> 由 Deployment 改成 DaemonSet
这样做的优点是
a. 能够保证每个Node节点上都运行一个 ingress-nginx Pod实例,实现全集群访问入口。(实际上我们会用label 只让ingress-nginx 部署在1台node上）
b. 节点故障转移时,新的Node上会自动启动Pod,保证入口始终可用。 -
controller.hostnetwork -> 由false 改成 true
这是DaemonSet的默认网络模式。每个Pod将直接共享Node的网络栈和资源,简化了网络配置。
但同时也会带来上面提到的一些隔离性和安全性问题。 -
controller.dnsPolic-> 由 ClusterFirst 改成 ClusterFirstWithHostNet
ClusterFirst表示使用集群提供的DNS服务进行解析,这也是默认策略。
但是DaemonSet使用hostNetwork模式后,Pod会直接使用宿主机的网络命名空间。
此时设置dnsPolicy=ClusterFirstWithHostNet表示:
首先使用集群的DNS服务来解析域名。
如果集群DNS无法响应,则解析请求会转发到宿主机的DNS服务上寻求响应。
这样可以实现两个目的:
使用集群的DNS服务提高可配置性和维护性。
同时如果集群DNSdown掉,Pod也可以fallback到宿主机本地DNS上,保证关键服务名能被解析到。
这相比直接使用宿主机DNS或仅使用集群DNS更加灵活。
因此对于使用hostNetwork的DaemonSet,dnsPolicy设置为ClusterFirstWithHostNet可以充分利用集群和宿主机的DNS功能,提高服务可用性。 -
controller.service.type -> 由 Loadbalancer 改成 ClusterIP
对于用Helm部署ingress-nginx,使用ClusterIP作为service类型通常是更好的选择。
LoadBalancer类型会要求底层基础设施如云平台自动为ingress分配公网IP地址,但这可能会产生额外的网络开支。
考虑到大多数K8S集群运行在内网环境,使用ClusterIP就可以满足需求。后续还可以配置支持内网访问的解决方案,如NodePort或External IPs。
一般来说,你可以直接在values.yaml里修改service.type为ClusterIP:
在部署 ingress-nginx controller 之前(我们已经把它改成1个DaemonSet)， 为它创建1个单独的namespace
kubectl create ns ingress-nginx
我们之后会单独地把 ingress-ingress 部署在这个 同名字的namespace 上面。
问题来了， 在命名空间A的 ingress controller 可以用与命名空间B的ingress 实例吗？
答案是yes：
关于Ingress controller namespace与Ingress资源namespace的关系:1. Ingress controller一般会部署在单独的namespace下,比如ingress-namespace。
2. Ingress资源可以定义在任何namespace中。
3. Kubernetes会检测所有namespace下是否有Ingress Controller的服务运行。如果有,则Ingress资源对应的namespace会自动重定向到Ingress controller所在的namespace。Ingress controller通过监视Apiserver,可以查询到集群内所有namespace下定义的Ingress资源。也就是说:1. Ingress controller和Ingress资源可以部署在不同的namespace。
2. Kubernetes会自动帮助Ingress资源找到部署在哪个namespace的Ingress controller。
3. Ingress controller 通过Apiserver可以查到所有namespace下的Ingress定义。
为所在node添加label
ingress-controller 作为1个类网关程序， 理论上只在其中1个node 部署就够了
由于我们修改了value.yaml, 把这个controll的类型改成了DaemonSet 所以， 理论上每个node 都会部署1个pod, 是有资源浪费的。
而实际上， value.yaml 有定义了1个规则- node selector, 只有具有某个label ingress=true 的node 才会部署
node selector 的定义：
controller.nodeSelector:
nodeSelector: # nodeSelector of DaemonSet we updatedkubernetes.io/os: linuxingress: "true" # use String instead of bool
所以我们只需要为1台node 添加这个label 就好
但是， 如果我们只给master node添加label是不work的， 因为master node 配置了污点， 会bypass 大部分 deployment/DaemonSet 的部署
这次我们只给k8s-node0 这个node 添加label
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/ingress-nginx/helm/ingress-nginx$ kubectl label no k8s-node0 ingress=true
node/k8s-node0 labeled
查看label
gateman@MoreFine-S500:~/tmp/ingress-nginx$ kubectl get nodes --show-labels=true
NAME STATUS ROLES AGE VERSION LABELS
k8s-master Ready control-plane,master 128d v1.23.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ingress=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=
k8s-node0 Ready <none> 128d v1.23.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ingress=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node0,kubernetes.io/os=linux
k8s-node1 Ready <none> 128d v1.23.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node1,kubernetes.io/os=linux
k8s-node3 Ready <none> 108d v1.23.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node3,kubernetes.io/os=linux部署ingress-inginx controller
进入之前解压的folder (value.yaml)的上一层
执行
helm install nginx
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/ingress-nginx/helm/ingress-nginx$ helm install ingress-nginx -n ingress-nginx .
NAME: ingress-nginx
LAST DEPLOYED: Tue Jun 25 02:16:57 2024
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
检查pods
gateman@MoreFine-S500:~/tmp/ingress-nginx$ kubectl get po -o wide -n ingress-nginx
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-controller-72dmb 1/1 Running 3 (2d18h ago) 6d20h 192.168.0.6 k8s-node0 <none> <none>
可见这个ingress-controller 已经运行在了 k8s-node0
创建1个ingress 资源
创建yaml
yaml file：
apiVersion: networking.k8s.io/v1
kind: Ingress # it is a resource name of k8s
metadata:name: ingress-nginx-instance-0 # the ingress namenamespace: default # the namespace of the ingressannotations: # Ingress frequently uses annotations to configure some options depending on the Ingress controller, # an example of which is the rewrite-target annotation. Different Ingress controllers support different annotations.kubernetes.io/ingress.class: nginx # it will search the ingress controller with the same class
spec: # spec means specification, it is the main part of the Ingress resourcerules: # rules is a list of host rules used to configure the Ingress controller- host: "www.example.com" # host is the domain name of the Ingress controller, it could be a wildcard, but must start with a wildcard * , e.g. *.example.comhttp: # http is a list of http rules used to configure the Ingress controller for HTTP traffic, the port is 80, cannot be changed for ingress-nginxpaths: # Equivalent to the location setting of nginx, cuold be multiple- pathType: Prefix # supports Prefix and Extract and ImplementationSpecificbackend: # backend is the service to forward the request toservice: # Usually the cluster-ipname: clusterip-bq-api-serviceport: number: 8080path: /actuator # Equivalent to the location setting of nginx
上面是1个最简单的ingress resource, 基本上我每一行都加上了注解
值得注意的有如下几点
- 既然有metadata, 为什么还需要annotations？
的确两者都可以用来存储一些元数据，但是metadata是k8s的内建字段，而annotations是用户自定义的字段, 而在ingress 中，annotation 的配置是非常重要的，因为不同的ingress controller 支持的annotation是不一样的，所以annotations是必须的 - pathType： 这里我只用了Prefix, prefix 也是sprint cloud gateway 常用的类型， 但是在ingress-nginx 中更常用的是ImplementationSpecific， 后面会表述原因
- 这个配置的规则是 ingress:80 -> clusterip:8080 其中ingress-nginx 的端口是不支持修改的， 要么是http 80, 要么是tls 443
执行创建命令
$ kubectl delete ingress ingress-nginx-instance-0
ingress.networking.k8s.io "ingress-nginx-instance-0" deleted
查看ingress 信息和测试
首先 浏览一下这个创建号的ingress
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/ingress-nginx/ingress-resource$ kubectl get ingress -o wide
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-nginx-instance-0 <none> www.example.com 10.107.95.68 80 53m
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/ingress-nginx/ingress-resource$ kubectl describe ingress ingress-nginx-instance-0
Name: ingress-nginx-instance-0
Labels: <none>
Namespace: default
Address: 10.107.95.68
Ingress Class: <none>
Default backend: <default>
Rules:Host Path Backends---- ---- --------www.example.com /actuator clusterip-bq-api-service:8080 (10.244.2.146:8080,10.244.2.147:8080,10.244.3.86:8080 + 1 more...)
Annotations: kubernetes.io/ingress.class: nginx
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Sync 52m (x2 over 53m) nginx-ingress-controller Scheduled for sync
可以见到这个ingress 被分配了容器内ip
我们试下是否能在容器内调用
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/ingress-nginx/ingress-resource$ kubectl exec -it dns-test -- /bin/sh
/ # ping 10.107.95.68
PING 10.107.95.68 (10.107.95.68): 56 data bytes
^C
--- 10.107.95.68 ping statistics ---
11 packets transmitted, 0 packets received, 100% packet loss
/ # curl 10.107.95.68/actuator/info
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
/ #
这个ip 在无法在容器内ping 通
但是能通过curl 调用， 因为nginx 规则， 必须用指定域名 www.example.com (见上面的yaml 配置）
无法调用成功
我们在容器内 set 一下hostname：
10.107.95.68 www.example.com
调用成功：
/ # nslookup www.example.com
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.localName: www.example.com
Address 1: 10.107.95.68 www.example.com
/ # curl www.example.com/actuator/info
{"app":"Sales API","version":"1.1.7","hostname":"deployment-bq-api-service-6f6ffc7866-g4854","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}
在集群外的server call ingress
上面我们 edit 容器内的/etc/hosts match了 10.107.95.68 www.example.com
但是这个ip 在容器外的server 特别是集群内的server 是无法访问的。
那么集群外的server 如何访问ingress呢
这时我们登陆 ingress-nginx controller 所create的pod, 所在的node
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/ingress-nginx/ingress-resource$ kubectl get pods --all-namespaces -o wide | grep ingress
ingress-nginx ingress-nginx-controller-72dmb 1/1 Running 4 (8d ago) 15d 192.168.0.6 k8s-node0 <none> <none>
查看k8s-node0 的端口占用
gateman@k8s-node0:~$ sudo ss -tunlp | grep nginx
tcp LISTEN 0 4096 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=67247,fd=23),("nginx",pid=67246,fd=23),("nginx",pid=67245,fd=23),("nginx",pid=67244,fd=23),("nginx",pid=3823,fd=23))
tcp LISTEN 0 4096 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=67247,fd=22),("nginx",pid=67246,fd=22),("nginx",pid=67245,fd=22),("nginx",pid=67244,fd=22),("nginx",pid=3823,fd=22))
tcp LISTEN 0 4096 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=67247,fd=21),("nginx",pid=67246,fd=21),("nginx",pid=67245,fd=21),("nginx",pid=67244,fd=21),("nginx",pid=3823,fd=21))
tcp LISTEN 0 4096 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=67247,fd=9),("nginx",pid=67246,fd=9),("nginx",pid=67245,fd=9),("nginx",pid=67244,fd=9),("nginx",pid=3823,fd=9))
tcp LISTEN 0 4096 127.0.0.1:10245 0.0.0.0:* users:(("nginx-ingress-c",pid=2207,fd=7))
tcp LISTEN 0 511 127.0.0.1:10246 0.0.0.0:* users:(("nginx",pid=67247,fd=13),("nginx",pid=67246,fd=13),("nginx",pid=67245,fd=13),("nginx",pid=67244,fd=13),("nginx",pid=3823,fd=13))
tcp LISTEN 0 511 127.0.0.1:10247 0.0.0.0:* users:(("nginx",pid=67247,fd=14),("nginx",pid=67246,fd=14),("nginx",pid=67245,fd=14),("nginx",pid=67244,fd=14),("nginx",pid=3823,fd=14))
tcp LISTEN 0 4096 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=67247,fd=17),("nginx",pid=67246,fd=17),("nginx",pid=67245,fd=17),("nginx",pid=67244,fd=17),("nginx",pid=3823,fd=17))
tcp LISTEN 0 4096 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=67247,fd=16),("nginx",pid=67246,fd=16),("nginx",pid=67245,fd=16),("nginx",pid=67244,fd=16),("nginx",pid=3823,fd=16))
tcp LISTEN 0 4096 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=67247,fd=15),("nginx",pid=67246,fd=15),("nginx",pid=67245,fd=15),("nginx",pid=67244,fd=15),("nginx",pid=3823,fd=15))
tcp LISTEN 0 4096 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=67247,fd=7),("nginx",pid=67246,fd=7),("nginx",pid=67245,fd=7),("nginx",pid=67244,fd=7),("nginx",pid=3823,fd=7))
tcp LISTEN 0 4096 0.0.0.0:8181 0.0.0.0:* users:(("nginx",pid=67247,fd=29),("nginx",pid=67246,fd=29),("nginx",pid=67245,fd=29),("nginx",pid=67244,fd=29),("nginx",pid=3823,fd=29))
tcp LISTEN 0 4096 0.0.0.0:8181 0.0.0.0:* users:(("nginx",pid=67247,fd=28),("nginx",pid=67246,fd=28),("nginx",pid=67245,fd=28),("nginx",pid=67244,fd=28),("nginx",pid=3823,fd=28))
tcp LISTEN 0 4096 0.0.0.0:8181 0.0.0.0:* users:(("nginx",pid=67247,fd=27),("nginx",pid=67246,fd=27),("nginx",pid=67245,fd=27),("nginx",pid=67244,fd=27),("nginx",pid=3823,fd=27))
tcp LISTEN 0 4096 0.0.0.0:8181 0.0.0.0:* users:(("nginx",pid=67247,fd=11),("nginx",pid=67246,fd=11),("nginx",pid=67245,fd=11),("nginx",pid=67244,fd=11),("nginx",pid=3823,fd=11))
tcp LISTEN 0 4096 *:8443 *:* users:(("nginx-ingress-c",pid=2207,fd=12))
tcp LISTEN 0 4096 [::]:443 [::]:* users:(("nginx",pid=67247,fd=10),("nginx",pid=67246,fd=10),("nginx",pid=67245,fd=10),("nginx",pid=67244,fd=10),("nginx",pid=3823,fd=10))
tcp LISTEN 0 4096 [::]:443 [::]:* users:(("nginx",pid=67247,fd=24),("nginx",pid=67246,fd=24),("nginx",pid=67245,fd=24),("nginx",pid=67244,fd=24),("nginx",pid=3823,fd=24))
tcp LISTEN 0 4096 [::]:443 [::]:* users:(("nginx",pid=67247,fd=25),("nginx",pid=67246,fd=25),("nginx",pid=67245,fd=25),("nginx",pid=67244,fd=25),("nginx",pid=3823,fd=25))
tcp LISTEN 0 4096 [::]:443 [::]:* users:(("nginx",pid=67247,fd=26),("nginx",pid=67246,fd=26),("nginx",pid=67245,fd=26),("nginx",pid=67244,fd=26),("nginx",pid=3823,fd=26))
tcp LISTEN 0 4096 *:10254 *:* users:(("nginx-ingress-c",pid=2207,fd=11))
tcp LISTEN 0 4096 [::]:80 [::]:* users:(("nginx",pid=67247,fd=8),("nginx",pid=67246,fd=8),("nginx",pid=67245,fd=8),("nginx",pid=67244,fd=8),("nginx",pid=3823,fd=8))
tcp LISTEN 0 4096 [::]:80 [::]:* users:(("nginx",pid=67247,fd=18),("nginx",pid=67246,fd=18),("nginx",pid=67245,fd=18),("nginx",pid=67244,fd=18),("nginx",pid=3823,fd=18))
tcp LISTEN 0 4096 [::]:80 [::]:* users:(("nginx",pid=67247,fd=19),("nginx",pid=67246,fd=19),("nginx",pid=67245,fd=19),("nginx",pid=67244,fd=19),("nginx",pid=3823,fd=19))
tcp LISTEN 0 4096 [::]:80 [::]:* users:(("nginx",pid=67247,fd=20),("nginx",pid=67246,fd=20),("nginx",pid=67245,fd=20),("nginx",pid=67244,fd=20),("nginx",pid=3823,fd=20))
tcp LISTEN 0 4096 [::]:8181 [::]:* users:(("nginx",pid=67247,fd=12),("nginx",pid=67246,fd=12),("nginx",pid=67245,fd=12),("nginx",pid=67244,fd=12),("nginx",pid=3823,fd=12))
tcp LISTEN 0 4096 [::]:8181 [::]:* users:(("nginx",pid=67247,fd=30),("nginx",pid=67246,fd=30),("nginx",pid=67245,fd=30),("nginx",pid=67244,fd=30),("nginx",pid=3823,fd=30))
tcp LISTEN 0 4096 [::]:8181 [::]:* users:(("nginx",pid=67247,fd=31),("nginx",pid=67246,fd=31),("nginx",pid=67245,fd=31),("nginx",pid=67244,fd=31),("nginx",pid=3823,fd=31))
tcp LISTEN 0 4096 [::]:8181 [::]:* users:(("nginx",pid=67247,fd=32),("nginx",pid=67246,fd=32),("nginx",pid=67245,fd=32),("nginx",pid=67244,fd=32),("nginx",pid=3823,fd=32))
可以见到已经有nginx 的worker 在监听80端口了
实际上通过 访问k8s-node0 80端口 来放问ingress
登陆一台集群外主机 (仍在同1个vpc-network）
gcloud compute ssh tf-vpc0-subnet0-main-server
edit /etc/host
gateman@tf-vpc0-subnet0-main-server:~$ sudo vi /etc/hosts
gateman@tf-vpc0-subnet0-main-server:~$ cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters192.168.1.5 tf-vpc0-subnet1-vm1
192.168.0.42 tf-vpc0-subnet0-mysql0
192.168.0.3 k8s-master
192.168.0.35 tf-vpc0-subnet0-main-server.c.jason-hsbc.internal tf-vpc0-subnet0-main-server # Added by Google
169.254.169.254 metadata.google.internal # Added by Google
192.168.0.6 www.example.com
gateman@tf-vpc0-subnet0-main-server:~$
其中192.168.0.6 就是k8s-node0的ip
测试
用ip 和 k8s-node0 这个域名都是无法call通的
gateman@tf-vpc0-subnet0-main-server:~$ curl 192.168.0.6/actuator/info
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
gateman@tf-vpc0-subnet0-main-server:~$ curl k8s-node0/actuator/info
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
但是用www.example.com 就可以了：
gateman@tf-vpc0-subnet0-main-server:~$ curl www.example.com/actuator/info
{"app":"Sales API","version":"1.1.7","hostname":"deployment-bq-api-service-6f6ffc7866-g4854","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}
框架图
到这里我们已经基本实现了ingress 的作用， 让1台k8s 内网外的server 访问k8s 内的pods 部署的api
注意k8s 外部网络主机访问ingress 并不是直接访问ingress 而是 访问Ingress controller 所在node 的ip
path 重定向
需求
假设上面的例子 1个ingress 只对1个service 上面的写法在此情况下是无问题的。
但是假如1个ingress 对两个service 就不行了
假如 service a 和 service b 都有两个相同的path 的api
e.g. 都有 /actuator/info
我希望
ingress 的 /service-a/actuator --> service a 的/actuator** , /service-b/actuator --> service b 的/actuator**
先试下简单粗暴的写法：
apiVersion: networking.k8s.io/v1
kind: Ingress # it is a resource name of k8s
metadata:name: ingress-nginx-instance-0 # the ingress namenamespace: default # the namespace of the ingressannotations: # Ingress frequently uses annotations to configure some options depending on the Ingress controller, # an example of which is the rewrite-target annotation. Different Ingress controllers support different annotations.kubernetes.io/ingress.class: nginx # it will search the ingress controller with the same class
spec: # spec means specification, it is the main part of the Ingress resourcerules: # rules is a list of host rules used to configure the Ingress controller- host: "www.example.com" # host is the domain name of the Ingress controller, it could be a wildcard, but must start with a wildcard * , e.g. *.example.comhttp: # http is a list of http rules used to configure the Ingress controller for HTTP traffic, the port is 80, cannot be changed for ingress-nginxpaths: # Equivalent to the location setting of nginx, cuold be multiple- pathType: Prefix # supports Prefix and Extract and ImplementationSpecificbackend: # backend is the service to forward the request toservice: # Usually the cluster-ipname: clusterip-bq-api-serviceport: number: 8080path: /bq-api-service/actuator # change here
测试， 是不行的， 因为 bq-api-service 根本没有/bq-api-service 开头的api
gateman@tf-vpc0-subnet0-main-server:~$ curl www.example.com/actuator/info
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
gateman@tf-vpc0-subnet0-main-server:~$ curl www.example.com/bq-api-service/actuator/info
{"timestamp":"2024-07-11T16:57:59.849+00:00","status":404,"error":"Not Found","message":"No message available","path":"/bq-api-service/actuator/info"}
nginx.ingress.kubernetes.io/rewrite-target
查阅官方文档
我们可以通过1个 annotation 实现这个功能
改写成
apiVersion: networking.k8s.io/v1
kind: Ingress # it is a resource name of k8s
metadata:name: ingress-nginx-instance-0 # the ingress namenamespace: default # the namespace of the ingressannotations: # Ingress frequently uses annotations to configure some options depending on the Ingress controller, # an example of which is the rewrite-target annotation. Different Ingress controllers support different annotations.kubernetes.io/ingress.class: nginx # it will search the ingress controller with the same classnginx.ingress.kubernetes.io/rewrite-target: /$2 # rewrite the url let ingress/bq-api-service/xxx point to backend:8080/xxx
spec: # spec means specification, it is the main part of the Ingress resourcerules: # rules is a list of host rules used to configure the Ingress controller- host: "www.example.com" # host is the domain name of the Ingress controller, it could be a wildcard, but must start with a wildcard * , e.g. *.example.comhttp: # http is a list of http rules used to configure the Ingress controller for HTTP traffic, the port is 80, cannot be changed for ingress-nginxpaths: # Equivalent to the location setting of nginx, cuold be multiple- pathType: ImplementationSpecific # supports Prefix and Extract and ImplementationSpecific# if we need to use rewrite-target, we must use ImplementationSpecific, because ony this could support regular expression pathbackend: # backend is the service to forward the request toservice: # Usually the cluster-ipname: clusterip-bq-api-serviceport: number: 8080path: /bq-api-service(/|$)(.*) # Equivalent to the location setting of nginx# means if the url is /bq-api-service/xxx, it will forward to backend:8080/xxx# (/|$) means / or line end# /bq-api-service(/|$)(.*) with /$2 , here $2 means the second Capture groups, the (.*) part# we could also use /bq-api-service(.*) with $1, please note here $1 do not have / before it
请注意我改了3个地方
- pathType 由 prefix -> ImplementationSpecific ， 因为只有ImplementationSpecific 支持正则表达式 的path
- 添加 annotation nginx.ingress.kubernetes.io/rewrite-target: /$2 这里表示 正则表达式path 的第2 捕获组
- path: /bq-api-service(/|$)(.*) 这里改成了正则表达式
这里再详细解释正则表达式的意思
(/|$) 表示 bq-api-service 后面可以跟着/ 或者 行结束符号， 注意这里的$ 并不是$字符， 而是表示行结束了 这里是第1个捕获组
(.*) 表示任何字符 第2个捕获组
当 传入/bq-api-service 时是可以匹配的 因为后面跟着行结束符号， 这时(.*)就什么都不是了 /$2 就是/
当掺入/bq-api-service/aaa/bb/ccc时 $2 就是aaa/bb/ccc 而/$2 就是 /aaa/bbb/cc
实际上 我们用 /bq-api-service(.*) 和 $1 (注意并不是/$1) 亲测效果是一样的， 由于官方推荐的是上面的写法， 我们就随官方文档
测试
测试通过
gateman@tf-vpc0-subnet0-main-server:~$ curl www.example.com/bq-api-service/actuator/info
{"app":"Sales API","version":"1.1.7","hostname":"deployment-bq-api-service-6f6ffc7866-lwxt7","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}
gateman@tf-vpc0-subnet0-main-server:~$ curl www.example.com/bq-api-service/actuator/info
{"app":"Sales API","version":"1.1.7","hostname":"deployment-bq-api-service-6f6ffc7866-mxwcq","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}
在公网访问ingress
思路
由于我的ingress 所在的k8s-node0 是没有公网ip的 所以从公网无法直接访问
简单粗暴的方法当然是为k8s-node0 配1个公网ip, 但这并不是1个stretagy solution
思路
找一台有公网ip的 ， 并且在同1个vpc-network 的主机(双网卡), tf-vpc0-subnet0-main-service ， 在上面配1个nginx 代理， 指向ingress 的node k8s-node0
gateman@MoreFine-S500:~/projects/coding/k8s-s/ingress/ingress-nginx/ingress-resource$ gcloud compute instances list
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
k8s-master europe-west2-c n2d-highmem-2 true 192.168.0.3 RUNNING
k8s-node0 europe-west2-c n2d-highmem-4 true 192.168.0.6 RUNNING
k8s-node1 europe-west2-c n2d-highmem-4 true 192.168.0.44 RUNNING
k8s-node2 europe-west2-c n2d-highmem-4 true 192.168.0.43 TERMINATED
k8s-node3 europe-west2-c n2d-highmem-4 true 192.168.0.45 RUNNING
tf-vpc0-subnet0-main-server europe-west2-c n2d-standard-4 true 192.168.0.35 34.39.2.90 RUNNING
框架图
配置域名
首先 为公网ip的主机配置1个域名
www.jp-gcp-vms.cloud
至于这个域名怎么来的， 当然是付费的了， 十几块第一年
前提是你的云主机有1个固定ip
在公网主机配置方向代理
前提要装个ingress 啦
8085_k8s_ingress.conf
upstream k8s-ingress-server {server k8s-node0:80;
}server {listen 8085;server_name www.jp-gvms.cloud;location / {proxy_pass http://k8s-ingress-server;proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;}
}
配置是很简单的
就是通过公网 域名 www.jp-gcp-vms.cloud 访问8085 端口时, http request 会转发到 k8s-node0:80
其中k8s-node0 就是gcp vm的创建是指定的默认内部dns name
具体配置反向代理入门教程参考我另一篇文章：
Nginx 配置反向代理 - part 3
重新配置ingress yaml
apiVersion: networking.k8s.io/v1
kind: Ingress # it is a resource name of k8s
metadata:name: ingress-nginx-instance-0 # the ingress namenamespace: default # the namespace of the ingressannotations: # Ingress frequently uses annotations to configure some options depending on the Ingress controller, # an example of which is the rewrite-target annotation. Different Ingress controllers support different annotations.kubernetes.io/ingress.class: nginx # it will search the ingress controller with the same classnginx.ingress.kubernetes.io/rewrite-target: /$2 # rewrite the url let ingress/bq-api-service/xxx point to backend:8080/xxx
spec: # spec means specification, it is the main part of the Ingress resourcerules: # rules is a list of host rules used to configure the Ingress controller- host: "www.example.com" # host is the domain name of the Ingress controller, it could be a wildcard, but must start with a wildcard * , e.g. *.example.comhttp: # http is a list of http rules used to configure the Ingress controller for HTTP traffic, the port is 80, cannot be changed for ingress-nginxpaths: # Equivalent to the location setting of nginx, cuold be multiple- pathType: ImplementationSpecific # supports Prefix and Extract and ImplementationSpecific# if we need to use rewrite-target, we must use ImplementationSpecific, because ony this could support regular expression pathbackend: # backend is the service to forward the request toservice: # Usually the cluster-ipname: clusterip-bq-api-serviceport: number: 8080path: /bq-api-service(/|$)(.*) # Equivalent to the location setting of nginx# means if the url is /bq-api-service/xxx, it will forward to backend:8080/xxx# (/|$) means / or line end# /bq-api-service(/|$)(.*) with /$2 , here $2 means the second Capture groups, the (.*) part# we could also use /bq-api-service(.*) with $1, please note here $1 do not have / before it- host: "k8s-node0"http:paths: # Equivalent to the location setting of nginx, cuold be multiple- pathType: ImplementationSpecific # supports Prefix and Exactbackend:service:name: clusterip-bq-api-serviceport: number: 8080path: /bq-api-service(/|$)(.*) # Equivalent to the location setting of nginx
既然公网nginx 规则是由 www.jp-gcp-vms.cloud:8085 转发到 k8s-node0:80
那么我在ingress 配置多1个rules 就行了？
其中host: “www.example.com” for 内网访问
host: “k8s-node0” for 公网访问
看起来完美
实际上：
并不能匹配
大坑
trouble shooting
原因是上面nginx 设置
proxy_set_header Host $host;
把公网browser 访问的域名带过去了
但是ingress 并没有这个域名的配置
解决方法一：
在ingress 设置中把
- host: “k8s-node0” 改成 - host: “www.jp-gcp-vms.cloud”
亲测可行
但是
这样把域名同时设置在nginx 反向代理 和 ingress 中， 太不完美了
解决方法二：
把proxy_set_header Host $host; 去掉
upstream k8s-ingress-server {server k8s-node0:80;
}server {listen 8085;server_name www.jp-gvms.cloud;location / {proxy_pass http://k8s-ingress-server;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;}
}
亲测失败， 默认就会有header HOST 把用户访问的域名带过去了， 大坑
解决方法三：
强制指定 带过去的header Host 的值为 k8s-node0
upstream k8s-ingress-server {server k8s-node0:80;
}server {listen 8085;server_name www.jp-gvms.cloud;location / {proxy_pass http://k8s-ingress-server;proxy_set_header Host k8s-node0;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;}
}
这次测试也可以
这是本人找到最合适的方法了！
最后， 一些问题
-
1个ingress controller 可以对应多个ingress 吗？
答： 可以的 -
如果1个ingress 的controller 配有多个ingress instance, 里面的配置会冲突吗？ 如果有相同的path 设置
答： 会冲突， 所以要保证多个ingress instance 的配置中 path 互相不重复
如果一个 Ingress 控制器对应两个 Ingress 资源，并且这两个 Ingress 的配置中具有相同的路径，那么会发生冲突。Ingress 资源的路径配置用于定义请求的路径匹配规则，以确定将请求转发到哪个后端服务。如果存在多个 Ingress 资源具有相同的路径配置，Ingress 控制器将无法确定将请求路由到哪个后端服务，从而导致冲突。为了避免冲突，应确保每个 Ingress 资源的路径配置是唯一且不重叠的。这可以通过更改路径配置，使其在不同的 Ingress 资源中具有不同的值，或者通过使用其他条件（例如主机名）来进一步区分路径配置。如果需要为相同的路径配置创建多个 Ingress 资源，您可以考虑使用不同的主机名来区分它们，或者使用其他条件来进行区分，例如使用不同的路径前缀来将请求路由到不同的后端服务。总之，确保每个 Ingress 资源的路径配置是唯一的，以避免冲突并使 Ingress 控制器能够正确地路由请求到相应的后端服务。
![[misc]-流量包-wireshark-icmp](https://i-blog.csdnimg.cn/direct/a7aa333a3e99427b835baf2eb35224b7.png)
