K8S暴露pod内多个端口

一、背景

公司统一用的某个底包跑jar服务，只暴露了8080端口

二、需求

由于有些服务在启动jar服务后，会启动多个端口，除了8080端口，还有别的端口需要暴露，我这里就还需要暴露9999端口。

注：解决办法其实是可以直接改底包就好了，在底包中多暴露几个端口，但是我这边因为无法改底包，所以只能通过下面的办法解决。

三、解决办法

我们平时在打版升级的过程中，会基于底包写dockerfile来替换最新的jar包得到最终的镜像，所以可以这个dockerfile中添加暴露9999端口，这样同样也可以增加容器端口暴露，如下

$ cat dockerfile 
FROM 10.0.8.56/basis-images/basis:tomcatARG jar_nameRUN rm -rf /usr/local/tomcat/*
ADD ./target/${jar_name}.jar /usr/local/tomcat
ADD ./start.sh /usr/local/tomcat
EXPOSE 9999
RUN chmod +x /usr/local/tomcat/start.sh$ docker build --build-arg jar_name=nsw-ai-video . -t 10.0.8.56/nsyai-test/nsw-ai-video:2023-07-28-15-40
$ docker push 10.0.8.56/nsyai-test/nsw-ai-video:2023-07-28-15-40

四、实验在docker上跑容器，验证是否暴露出8080和9999端口（10.0.8.56是我的harbor私有镜像仓库）

[ yukw @ docker-work01 10.0.8.59 ] ~
$ docker login 10.0.8.56
Username: yukw
Password: 
WARNING! Your password will be stored unencrypted in /home/yukw/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded
[ yukw @ docker-work01 10.0.8.59 ] ~
$ docker run -d -P --name nsyai-test 10.0.8.56/nsyai-test/nsw-ai-video:2023-07-28-15-40
Unable to find image '10.0.8.56/nsyai-test/nsw-ai-video:2023-07-28-15-40' locally
2023-07-28-15-40: Pulling from nsyai-test/nsw-ai-video
a8c7037c15e9: Pull complete 
7f59206c4cb3: Pull complete 
d6593d2ee432: Pull complete 
47613084598b: Pull complete 
7ef22be88035: Pull complete 
edf70be6f818: Pull complete 
af72e686cb89: Pull complete 
376658e1b07e: Pull complete 
6991c8295d7f: Pull complete 
f0a023d2bec5: Pull complete 
9356db0572c6: Pull complete 
1cbc500b22f4: Pull complete 
2a8383c1d611: Pull complete 
962207b93da3: Pull complete 
9fdef278ff07: Pull complete 
8cc25cf21f3b: Pull complete 
Digest: sha256:e07a648e671746f4408565b2237584303cfdfb7d5a451adfa707dda3fc87d670
Status: Downloaded newer image for 10.0.8.56/nsyai-test/nsw-ai-video:2023-07-28-15-40
e11553520d6c4d94c71d8d11a699bd4d1c6df8202d4e1ec15b28ca1bcd21ff25
[ yukw @ docker-work01 10.0.8.59 ] ~
$ docker ps -a |grep 'nsyai-test'
e11553520d6c   10.0.8.56/nsyai-test/nsw-ai-video:2023-07-28-15-40   "/usr/local/tomcat/s…"   8 seconds ago   Up 7 seconds   0.0.0.0:49154->8080/tcp, 0.0.0.0:49153->9999/tcp   nsyai-test
9fd678ee8eeb   10.0.8.56/nsyai-test/nsyai-web:2023-07-12-12-01      "/docker-entrypoint.…"   2 weeks ago     Up 2 weeks     80/tcp                                             my-nsyai-test
[ yukw @ docker-work01 10.0.8.59 ] ~
$ docker port e11553520d6c
8080/tcp -> 0.0.0.0:49154
9999/tcp -> 0.0.0.0:49153

实验发现，端口暴露成功
容器8080端口随机映射到了宿主机49154端口
容器9999端口随机映射到了宿主机49153端口

五、编写dp.yaml

# cat dp.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:annotations:deployment.kubernetes.io/revision: "63"description: video模块labels:k8s-app: nsw-ai-videoqcloud-app: nsw-ai-videoname: nsw-ai-videonamespace: nsyai-test
spec:progressDeadlineSeconds: 600replicas: 1revisionHistoryLimit: 10selector:matchLabels:k8s-app: nsw-ai-videoqcloud-app: nsw-ai-videotemplate:metadata:labels:k8s-app: nsw-ai-videoqcloud-app: nsw-ai-videospec:containers:- name: nsw-ai-videoimage: 10.0.8.56/nsyai-test/nsw-ai-video:2023-07-28-15-40imagePullPolicy: AlwayslivenessProbe:failureThreshold: 5initialDelaySeconds: 180periodSeconds: 10successThreshold: 1tcpSocket:port: 8080timeoutSeconds: 6readinessProbe:failureThreshold: 5initialDelaySeconds: 60periodSeconds: 10successThreshold: 1tcpSocket:port: 8080timeoutSeconds: 60resources:limits:cpu: 2000mmemory: 2Girequests:cpu: "1"memory: 512Miports:- containerPort: 8080name: image-portprotocol: TCP- containerPort: 9999name: xxl-job-portprotocol: TCPimagePullSecrets:- name: nsw-harbor-secret

containerPort是在pod控制器中定义的、pod中的容器需要暴露的端口

六、编写svc.yaml

# cat svc.yaml 
apiVersion: v1
kind: Service
metadata:name: nsw-ai-videonamespace: nsyai-test
spec:externalTrafficPolicy: Clusterports:- name: 8080-8080-tcpnodePort: 30083port: 8080protocol: TCPtargetPort: 8080- name: 9999-9999-tcpnodePort: 30084port: 9999protocol: TCPtargetPort: 9999selector:k8s-app: nsw-ai-videoqcloud-app: nsw-ai-videotype: NodePort

七、应用配置清单

# kubectl apply -f dp.yaml
# kubectl apply -f svc.yaml# kubectl get svc -n nsyai-test
NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                         AGE
nsw-ai-gateway   ClusterIP   10.0.0.107   <none>        8080/TCP                        16d
nsw-ai-video     NodePort    10.0.0.119   <none>        8080:30083/TCP,9999:30084/TCP   3h25m
nsyai-pc-nginx   NodePort    10.0.0.185   <none>        80:30082/TCP                    16d

总结：

1、从上面可以发现，在制作pod镜像中EXPOSE暴露了两个端口，这个是容器本身需要暴露的端口，在dp.yaml中配置了两个containerPort，这个是pod中的容器需要暴露的端口，在svc.yaml中配置了nodePort，port，targetport，分别代表宿主机端口，service端口和容器端口。