Kubernetes 离线部署 Spinnaker

离线部署 Spinnaker

离线部署 spinnaker 需要提前准备以下依赖项

halyard 安装工具：该hal命令的apt源地址https://us-apt.pkg.dev/projects/spinnaker-community位于国外
halyard boms物料清单：Spinnaker 将其halyard boms配置存储在公共谷歌云存储 ( gs://halconfig ) 存储桶中。该位置无法访问，并且导致 hal 命令超时。
Spinnaker 微服务docker镜像：微服务的镜像存储在 us-docker.pkg.dev/spinnaker-community/docker/ 镜像仓库中，从国内访问该存储库同样很困难。

在这里插入图片描述

准备kubernetes集群

查看节点信息，kubernetes版本为v1.27.7，容器运行时为containerd.

root@node40:~# kubectl get nodes -o wide
NAME     STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node40   Ready    control-plane   11d   v1.27.7   192.168.72.40   <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd://1.6.24

已通过 helm 方式安装 openebs，用于通过 storageclass 为minio 动态提供存储卷

root@node40:~# kubectl -n openebs get pods
NAME                                           READY   STATUS    RESTARTS      AGE
openebs-localpv-provisioner-658c87d4bb-rtsc7   1/1     Running   2 (22m ago)   11d
root@node40:~# 
root@node40:~# kubectl get sc
NAME                         PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
openebs-hostpath (default)   openebs.io/local   Delete          WaitForFirstConsumer   false                  11d

已通过 helm 方式安装 minio，spinnaker依赖S3存储保存数据

root@node40:~# kubectl -n minio get pods
NAME                    READY   STATUS    RESTARTS   AGE
minio-8976fd4b7-42vkt   1/1     Running   0          103mroot@node40:~# kubectl -n minio get svc
NAME            TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
minio           NodePort   10.96.2.159   <none>        9000:32000/TCP   103m
minio-console   NodePort   10.96.1.5     <none>        9001:32001/TCP   103m

备注：记录minio service连接地址及凭证，后续spinnaker指定存储时需要使用，以上示例在集群内连接地址为http://minio.minio:9000。

已通过 helm 方式安装 metallb，为ingress-nginx提供LoadBalancer类型IP地址

root@node40:~# kubectl -n metallb-system get pods
NAME                                  READY   STATUS    RESTARTS   AGE
metallb-controller-5cd9b4944b-jwjbm   1/1     Running   0          104m
metallb-speaker-mqdh9                 4/4     Running   0          104m

已通过 helm 方式安装 ingress-nginx 网关，后续通过创建ingress规则，连接spinnaker UI.

root@node40:~# kubectl -n ingress-nginx get pods
NAME                                      READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-8db7bbbc-mdpsv   1/1     Running   0          107m
root@node40:~# kubectl -n ingress-nginx get svc
NAME                                 TYPE           CLUSTER-IP    EXTERNAL-IP      PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.96.3.108   192.168.72.191   80:31306/TCP,443:32253/TCP   107m
ingress-nginx-controller-admission   ClusterIP      10.96.2.255   <none>           443/TCP                      107m

备注：记录metallb为 ingress-nginx 分配的EXTERNAL-IP: 192.168.72.191，后续访问spinnaker解析ingress域名时需要使用。

下载spinnaker boms 依赖

halyard 配置存储在google gcs 存储桶中。所有标准安装都将引用此位置的元数据，例如可用版本列表、各个微服务每个版本的物料清单 (bom) 等。

可以使用以下命令查看存储桶中的文件，注意，执行该命令需要能够访问外网或配置代理。

docker run -it --rm --name gloud-cli docker.io/google/cloud-sdk:alpine \gcloud storage ls gs://halconfig

示例输出如下：

$ docker run -it --rm --name gloud-cli docker.io/google/cloud-sdk:alpine \gcloud storage ls gs://halconfig
gs://halconfig/versions.yml
gs://halconfig/bom/
gs://halconfig/clouddriver/
gs://halconfig/deck/
gs://halconfig/echo/
gs://halconfig/fiat/
gs://halconfig/front50/
gs://halconfig/gate/
gs://halconfig/igor/
gs://halconfig/kayenta/
gs://halconfig/monitoring-daemon/
gs://halconfig/orca/
gs://halconfig/rosco/

创建 google-cloud-sdk 容器

docker run -d --name gloud-cli -v /workspace:/workspace -w /workspace \--env HTTP_PROXY=http://192.168.72.1:7890 \--env HTTPS_PROXY=http://192.168.72.1:7890 \docker.io/google/cloud-sdk:alpine sleep infinity

下载整个halconfig文件

可以下载整个halconfig存储桶到本地，请注意， gsutil cp 命令可能需要相当长的时间，虽然大小仅有150M左右，但它有超过 55k 个文件需要拉取。

$ docker exec -it gloud-cli bash
gsutil cp -R -n gs://halconfig .
# gcloud storage cp -R -n gs://halconfig .
tar -zcvf halconfig.tar.gz halconfig

下载指定halconfig文件

实际我们只需要对应版本的boms文件，需要下载的文件列表如下：

gs://halconfig/versions.yml
gs://halconfig/bom/${spinnaker_version}.yml
gs://halconfig/clouddriver/${clouddriver_version}/
gs://halconfig/deck/${deck_version}/
gs://halconfig/echo/${echo_version}/
gs://halconfig/fiat/${fiat_version}/
gs://halconfig/front50/${front50_version}/
gs://halconfig/gate/${gate_version}/
gs://halconfig/igor/${igor_version}/
gs://halconfig/kayenta/${kayenta_version}/
gs://halconfig/monitoring-daemon/${monitoring-daemon_version}/
gs://halconfig/orca/${orca_version}/
gs://halconfig/rosco/${rosco_version}/

下面创建一个boms下载脚本

$ cat /workspace/download_boms.sh
#!/bin/bash
spinnaker_version=1.32.2
halconfig_path=./halconfig
spinnaker_bom_path=${halconfig_path}/bom
spinnaker_bom_file=${spinnaker_version}.ymlcurl -sL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.40.2/yq_linux_amd64
chmod a+x /usr/local/bin/yqmkdir -p ${spinnaker_bom_path}
gsutil -m cp -R gs://halconfig/versions.yml ./halconfig
gsutil -m cp -R gs://halconfig/bom/${spinnaker_bom_file} ${spinnaker_bom_path}
services=$(yq e '.services | keys | .[]' ${spinnaker_bom_path}/${spinnaker_bom_file})for service in ${services}; domkdir -p ${halconfig_path}/${service}spinnaker_service_path=${halconfig_path}/${service}version=$(service=${service} yq e '.services.[env(service)].version' ${spinnaker_bom_path}/${spinnaker_bom_file})gsutil -m cp -R gs://halconfig/${service}/${version} ${spinnaker_service_path}
done
yq e -i '.services.*.version |= "local:" + .' ${spinnaker_bom}
tar -zcvf halconfig.tar.gz ./halconfig

进入容器执行脚本下载boms

docker exec -it gloud-cli bash /workspace/download_boms.sh

查看下载并打包好的boms文件

root@ubuntu:~# ls /workspace/
download_boms.sh  halconfig  halconfig.tar.gz

下载docker镜像

创建 skopeo 容器，使用skopeo工具在两个远程仓库之间同步镜像，将spinnaker镜像转存到国内阿里云acr镜像仓库。

docker run -d --name skopeo -v /workspace:/workspace -w /workspace --entrypoint="" \--env HTTP_PROXY=http://192.168.72.1:7890 \--env HTTPS_PROXY=http://192.168.72.1:7890 \quay.io/skopeo/stable:latest sleep infinity

创建镜像下载脚本

$ cat /workspace/download_images.sh
#!/bin/bash
export spinnaker_version=1.32.2
export spinnaker_dockerRegistry="us-docker.pkg.dev/spinnaker-community/docker/"
export dockerhub_dockerRegistry="docker.io/library/"
export spinnaker_bom="./halconfig/bom/${spinnaker_version}.yml"export dest_registry=registry.cn-shenzhen.aliyuncs.com/cnmirror
export registry_username=xxxxxx
export registry_password=xxxxxxcurl -sL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.40.2/yq_linux_amd64
chmod a+x /usr/local/bin/yqyq -r '.services | to_entries | .[] | env(spinnaker_dockerRegistry) + .key + ":" + .value.version' ${spinnaker_bom} > spinnaker_images.txt
yq -r '.dependencies | to_entries | .[] | env(dockerhub_dockerRegistry) + .key + ":" + .value.version' ${spinnaker_bom} >> spinnaker_images.txt
echo "us-docker.pkg.dev/spinnaker-community/docker/halyard:stable" >> spinnaker_images.txtsed -i "s#docker.io/library/redis:.*#docker.io/library/redis:6.2#g" spinnaker_images.txt
sed -i '/monitoring-third-party/d' spinnaker_images.txtfor image in $(cat spinnaker_images.txt);doimage_name=$(echo $image | awk -F '/' '{print $NF}')skopeo sync --src docker --dest docker \--dest-username ${registry_username} --dest-password ${registry_password} \$image ${dest_registry}
done

进入容器执行脚本下载spinnaker镜像

docker exec -it skopeo bash /workspace/download_images.sh

查看生成的镜像清单，可能个别镜像并不需要，另外redis镜像tag需要从halyard源码仓库搜索获取。

root@ubuntu:~# cat /workspace/spinnaker_images.txt 
us-docker.pkg.dev/spinnaker-community/docker/clouddriver:5.82.0
us-docker.pkg.dev/spinnaker-community/docker/deck:3.15.1
us-docker.pkg.dev/spinnaker-community/docker/echo:2.39.0
us-docker.pkg.dev/spinnaker-community/docker/fiat:1.42.0
us-docker.pkg.dev/spinnaker-community/docker/front50:2.30.1
us-docker.pkg.dev/spinnaker-community/docker/gate:6.60.1
us-docker.pkg.dev/spinnaker-community/docker/igor:4.13.3
us-docker.pkg.dev/spinnaker-community/docker/kayenta:2.38.0
us-docker.pkg.dev/spinnaker-community/docker/monitoring-daemon:1.4.0
us-docker.pkg.dev/spinnaker-community/docker/orca:8.36.0
us-docker.pkg.dev/spinnaker-community/docker/rosco:1.18.0
docker.io/library/consul:0.7.5
docker.io/library/redis:6.2
docker.io/library/vault:0.7.0
us-docker.pkg.dev/spinnaker-community/docker/halyard:stable

docker安装halyard

以docker方式安装halyard

docker run -d --name halyard \-p 8084:8084 -p 9000:9000 \-w /home/spinnaker \-v /home/spinnaker/.hal:/home/spinnaker/.hal \-v /home/spinnaker/.kube:/home/spinnaker/.kube \-e KUBECONFIG=/home/spinnaker/.kube/config \registry.cn-shenzhen.aliyuncs.com/cnmirror/halyard:1.62.0

复制kubeconfig文件

使用scp命令复制kubernetes节点上的/root/.kube/config文件到halyard节点/home/spinnaker/.kube/目录下

cp config /home/spinnaker/.kube/config

修改主机目录权限，避免后续安装出现权限问题

uid=$(docker exec -it halyard id -u)
chown -R ${uid}:${uid} /home/spinnaker

进入halyard容器

docker exec -it halyard bash

验证与集群连接是否正常

kubectl get nodes

复制boms到halyard目录

解压halconfig.tar.gz到/home/spinnaker/.hal/.boms/目录下

mkdir -p /home/spinnaker/.hal/.boms/
tar -zxvf halconfig.tar.gz -C /home/spinnaker/.hal/.boms/ --strip=2
chown -R 1000:1000 /home/spinnaker/.hal/.boms/

boms目录结构如下：

root@ubuntu:~# tree /home/spinnaker/.hal/.boms/
/home/spinnaker/.hal/.boms/
├── bom
│   └── 1.32.2.yml
├── clouddriver
│   └── 5.82.0
│       ├── clouddriver-bootstrap.yml
│       ├── clouddriver-caching.yml
│       ├── clouddriver-ro-deck.yml
│       ├── clouddriver-ro.yml
│       ├── clouddriver-rw.yml
│       ├── clouddriver.yml
│       └── README.md
├── deck
│   └── 3.15.1
│       ├── README.md
│       └── settings.js
├── echo
│   └── 2.39.0
│       ├── echo-scheduler.yml
│       ├── echo-worker.yml
│       ├── echo.yml
│       └── README.md
├── fiat
│   └── 1.42.0
│       ├── fiat.yml
│       └── README.md
├── front50
│   └── 2.30.1
│       ├── front50.yml
│       └── README.md
├── gate
│   └── 6.60.1
│       ├── gate.yml
│       └── README.md
├── igor
│   └── 4.13.3
│       ├── igor.yml
│       └── README.md
├── kayenta
│   └── 2.38.0
│       ├── kayenta.yml
│       └── README.md
├── monitoring-daemon
│   └── 1.4.0
│       ├── README.md
│       └── spinnaker-monitoring.yml
├── monitoring-third-party
├── orca
│   └── 8.36.0
│       ├── orca-bootstrap.yml
│       ├── orca.yml
│       └── README.md
├── rosco
│   └── 1.18.0
│       ├── images.yml
│       ├── packer.tar.gz
│       ├── README.md
│       └── rosco.yml
└── versions.yml

修改/home/spinnaker/.hal/.boms/bom/<version>.yml文件，其中services.*.version版本字段前需要加local。配置完成之后，hal在deploy的时候不会去google拉取版本配置，而是从本地读取。

以下命令在download_boms.sh脚本中已执行，无需重复执行。

yq e -i '.services.*.version |= "local:" + .' /home/spinnaker/.hal/.boms/bom/1.32.2.yml

查看修改后的配置文件

root@ubuntu:~# cat /home/spinnaker/.hal/.boms/bom/1.32.2.yml
artifactSources:debianRepository: https://us-apt.pkg.dev/projects/spinnaker-communitydockerRegistry: registry.cn-shenzhen.aliyuncs.com/cnmirrorgitPrefix: https://github.com/spinnakergoogleImageProject: marketplace-spinnaker-release
dependencies:consul:version: 0.7.5redis:version: 2:2.8.4-2vault:version: 0.7.0
services:clouddriver:commit: 98204860175e00f8b1e01b43f8fa06670cc4cd2eversion: local:5.82.0deck:commit: 4bc0ae68f578f94fb5473a90d2db18d4c580944aversion: local:3.15.1echo:commit: c24b570673fc0f441a65fb69f6936027905c686dversion: local:2.39.0fiat:commit: 37a686b46d1bfe5569fff487151d6b8e84295169version: local:1.42.0front50:commit: f8d4d344e3858d9c3d45534d34a10e51181ece8eversion: local:2.30.1gate:commit: 431b73f62d5d2caec4619e64ca7084c0d61f7b9eversion: local:6.60.1igor:commit: 95c19bd7f49d5fd831104c8a0a92195a6eded154version: local:4.13.3kayenta:commit: 137036d870f9f4834eccd60d7895b212a42b9080version: local:2.38.0monitoring-daemon:commit: 96d510cb22f65dcf788324ed8b68447c31de255aversion: local:1.4.0monitoring-third-party:commit: 96d510cb22f65dcf788324ed8b68447c31de255aversion: local:1.4.0orca:commit: cb269b64cf2883317cd169cc7d82b75093ad6754version: local:8.36.0rosco:commit: a035e26d7fadeb258cb3c4206e386203c6ab1ac3version: local:1.18.0
timestamp: '2023-09-20 20:55:47'
version: 1.32.2

配置 halyard 以引用本地镜像

配置 halyard 从国内镜像仓库拉取 Spinnaker 微服务镜像。编辑 ~/.hal/.boms/bom 文件夹中的 <version>.yml 文件，修改dockerRegistry参数，例如：

$ vim /home/spinnaker/.hal/.boms/bom/1.32.2.yml
artifactSources:debianRepository: https://us-apt.pkg.dev/projects/spinnaker-communitydockerRegistry: registry.cn-shenzhen.aliyuncs.com/cnmirrorgitPrefix: https://github.com/spinnakergoogleImageProject: marketplace-spinnaker-release

本地 BOM 仅引用 Spinnaker 服务的私有注册表，而不引用 Redis 等依赖服务。如果想从自己的私有注册表中提取 Redis 映像，需要创建一个~/.hal/$DEPLOYMENT/service-settings/redis.yml文件，其中 $DEPLOYMENT 通常是默认值。

DEPLOYMENT=default
mkdir -p /home/spinnaker/.hal/$DEPLOYMENT/service-settings/
cat >/home/spinnaker/.hal/$DEPLOYMENT/service-settings/redis.yml<<EOF
artifactId: registry.cn-shenzhen.aliyuncs.com/cnmirror/redis:6.2
EOF
chown -R 1000:1000 /home/spinnaker

在单独的 shell 中，以root身份连接到 Halyard：

docker exec -it -u root halyard bash

禁用gcs，配置halyard使用本地boms文件，可以新建halyard-local.yml或修改halyard.yml

cat >/opt/halyard/config/halyard-local.yml<<EOF
spinnaker:config:input:gcs:enabled: false
EOF

重新启动halyard

docker restart halyard

halyard 配置 spinnaker

后续所有操作都在halyard容器中执行。

docker exec -it halyard bash

指定 Spinnaker 版本

列出当前halyard可用的spinnaker版本。

hal version list

选择一个要部署的 spinnaker 版本，这里以 spinnaker 最新版本为例，读取本地boms文件需要在版本号前加上local。

hal config version edit --version local:1.32.2

添加spinnaker账户

使用kubernetes cluster-admin作为spinnaker账户

hal config provider kubernetes account add spinnaker-admin \
--context $(kubectl config current-context)

为Spinnaker UI设置时区

hal config edit --timezone Asia/Shanghai

指定providers

配置提供程序，指定为kubernetes

hal config provider kubernetes enable

指定安装环境

分布式安装适用于资源占用量较大的开发组织，以及那些无法承受 Spinnaker 更新期间停机的情况。

使用您在配置提供程序时创建的$ACCOUNT名称运行以下命令：

hal config deploy edit --type distributed --account-name spinnaker-admin

指定外置存储

记录值如下:

export ENDPOINT="http://minio.minio:9000"
export MINIO_ACCESS_KEY=admin
export MINIO_SECRET_KEY=minio123

鉴于 Minio 不支持版本控制对象，需要在 Spinnaker 中禁用它。将以下行添加到 ~/.hal/$DEPLOYMENT/profiles/front50-local.yml ：

DEPLOYMENT=default
mkdir -p /home/spinnaker/.hal/$DEPLOYMENT/profiles
echo "spinnaker.s3.versioning: false" > /home/spinnaker/.hal/$DEPLOYMENT/profiles/front50-local.yml

该$DEPLOYMENT 通常是 default 。在这里内容。如果该文件不存在，则可能需要创建它。

编辑存储设置，运行以下命令（请注意，我们选择 S3 作为存储类型，因为 Minio 实现了 S3 API）：

echo $MINIO_SECRET_KEY | hal config storage s3 edit --path-style-access=true \--endpoint $ENDPOINT \--access-key-id $MINIO_ACCESS_KEY \--secret-access-key
hal config storage edit --type s3

配置spinnaker UI URL

修改Spinnaker Deck和Gate的域名配置，与后续创建的ingress规则域名一致。

hal config security ui edit --override-base-url http://spinnaker.example.com
hal config security api edit --override-base-url http://spinnaker.example.com/api/v1

修改gate组件配置，与override-base-url和ingress规则一致

mkdir -p /home/spinnaker/.hal/default/profiles
cat >/home/spinnaker/.hal/default/profiles/gate-local.yml<<EOF
server:servlet:context-path: /api/v1
EOFmkdir -p /home/spinnaker/.hal/default/service-settings
cat >/home/spinnaker/.hal/default/service-settings/gate.yml<<EOF
healthEndpoint: /api/v1/health
EOF

部署 Spinnaker

执行以下命令开始部署 Spinnaker 到 kubernetes 集群

hal deploy apply

命令输出结果如下：

spinnaker@ubuntu:~$ hal deploy apply
+ Get current deploymentSuccess
+ Prep deploymentSuccess
Validation in default.stats:
- INFO Stats are currently ENABLED. Usage statistics are beingcollected. Thank you! These stats inform improvements to the product, and thathelps the community. To disable, run `hal config stats disable`. To learn moreabout what and how stats data is used, please seehttps://spinnaker.io/docs/community/stay-informed/stats.+ Preparation complete... deploying Spinnaker
+ Get current deploymentSuccess
+ Apply deploymentSuccess
+ Deploy spin-redisSuccess
+ Deploy spin-clouddriverSuccess
+ Deploy spin-front50Success
+ Deploy spin-orcaSuccess
+ Deploy spin-deckSuccess
+ Deploy spin-echoSuccess
+ Deploy spin-gateSuccess
+ Deploy spin-roscoSuccess
+ Run `hal deploy connect` to connect to Spinnaker.

查看创建的pods，所有pod完全运行可能需要5~10m，需要耐心等待。

root@node40:~# kubectl -n spinnaker get pods
NAME                                READY   STATUS    RESTARTS   AGE
spin-clouddriver-77b9989787-962xc   1/1     Running   0          5m59s
spin-deck-7b6d795dcb-8pxfk          1/1     Running   0          6m
spin-echo-bbcd4f8f-nnnsf            1/1     Running   0          6m
spin-front50-58bf57cbcf-hd55d       1/1     Running   0          5m58s
spin-gate-5d4f6c9dd9-ks6m6          1/1     Running   0          6m
spin-orca-5cbf687487-bgbvx          1/1     Running   0          5m59s
spin-redis-6d475cc85c-mcv2v         1/1     Running   0          6m1s
spin-rosco-5df7654856-8pd7q         1/1     Running   0          5m58s

查看创建的service

root@node40:~# kubectl -n spinnaker get svc
NAME               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
spin-clouddriver   ClusterIP   10.96.0.171   <none>        7002/TCP   6m7s
spin-deck          ClusterIP   10.96.3.245   <none>        9000/TCP   6m6s
spin-echo          ClusterIP   10.96.2.106   <none>        8089/TCP   6m6s
spin-front50       ClusterIP   10.96.1.59    <none>        8080/TCP   6m7s
spin-gate          ClusterIP   10.96.1.2     <none>        8084/TCP   6m6s
spin-orca          ClusterIP   10.96.0.69    <none>        8083/TCP   6m7s
spin-redis         ClusterIP   10.96.3.62    <none>        6379/TCP   6m7s
spin-rosco         ClusterIP   10.96.2.66    <none>        8087/TCP   6m6s

查看pods镜像，默认从阿里云镜像仓库拉取：

root@node40:~# kubectl -n spinnaker get pods  -o yaml  | grep image: | sort -uimage: registry.cn-shenzhen.aliyuncs.com/cnmirror/redis:6.2image: registry.cn-shenzhen.aliyuncs.com/cnmirror/clouddriver:5.82.0image: registry.cn-shenzhen.aliyuncs.com/cnmirror/deck:3.15.1- image: registry.cn-shenzhen.aliyuncs.com/cnmirror/deck:3.15.1image: registry.cn-shenzhen.aliyuncs.com/cnmirror/echo:2.39.0image: registry.cn-shenzhen.aliyuncs.com/cnmirror/front50:2.30.1image: registry.cn-shenzhen.aliyuncs.com/cnmirror/gate:6.60.1image: registry.cn-shenzhen.aliyuncs.com/cnmirror/orca:8.36.0image: registry.cn-shenzhen.aliyuncs.com/cnmirror/rosco:1.18.0

连接到 Spinnaker UI

这里使用ingress连接spinnaker UI，创建如下Ingress规则

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: spin-ingressnamespace: spinnaker
spec:rules:- host: spinnaker.example.comhttp:paths:- path: /pathType: ImplementationSpecificbackend:service:name: spin-deckport:number: 9000- host: spinnaker.example.comhttp:paths:- path: /api/v1pathType: ImplementationSpecificbackend:service:name: spin-gateport:number: 8084
EOF

查看创建的ingress规则

root@node40:~# kubectl -n spinnaker get ingress
NAME           CLASS   HOSTS                                         ADDRESS          PORTS   AGE
spin-ingress   nginx   spinnaker.example.com,spinnaker.example.com   192.168.72.191   80      75s

本地配置hosts解析，其中 192.168.72.191 为ingress服务绑定的 EXTERNAL-IP