摆脱Jenkins - 使用google cloudbuild 部署 java service 到 compute engine VM

在之前介绍 cloud build 的文章中

初探 Google 云原生的CICD - CloudBuild

已经介绍过，用cloud build 去部署1个 spring boot service 到 cloud run 是很简单的，因为部署cloud run 无非就是用gcloud 去部署1个 GAR 上的docker image 到cloud run 容器
yaml file 例子：

steps:- id: check maven and jdk versionname: maven:3.9.6-sapmachine-17 # https://hub.docker.com/_/mavenentrypoint: mvnargs: ['--version']- id: run maven testname: maven:3.9.6-sapmachine-17 # https://hub.docker.com/_/mavenentrypoint: mvnargs: ['test']- id: run maven packagename: maven:3.9.6-sapmachine-17 # https://hub.docker.com/_/mavenentrypoint: mvnargs: ['package', '-Dmaven.test.skip=true']# https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values- id: build docker imagename: 'gcr.io/cloud-builders/docker'args: ['build', '-t', 'europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/demo-cloud-user', '.']- id: upload docker image to GARname: 'gcr.io/cloud-builders/docker'args: [ 'push', 'europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/demo-cloud-user']# deploy to Cloud run- id: deploy image to cloud runname: 'gcr.io/cloud-builders/gcloud'args: ['run', 'deploy', 'demo-cloud-user','--image=europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/demo-cloud-user','--port=8080','--platform=managed','--region=europe-west2','--no-allow-unauthenticated','--service-account=vm-common@jason-hsbc.iam.gserviceaccount.com','--key=projects/$PROJECT_ID/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek']
# https://stackoverflow.com/questions/68779751/error-publishing-source-code-from-cloud-build-to-a-bucket-using-triggers
logsBucket: gs://jason-hsbc_cloudbuild/logs/
options: # https://cloud.google.com/cloud-build/docs/build-config#optionslogging: GCS_ONLY # or CLOUD_LOGGING_ONLY https://cloud.google.com/cloud-build/docs/build-config#logging

部署到GCE的问题

但是cloud build 本身是1个non-vpc product, 是无法直接通过 GCE vm的subnet ip address 去连接vm的.

但是
gcloud compute ssh 本身是可以直接用 ssh key file 验证的
而且
gcloud compute ssh 后面可以带 – 参数执行1段指定的命令

所以实际上不同过内网ip 连接. 那就是讲用cloud build 部署service 到 vm的方案是可行的

部署到GCE的思路

首先准备1对ssh key pair, 并把public key安装在对应的vm中，确保可以用private key登陆
把这对key pari 放在 google security manager 中
在 cloudbuild yaml 中把这对key pari 引入，虽然理论上只需要私钥就可以，但是cloudbuild 也需要校验public key 奇怪了

编写cloudbuild yaml

a. mvn build
b. build docker image
c. push docker image to GAR
d. 利用 gcloud compute ssh 连接vm执行：1. docker stop current container2. docker pull latest image3. docker run container

创建1个cloud build trigger, 当有新的代码push 到指定branch的时候自动出发cloud build, 有1个前提，这个 code repository 必须是github, github Enterprise, GitLab, BitBucket 之一，国内的一些代码仓库就算了

实现

1. 准备一对key pair

具体步骤忽略
确保可以用ssh key 登陆对应的vm

[gateman@manjaro-x13 keys]$ gcloud compute ssh tf-vpc0-subnet0-vm0 --ssh-key-file=/home/gateman/.ssh/id_rsa
No zone specified. Using zone [europe-west2-c] for instance: [tf-vpc0-subnet0-vm0].
External IP address was not found; defaulting to using IAP tunneling.
WARNING: To increase the performance of the tunnel, consider installing NumPy. For instructions,
please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidthLinux tf-vpc0-subnet0-vm0 5.10.0-30-cloud-amd64 #1 SMP Debian 5.10.218-1 (2024-06-01) x86_64The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Jun  8 16:15:23 2024 from 35.235.242.17
gateman@tf-vpc0-subnet0-vm0:~$

2. 把这对key pari 放在 google security manager 中

key pair 是敏感数据，建议不要用terraform, 避免敏感数据check in 到代码仓库
在这里插入图片描述

3. 编写cloudbuild.yaml

在这里插入图片描述

注意这里的第一个步是多余的并不起作用，因为在
deploy image to GCE 这个step中

会利用gcloud 命令重新下载key file pairs
只是简单介绍下引入 secret manager item的方法

至于为何不用第一步的方法，因为遇到了1个奇怪的错误, prviate key format is not valid.
我已经raised 1个ticket 给google support, 等下文

在这里插入图片描述

cloudbuild-gce.yaml:

steps:# to prepare ssh private key file- id: prepare ssh private key filename: 'ubuntu'entrypoint: bashargs:- '-c'- |echo $$SSH_PRIVATE_KEY | cut -c 1-30echo $$SSH_PRIVATE_KEY > /workspace/ssh_key_fileecho $$SSH_PUBLIC_KEY > /workspace/ssh_key_file.pubchmod 600 /workspace/ssh_key_filechmod 600 /workspace/ssh_key_file.pubsecretEnv:- 'SSH_PRIVATE_KEY'- 'SSH_PUBLIC_KEY'- id: run maven packagename: maven:3.9.6-sapmachine-17 # https://hub.docker.com/_/mavenentrypoint: mvnargs: [ 'package' ]# https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values# i guess I should combine the two steps into one- id: build docker imagename: 'gcr.io/cloud-builders/docker'args: [ 'build', '-t', 'europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/${_APP_NAME}', '.' ]- id: upload docker image to GARname: 'gcr.io/cloud-builders/docker'args: [ 'push', 'europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/${_APP_NAME}' ]- id: deploy image to GCEname: 'gcr.io/cloud-builders/gcloud'entrypoint: bashargs:- '-c'- |whoamiset -xmkdir -p /root/.sshgcloud secrets versions access latest --secret=gateman-private-ssh-key > /root/.ssh/id_rsagcloud secrets versions access latest --secret=gateman-public-ssh-key > /root/.ssh/id_rsa.pubchmod 600 /root/.ssh/id_rsachmod 600 /root/.ssh/id_rsa.pubgcloud compute ssh gateman@${_VM_HOST} --zone=europe-west2-c --quiet --ssh-key-file=/root/.ssh/id_rsa -- "whoami" gcloud compute ssh gateman@${_VM_HOST} --zone=europe-west2-c --quiet --ssh-key-file=/root/.ssh/id_rsa -- "sudo docker container prune -f; sudo docker ps -a"  gcloud compute ssh gateman@${_VM_HOST} --zone=europe-west2-c --quiet --ssh-key-file=/root/.ssh/id_rsa -- "sudo docker stop ${_APP_NAME} && sudo docker rm ${_APP_NAME}" gcloud compute ssh gateman@${_VM_HOST} --zone=europe-west2-c --quiet --ssh-key-file=/root/.ssh/id_rsa -- "sudo docker pull europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/${_APP_NAME}:${_APP_TAG}"gcloud compute ssh gateman@${_VM_HOST} --zone=europe-west2-c --quiet --ssh-key-file=/root/.ssh/id_rsa -- "sudo docker run -d -p ${_PORT}:8080 --name ${_APP_NAME} europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/${_APP_NAME}:${_APP_TAG}"echo oklogsBucket: gs://jason-hsbc_cloudbuild/logs/
options: # https://cloud.google.com/cloud-build/docs/build-config#optionslogging: GCS_ONLY # or CLOUD_LOGGING_ONLY https://cloud.google.com/cloud-build/docs/build-config#logging# to define
availableSecrets:secretManager:- versionName: projects/$PROJECT_ID/secrets/gateman-private-ssh-key/versions/latestenv: 'SSH_PRIVATE_KEY'- versionName: projects/$PROJECT_ID/secrets/gateman-public-ssh-key/versions/latestenv: 'SSH_PUBLIC_KEY'substitutions:_APP_NAME: demo-cloud-user_APP_TAG: latest_PORT: "8081"

4. 创建1个cloudbuild trigger


# difference between data and resource: data is read only, resource is read and write
data google_service_account "cloudbuild_sa" {project = var.project_idaccount_id = "terraform"
}# referring https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloudbuild_trigger
resource "google_cloudbuild_trigger" "demo_cloud_user-gce-trigger" {name = "demo-cloud-user-gce-trigger" # could not contains underscorelocation = var.region_id# when use github then should use trigger_templategithub {name = "demo_cloud_user"owner = "nvd11"push {branch = "main"invert_regex = false # means trigger on branch}}# thesubstitutions = {_VM_HOST = "tf-vpc0-subnet0-vm0"}filename = "cloudbuild-gce.yaml"# projects/jason-hsbc/serviceAccounts/terraform@jason-hsbc.iam.gserviceaccount.comservice_account = data.google_service_account.cloudbuild_sa.id 
}