IaC基础设施即代码：Terraform 创建ACK集群与部署应用

一、实验

1.环境

2.Terraform 创建网络资源

3. 阿里云给RAM添加权限

4.Terraform 创建 ACK集群

5.在ACK集群中部署应用

6.销毁资源

二、问题

1.Terraform 验证失败

2.Terraform申请资源失败

一、实验

1.环境

（1）主机

表1-1 主机

主机	系统	软件	工具	备注
jia	Windows	Terraform 1.6.6	VS Code、 PowerShell、 Chocolatey

2.Terraform 创建网络资源

（1）查看项目

（2）网络配置文件

network.tf

//VPC 专有网络
resource "alicloud_vpc" "vpc" {vpc_name   = "k8s_vpc"cidr_block = "172.16.0.0/12"
}//switch 交换机
resource "alicloud_vswitch" "vsw" {vpc_id     = alicloud_vpc.vpc.idcidr_block = "172.16.0.0/16"zone_id    = "cn-hangzhou-j"
}

(3) 版本配置文件

versions.tf

terraform {required_providers {alicloud = {source  = "aliyun/alicloud"version = "1.214.1"}}
}# Configure the Alicloud Provider 默认供应商
provider "alicloud" {access_key = var.access_keysecret_key = var.secret_keyregion     = "cn-hangzhou"
}

（4）变量配置文件

variables.tf

variable "access_key" {description = "access_key"}variable "secret_key" {description = "secret_key"
}

(5) 密钥配置文件

terraform.tfvars

（6）初始化

terraform init

（7）格式化代码

terraform fmt

（8）验证代码

terraform validate

（9）计划与预览

 terraform plan

（10）申请资源

terraform apply

（11）登录阿里云系统查看VPC

VPC已新增1个（cn-hangzhou）

交换机已新增1个（cn-hangzhou）

3. 阿里云给RAM添加权限

（1）AliyunCSFullAcess

（2）AliyunApiGatewayFullAcess

（3）NATGatewayFullAcess

4.Terraform 创建 ACK集群

（1）查看alicloud provider 示例

Terraform Registry

托管版K8S 示例

……resource "alicloud_cs_managed_kubernetes" "k8s" {name         = var.namecluster_spec = "ack.pro.small"# version can not be defined in variables.tf.version            = "1.26.3-aliyun.1"worker_vswitch_ids = length(var.vswitch_ids) > 0 ? split(",", join(",", var.vswitch_ids)) : length(var.vswitch_cidrs) < 1 ? [] : split(",", join(",", alicloud_vswitch.vswitches.*.id))pod_vswitch_ids    = length(var.terway_vswitch_ids) > 0 ? split(",", join(",", var.terway_vswitch_ids)) : length(var.terway_vswitch_cidrs) < 1 ? [] : split(",", join(",", alicloud_vswitch.terway_vswitches.*.id))new_nat_gateway    = truenode_cidr_mask     = var.node_cidr_maskproxy_mode         = var.proxy_modeservice_cidr       = var.service_cidrdynamic "addons" {for_each = var.cluster_addonscontent {name   = lookup(addons.value, "name", var.cluster_addons)config = lookup(addons.value, "config", var.cluster_addons)}}
}

(2) 修改主配置文件

main.tf

locals {cluster_version = "1.26.3-aliyun.1"service_cidr    = "192.168.0.0/16"pod_cidr        = "10.212.0.0/16"
}resource "alicloud_cs_managed_kubernetes" "k8s" {name                 = var.cluster_nameversion              = local.cluster_version  cluster_spec         = "ack.standard"worker_vswitch_ids   = [alicloud_vswitch.vsw.id] new_nat_gateway      = truepod_cidr             = local.service_cidrservice_cidr         = local.pod_cidrload_balancer_spec   = "slb.s1.small"slb_internet_enabled = truedynamic "addons" {for_each = var.cluster_addonscontent {name   = lookup(addons.value, "name", var.cluster_addons)config = lookup(addons.value, "config", var.cluster_addons)}}
}resource "alicloud_cs_kubernetes_node_pool" "default" {name                 = var.nodepool_namecluster_id           = alicloud_cs_managed_kubernetes.k8s.idvswitch_ids          = [alicloud_vswitch.vsw.id]instance_types       = ["ecs.g6.xlarge"]system_disk_category = "cloud_efficiency"system_disk_size     = 40desired_size         = 1password             = "Admin@123"runtime_name         = "containerd"runtime_version      = "1.6.20"
}

(3) 修改变量配置文件

variables.tf

variable "access_key" {description = "access_key"}variable "secret_key" {description = "secret_key"
}variable "cluster_name" {default = "k8s_cluster_01"
}variable "nodepool_name" {default = "k8s-nodepool"
}variable "cluster_addons" {type = list(object({name   = stringconfig = string}))default = [{"name"   = "flannel","config" = "",},{"name"   = "csi-plugin","config" = "",},{"name"   = "csi-provisioner","config" = "",},{"name"   = "logtail-ds","config" = "{'IngressDashboardEnabled':'true'}",},{"name"   = "nginx-ingress-controller","config" = "{'IngressSlbNetworkType':'internet'}",},{"name"   = "arms-prometheus","config" = "",},{"name"   = "ack-node-problem-detector","config" = "{'sls_project_name':''}",}]
}

(4) 验证代码

terraform validate

(5) 计划与预览

 terraform plan

(6)申请资源

terraform apply

yes ,用时大约6分钟

(7) 登录阿里云系统查看ACK集群

初始化中

运行中

（8）查看节点池

节点池

伸缩活动

（9）查看命名空间

（10）查看网络

服务 service

5.在ACK集群中部署应用

（1）查看目录

（2）Terraform模板(docker)

Terraform Registry

USE PROVIDER

terraform {required_providers {kubernetes = {source = "hashicorp/kubernetes"version = "2.25.2"}}
}provider "kubernetes" {# Configuration options
}

(3)下载软件包

https://github.com/hashicorp/terraform-provider-kubernetes/releases

（3）修改K8S集群配置文件

阿里云系统查看连接集群信息

复制上面的连接集群信息到clustera.config

（5）修改主配置文件

provider "kubernetes" {# Configuration optionsconfig_path    = "../config/clustera.config"config_context = "kubernetes-admin-c718a5ce282f94d539ee5ce1986370194"alias          = "clustera"insecure       = true
}resource "kubernetes_namespace" "jenkins" {provider = kubernetes.clusterametadata {name = "devops"}
}

（6）修改版本配置文件

terraform {required_providers {kubernetes = {source = "hashicorp/kubernetes"version = "2.25.2"}}
}provider "kubernetes" {# Configuration options
}

（7）修改输出配置文件

output "service_name" {value = kubernetes_service_v1.jenkins.metadata[0].name}

（8）修改服务配置文件

jenkins.tf

resource "kubernetes_deployment_v1" "jenkins" {provider = kubernetes.clusterametadata {name = "jenkins"labels = {app = "jenkins"}namespace = kubernetes_namespace.jenkins.id}spec {replicas = 1selector {match_labels = {app = "jenkins"}}template {metadata {labels = {app = "jenkins"}}spec {container {image             = "jenkins/jenkins:latest"name              = "jenkins"image_pull_policy = "IfNotPresent"port {container_port = 8080}resources {limits = {cpu    = "1000m"memory = "4096Mi"}requests = {cpu    = "250m"memory = "1024Mi"}}#   liveness_probe {#     http_get {#       path = "/"#       port = 8080#     }#     initial_delay_seconds = 30#     period_seconds        = 3#   }}}}}
}resource "kubernetes_service_v1" "jenkins" {provider = kubernetes.clusterametadata {name      = "jenkins-service"namespace = kubernetes_namespace.jenkins.id}spec {selector = {app = kubernetes_deployment_v1.jenkins.metadata[0].labels.app}port {port        = 8080target_port = 8080}type = "ClusterIP"}
}resource "kubernetes_ingress_v1" "jenkins_ingress" {provider = kubernetes.clusterametadata {name      = "jenkins-ingress"namespace = kubernetes_namespace.jenkins.id}spec {rule {host = "jenkins.maojing.site"http {path {backend {service {name = kubernetes_service_v1.jenkins.metadata[0].nameport {number = 8080}}}path_type = "Prefix"path = "/"}}}}
}

(9)初始化

terraform init

(10)格式化代码

terraform fmt

(11)验证代码

terraform validate

(12)计划与预览

 terraform plan

(13) 申请资源

terraform apply

yes , 4个资源将被添加

（14）登录阿里云系统查看

命名空间新增1个 devops

工作负载（无状态deployment）新增1个jenkins

进入jenkins，状态为running

服务service

service关联路由

（15）修改输出配置文件

outputs.tf,添加如下代码

output "ingress_ip" {value = kubernetes_ingress_v1.jenkins_ingress.status[0].load_balancer[0].ingress[0].ip
}

（16）计划与预览

 terraform plan

成功拿到ingress的ip

（17）添加DNS配置文件

dns.tf

#  DNS
resource "alicloud_dns_record" "record" {name        = "maojing.site"host_record = "jenkins"type        = "A"value       = kubernetes_ingress_v1.jenkins_ingress.status[0].load_balancer[0].ingress[0].ip
}

(18) 添加变量配置文件

variables.tf

variable "access_key" {description = "access_key"}variable "secret_key" {description = "secret_key"
}

(19) 修改版本配置文件

terraform {required_providers {kubernetes = {source  = "hashicorp/kubernetes"version = "2.25.2"}alicloud = {source  = "aliyun/alicloud"version = "1.214.1"}}
}provider "kubernetes" {# Configuration options
}# Configure the Alicloud Provider 默认供应商
provider "alicloud" {access_key = var.access_keysecret_key = var.secret_keyregion     = "cn-hangzhou"
}

（20）初始化

terraform init

（21）格式化代码

terraform fmt

（22）验证代码

terraform validate

（23）计划与预览

 terraform plan

（24）申请资源

terraform apply

yes

（25）阿里云系统查看

域名解析已新增

（26）dig测试DNS

dig jenkins.maojing.site

(27) 浏览器测试

显示Jenkins安装界面

(28) 查看集群监控

6.销毁资源

（1）销毁服务资源

terraform destroy

yes

（2）登录阿里云系统

DNS解析已删除

devops命名空间已删除

（3）销毁集群资源

terraform destroy

yes ，用时大约5分钟

(4)登录阿里云系统查看集群

删除中

已删除

二、问题

1.Terraform 验证失败

（1）报错

╷
│ Error: "availability_zone": [REMOVED] Field 'availability_zone' has been removed from provider version 1.212.0.
│
│   with alicloud_cs_managed_kubernetes.k8s,
│   on main.tf line 7, in resource "alicloud_cs_managed_kubernetes" "k8s":
│    7: resource "alicloud_cs_managed_kubernetes" "k8s" {

│ Error: "availability_zone": [REMOVED] Field 'availability_zone' has been removed from provider version 1.212.0.
│
│   with alicloud_cs_managed_kubernetes.k8s,
│   on main.tf line 7, in resource "alicloud_cs_managed_kubernetes" "k8s":
│    7: resource "alicloud_cs_managed_kubernetes" "k8s" {

│ Error: "runtime": [REMOVED] Field 'runtime' has been removed from provider version 1.212.0. Please use resource 'alicloud_cs_kubernetes_node_pool' to manage cluster nodes, by using field 'runtime_name' and 'runtime_version' to replace it.
│
│   with alicloud_cs_managed_kubernetes.k8s,
│   on main.tf line 7, in resource "alicloud_cs_managed_kubernetes" "k8s":
│    7: resource "alicloud_cs_managed_kubernetes" "k8s" {

（2）原因分析

Terraform Registry

从1.212版本开始，部分关键地段被移除，推荐使用alicloud_cs_kubernetes_node_pool 管理工作节点。

From version 1.212.0, runtime,enable_ssh,rds_instances,exclude_autoscaler_nodes,worker_number,worker_instance_types,password,key_name,kms_encrypted_password,kms_encryption_context,worker_instance_charge_type,worker_period,worker_period_unit,worker_auto_renew,worker_auto_renew_period,worker_disk_category,worker_disk_size,worker_data_disks,node_name_mode,node_port_range,os_type,platform,image_id,cpu_policy,user_data,taints,worker_disk_performance_level,worker_disk_snapshot_policy_id,install_cloud_monitor,kube_config,availability_zone are removed. Please use resource alicloud_cs_kubernetes_node_pool to manage your cluster worker nodes.

（3）解决方法

修改配置文件。

2.Terraform申请资源失败

（1）报错

Error: [ERROR] terraform-provider-alicloud/alicloud/resource_alicloud_cs_kubernetes.go:1230: Resource c28e6d5ac0cf64922a476e6963f1239b8 DescribeNatGateways Failed!!! [SDK alibaba-cloud-sdk-go ERROR]:
│ SDK.ServerError
│ ErrorCode: Forbidden.RAM
│ Recommend: https://api.aliyun.com/troubleshoot?q=Forbidden.RAM&product=Vpc&requestId=0254494A-FE5F-51C9-96DA-394123C37E13
│ RequestId: 0254494A-FE5F-51C9-96DA-394123C37E13
│ Message: User not authorized to operate on the specified resource, or this API doesn't support RAM.
│ RespHeaders: map[Access-Control-Allow-Origin:[*] Access-Control-Expose-Headers:[*] Connection:[keep-alive] Content-Length:[568] Content-Type:[application/json;charset=utf-8] Date:[Tue, 23 Jan 2024 05:11:28 GMT] Keep-Alive:[timeout=25] X-Acs-Request-Id:[0254494A-FE5F-51C9-96DA-394123C37E13] X-Acs-Trace-Id:[740d51a284c42eb37e67556a9d62faa6]]
│ AccessDeniedDetail: map[AuthPrincipalDisplayName:205814005146961779 AuthPrincipalOwnerId:1889388625243280 AuthPrincipalType:SubUser EncodedDiagnosticMessage:AQEAAAAAZa9KgzAyNTQ0OTRBLUZFNUYtNTFDOS05NkRBLTM5NDEyM0MzN0UxMw==]
│
│   with alicloud_cs_managed_kubernetes.k8s,
│   on main.tf line 7, in resource "alicloud_cs_managed_kubernetes" "k8s":
│    7: resource "alicloud_cs_managed_kubernetes" "k8s" {