1. 预备条件

9 个节点：

角色	ip	cpu	mem	disk	ios	kernel
download	10.168.2.10、10.90.0.10	4	8G	100G	rhel 8.7	4.18+
bastion01	10.90.0.11	4	8G	100G	rhel 8.7	4.18+
kube-master01	10.90.0.12	4	8G	100G	rhel 8.7	4.18+
kube-node01	10.90.0.13	4	8G	100G	rhel 8.7	4.18+
kube-node02	10.90.0.14	4	8G	100G	rhel 8.7	4.18+
kube-node03	10.90.0.15	4	8G	100G	rhel 8.7	4.18+

配置网卡

download01

cat /etc/sysconfig/network-scripts/ifcfg-ens192 
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=eui64
NAME=ens192
DEVICE=ens192
ONBOOT=yes
IPADDR=10.168.2.10
PREFIX=16
GATEWAY=10.168.1.1
DNS1=8.8.8.8[root@download ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens224 
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=eui64
NAME=ens224
DEVICE=ens224
ONBOOT=yes
IPADDR=10.90.0.10
PREFIX=16

download01 节点安装 nerdctl

参考：nerdctl install【nerdctl 安装】

配置容器代理

 cat /etc/systemd/system/containerd.service.d/http-proxy.conf 
[Service]
Environment="HTTP_PROXY=http://192.168.21.101:7890" "HTTPS_PROXY=http://192.168.21.101:7890" "NO_PROXY=localhost,127.0.0.0/8,169.0.0.0/8,10.0.0.0/8,192.168.0.0/16,*.coding.net,*.tencentyun.com,*.myqcloud.com"

重启 containerd

systemctl daemon-reload && systemctl restart containerd

3. download01 节点 介质下载

（download）
介质列表：

• rhel-8.6-x86_64-dvd.iso （用于安装系统和配置yum源）
• kubespray-offline-2.23.0-0.tar.gz（离线安装 kubernetes）

在下载所有介质之前需除了容器拉取镜像代理，主机也要配置代理否则，是否无法下载成功的。

主机代理：

export https_proxy=http://192.168.21.101:7890
export http_proxy=http://192.168.21.101:7890
export no_proxy=192.168.21.2,10.0.0.0/8,192.168.0.0/16,localhost,127.0.0.0/8,.coding.net,.tencentyun.com,.myqcloud.com

下载最新版本 2.23.0-0： https://github.com/kubespray-offline/kubespray-offline/releases

wget https://github.com/kubespray-offline/kubespray-offline/archive/refs/tags/v2.23.0-0.zipunzip v2.23.0-0.zip

修改默认下载的kubernetes 版本

$ vim config.sh 
#!/bin/bash# Kubespray version to download. Use "master" for latest master branch.
KUBESPRAY_VERSION=${KUBESPRAY_VERSION:-2.23.1}  #修改
#KUBESPRAY_VERSION=${KUBESPRAY_VERSION:-master}# container runtime for preparation node
docker=${docker:-docker}
#docker=${docker:-/usr/local/bin/nerdctl}# Run ansible in container?
ansible_in_container=${ansible_in_container:-false}

安装成功后，下载介质执行：

./download-all.sh

所有工件都存储在./outputs 目录。
此脚本调用以下所有脚本：

• prepare-pkgs.sh
  Setup python, etc.
• prepare-py.sh
  Setup python venv, install required python packages.
• get-kubespray.sh
  Download and extract kubespray, if KUBESPRAY_DIR does not exist.
• pypi-mirror.sh
  Download PyPI mirror files
• download-kubespray-files.sh
  Download kubespray offline files (containers, files, etc)
• download-additional-containers.sh
  Download additional containers.
  You can add any container image repoTag to imagelists/*.txt.
• create-repo.sh
  Download RPM or DEB repositories.
• copy-target-scripts.sh
  Copy scripts for target node.

用时 30分钟 下载完成介质。也许一次会下载失败，需要调试反复执行，或者根据环境改动一些内容。

注意：如果你喜欢通过容器部署集群，需要下载 kubespray 镜像，默认 quay.io/kubespray/kubespray:v2.21.0 ，存在一个 pip 包依赖（jmespath 1.0.1），下面镜像已重新编译。

nerdctl pull ghostwritten/kubespray:v2.21-a94b893e2
nerdctl save -o kubespray:v2.21-a94b893e2.tar ghostwritten/kubespray:v2.21-a94b893e2

打包下载的介质：

tar zcvf kubespray-offline-2.23.0-0-kubernetes-v1.27.7-origin.tar.gz kubespray-offline-2.23.0-0

将介质传输至离线的 bastion01

4. bastion01节点配置 yum 源

（bastion01离线状态）

参考：https://www.redhat.com/sysadmin/apache-yum-dnf-repo

mkdir /mnt/cdrom
mount -o loop /root/kubernetes-offline/rhel-8.6-x86_64-dvd.iso /mnt/cdrom

本地配置yum源

cat local.repo 
[BaseOS]
name=BaseOS
baseurl=file:///mnt/cdrom/BaseOS
enable=1
gpgcheck=0[AppStream]
name=AppStream
baseurl=file:///mnt/cdrom/AppStream
enabled=1
gpgcheck=0

测试

$ yum repolist --verbose
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, kpatch, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
Updating Subscription Management repositories.
Unable to read consumer identityThis system is not registered with an entitlement server. You can use subscription-manager to register.YUM version: 4.7.0
cachedir: /var/cache/dnf
Last metadata expiration check: 0:35:00 ago on Thu 19 Oct 2023 05:45:22 PM CST.
Repo-id            : AppStream
Repo-name          : AppStream
Repo-revision      : 1656402327
Repo-updated       : Tue 28 Jun 2022 03:45:28 PM CST
Repo-pkgs          : 6,609
Repo-available-pkgs: 5,345
Repo-size          : 8.6 G
Repo-baseurl       : file:///mnt/cdrom/AppStream
Repo-expire        : 172,800 second(s) (last: Thu 19 Oct 2023 05:45:22 PM CST)
Repo-filename      : /etc/yum.repos.d/local.repoRepo-id            : BaseOS
Repo-name          : BaseOS
Repo-revision      : 1656402376
Repo-updated       : Tue 28 Jun 2022 03:46:17 PM CST
Repo-pkgs          : 1,714
Repo-available-pkgs: 1,712
Repo-size          : 1.3 G
Repo-baseurl       : file:///mnt/cdrom/BaseOS
Repo-expire        : 172,800 second(s) (last: Thu 19 Oct 2023 05:45:21 PM CST)
Repo-filename      : /etc/yum.repos.d/local.repo
Total packages: 8,323

yum clean all
yum install ntp

其他节点使用源

$ yum -y install httpd$ vim /etc/httpd/conf/httpd.conflisten 81# vim  /etc/httpd/conf.d/define.conf
<VirtualHost *:81>ServerName  www.XXX-ym.comDocumentRoot /mnt/cdrom
</VirtualHost># vim  /etc/httpd/conf.d/permission.conf
<Directory /mnt/cdrom>Require all granted      
</Directory>$ systemctl  restart  httpd
$ chcon -R --reference=/var/www  /mnt/cdrom  //调整SELinux属性$ systemctl  restart  httpd

远程节点配置yum源

$ cat http_iso.repo 
[BaseOS] 
name=http_iso 
baseurl=http://10.70.0.254/BaseOS
gpgcheck=0 
enabled=1
[AppStream] 
name=http_iso 
baseurl=http://10.70.0.254/AppStream
gpgcheck=0 
enabled=1

5. bastion01 离线安装 nerdctl

打包下载的介质传输到离线环境：

cd ../
tar zcvf kubespray-offline-2.23.0-0-kubernetes-v1.27.7-origin.tar.gz kubespray-offline-2.23.0-0

登陆部署节点（registry01），然后在输出目录中运行以下脚本：

mkdir /root/kubernetes-offline && cd kubespray-offline-2.23.0-0.tar.gz
tar zxvf kubespray-offline-2.23.0-0.tar.gz

cd /root/kubespray-offline-2.23.0-0/outputs && vim offline-install-nerdctl-containerd.sh

#!/bin/bashBASE_DIR="$(dirname "$(readlink -f "${0}")")"
name=`basename $0`tar xvf ${BASE_DIR}/files/containerd-1.7.5-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin
cp ${BASE_DIR}/files/runc/v1.1.9/runc.amd64 /usr/local/bin/runc
chmod 755 /usr/local/bin/runctar zxvf ${BASE_DIR}/files/nerdctl-1.5.0-linux-amd64.tar.gz -C /tmp/
cp /tmp/nerdctl /usr/local/binsudo mkdir -p \/etc/systemd/system/containerd.service.d \/etc/containerd \/var/lib/containerd \/run/containerd#配置 containerdcat <<EOF> /etc/systemd/system/containerd.service
# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerdType=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999[Install]
WantedBy=multi-user.target
EOFcat <<EOF> /etc/containerd/config.toml
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0[grpc]address = "/run/containerd/containerd.sock"uid = 0gid = 0[debug]address = "/run/containerd/debug.sock"uid = 0gid = 0level = "info"[metrics]address = ""grpc_histogram = false[cgroup]path = ""[plugins][plugins."io.containerd.grpc.v1.cri".containerd]default_runtime_name = "runc"snapshotter = "overlayfs"[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]runtime_type = "io.containerd.runc.v2"[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]systemdCgroup = true
EOF# restart containerd
sudo systemctl daemon-reload && sudo systemctl enable --now containerd &&  systemctl status  containerd | grep Active#CNI plugins installsudo mkdir -p /opt/cni/bin && tar xvzf  ${BASE_DIR}/files/kubernetes/cni/cni-plugins-linux-amd64-v1.3.0.tgz -C /opt/cni/bin

执行：

sh -x offline-install-nerdctl-containerd.sh

安装l insecure registry

修改 config.sh

$ vim /root/kubespray-offline-2.23.0-0/outputs/config.sh 
#!/bin/bash
KUBESPRAY_VERSION=${KUBESPRAY_VERSION:-2.23.1}  #修改
#KUBESPRAY_VERSION=${KUBESPRAY_VERSION:-master}REGISTRY_PORT=${REGISTRY_PORT:-35000}

确保 /etc/resolv.conf 存在，否则会报错，touch /etc/resolv.conf

载入 registry 镜像

$ nerdctl load -i images/docker.io_library_registry-2.8.2.tar.gz 
unpacking docker.io/library/registry:2.8.2 (sha256:43408412c6b1e1ec25a0d8ac2ff65e8e7c4e5d9d1ee1bab6ea513589b9e841a0)...
Loaded image: registry:2.8.2

sh start-registry.sh 
===> Start registry
be7b2cd72cd0f39588bb6083f730ac840e08de766143010d61cd8d9f4215289f

配置 /etc/hosts

$ cat /etc/hosts
10.70.0.73 registry.upm

测试

nerdctl tag registry:2.8.2 registry.upm:35000/registry:2.8.2
nerdctl --insecure-registry=true push  registry.upm:35000/registry:2.8.2
nerdctl --insecure-registry=true pull  registry.upm:35000/registry:2.8.2

输出：

$ nerdctl --insecure-registry=true push  registry.upm:35000/registry:2.8.2
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.v2+json, sha256:43408412c6b1e1ec25a0d8ac2ff65e8e7c4e5d9d1ee1bab6ea513589b9e841a0) 
WARN[0000] skipping verifying HTTPS certs for "registry.upm:35000" 
manifest-sha256:43408412c6b1e1ec25a0d8ac2ff65e8e7c4e5d9d1ee1bab6ea513589b9e841a0: waiting        |--------------------------------------| 
config-sha256:fb59138d7af16a5974de721cb6a4399b9700f0ade6a1bc91209a174245cfb789:   waiting        |--------------------------------------| 
elapsed: 0.1 s                                                                    total:   0.0 B (0.0 B/s)                                         
WARN[0000] server "registry.upm:35000" does not seem to support HTTPS, falling back to plain HTTP  error="failed to do request: Head \"https://registry.upm:35000/v2/registry/blobs/sha256:6db7f901dc3b0c738464ace0ea79250b5341508cf23d091bcaf37332befa1209\": http: server gave HTTP response to HTTPS client"
manifest-sha256:43408412c6b1e1ec25a0d8ac2ff65e8e7c4e5d9d1ee1bab6ea513589b9e841a0: waiting        |--------------------------------------| 
config-sha256:fb59138d7af16a5974de721cb6a4399b9700f0ade6a1bc91209a174245cfb789:   waiting        |--------------------------------------| 
elapsed: 0.6 s                                                                    total:   0.0 B (0.0 B/s)           

配置镜像入库

配置但不执行：

$ cd /root/kubernetes-offline/kubespray-offline-2.23.0-0/outputs
$ cat load-push-all-images.sh 
#!/bin/bashsource ./config.shLOCAL_REGISTRY=${LOCAL_REGISTRY:-"registry.upm:${REGISTRY_PORT}"}  #修改 registry.upm
NERDCTL=/usr/local/bin/nerdctlBASEDIR="."
if [ ! -d images ] && [ -d ../outputs ]; thenBASEDIR="../outputs"  # for tests
fiload_images() {for image in $BASEDIR/images/*.tar.gz; doecho "===> Loading $image"sudo $NERDCTL load -i $imagedone
}push_images() {images=$(cat $BASEDIR/images/*.list)for image in $images; do# Removes specific repo parts from each image for kubespraynewImage=$imagefor repo in registry.k8s.io k8s.gcr.io gcr.io docker.io quay.io; donewImage=$(echo ${newImage} | sed s@^${repo}/@@)donenewImage=${LOCAL_REGISTRY}/${newImage}echo "===> Tag ${image} -> ${newImage}"sudo $NERDCTL tag ${image} ${newImage}echo "===> Push ${newImage}"sudo $NERDCTL --insecure-registry=true push ${newImage}done
}load_images
push_images

执行 set-all.sh

[root@bastion01 outputs]# cat setup-all.sh 
#!/bin/bashrun() {echo "=> Running: $*"$* || {echo "Failed in : $*"exit 1}
}# prepare
#run ./setup-container.sh# start web server
run ./start-nginx.sh# setup local repositories
run ./setup-offline.sh# setup python
run ./setup-py.sh# start private registry
#run ./start-registry.sh# load and push all images to registry
run ./load-push-all-images.sh

执行：

sh -x set-all.sh

7. bastion01 配置互信

ssh-keygen
for i in {71..78};do ssh-copy-id root@10.70.0.$i;done

8. 启动容器部署环境

nerdctl load -i quay.io_kubespray_kubespray_v2.23.1.tar 
nerdctl  run --name kubespray-offline-ansible --network=host  -it -v "$(pwd)"/kubespray-offline-2.23.0-0:/kubespray -v "${HOME}"/.ssh:/root/.ssh  -v /etc/hosts:/etc/hosts quay.io/kubespray/kubespray:v2.23.1  bash

9. 部署前准备

检查批量通信、镜像仓库地址域名解析、可能存在缺失的内核加载模块。

9.1 配置 extract-kubespray.sh

保持 kubespray-v2.22.1 内容不变，注释掉 apply patches内容。

cd /root/kubernetes-offline/kubespray-offline-2.23.0-0/outputs
./extract-kubespray.sh

9.2 编写 inventory.ini

vim kubespray-2.23.0/inventory/sample/inventory.ini

[all]
kube-master01 ansible_host=10.70.0.71
kube-master02 ansible_host=10.70.0.72
kube-master03 ansible_host=10.70.0.73
kube-node01 ansible_host=10.70.0.74
kube-node02 ansible_host=10.70.0.75
kube-node03 ansible_host=10.70.0.76[bastion]
bastion01 ansible_host=10.70.0.78 ansible_user=root[kube_control_plane]
kube-master01
kube-master02
kube-master03[etcd]
kube-master01
kube-master02
kube-master03[kube_node]
kube-node01
kube-node02
kube-node03[calico_rr][k8s_cluster:children]
kube_control_plane
kube_node
calico_rr

执行：

ansible -i inventory/sample/inventory.ini all  -m ping
ansible -i inventory/sample/inventory.ini all -m lineinfile -a "path=/etc/hosts line='10.70.0.78 registry.upm'"
ansible -i inventory/sample/inventory.ini all -m lineinfile -a "path=/etc/rc.local line='modprobe br_netfilter\nmodprobe ip_conntrack'"
echo "nameserver 8.8.8.8" >  /etc/resolv.conf
ansible -i inventory/sample/inventory.ini all -m copy -a "src=/etc/resolv.conf dest=/etc/"

9.3 编写 offline.yml

vim  outputs/kubespray-2.23.1/inventry/sample/group_vars/all/offline.yml

http_server: "http://10.60.0.30"
registry_host: "registry.demo:35000"containerd_insecure_registries: # Kubespray #8340"registry.demo:35000": "http://registry.demo:35000"files_repo: "{{ http_server }}/files"
yum_repo: "{{ http_server }}/rpms"
ubuntu_repo: "{{ http_server }}/debs"# Registry overrides
kube_image_repo: "{{ registry_host }}"
gcr_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"# Download URLs: See roles/download/defaults/main.yml of kubespray.
kubeadm_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubeadm"
kubectl_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubectl"
kubelet_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubelet"
# etcd is optional if you **DON'T** use etcd_deployment=host
etcd_download_url: "{{ files_repo }}/kubernetes/etcd/etcd-{{ etcd_version }}-linux-amd64.tar.gz"
cni_download_url: "{{ files_repo }}/kubernetes/cni/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
crictl_download_url: "{{ files_repo }}/kubernetes/cri-tools/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
# If using Calico
calicoctl_download_url: "{{ files_repo }}/kubernetes/calico/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
# If using Calico with kdd
calico_crds_download_url: "{{ files_repo }}/kubernetes/calico/{{ calico_version }}.tar.gz"runc_download_url: "{{ files_repo }}/runc/{{ runc_version }}/runc.{{ image_arch }}"
nerdctl_download_url: "{{ files_repo }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
containerd_download_url: "{{ files_repo }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"#containerd_insecure_registries:
#    "{{ registry_addr }}"："{{ registry_host }}"# CentOS/Redhat/AlmaLinux/Rocky Linux
## Docker / Containerd
docker_rh_repo_base_url: "{{ yum_repo }}/docker-ce/$releasever/$basearch"
docker_rh_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"# Fedora
## Docker
docker_fedora_repo_base_url: "{{ yum_repo }}/docker-ce/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}"
docker_fedora_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
## Containerd
containerd_fedora_repo_base_url: "{{ yum_repo }}/containerd"
containerd_fedora_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"

9.4 配置 containerd.yml

$ vim  inventory/sample/group_vars/all/containerd.yml
......
containerd_registries_mirrors:- prefix: registry.upm:35000mirrors:- host: http://registry.upm:35000capabilities: ["pull", "resolve"]skip_verify: true
....

9.5 配置 nerdctl

解决镜像拉取失败

vim roles/container-engine/nerdctl/templates/nerdctl.toml.j2 
...
insecure_registry = true #添加

9.6 配置 nf_conntrack

123-148行 nf_conntrack_ipv4修改为 nf_conntrack

$ vim roles/kubernetes/node/tasks/main.yml
- name: Modprobe nf_conntrackcommunity.general.modprobe:name: nf_conntrackstate: presentregister: modprobe_nf_conntrackignore_errors: true  # noqa ignore-errorswhen:- kube_proxy_mode == 'ipvs'tags:- kube-proxy- name: Persist ip_vs modulescopy:dest: /etc/modules-load.d/kube_proxy-ipvs.confmode: 0644content: |ip_vsip_vs_rrip_vs_wrrip_vs_sh{% if modprobe_nf_conntrack is success -%}nf_conntrack{%-   endif -%}when: kube_proxy_mode == 'ipvs'tags:- kube-proxy

10. 部署 offline repo

Deploy offline repo configurations which use your yum_repo/ubuntu_repo to all target nodes using ansible.
First, copy offline setup playbook to kubespray directory.

ansible -i inventory/sample/inventory.ini all -m shell -a  "mv /etc/yum.repos.d /tmp"
ansible -i inventory/sample/inventory.ini all -m shell -a  "mkdir /etc/yum.repos.d"
cp -r ../playbook .
ansible-playbook -i  inventory/sample/inventory.ini playbook/offline-repo.yml 
ansible -i inventory/sample/inventory.ini all  -m shell -a "yum repolist --verbose"

开始部署

ansible-playbook -i inventory/sample/inventory.ini --become --become-user=root cluster.yml

输出：

PLAY RECAP ***************************************************************************************************
bastion01                  : ok=6    changed=0    unreachable=0    failed=0    skipped=14   rescued=0    ignored=0   
kube-master01              : ok=765  changed=153  unreachable=0    failed=0    skipped=1273 rescued=0    ignored=8   
kube-node01                : ok=525  changed=95   unreachable=0    failed=0    skipped=773  rescued=0    ignored=1   
kube-node02                : ok=525  changed=95   unreachable=0    failed=0    skipped=772  rescued=0    ignored=1   
kube-node03                : ok=525  changed=95   unreachable=0    failed=0    skipped=772  rescued=0    ignored=1   
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   Tuesday 21 November 2023  05:08:21 +0000 (0:00:01.427)       0:55:24.797 ****** 
=============================================================================== 
container-engine/containerd : Download_file | Download item ------------------------------------------ 55.86s
container-engine/crictl : Download_file | Download item ---------------------------------------------- 54.44s
container-engine/nerdctl : Download_file | Download item --------------------------------------------- 52.61s
bootstrap-os : Assign inventory name to unconfigured hostnames (non-CoreOS, non-Flatcar, Suse and ClearLinux, non-Fedora) -- 45.77s
container-engine/runc : Download_file | Download item ------------------------------------------------ 45.30s
container-engine/runc : Download_file | Validate mirrors --------------------------------------------- 43.93s
container-engine/nerdctl : Download_file | Validate mirrors ------------------------------------------ 41.38s
container-engine/crictl : Download_file | Validate mirrors ------------------------------------------- 39.49s
container-engine/containerd : Download_file | Validate mirrors --------------------------------------- 39.12s
etcdctl_etcdutl : Download_file | Download item ------------------------------------------------------ 38.89s
container-engine/crictl : Extract_file | Unpacking archive ------------------------------------------- 38.35s
download : Download_file | Download item ------------------------------------------------------------- 33.73s
kubernetes-apps/ansible : Kubernetes Apps | Lay Down CoreDNS templates ------------------------------- 32.09s
kubernetes/control-plane : Kubeadm | Initialize first master ----------------------------------------- 31.33s
etcdctl_etcdutl : Extract_file | Unpacking archive --------------------------------------------------- 28.66s
container-engine/nerdctl : Extract_file | Unpacking archive ------------------------------------------ 27.99s
kubernetes-apps/ansible : Kubernetes Apps | Start Resources ------------------------------------------ 26.80s
Gather necessary facts (hardware) -------------------------------------------------------------------- 25.67s
bootstrap-os : Gather host facts to get ansible_distribution_version ansible_distribution_major_version -- 24.51s
download : Download | Download files / images -------------------------------------------------------- 24.30s#登陆master01节点
[root@kube-master01 ~]# kubectl get node 
NAME            STATUS   ROLES           AGE     VERSION
kube-master01   Ready    control-plane   7h53m   v1.27.7
kube-node01     Ready    <none>          7h51m   v1.27.7
kube-node02     Ready    <none>          7h51m   v1.27.7
kube-node03     Ready    <none>          7h51m   v1.27.7