信创:鲲鹏(arm64)+麒麟(kylin v10)离线部署k8s和kubesphere(含离线部署新方式)

本文将详细介绍,如何基于鲲鹏CPU(arm64)和操作系统 Kylin V10 SP2/SP3,利用 KubeKey 制作 KubeSphere 和 Kubernetes 离线安装包,并实战部署 KubeSphere 3.3.1 和 Kubernetes 1.22.12 集群。

服务器配置

主机名IPCPUOS用途
master-1192.168.10.2Kunpeng-920Kylin V10 SP2离线环境 KubeSphere/k8s-master
master-2192.168.10.3Kunpeng-920Kylin V10 SP2离线环境 KubeSphere/k8s-master
master-3192.168.10.4Kunpeng-920Kylin V10 SP2离线环境 KubeSphere/k8s-master
deploy192.168.200.7Kunpeng-920Kylin V10 SP3联网主机用于制作离线包

实战环境涉及软件版本信息

  • 服务器芯片:Kunpeng-920

  • 操作系统:麒麟V10 SP2/SP3 aarch64

  • Docker: 24.0.7

  • Harbor: v2.7.1

  • KubeSphere:v3.3.1

  • Kubernetes:v1.22.12

  • KubeKey: v2.3.1

本文介绍

本文介绍了如何在 麒麟 V10 aarch64 架构服务器上制品和离线部署 KubeSphere 和 Kubernetes 集群。我们将使用 KubeSphere 开发的 KubeKey 工具实现自动化部署,在三台服务器上实现高可用模式最小化部署 Kubernetes 集群和 KubeSphere。

KubeSphere 和 Kubernetes 在 ARM 架构 和 X86 架构的服务器上部署,最大的区别在于所有服务使用的容器镜像架构类型的不同,KubeSphere 开源版对于 ARM 架构的默认支持可以实现 KubeSphere-Core 功能,即可以实现最小化的 KubeSphere 和完整的 Kubernetes 集群的部署。当启用了 KubeSphere 可插拔组件时,会遇到个别组件部署失败的情况,需要我们手工替换官方或是第三方提供的 ARM 版镜像或是根据官方源码手工构建 ARM 版镜像。如果需要实现开箱即用及更多的技术支持,则需要购买企业版的 KubeSphere。

1.1 确认操作系统配置

在执行下文的任务之前,先确认操作系统相关配置。

  • 操作系统类型

    [root@localhost ~]# cat /etc/os-release
    NAME="Kylin Linux Advanced Server"
    VERSION="V10 (Halberd)"
    ID="kylin"
    VERSION_ID="V10"
    PRETTY_NAME="Kylin Linux Advanced Server V10 (Halberd)"
    ANSI_COLOR="0;31
  • 操作系统内核

    [root@node1 ~]# uname -r
    Linux node1 4.19.90-52.22.v2207.ky10.aarch64 
  • 服务器 CPU 信息

    [root@node1 ~]# lscpu
    Architecture:                    aarch64
    CPU op-mode(s):                  64-bit
    Byte Order:                      Little Endian
    CPU(s):                          32
    On-line CPU(s) list:             0-31
    Thread(s) per core:              1
    Core(s) per socket:              1
    Socket(s):                       32
    NUMA node(s):                    2
    Vendor ID:                       HiSilicon
    Model:                           0
    Model name:                      Kunpeng-920
    Stepping:                        0x1
    BogoMIPS:                        200.00
    NUMA node0 CPU(s):               0-15
    NUMA node1 CPU(s):               16-31
    Vulnerability Itlb multihit:     Not affected
    Vulnerability L1tf:              Not affected
    Vulnerability Mds:               Not affected
    Vulnerability Meltdown:          Not affected
    Vulnerability Spec store bypass: Not affected
    Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
    Vulnerability Spectre v2:        Not affected
    Vulnerability Srbds:             Not affected
    Vulnerability Tsx async abort:   Not affected
    Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm

    2. 安装k8s依赖服务

    这里使用能联网的 deploy 节点,用来制作离线部署资源包。由于harbor官方不支持arm,先使用在线安装kubesphere,后续根据kubekey生成的文件作为伪制品。故在192.168.200.7服务器以单节点形式部署ks。

    以下为多阶段部署,目的是方便制作离线安装包

    2.1 部署docker和docker-compose

    具体可参考以下文章第一部分

    天行1st,公众号:编码如写诗鲲鹏+欧拉部署KubeSphere3.4

    这里采用制作好的安装包形式,直接安装,解压后执行其中的install.sh

链接: https://pan.baidu.com/s/1j9EUonkZPfwQlDC9U1hRMQ?pwd=7y53

2.2 部署harbor

上传安装包后解压后执行其中的install.sh

链接: https://pan.baidu.com/s/1diy4-vxmWXDMp5MWYpaqMw?pwd=nr9y 

2.3 下载麒麟系统k8s依赖包

mkdir -p /root/kubesphere/k8s-init
# 该命令将下载相关依赖到/root/kubesphere/k8s-init目录
yum -y install openssl socat conntrack ipset ebtables chrony ipvsadm --downloadonly --downloaddir /root/kubesphere/k8s-init
# 编写安装脚本
cat install.sh
#!/bin/bash
# rpm -ivh *.rpm --force --nodeps# 打成压缩包,方便离线部署使用
tar -czvf k8s-init-Kylin_V10-arm.tar.gz ./k8s-init/*

2.4 下载ks相关镜像

下载kubesphere3.3.1所需要的arm镜像


#!/bin/bash
# 
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/ks-console:v3.3.1
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.1
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/ks-installer:v3.3.1
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/ks-apiserver:v3.3.1
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/openpitrix-jobs:v3.3.1
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/alpine:3.14
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.22.12
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.22.12
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.22.12
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.22.12
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/provisioner-localpv:3.3.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils:3.3.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.23.2
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.23.2
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.23.2
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.23.2
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/kube-state-metrics:v2.5.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/fluent-bit:v1.8.11
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-config-reloader:v0.55.1
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-operator:v0.55.1
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/thanos:v0.25.2
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus:v2.34.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/fluentbit-operator:v0.13.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/node-exporter:v1.3.1
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.22.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager:v1.4.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/notification-tenant-sidecar:v3.2.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager-operator:v1.4.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/alertmanager:v0.23.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.11.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/docker:19.03
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.5
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/configmap-reload:v0.5.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/snapshot-controller:v4.0.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.8.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/log-sidecar-injector:1.1
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.15.12
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/mc:RELEASE.2019-08-07T23-14-43Z
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/defaultbackend-amd64:1.4
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/redis:5.0.14-alpine
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.3
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/opensearch:2.6.0
docker pull registry.cn-beijing.aliyuncs.com/kubesphereio/busybox:latest
docker pull kubesphere/fluent-bit:v2.0.6

这里使用ks阿里云镜像,其中有些镜像会下载失败,具体可查看运维有术文章。对于下载失败的镜像,可通过本地电脑,直接去hub.docker.com下载。例如:

docker pull kubesphere/fluent-bit:v2.0.6 --platform arm64
#官方ks-console:v3.3.1(arm版)在麒麟中跑不起来,据运维有术介绍,需要使用node14基础镜像。当在鲲鹏服务器准备自己构建时报错淘宝源https过期,使用https://registry.npmmirror.com仍然报错,于是放弃使用该3.3.0镜像,重命名为3.3.1
docker pull zl862520682/ks-console:v3.3.0
docker tag zl862520682/ks-console:v3.3.0 dockerhub.kubekey.local/kubesphereio/ks-console:v3.3.1
## mc和minio也需要重新拉取打tag
docker pull minio/minio:RELEASE.2020-11-25T22-36-25Z-arm64
docker tag  minio/minio:RELEASE.2020-11-25T22-36-25Z-arm64 dockerhub.kubekey.local/kubesphereio/minio:RELEASE

2.5 重命名镜像

重新给镜像打tag,标记为私有仓库镜像

docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.27.3  dockerhub.kubekey.local/kubesphereio/kube-controllers:v3.27.3docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.3  dockerhub.kubekey.local/kubesphereio/cni:v3.27.3docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.27.3  dockerhub.kubekey.local/kubesphereio/pod2daemon-flexvol:v3.27.3docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.3  dockerhub.kubekey.local/kubesphereio/node:v3.27.3docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/ks-console:v3.3.1  dockerhub.kubekey.local/kubesphereio/ks-console:v3.3.1docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/alpine:3.14  dockerhub.kubekey.local/kubesphereio/alpine:3.14docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.22.20  dockerhub.kubekey.local/kubesphereio/k8s-dns-node-cache:1.22.20docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.3.1  dockerhub.kubekey.local/kubesphereio/ks-controller-manager:v3.3.1docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/ks-installer:v3.3.1  dockerhub.kubekey.local/kubesphereio/ks-installer:v3.3.1docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/ks-apiserver:v3.3.1  dockerhub.kubekey.local/kubesphereio/ks-apiserver:v3.3.1docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/openpitrix-jobs:v3.3.1  dockerhub.kubekey.local/kubesphereio/openpitrix-jobs:v3.3.1docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.22.12  dockerhub.kubekey.local/kubesphereio/kube-apiserver:v1.22.12docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.22.12  dockerhub.kubekey.local/kubesphereio/docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.22.12  dockerhub.kubekey.local/kubesphereio/kube-controller-manager:v1.22.12docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.22.12  dockerhub.kubekey.local/kubesphereio/kube-scheduler:v1.22.12docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/provisioner-localpv:3.3.0  dockerhub.kubekey.local/kubesphereio/provisioner-localpv:3.3.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils:3.3.0  dockerhub.kubekey.local/kubesphereio/linux-utils:3.3.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/kube-state-metrics:v2.5.0  dockerhub.kubekey.local/kubesphereio/kube-state-metrics:v2.5.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/fluent-bit:v1.8.11  dockerhub.kubekey.local/kubesphereio/fluent-bit:v1.8.11docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-config-reloader:v0.55.1  dockerhub.kubekey.local/kubesphereio/prometheus-config-reloader:v0.55.1docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-operator:v0.55.1  dockerhub.kubekey.local/kubesphereio/prometheus-operator:v0.55.1docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/thanos:v0.25.2  dockerhub.kubekey.local/kubesphereio/thanos:v0.25.2docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus:v2.34.0  dockerhub.kubekey.local/kubesphereio/prometheus:v2.34.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/fluentbit-operator:v0.13.0  dockerhub.kubekey.local/kubesphereio/fluentbit-operator:v0.13.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/node-exporter:v1.3.1  dockerhub.kubekey.local/kubesphereio/node-exporter:v1.3.1docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.22.0  dockerhub.kubekey.local/kubesphereio/kubectl:v1.22.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager:v1.4.0  dockerhub.kubekey.local/kubesphereio/notification-manager:v1.4.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/notification-tenant-sidecar:v3.2.0  dockerhub.kubekey.local/kubesphereio/notification-tenant-sidecar:v3.2.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager-operator:v1.4.0  dockerhub.kubekey.local/kubesphereio/notification-manager-operator:v1.4.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/alertmanager:v0.23.0  dockerhub.kubekey.local/kubesphereio/alertmanager:v0.23.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.11.0  dockerhub.kubekey.local/kubesphereio/kube-rbac-proxy:v0.11.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/docker:19.03  dockerhub.kubekey.local/kubesphereio/docker:19.03docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/metrics-server:v0.4.2  dockerhub.kubekey.local/kubesphereio/metrics-server:v0.4.2docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.5  dockerhub.kubekey.local/kubesphereio/pause:3.5docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/configmap-reload:v0.5.0  dockerhub.kubekey.local/kubesphereio/configmap-reload:v0.5.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/snapshot-controller:v4.0.0  dockerhub.kubekey.local/kubesphereio/snapshot-controller:v4.0.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/mc:RELEASE.2019-08-07T23-14-43Z  dockerhub.kubekey.local/kubesphereio/mc:RELEASE.2019-08-07T23-14-43Zdocker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z  dockerhub.kubekey.local/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Zdocker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.8.0  dockerhub.kubekey.local/kubesphereio/kube-rbac-proxy:v0.8.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.0  dockerhub.kubekey.local/kubesphereio/coredns:1.8.0docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/log-sidecar-injector:1.1  dockerhub.kubekey.local/kubesphereio/log-sidecar-injector:1.1docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/defaultbackend-amd64:1.4  dockerhub.kubekey.local/kubesphereio/defaultbackend-amd64:1.4docker tag  registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.22.12  dockerhub.kubekey.local/kubesphereio/kube-proxy:v1.22.12
docker tag registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.22.20 dockerhub.kubekey.local/kubesphereio/k8s-dns-node-cache:1.15.12
docker tag registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.23.2    dockerhub.kubekey.local/kubesphereio/kube-controllers:v3.23.2
docker tag registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.23.2   dockerhub.kubekey.local/kubesphereio/cni:v3.23.2
docker tag registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.23.2   dockerhub.kubekey.local/kubesphereio/pod2daemon-flexvol:v3.23.2
docker tag registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.23.2  dockerhub.kubekey.local/kubesphereio/node:v3.23.2
docker tag registry.cn-beijing.aliyuncs.com/kubesphereio/opensearch:2.6.0 dockerhub.kubekey.local/kubesphereio/opensearch:2.6.0
docker tag registry.cn-beijing.aliyuncs.com/kubesphereio/busybox:latest dockerhub.kubekey.local/kubesphereio/busybox:latest
docker tag kubesphere/fluent-bit:v2.0.6 dockerhub.kubekey.local/kubesphereio/fluent-bit:v2.0.6 # 也可重命名为v1.8.11,可省下后续修改fluent的yaml,这里采用后修改方式

2.6 推送至harbor私有仓库

#!/bin/bash
# docker load < ks3.3.1-images.tar.gzdocker login -u admin -p Harbor12345 dockerhub.kubekey.localdocker push dockerhub.kubekey.local/kubesphereio/ks-console:v3.3.1
docker push dockerhub.kubekey.local/kubesphereio/ks-controller-manager:v3.3.1
docker push dockerhub.kubekey.local/kubesphereio/ks-installer:v3.3.1
docker push dockerhub.kubekey.local/kubesphereio/ks-apiserver:v3.3.1
docker push dockerhub.kubekey.local/kubesphereio/openpitrix-jobs:v3.3.1
docker push dockerhub.kubekey.local/kubesphereio/alpine:3.14
docker push dockerhub.kubekey.local/kubesphereio/kube-apiserver:v1.22.12
docker push dockerhub.kubekey.local/kubesphereio/kube-scheduler:v1.22.12
docker push dockerhub.kubekey.local/kubesphereio/kube-proxy:v1.22.12
docker push dockerhub.kubekey.local/kubesphereio/kube-controller-manager:v1.22.12
docker push dockerhub.kubekey.local/kubesphereio/provisioner-localpv:3.3.0
docker push dockerhub.kubekey.local/kubesphereio/linux-utils:3.3.0
docker push dockerhub.kubekey.local/kubesphereio/kube-controllers:v3.23.2
docker push dockerhub.kubekey.local/kubesphereio/cni:v3.23.2
docker push dockerhub.kubekey.local/kubesphereio/pod2daemon-flexvol:v3.23.2
docker push dockerhub.kubekey.local/kubesphereio/node:v3.23.2
docker push dockerhub.kubekey.local/kubesphereio/kube-state-metrics:v2.5.0
docker push dockerhub.kubekey.local/kubesphereio/fluent-bit:v1.8.11
docker push dockerhub.kubekey.local/kubesphereio/prometheus-config-reloader:v0.55.1
docker push dockerhub.kubekey.local/kubesphereio/prometheus-operator:v0.55.1
docker push dockerhub.kubekey.local/kubesphereio/thanos:v0.25.2
docker push dockerhub.kubekey.local/kubesphereio/prometheus:v2.34.0
docker push dockerhub.kubekey.local/kubesphereio/fluentbit-operator:v0.13.0
docker push dockerhub.kubekey.local/kubesphereio/node-exporter:v1.3.1
docker push dockerhub.kubekey.local/kubesphereio/kubectl:v1.22.0
docker push dockerhub.kubekey.local/kubesphereio/notification-manager:v1.4.0
docker push dockerhub.kubekey.local/kubesphereio/notification-tenant-sidecar:v3.2.0
docker push dockerhub.kubekey.local/kubesphereio/notification-manager-operator:v1.4.0
docker push dockerhub.kubekey.local/kubesphereio/alertmanager:v0.23.0
docker push dockerhub.kubekey.local/kubesphereio/kube-rbac-proxy:v0.11.0
docker push dockerhub.kubekey.local/kubesphereio/docker:19.03
docker push dockerhub.kubekey.local/kubesphereio/pause:3.5
docker push dockerhub.kubekey.local/kubesphereio/configmap-reload:v0.5.0
docker push dockerhub.kubekey.local/kubesphereio/snapshot-controller:v4.0.0
docker push dockerhub.kubekey.local/kubesphereio/kube-rbac-proxy:v0.8.0
docker push dockerhub.kubekey.local/kubesphereio/coredns:1.8.0
docker push dockerhub.kubekey.local/kubesphereio/log-sidecar-injector:1.1
docker push dockerhub.kubekey.local/kubesphereio/k8s-dns-node-cache:1.15.12
docker push dockerhub.kubekey.local/kubesphereio/mc:RELEASE.2019-08-07T23-14-43Z
docker push dockerhub.kubekey.local/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z
docker push dockerhub.kubekey.local/kubesphereio/defaultbackend-amd64:1.4
docker push dockerhub.kubekey.local/kubesphereio/redis:5.0.14-alpine
docker push dockerhub.kubekey.local/kubesphereio/haproxy:2.3
docker push dockerhub.kubekey.local/kubesphereio/opensearch:2.6.0
docker push dockerhub.kubekey.local/kubesphereio/busybox:latest
docker push dockerhub.kubekey.local/kubesphereio/fluent-bit:v2.0.6

3. 使用kk部署kubesphere

3.1 移除麒麟系统自带的podman

podman是麒麟系统自带的容器引擎,为避免后续与docker冲突,直接卸载。否则后续coredns/nodelocaldns也会受影响无法启动以及各种docker权限问题。

yum remove podman

3.2 下载kubekey

下载 kubekey-v2.3.1-linux-arm64.tar.gz。具体 KubeKey 版本号可以在 KubeKey 发行页面([1]) 查看。

  • 方式一

    cd ~
    mkdir kubesphere
    cd kubesphere/# 选择中文区下载(访问 GitHub 受限时使用)
    export KKZONE=cn# 执行下载命令,获取最新版的 kk(受限于网络,有时需要执行多次)
    curl -sfL https://get-kk.kubesphere.io/v2.3.1/kubekey-v2.3.1-linux-arm64.tar.gz | tar xzf -
    
  • 方式二

        使用本地电脑,直接去github下载 Releases · kubesphere/kubekey

        上传至服务器/root/kubesphere目录解压 

tar zxf kubekey-v2.3.1-linux-arm64.tar.gz

3.3  生成集群创建配置文件

创建集群配置文件,本示例中,选择 KubeSphere v3.3.1 和 Kubernetes v1.22.12。

./kk create config -f kubesphere-v331-v12212.yaml --with-kubernetes v1.22.12 --with-kubesphere v3.3.1

命令执行成功后,在当前目录会生成文件名为 kubesphere-v331-v12212.yaml 的配置文件。

注意: 生成的默认配置文件内容较多,这里就不做过多展示了,更多详细的配置参数请参考 官方配置示例。

本文示例采用 3 个节点同时作为 control-plane、etcd 节点和 worker 节点。

编辑配置文件 kubesphere-v331-v12212.yaml,主要修改 kind: Cluster 和 kind: ClusterConfiguration 两小节的相关配置

修改 kind: Cluster 小节中 hosts 和 roleGroups 等信息,修改说明如下。

  • hosts:指定节点的 IP、ssh 用户、ssh 密码、ssh 端口。特别注意 一定要手工指定 arch: arm64,否则部署的时候会安装 X86 架构的软件包。

  • roleGroups:指定 3 个 etcd、control-plane 节点,复用相同的机器作为 3 个 worker 节点。

  • internalLoadbalancer:启用内置的 HAProxy 负载均衡器。

  • domain:自定义了一个 opsman.top

  • containerManager:使用了 containerd

  • storage.openebs.basePath:新增配置,指定默认存储路径为 /data/openebs/local

修改后的示例如下:

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:name: sample
spec:hosts:- {name: node1, address: 192.168.200.7, internalAddress: 192.168.200.7, user: root, password: "123456", arch: arm64}roleGroups:etcd:- node1control-plane: - node1worker:- node1registry:- node1controlPlaneEndpoint:## Internal loadbalancer for apiservers # internalLoadbalancer: haproxydomain: lb.kubesphere.localaddress: ""port: 6443kubernetes:version: v1.22.12clusterName: cluster.localautoRenewCerts: truecontainerManager: dockeretcd:type: kubekeynetwork:plugin: calicokubePodsCIDR: 10.233.64.0/18kubeServiceCIDR: 10.233.0.0/18## multus support. https://github.com/k8snetworkplumbingwg/multus-cnimultusCNI:enabled: falseregistry:type: harborauths:"dockerhub.kubekey.local":username: adminpassword: Harbor12345privateRegistry: "dockerhub.kubekey.local"namespaceOverride: "kubesphereio"registryMirrors: []insecureRegistries: []addons: []---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:name: ks-installernamespace: kubesphere-systemlabels:version: v3.3.1
spec:persistence:storageClass: ""authentication:jwtSecret: ""zone: ""local_registry: ""namespace_override: ""# dev_tag: ""etcd:monitoring: trueendpointIps: localhostport: 2379tlsEnable: truecommon:core:console:enableMultiLogin: trueport: 30880type: NodePort# apiserver:#  resources: {}# controllerManager:#  resources: {}redis:enabled: falsevolumeSize: 2Giopenldap:enabled: falsevolumeSize: 2Giminio:volumeSize: 20Gimonitoring:# type: externalendpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090GPUMonitoring:enabled: falsegpu:kinds:- resourceName: "nvidia.com/gpu"resourceType: "GPU"default: truees:# master:#   volumeSize: 4Gi#   replicas: 1#   resources: {}# data:#   volumeSize: 20Gi#   replicas: 1#   resources: {}logMaxAge: 7elkPrefix: logstashbasicAuth:enabled: falseusername: ""password: ""externalElasticsearchHost: ""externalElasticsearchPort: ""alerting:enabled: true# thanosruler:#   replicas: 1#   resources: {}auditing:enabled: false# operator:#   resources: {}# webhook:#   resources: {}devops:enabled: false# resources: {}jenkinsMemoryLim: 8GijenkinsMemoryReq: 4GijenkinsVolumeSize: 8Gievents:enabled: false# operator:#   resources: {}# exporter:#   resources: {}# ruler:#   enabled: true#   replicas: 2#   resources: {}logging:enabled: truelogsidecar:enabled: truereplicas: 2# resources: {}metrics_server:enabled: falsemonitoring:storageClass: ""node_exporter:port: 9100# resources: {}# kube_rbac_proxy:#   resources: {}# kube_state_metrics:#   resources: {}# prometheus:#   replicas: 1#   volumeSize: 20Gi#   resources: {}#   operator:#     resources: {}# alertmanager:#   replicas: 1#   resources: {}# notification_manager:#   resources: {}#   operator:#     resources: {}#   proxy:#     resources: {}gpu:nvidia_dcgm_exporter:enabled: false# resources: {}multicluster:clusterRole: nonenetwork:networkpolicy:enabled: falseippool:type: nonetopology:type: noneopenpitrix:store:enabled: trueservicemesh:enabled: falseistio:components:ingressGateways:- name: istio-ingressgatewayenabled: falsecni:enabled: falseedgeruntime:enabled: falsekubeedge:enabled: falsecloudCore:cloudHub:advertiseAddress:- ""service:cloudhubNodePort: "30000"cloudhubQuicNodePort: "30001"cloudhubHttpsNodePort: "30002"cloudstreamNodePort: "30003"tunnelNodePort: "30004"# resources: {}# hostNetWork: falseiptables-manager:enabled: truemode: "external"# resources: {}# edgeService:#   resources: {}terminal:timeout: 600

3.4 执行安装

./kk create cluster -f kubesphere-v331-v122123.yaml

此节点之所以安装kubesphere是因为kk在安装过程中会产生kubekey文件夹并将k8s所需要的依赖都下载到kubekey目录。后续我们离线安装主要使用kubekey文件夹,配合一下脚本代替之前的制品。

4. 制作离线部署资源

4.1 导出k8s基础依赖包

yum -y install openssl socat conntrack ipset ebtables chrony ipvsadm --downloadonly --downloaddir /root/kubesphere/k8s-init
# 打成压缩包
tar -czvf k8s-init-Kylin_V10-arm.tar.gz ./k8s-init/*

4.2 导出ks需要的镜像

导出ks相关的镜像至ks3.3.1-images.tar

docker save -o ks3.3.1-images.tar  dockerhub.kubekey.local/kubesphereio/kube-controllers:v3.27.3  dockerhub.kubekey.local/kubesphereio/cni:v3.27.3  dockerhub.kubekey.local/kubesphereio/pod2daemon-flexvol:v3.27.3  dockerhub.kubekey.local/kubesphereio/node:v3.27.3  dockerhub.kubekey.local/kubesphereio/ks-console:v3.3.1  dockerhub.kubekey.local/kubesphereio/alpine:3.14  dockerhub.kubekey.local/kubesphereio/k8s-dns-node-cache:1.22.20  dockerhub.kubekey.local/kubesphereio/ks-controller-manager:v3.3.1  dockerhub.kubekey.local/kubesphereio/ks-installer:v3.3.1  dockerhub.kubekey.local/kubesphereio/ks-apiserver:v3.3.1  dockerhub.kubekey.local/kubesphereio/openpitrix-jobs:v3.3.1  dockerhub.kubekey.local/kubesphereio/kube-apiserver:v1.22.12  dockerhub.kubekey.local/kubesphereio/kube-proxy:v1.22.12  dockerhub.kubekey.local/kubesphereio/kube-controller-manager:v1.22.12  dockerhub.kubekey.local/kubesphereio/kube-scheduler:v1.22.12  dockerhub.kubekey.local/kubesphereio/provisioner-localpv:3.3.0  dockerhub.kubekey.local/kubesphereio/linux-utils:3.3.0  dockerhub.kubekey.local/kubesphereio/kube-state-metrics:v2.5.0  dockerhub.kubekey.local/kubesphereio/fluent-bit:v2.0.6  dockerhub.kubekey.local/kubesphereio/prometheus-config-reloader:v0.55.1  dockerhub.kubekey.local/kubesphereio/prometheus-operator:v0.55.1  dockerhub.kubekey.local/kubesphereio/thanos:v0.25.2  dockerhub.kubekey.local/kubesphereio/prometheus:v2.34.0  dockerhub.kubekey.local/kubesphereio/fluentbit-operator:v0.13.0   dockerhub.kubekey.local/kubesphereio/node-exporter:v1.3.1  dockerhub.kubekey.local/kubesphereio/kubectl:v1.22.0  dockerhub.kubekey.local/kubesphereio/notification-manager:v1.4.0  dockerhub.kubekey.local/kubesphereio/notification-tenant-sidecar:v3.2.0  dockerhub.kubekey.local/kubesphereio/notification-manager-operator:v1.4.0  dockerhub.kubekey.local/kubesphereio/alertmanager:v0.23.0  dockerhub.kubekey.local/kubesphereio/kube-rbac-proxy:v0.11.0  dockerhub.kubekey.local/kubesphereio/docker:19.03  dockerhub.kubekey.local/kubesphereio/metrics-server:v0.4.2  dockerhub.kubekey.local/kubesphereio/pause:3.5  dockerhub.kubekey.local/kubesphereio/configmap-reload:v0.5.0  dockerhub.kubekey.local/kubesphereio/snapshot-controller:v4.0.0  dockerhub.kubekey.local/kubesphereio/mc:RELEASE.2019-08-07T23-14-43Z  dockerhub.kubekey.local/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z  dockerhub.kubekey.local/kubesphereio/kube-rbac-proxy:v0.8.0  dockerhub.kubekey.local/kubesphereio/coredns:1.8.0   dockerhub.kubekey.local/kubesphereio/defaultbackend-amd64:1.4 dockerhub.kubekey.local/kubesphereio/redis:5.0.14-alpine dockerhub.kubekey.local/kubesphereio/k8s-dns-node-cache:1.15.12 dockerhub.kubekey.local/kubesphereio/node:v3.23.2 dockerhub.kubekey.local/kubesphereio/pod2daemon-flexvol:v3.23.2 dockerhub.kubekey.local/kubesphereio/cni:v3.23.2 dockerhub.kubekey.local/kubesphereio/kube-controllers:v3.23.2 dockerhub.kubekey.local/kubesphereio/haproxy:2.3 dockerhub.kubekey.local/kubesphereio/busybox:latest dockerhub.kubekey.local/kubesphereio/opensearch:2.6.0 dockerhub.kubekey.local/kubesphereio/fluent-bit:v2.0.6

压缩

gzip ks3.3.1-images.tar

4.3 导出kubesphere文件夹

[root@node1 ~]# cd /root/kubesphere
[root@node1 kubesphere]# ls
create_project_harbor.sh  docker-24.0.7-arm.tar.gz  fluent-bit-daemonset.yaml  harbor-arm.tar.gz  harbor.tar.gz  install.sh  k8s-init-Kylin_V10-arm.tar.gz  ks3.3.1-images.tar.gz  ks3.3.1-offline  push-images.shtar -czvf kubeshpere.tar.gz ./kubesphere/*
编写install.sh用于后续一键离线安装kk

#!/usr/bin/env bash
read -p "请先修改机器配置文件ks3.3.1-offline/kubesphere-v331-v12212.yaml中相关IP地址,是否已修改(yes/no)" B 
do_k8s_init(){echo "--------开始进行依赖包初始化------"yum remove podman -ytar zxf k8s-init-Kylin_V10-arm.tar.gzcd k8s-init && ./install.shcd -rm -rf k8s-init
}install_docker(){echo "--------开始安装docker--------"tar zxf docker-24.0.7-arm.tar.gzcd docker && ./install.shcd -}
install_harbor(){echo "-------开始安装harbor----------"tar zxf  harbor-arm.tar.gzcd harbor && ./install.shcd -echo "--------开始推送镜像----------"source create_project_harbor.shsource push-images.shecho "--------镜像推送完成--------"
}
install_ks(){echo "--------开始安装kubesphere--------"
#        tar zxf ks3.3.1-offline.tar.gzcd ks3.3.1-offline && ./install.sh
}if [ "$B" = "yes" ] || [ "$B" = "y" ]; thendo_k8s_initinstall_dockerinstall_harborinstall_ks
elseecho "请先配置集群配置文件"exit 1
fi

5. 离线环境安装k8s和kubesphere

5.1 卸载podman和安装k8s依赖

所有节点都需要操作,

上传k8s-init-Kylin_V10-arm.tar.gz并解压后执行install.sh,如果单节点离线部署可直接使用下一步。

yum remove podman -y

5.2 安装ks集群

上传kubeshpere.tar.gz并解压,修改./kubesphere/ks3.3.1-offline/kubesphere-v331-v12212.yaml集群配置文件中相关ip,密码等信息。修改后执行install.sh,等待十分钟左右可看到如下消息:

**************************************************
Waiting for all tasks to be completed ...
task alerting status is successful  (1/6)
task network status is successful  (2/6)
task multicluster status is successful  (3/6)
task openpitrix status is successful  (4/6)
task logging status is successful  (5/6)
task monitoring status is successful  (6/6)
**************************************************
Collecting installation results ...
#####################################################
###              Welcome to KubeSphere!           ###
#####################################################Console: http://192.168.10.2:30880
Account: admin
Password: P@88w0rd
NOTES:1. After you log into the console, please check themonitoring status of service components in"Cluster Management". If any service is notready, please wait patiently until all components are up and running.2. Please change the default password after login.#####################################################
https://kubesphere.io             2024-07-03 11:10:11
#####################################################

5.3 其他修改

由于开启了日志功能,arm-麒麟版本fluent-bit一直报错,需更改为2.0.6

kubectl edit daemonsets fluent-bit -n kubesphere-logging-system#修改其中fluent-bit版本1.8.11为2.0.6

如果不需要日志,可以修改ks创建集群配置文件不安装log插件,镜像也可以更加简化

6.测试查看

6.1 验证集群状态

[root@node1 ~]# kubectl get nodes 
NAME    STATUS   ROLES                         AGE   VERSION
node1   Ready    control-plane,master,worker   25h   v1.22.12
node2   Ready    control-plane,master,worker   25h   v1.22.12
node3   Ready    control-plane,master,worker   25h   v1.22.12

页面查看

图片

7.总结

本文主要实战演示了ARM 版 麒麟 V10服务器通过在线环境部署k8s和kubesphere,并将基础依赖,需要的docker镜像和harbor,以及kubekey部署ks下载的各类包一起打包。通过shell脚本编写简单的部署过程,实现离线环境安装k8s和kubesphere。

离线安装主要知识点

  • 卸载podman

  • 安装k8s依赖包

  • 安装Docker

  • 安装harbor

  • 将k8s和ks需要的镜像推送到harbor

  • 使用kk部署集群

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.mzph.cn/bicheng/43288.shtml

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

【linux高级IO(二)】多路转接之select详解

&#x1f493;博主CSDN主页:杭电码农-NEO&#x1f493;   ⏩专栏分类:Linux从入门到精通⏪   &#x1f69a;代码仓库:NEO的学习日记&#x1f69a;   &#x1f339;关注我&#x1faf5;带你学更多操作系统知识   &#x1f51d;&#x1f51d; Linux高级IO 1. 前言2. 初识s…

TEE共享内存的介绍-共享内存的限制是什么? TA堆内存/栈内存有限制吗?Trustzone/TEE

快速链接: . 👉👉👉 个人博客笔记导读目录(全部) 👈👈👈 付费专栏-付费课程 【购买须知】:【精选】TEE从入门到精通-[目录] 👈👈👈思考: 如何开发一个TA? sdk又是什么?开发一个TA的流程是怎样的?How to do?有关TA的签名介绍TEE开发Secure driver介绍RP…

昇思25天学习打卡营第21天|基于MindSpore实现BERT对话情绪识别

模型简介 BERT全称是来自变换器的双向编码器表征量。与BERT模型相似的预训练语言模型例如问答、命名实体识别、自然语言推理、文本分类等在许多自然语言处理任务中发挥着重要作用。模型是基于Transformer中的Encoder并加上双向的结构。 BERT模型的主要创新点都在pre-train方法…

SCI丨返修一作+通讯

中科四区&#xff0c;JCR2 返修转让一作通讯&#xff0c;5个月左右录用 题目&#xff1a;通过机器学习算法XXXXXXXxxx混凝土力学性能的可靠方法

Nginx 配置ssl证书

1. 准备 SSL 证书文件 确保您有以下文件&#xff1a; SSL 证书文件&#xff08;通常是 .crt 或 .pem 文件&#xff09; 私钥文件&#xff08;通常是 .key 文件&#xff09; 中间证书文件&#xff08;如果适用&#xff0c;通常是 .crt 或 .pem 文件&#xff09; 将这些文件上传…

苍穹外卖--完善登录功能:进行MD5加密

目标 TODO&#xff1a;使用MD5加密方式对明文密码。 实现 password DigestUtils.md5DigestAsHex(password.getBytes());

Face_recognition实现人脸识别

这里写自定义目录标题 欢迎使用Markdown编辑器一、安装人脸识别库face_recognition1.1 安装cmake1.2 安装dlib库1.3 安装face_recognition 二、3个常用的人脸识别案例2.1 识别并绘制人脸框2.2 提取并绘制人脸关键点2.3 人脸匹配及标注 欢迎使用Markdown编辑器 本文基于face_re…

双向链表+Map实现LRU

LRU: LRU是Least Recently Used的缩写&#xff0c;即最近最少使用&#xff0c;是一种常用的页面置换算法&#xff0c;选择最近最久未使用的页面予以淘汰。 核心思想&#xff1a; 基于Map实现k-v存储&#xff0c;双向链表中使用一个虚拟头部和虚拟尾部&#xff0c;虚拟头部的…

BioXcell—InVivoMAb anti-West Nile/dengue virus E protein

研发背景&#xff1a; 西尼罗河病毒(WNV)是一种由蚊虫类介导传播的黄病毒&#xff0c;与引起人类感染性流行病的登革热病毒、黄热病病毒和日本脑炎病毒密切相关。 WNV和登革热病毒(DENV)同属黄病毒科&#xff08;Flaviviridae&#xff09;黄热病毒属&#xff0c;是具有小包膜单…

实现llava的【单轮对话】调整成【多轮对话】 (输入图片/多模态/多轮对话/llava)

使用llava时&#xff0c;将llava的单轮对话调整成多轮对话 先说好方法一&#xff0c;直接对官方网站的quick start代码进行修改方法二&#xff1a;使用bash结合基础的单轮对话代码进行修改2.1 首先是最基础的llava官网代码回顾2.2 伪多轮对话方法&#xff08;每次新的prompt输入…

AEC10 SA计算整理 --- 基础SA

LuxSA: LuxSALumaAvgLumaBE16x16 LuxSATarget[setparam/tr:luxlux] LuxSAAdjRatioLuxSATarget/LuxSALumaLuxSALuma: 计算16x16区域的平均亮度&#xff08;Luma值&#xff09;。 LuxSATarget: 通过参数设置获取目标亮度值&#xff08;通常与当前光线条件相关&#xff09;。 Lux…

【多模态】41、VILA | 打破常规多模态模型训练策略,在预训练阶段就微调 LLM 被证明能取得更好的效果!

论文&#xff1a;VILA: On Pre-training for Visual Language Models 代码&#xff1a;https://github.com/NVlabs/VILA 出处&#xff1a;NVLabs 时间&#xff1a;2024.05 贡献&#xff1a; 证明在预训练阶段对 LLM 进行微调能够提升模型对上下文任务的效果在 SFT 阶段混合…

同三维T80006解码器视频使用操作说明书:高清HDMI解码器,高清SDI解码器,4K超清HDMI解码器,双路4K超高清解码器

同三维T80006解码器视频使用操作说明书&#xff1a;高清HDMI解码器&#xff0c;高清SDI解码器&#xff0c;4K超清HDMI解码器&#xff0c;双路4K超高清解码器 解码器T80006 同三维&#xff0c;十多年老品牌&#xff0c;我们一直专注&#xff1a;视频采集卡、视频编码器、转码器、…

Centos7离线安装ElasticSearch7.4.2

一、官网下载相关的安装包 ElasticSearch7.4.2&#xff1a; elasticsearch-7.4.2-linux-x86_64.tar.gz 下载中文分词器&#xff1a; elasticsearch-analysis-ik-7.4.2.zip 二、上传解压文件到服务器 上传到目录&#xff1a;/home/data/elasticsearch 解压文件&#xff1…

免费无限白嫖阿里云服务器

今天&#xff0c;我来分享一个免费且无限使用阿里云服务器的方法&#xff0c;零成本&#xff01;这适用于日常测试学习&#xff0c;比如测试 Shell 脚本、学习 Docker 安装、MySQL 等等。跟着我的步骤&#xff0c;你将轻松拥有一个稳定可靠的服务器&#xff0c;为你的学习和实践…

数据库的优点和缺点分别是什么

数据库作为数据存储和管理的核心组件&#xff0c;具有一系列显著的优点&#xff0c;同时也存在一些潜在的缺点。以下是对数据库优点和缺点的详细分析&#xff1a; 优点 数据一致性&#xff1a;数据库通过事务处理和锁机制等手段&#xff0c;确保数据的一致性和完整性。这意味着…

错误记录-SpringCloud-OpenFeign测试远程调用

文章目录 1&#xff0c;org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name memberController: Unsatisfied dependency expressed through field couponFeign2&#xff0c; Receiver class org.springframework.cloud.netflix…

几种不同的方式禁止IP访问网站(PHP、Nginx、Apache设置方法)

1、PHP禁止IP和IP段访问 <?//禁止某个IP$banned_ip array ("127.0.0.1",//"119.6.20.66","192.168.1.4");if ( in_array( getenv("REMOTE_ADDR"), $banned_ip ) ){die ("您的IP禁止访问&#xff01;");}//禁止某个IP段…

SAP S4 环境下,KSU1 Ob52 转为前台操作,不产生传输请求号

参考 OB52/KSV1/KSU1等与Client状态相关的前台操作-y_q_yang se16n 编辑表 T811FLAGS 新增行 CYCLES MAINTENANCE X 即可

WAIC 2024 AI盛宴大会亮点回顾

&#x1f389; 刚刚落幕的WAIC 2024大会&#xff0c;简直是科技迷们的狂欢节&#xff01;这场全球瞩目的人工智能盛会&#xff0c;不仅汇聚了全球顶尖的智慧&#xff0c;还带来了无数令人惊叹的创新成果。让我们一起回顾那些抓人眼球的亮点吧&#xff01; &#x1f525; 超燃开…