Kubernetes云原生存储解决方案openebs部署实践-4.0.1版本（helm部署）

简介

OpenEBS 是一种开源云原生存储解决方案。OpenEBS 可以将 Kubernetes 工作节点可用的任何存储转化为本地或复制的 Kubernetes 持久卷。OpenEBS 帮助应用和平台团队轻松地部署需要快速、持久耐用、可靠且可扩展的容器原生存储的 Kubernetes 有状态工作负载。

OpenEBS 也是基于 NVMe 存储部署的首选方案之一。

OpenEBS 最初由 MayaData 构建，并捐赠给了云原生计算基金会（Cloud Native Computing Foundation）。

OpenEBS 工作流:

在这里插入图片描述

安装部署

另一篇文章部署实践了3.10.0版本，本文部署最新的版本，新版本提供了更多的功能。

设置helm仓库

helm repo add openebs https://openebs.github.io/openebs
helm repo update

使用默认值安装OpenEBS helm chart

helm install openebs --namespace openebs openebs/openebs --create-namespace

上面的命令会安装OpenEBS Local PV Hostpath, OpenEBS Local PV LVM, OpenEBS Local PV ZFS, and OpenEBS Replicated Storage组件。

下面的命令选择不安装openebs replicated Storage:

helm install openebs --namespace openebs openebs/openebs --set engines.replicated.mayastor.enabled=false --create-namespace

查看安装的chart

helm ls -n openebs

安装情况确认

kubectl get pods -n openebs

如果是全部组件安装：

NAME                                              READY   STATUS    RESTARTS   AGE
openebs-agent-core-674f784df5-7szbm               2/2     Running   0          11m
openebs-agent-ha-node-nnkmv                       1/1     Running   0          11m
openebs-agent-ha-node-pvcrr                       1/1     Running   0          11m
openebs-agent-ha-node-rqkkk                       1/1     Running   0          11m
openebs-api-rest-79556897c8-b824j                 1/1     Running   0          11m
openebs-csi-controller-b5c47d49-5t5zd             6/6     Running   0          11m
openebs-csi-node-flq49                            2/2     Running   0          11m
openebs-csi-node-k8d7h                            2/2     Running   0          11m
openebs-csi-node-v7jfh                            2/2     Running   0          11m
openebs-etcd-0                                    1/1     Running   0          11m
openebs-etcd-1                                    1/1     Running   0          11m
openebs-etcd-2                                    1/1     Running   0          11m
openebs-io-engine-7t6tf                           2/2     Running   0          11m
openebs-io-engine-9df6r                           2/2     Running   0          11m
openebs-io-engine-rqph4                           2/2     Running   0          11m
openebs-localpv-provisioner-6ddf7c7978-4fkvs      1/1     Running   0          11m
openebs-loki-0                                    1/1     Running   0          11m
openebs-lvm-localpv-controller-7b6d6b4665-fk78q   5/5     Running   0          11m
openebs-lvm-localpv-node-mcch4                    2/2     Running   0          11m
openebs-lvm-localpv-node-pdt88                    2/2     Running   0          11m
openebs-lvm-localpv-node-r9jn2                    2/2     Running   0          11m
openebs-nats-0                                    3/3     Running   0          11m
openebs-nats-1                                    3/3     Running   0          11m
openebs-nats-2                                    3/3     Running   0          11m
openebs-obs-callhome-854bc967-5f879               2/2     Running   0          11m
openebs-operator-diskpool-5586b65c-cwpr8          1/1     Running   0          11m
openebs-promtail-2vrzk                            1/1     Running   0          11m
openebs-promtail-mwxk8                            1/1     Running   0          11m
openebs-promtail-w7b8k                            1/1     Running   0          11m
openebs-zfs-localpv-controller-f78f7467c-blr7q    5/5     Running   0          11m
openebs-zfs-localpv-node-h46m5                    2/2     Running   0          11m
openebs-zfs-localpv-node-svfgq                    2/2     Running   0          11m
openebs-zfs-localpv-node-wm9ks                    2/2     Running   0          11m

问题记录：

本文在进行实际部署后，出现了openebs-csi-node无法正常启动的问题，详情如下：

root@master1:~# kubectl get pod -n openebs
...
openebs-csi-node-jk7wj                            0/2     Init:0/1   1              23h
openebs-csi-node-k98dr                            0/2     Init:0/1   1              23h
openebs-csi-node-vrhsj                            0/2     Init:0/1   1              23h
openebs-csi-node-wp7z2                            0/2     Init:0/1   1              23h
...# 查看日志pod卡在初始化阶段
root@master1:/etc# kubectl logs -n openebs openebs-csi-node-jk7wj
Defaulted container "csi-node" out of: csi-node, csi-driver-registrar, nvme-tcp-probe (init)
Error from server (BadRequest): container "csi-node" in pod "openebs-csi-node-jk7wj" is waiting to start: PodInitializing

出现了该问题的原因为linux内核没有加载nvme_tcp模块，各个节点使用命令modprobe nvme_tcp加载后，删除pod重建启动正常。

将模块加载命令写入到配置文件中，重启系统后自动加载：

echo "nvme_tcp" | sudo tee /etc/modules-load.d/openebs.conf

如果禁用了Replicated Storage，输出类似如下：

NAME                                              READY   STATUS    RESTARTS   AGE
openebs-localpv-provisioner-6ddf7c7978-jsstg      1/1     Running   0          3m9s
openebs-lvm-localpv-controller-7b6d6b4665-wfw64   5/5     Running   0          3m9s
openebs-lvm-localpv-node-62lnq                    2/2     Running   0          3m9s
openebs-lvm-localpv-node-lhndx                    2/2     Running   0          3m9s
openebs-lvm-localpv-node-tlcqv                    2/2     Running   0          3m9s
openebs-zfs-localpv-controller-f78f7467c-k7ldb    5/5     Running   0          3m9s
openebs-zfs-localpv-node-5mwbz                    2/2     Running   0          3m9s
openebs-zfs-localpv-node-g45ft                    2/2     Running   0          3m9s
openebs-zfs-localpv-node-g77g6                    2/2     Running   0          3m9s

说明：

OpenEBS中的Replicated Storage是指通过将数据复制到多个存储节点来提供高可用性和数据冗余的一种存储方案。

demo测试

openebs部署完成后会自动创建存储类，我们使用openebs-hostpath这个StorageClass来创建PVC。

存储类：

root@master1:~# kubectl get sc -n openebs
NAME                     PROVISIONER               RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
mayastor-etcd-localpv    openebs.io/local          Delete          WaitForFirstConsumer   false                  2d
mayastor-loki-localpv    openebs.io/local          Delete          WaitForFirstConsumer   false                  2d
openebs-hostpath         openebs.io/local          Delete          WaitForFirstConsumer   false                  2d
openebs-single-replica   io.openebs.csi-mayastor   Delete          Immediate              true                   2d

创建一个示例应用挂载openebs提供的存储进行测试。资源清单文件openebs-test.yaml，包括pvc和pod：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: local-hostpath-pvc
spec:storageClassName: openebs-hostpathaccessModes:- ReadWriteOnceresources:requests:storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:name: hello-local-hostpath-pod
spec:volumes:- name: local-storagepersistentVolumeClaim:claimName: local-hostpath-pvccontainers:- name: hello-containerimage: busyboxcommand:- sh- -c- 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'volumeMounts:- mountPath: /mnt/storename: local-storage

创建资源：kubectl create -f openebs-test.yaml，等待创建完成。

root@master1:~# kubectl get pod -o wide | grep hello
hello-local-hostpath-pod           1/1     Running   0              33s    10.244.2.35    master3   <none>           <none>root@master1:~# kubectl get pvc
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
local-hostpath-pvc   Bound    pvc-8b06018d-4222-49e0-adf2-4cbf0b4f3410   5Gi        RWO            openebs-hostpath   78sroot@master1:~# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS      REASON   AGE
pvc-8b06018d-4222-49e0-adf2-4cbf0b4f3410   5Gi        RWO            Delete           Bound    default/local-hostpath-pvc  openebs-hostpath        77s# 检查node1节点的本地路径
test@master3:~$  cat /var/openebs/local/pvc-8b06018d-4222-49e0-adf2-4cbf0b4f3410/greet.txt
Wed Jul  3 14:04:49 UTC 2024 [hello-local-hostpath-pod] Hello from OpenEBS Local PV.