简介

k8s集群中搭建rabbitmq集群服务一般都会用到pvc，但是考虑到有些自建k8s环境下，搭建的共享存储可能会存在稳定性及性能问题，所以这次是通过采用节点亲和性和hostpath来实现，目前的operator的基本都是采用共享存储的方法。本文将根据现有环境及不同需求将rabbitmq镜像集群的搭建采用hostpath+亲和性的权重+多副本来实现数据持久化和高可用。

一.条件及环境说明

k8s版本k8s-1.29.4，环境搭建在电信机房,六个worker节点，每个节点有一块非系统盘的ssd盘挂载到/data/路径，不采用ceph或nas之类的共享存储，也未采用operator，目前rabbitmq-operator基本都是采用了共享存储。

二.需求说明

• 搭建rabbitmq镜像模式集群。
• 高可用：有一个k8s节点死掉或重启pod之后也不影响使用。
• 高性能：读写本地磁盘实现高性能的io，共享存储如果资源及硬件性能不够的话，io将会是性能瓶颈。
• 数据安全：在发生切换,节点故障以后，数据要尽可能的保证完整
• 安装简单、管理维护容易

三.实现原理及说明

四.详细步骤

4.1.规划节点标签

节点名称	角色	标签
k8s-worker-120-81	主	storage-selector=node-a-master
k8s-worker-120-82	主	storage-selector=node-b-master
k8s-worker-120-83	主	storage-selector=node-c-master
k8s-worker-120-84	备	storage-selector=node-a-slve
k8s-worker-120-85	备	storage-selector=node-a-slve
k8s-worker-120-86	备	storage-selector=node-a-slve

注：这里的rabbitmq是三节点的镜像集群，每个redis的pod分别落在三个主的节点上，当主节点发生故障是，就通过权重分配切换到备节点，例如：名称为autotest-rabbitmq-a-0的pod，默认落在标签为storage-selector=node-a-master的节点上，当该节点发生故障是，就会根据权重匹配罗在storage-selector=node-a-slve的节点上。

4.2.创建configmap配置

将如下配置保存到autotest-rabbitmq-config.yaml，配置中配置默认的vhost和用户密码，并初始化好集群节点【安装前规划好节点名称】信息。

apiVersion: v1
kind: ConfigMap
metadata:name: autotest-rabbitmq-confignamespace: zx-applabels:appname: pcauto-zxapp: autotest-rabbitmq-config 
data:enabled_plugins: |[rabbitmq_management,rabbitmq_prometheus].rabbitmq.conf: |#listeners.tcp.default = 5672default_vhost = /default_user = admindefault_pass = admindefault_user_tags.administrator = truedefault_user_tags.management = truedefault_user_tags.custom_tag = truechannel_max = 1024tcp_listen_options.backlog = 2048cluster_partition_handling = autohealcluster_partition_handling.pause_if_all_down.recover = autohealcluster_formation.peer_discovery_backend = classic_configcluster_formation.classic_config.nodes.1 = rabbit@autotest-rabbitmq-a-0.autotest-rabbitmq-a.zx-app.svc.cluster.localcluster_formation.classic_config.nodes.2 = rabbit@autotest-rabbitmq-b-0.autotest-rabbitmq-b.zx-app.svc.cluster.localcluster_formation.classic_config.nodes.3 = rabbit@autotest-rabbitmq-c-0.autotest-rabbitmq-c.zx-app.svc.cluster.local# cluster_formation.node_type = disc# cluster_keepalive_interval = 10000log.file.level = errorlog.console = true

4.3.创建三个statefulset和service headless配置

1.autotest-a.yaml 节点a的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:name: autotest-rabbitmq-a namespace: zx-app labels:appname: pcauto-zxapp: autotest-rabbitmq-aappgroup: autotest-rabbitmq
spec:serviceName: "autotest-rabbitmq-a"replicas: 1selector:matchLabels:app: autotest-rabbitmq-aappgroup: autotest-rabbitmqtemplate:metadata:labels:app: autotest-rabbitmq-aappgroup: autotest-rabbitmqspec:affinity:nodeAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 80preference:matchExpressions:- key: storage-selectoroperator: Invalues:- node-a-master- weight: 20preference:matchExpressions:- key: storage-selectoroperator: Invalues:- node-a-slavecontainers:- name: rabbitmq-serverimage: xxx-vpc.cn-shenzhen.cr.aliyuncs.com/public/rabbitmq:3.12.14-management imagePullPolicy: IfNotPresentcommand: ["/bin/sh","-c"]args: - mkdir -p /home/rabbitmq/${POD_NAME};chown rabbitmq.rabbitmq -R /home/rabbitmq/${POD_NAME};rabbitmq-serverenv:- name: RABBITMQ_ERLANG_COOKIEvalue: "YZSDHWMFSMKEMBDHSGGZ"- name: K8S_SERVICE_NAMEvalue: autotest-rabbitmq-a- name: POD_IPvalueFrom:fieldRef:fieldPath: status.podIP- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: RABBITMQ_MNESIA_BASEvalue: /home/rabbitmq/$(POD_NAME)/mnesia- name: RABBITMQ_USE_LONGNAMEvalue: "true"- name: RABBITMQ_NODENAMEvalue: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.localports:- name: httpcontainerPort: 15672 - name: amqpcontainerPort: 5672 volumeMounts:- name: rbmq-datamountPath: /home/rabbitmq - name: rabbitmq-config-volumemountPath: /etc/rabbitmq/ restartPolicy: Always terminationGracePeriodSeconds: 30volumes:- name: rbmq-datahostPath:path: /data/rabbitmqtype: ""- name: rabbitmq-config-volumeconfigMap:name: autotest-rabbitmq-configtolerations:- key: node.kubernetes.io/not-readyoperator: Existseffect: NoExecutetolerationSeconds: 3600- key: node.kubernetes.io/unreachableoperator: Existseffect: NoExecutetolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:name: autotest-rabbitmq-anamespace: zx-applabels:appname: pcauto-zxapp: autotest-rabbitmq-a
spec:ports:- port: 5672clusterIP: Noneselector:app: autotest-rabbitmq-a

2.autotest-b.yaml 节点b的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:name: autotest-rabbitmq-b namespace: zx-app labels:appname: pcauto-zxapp: autotest-rabbitmq-bappgroup: autotest-rabbitmq
spec:serviceName: "autotest-rabbitmq-b"replicas: 1selector:matchLabels:app: autotest-rabbitmq-bappgroup: autotest-rabbitmqtemplate:metadata:labels:app: autotest-rabbitmq-bappgroup: autotest-rabbitmqspec:affinity:nodeAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 80preference:matchExpressions:- key: storage-selectoroperator: Invalues:- node-b-master- weight: 20preference:matchExpressions:- key: storage-selectoroperator: Invalues:- node-b-slavecontainers:- name: rabbitmq-serverimage: xxx-vpc.cn-shenzhen.cr.aliyuncs.com/public/rabbitmq:3.12.14-management imagePullPolicy: IfNotPresentcommand: ["/bin/sh","-c"]args: - mkdir -p /home/rabbitmq/${POD_NAME};chown rabbitmq.rabbitmq -R /home/rabbitmq/${POD_NAME};rabbitmq-serverenv:- name: RABBITMQ_ERLANG_COOKIEvalue: "YZSDHWMFSMKEMBDHSGGZ"- name: K8S_SERVICE_NAMEvalue: autotest-rabbitmq-b- name: POD_IPvalueFrom:fieldRef:fieldPath: status.podIP- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: RABBITMQ_MNESIA_BASEvalue: /home/rabbitmq/$(POD_NAME)/mnesia- name: RABBITMQ_USE_LONGNAMEvalue: "true"- name: RABBITMQ_NODENAMEvalue: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.localports:- name: httpcontainerPort: 15672 - name: amqpcontainerPort: 5672 volumeMounts:- name: rbmq-datamountPath: /home/rabbitmq - name: rabbitmq-config-volumemountPath: /etc/rabbitmq/ restartPolicy: Always terminationGracePeriodSeconds: 30volumes:- name: rbmq-datahostPath:path: /data/rabbitmqtype: ""- name: rabbitmq-config-volumeconfigMap:name: autotest-rabbitmq-configtolerations:- key: node.kubernetes.io/not-readyoperator: Existseffect: NoExecutetolerationSeconds: 3600- key: node.kubernetes.io/unreachableoperator: Existseffect: NoExecutetolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:name: autotest-rabbitmq-bnamespace: zx-applabels:appname: pcauto-zxapp: autotest-rabbitmq-b
spec:ports:- port: 5672clusterIP: Noneselector:app: autotest-rabbitmq-b

3.autotest-c.yaml 节点c的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:name: autotest-rabbitmq-c namespace: zx-app labels:appname: pcauto-zxapp: autotest-rabbitmq-cappgroup: autotest-rabbitmq
spec:serviceName: "autotest-rabbitmq-c"replicas: 1selector:matchLabels:app: autotest-rabbitmq-cappgroup: autotest-rabbitmqtemplate:metadata:labels:app: autotest-rabbitmq-cappgroup: autotest-rabbitmqspec:affinity:nodeAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 80preference:matchExpressions:- key: storage-selectoroperator: Invalues:- node-c-master- weight: 20preference:matchExpressions:- key: storage-selectoroperator: Invalues:- node-c-slavecontainers:- name: rabbitmq-serverimage: xxx-vpc.cn-shenzhen.cr.aliyuncs.com/public/rabbitmq:3.12.14-management imagePullPolicy: IfNotPresentcommand: ["/bin/sh","-c"]args: - mkdir -p /home/rabbitmq/${POD_NAME};chown rabbitmq.rabbitmq -R /home/rabbitmq/${POD_NAME};rabbitmq-serverenv:- name: RABBITMQ_ERLANG_COOKIEvalue: "YZSDHWMFSMKEMBDHSGGZ"- name: K8S_SERVICE_NAMEvalue: autotest-rabbitmq-c- name: POD_IPvalueFrom:fieldRef:fieldPath: status.podIP- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: RABBITMQ_MNESIA_BASEvalue: /home/rabbitmq/$(POD_NAME)/mnesia- name: RABBITMQ_USE_LONGNAMEvalue: "true"- name: RABBITMQ_NODENAMEvalue: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.localports:- name: httpcontainerPort: 15672 - name: amqpcontainerPort: 5672 volumeMounts:- name: rbmq-datamountPath: /home/rabbitmq - name: rabbitmq-config-volumemountPath: /etc/rabbitmq/ restartPolicy: Always terminationGracePeriodSeconds: 30volumes:- name: rbmq-datahostPath:path: /data/rabbitmqtype: ""- name: rabbitmq-config-volumeconfigMap:name: autotest-rabbitmq-configtolerations:- key: node.kubernetes.io/not-readyoperator: Existseffect: NoExecutetolerationSeconds: 3600- key: node.kubernetes.io/unreachableoperator: Existseffect: NoExecutetolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:name: autotest-rabbitmq-cnamespace: zx-applabels:appname: pcauto-zxapp: autotest-rabbitmq-c
spec:ports:- port: 5672clusterIP: Noneselector:app: autotest-rabbitmq-c

4.4.创建service配置

给amqp端口5672和管理端口15672配置一个service的代理服务，注意其中的selector是指向到appgroup,这是将三个statefulset定义了一个appgroup的标签。

apiVersion: v1
kind: Service
metadata:name: autotest-rabbitmq-managenamespace: zx-applabels:app: autotest-rabbitmq-manageappname: pcauto-zx
spec:ports:- port: 5672name: amqp- port: 15672name: httpselector:appgroup: autotest-rabbitmqsessionAffinity: ClientIPtype: LoadBalancer 

五.安装完后的配置

将rabbitmq的集群设置成3节点的镜像集群，在4.4步骤操作完以后就可以通过loadbalancer的IP进行登录控制台了：http://ip:15672,用configmap中设置的 default_user和default_pass来登录。

镜像模式设置：

设置后：

镜像模式可以通过命令设置：
设置demo的vhost的镜像执行命令：
rabbitmqctl set_policy -p demo ha-all "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'
设置默认vhost /的镜像模式：
rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

六.安装说明

• 需要先在k8s的节点创建/data/rabbitmq的路径来存放数据。
• 使用配置前将配置中的autotest替换成自己需要的命名,配置的命名空间为：zx-app，修改成自己的命名空间，appname为pcauto-zx，修改成自己的应用名称，该标签也可以删除。
• 配置中的镜像地址采用了私有的镜像地址，镜像是dockerhub上下载的rabbitmq:3.12.14-management 镜像到私有镜像仓库的，如果能直通外网的k8s可以直接用rabbitmq:3.12.14-management 。否则配置一个可以通的镜像地址。
• 默认的用户名和密码是admin。
• 节点标签的命名规则若有更改需要在每个statefulset中的affinity: 中的标签名称进行修改。