K8S 部署peometheus + grafana 监控

安装说明

如果有下载不下来的docker镜像可以私信我免费下载。
系统版本为 Centos7.9
内核版本为 6.3.5-1.el7
K8S版本为 v1.26.14
动态存储：部署文档
GitHub地址

下载yaml 文件

## 因为我的K8S 版本比较新，我下载的是当前的最新版本，你的要是老版本K8集群最好看一下那个版本是支持你的集群的。
git clone -b release-0.13 https://github.com/prometheus-operator/kube-prometheus.git

安装operator

cd ./kube-prometheus-release-0.13/manifests/setup
kubectl create -f ./cd /root/kube-prometheus-release-0.13/manifests
kubectl create -f ./

注意事项

默认的 yaml 文件里面有网络隔离配置，配置文件名字如下

[root@master01 manifests]#ls *networkPolicy.yaml
alertmanager-networkPolicy.yaml      grafana-networkPolicy.yaml           nodeExporter-networkPolicy.yaml       prometheus-networkPolicy.yaml
blackboxExporter-networkPolicy.yaml  kubeStateMetrics-networkPolicy.yaml  prometheusAdapter-networkPolicy.yaml  prometheusOperator-networkPolicy.yaml

正常按照默认的部署是不能访问这个监控的，需要修改配置，
举例：以 grafana-networkPolicy.yaml 配置文件为例（ingress-nginx 方式暴露服务，所以添加ingress-nginx 命名空间的pod允许访问Grafana即可。）

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:labels:app.kubernetes.io/component: grafanaapp.kubernetes.io/name: grafanaapp.kubernetes.io/part-of: kube-prometheusapp.kubernetes.io/version: 11.1.0name: grafananamespace: monitoring
spec:egress:- {}ingress:- from:- podSelector:matchLabels:app.kubernetes.io/name: prometheus- namespaceSelector: ## 添加允许访问策略matchLabels:name: ingress-nginxports:- port: 3000protocol: TCPpodSelector:matchLabels:app.kubernetes.io/component: grafanaapp.kubernetes.io/name: grafanaapp.kubernetes.io/part-of: kube-prometheuspolicyTypes:- Egress- Ingress

保存后重新部署，其他的网络隔离也是这样修改，此处不再一一举例。

kubectl replace -f grafana-networkPolicy.yaml

或者嫌麻烦可以直接删除所有 networkPolicy

## 谨慎使用，当前命令会删除指定命名空间中所有networkPolicy 配置。
kubectl delete netpol -n monitoring --all

修改 prometheus-prometheus.yaml 文件

默认这个是没有持久化存储的并且prometheus 监控数据只能保存24小时，下面这个是我修改过的，修改监控数据保存时间，添加动态存储。

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:labels:app.kubernetes.io/component: prometheusapp.kubernetes.io/instance: k8sapp.kubernetes.io/name: prometheusapp.kubernetes.io/part-of: kube-prometheusapp.kubernetes.io/version: 2.46.0name: k8snamespace: monitoring
spec:alerting:alertmanagers:- apiVersion: v2name: alertmanager-mainnamespace: monitoringport: webenableFeatures: []externalLabels: {}image: quay.io/prometheus/prometheus:v2.46.0nodeSelector:kubernetes.io/os: linuxpodMetadata:labels:app.kubernetes.io/component: prometheusapp.kubernetes.io/instance: k8sapp.kubernetes.io/name: prometheusapp.kubernetes.io/part-of: kube-prometheusapp.kubernetes.io/version: 2.46.0podMonitorNamespaceSelector: {}podMonitorSelector: {}probeNamespaceSelector: {}probeSelector: {}replicas: 1resources:requests:memory: 1000MiruleNamespaceSelector: {}ruleSelector: {}securityContext:fsGroup: 2000runAsNonRoot: truerunAsUser: 1000serviceAccountName: prometheus-k8sserviceMonitorNamespaceSelector: {}serviceMonitorSelector: {}## 在这里添加持久化存储，我这里用的是动态存储。storage:volumeClaimTemplate:metadata:name: prometheus-k8s-dbspec:accessModes: [ "ReadWriteOnce" ]storageClassName: rook-ceph-rbdresources:requests:storage: 50Gi
## 修改默认保存的监控数据为180天retention: 180d   version: 2.46.0

修改好以后重新部署

添加 Prometheus Ingress访问

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: prometheus-ingressnamespace: monitoringlabels:app.kubernetes.io/component: prometheusapp.kubernetes.io/instance: k8sapp.kubernetes.io/name: prometheusapp.kubernetes.io/part-of: kube-prometheusapp.kubernetes.io/version: 2.46.0operator.prometheus.io/mode: server
spec:ingressClassName: nginxrules:- host: prometheus.demo.cnhttp:paths:- backend:service:name: prometheus-k8sport:name: webpath: /pathType: Prefixtls:- hosts:- prometheus.demo.cnsecretName: prometheus.demo.cn

配置好host 直接用域名访问

查看prometheus Targets

没问题的话就继续。
在这里插入图片描述

metrics-server 服务异常

[root@master01 data]# kubectl top node
error: Metrics API not available

解决办法

部署 Grafana 监控面板

修改 Grafana yaml 文件添加存储，默认的配置是没有存储的。

## 创建PVC存储给 Grafana 使用
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: grafana-data-pvcnamespace: monitoring
spec:accessModes:- ReadWriteOnceresources:requests:storage: 5GistorageClassName: rook-ceph-rbd
---
apiVersion: apps/v1
kind: Deployment
metadata:labels:app.kubernetes.io/component: grafanaapp.kubernetes.io/name: grafanaapp.kubernetes.io/part-of: kube-prometheusapp.kubernetes.io/version: 9.5.3name: grafananamespace: monitoring
spec:replicas: 1selector:matchLabels:app.kubernetes.io/component: grafanaapp.kubernetes.io/name: grafanaapp.kubernetes.io/part-of: kube-prometheustemplate:metadata:annotations:checksum/grafana-config: 5c598ba58d9b65011bdbb3864138399achecksum/grafana-dashboardproviders: c9c1743868aa1c3dab60d2c402e2dcf0checksum/grafana-datasources: 5ef0e6acaa5b4e8603740fbad440717dlabels:app.kubernetes.io/component: grafanaapp.kubernetes.io/name: grafanaapp.kubernetes.io/part-of: kube-prometheusapp.kubernetes.io/version: 9.5.3spec:automountServiceAccountToken: falsecontainers:- env: []image: grafana/grafana:9.5.3name: grafanaports:- containerPort: 3000name: httpreadinessProbe:httpGet:path: /api/healthport: httpresources:limits:cpu: 200mmemory: 200Mirequests:cpu: 100mmemory: 100MisecurityContext:allowPrivilegeEscalation: falsecapabilities:drop:- ALLreadOnlyRootFilesystem: trueseccompProfile:type: RuntimeDefaultvolumeMounts:- mountPath: /var/lib/grafananame: grafana-storagereadOnly: false- mountPath: /etc/grafana/provisioning/datasourcesname: grafana-datasourcesreadOnly: false- mountPath: /etc/grafana/provisioning/dashboardsname: grafana-dashboardsreadOnly: false- mountPath: /tmpname: tmp-pluginsreadOnly: false- mountPath: /grafana-dashboard-definitions/0/alertmanager-overviewname: grafana-dashboard-alertmanager-overviewreadOnly: false- mountPath: /grafana-dashboard-definitions/0/apiservername: grafana-dashboard-apiserverreadOnly: false- mountPath: /grafana-dashboard-definitions/0/cluster-totalname: grafana-dashboard-cluster-totalreadOnly: false- mountPath: /grafana-dashboard-definitions/0/controller-managername: grafana-dashboard-controller-managerreadOnly: false- mountPath: /grafana-dashboard-definitions/0/grafana-overviewname: grafana-dashboard-grafana-overviewreadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-clustername: grafana-dashboard-k8s-resources-clusterreadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-multiclustername: grafana-dashboard-k8s-resources-multiclusterreadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-namespacename: grafana-dashboard-k8s-resources-namespacereadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-nodename: grafana-dashboard-k8s-resources-nodereadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-podname: grafana-dashboard-k8s-resources-podreadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-workloadname: grafana-dashboard-k8s-resources-workloadreadOnly: false- mountPath: /grafana-dashboard-definitions/0/k8s-resources-workloads-namespacename: grafana-dashboard-k8s-resources-workloads-namespacereadOnly: false- mountPath: /grafana-dashboard-definitions/0/kubeletname: grafana-dashboard-kubeletreadOnly: false- mountPath: /grafana-dashboard-definitions/0/namespace-by-podname: grafana-dashboard-namespace-by-podreadOnly: false- mountPath: /grafana-dashboard-definitions/0/namespace-by-workloadname: grafana-dashboard-namespace-by-workloadreadOnly: false- mountPath: /grafana-dashboard-definitions/0/node-cluster-rsrc-usename: grafana-dashboard-node-cluster-rsrc-usereadOnly: false- mountPath: /grafana-dashboard-definitions/0/node-rsrc-usename: grafana-dashboard-node-rsrc-usereadOnly: false- mountPath: /grafana-dashboard-definitions/0/nodes-darwinname: grafana-dashboard-nodes-darwinreadOnly: false- mountPath: /grafana-dashboard-definitions/0/nodesname: grafana-dashboard-nodesreadOnly: false- mountPath: /grafana-dashboard-definitions/0/persistentvolumesusagename: grafana-dashboard-persistentvolumesusagereadOnly: false- mountPath: /grafana-dashboard-definitions/0/pod-totalname: grafana-dashboard-pod-totalreadOnly: false- mountPath: /grafana-dashboard-definitions/0/prometheus-remote-writename: grafana-dashboard-prometheus-remote-writereadOnly: false- mountPath: /grafana-dashboard-definitions/0/prometheusname: grafana-dashboard-prometheusreadOnly: false- mountPath: /grafana-dashboard-definitions/0/proxyname: grafana-dashboard-proxyreadOnly: false- mountPath: /grafana-dashboard-definitions/0/schedulername: grafana-dashboard-schedulerreadOnly: false- mountPath: /grafana-dashboard-definitions/0/workload-totalname: grafana-dashboard-workload-totalreadOnly: false- mountPath: /etc/grafananame: grafana-configreadOnly: falsenodeSelector:kubernetes.io/os: linuxsecurityContext:fsGroup: 65534runAsNonRoot: truerunAsUser: 65534serviceAccountName: grafanavolumes:## 添加PVC存储- name: grafana-storagepersistentVolumeClaim:claimName: grafana-data-pvc- name: grafana-datasourcessecret:secretName: grafana-datasources- configMap:name: grafana-dashboardsname: grafana-dashboards- emptyDir:medium: Memoryname: tmp-plugins- configMap:name: grafana-dashboard-alertmanager-overviewname: grafana-dashboard-alertmanager-overview- configMap:name: grafana-dashboard-apiservername: grafana-dashboard-apiserver- configMap:name: grafana-dashboard-cluster-totalname: grafana-dashboard-cluster-total- configMap:name: grafana-dashboard-controller-managername: grafana-dashboard-controller-manager- configMap:name: grafana-dashboard-grafana-overviewname: grafana-dashboard-grafana-overview- configMap:name: grafana-dashboard-k8s-resources-clustername: grafana-dashboard-k8s-resources-cluster- configMap:name: grafana-dashboard-k8s-resources-multiclustername: grafana-dashboard-k8s-resources-multicluster- configMap:name: grafana-dashboard-k8s-resources-namespacename: grafana-dashboard-k8s-resources-namespace- configMap:name: grafana-dashboard-k8s-resources-nodename: grafana-dashboard-k8s-resources-node- configMap:name: grafana-dashboard-k8s-resources-podname: grafana-dashboard-k8s-resources-pod- configMap:name: grafana-dashboard-k8s-resources-workloadname: grafana-dashboard-k8s-resources-workload- configMap:name: grafana-dashboard-k8s-resources-workloads-namespacename: grafana-dashboard-k8s-resources-workloads-namespace- configMap:name: grafana-dashboard-kubeletname: grafana-dashboard-kubelet- configMap:name: grafana-dashboard-namespace-by-podname: grafana-dashboard-namespace-by-pod- configMap:name: grafana-dashboard-namespace-by-workloadname: grafana-dashboard-namespace-by-workload- configMap:name: grafana-dashboard-node-cluster-rsrc-usename: grafana-dashboard-node-cluster-rsrc-use- configMap:name: grafana-dashboard-node-rsrc-usename: grafana-dashboard-node-rsrc-use- configMap:name: grafana-dashboard-nodes-darwinname: grafana-dashboard-nodes-darwin- configMap:name: grafana-dashboard-nodesname: grafana-dashboard-nodes- configMap:name: grafana-dashboard-persistentvolumesusagename: grafana-dashboard-persistentvolumesusage- configMap:name: grafana-dashboard-pod-totalname: grafana-dashboard-pod-total- configMap:name: grafana-dashboard-prometheus-remote-writename: grafana-dashboard-prometheus-remote-write- configMap:name: grafana-dashboard-prometheusname: grafana-dashboard-prometheus- configMap:name: grafana-dashboard-proxyname: grafana-dashboard-proxy- configMap:name: grafana-dashboard-schedulername: grafana-dashboard-scheduler- configMap:name: grafana-dashboard-workload-totalname: grafana-dashboard-workload-total- name: grafana-configsecret:secretName: grafana-config

修改好以后重新部署

添加 Grafana Ingress 访问

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: grafana-ingressnamespace: monitoringlabels:app.kubernetes.io/component: grafanaapp.kubernetes.io/name: grafanaapp.kubernetes.io/part-of: kube-prometheusapp.kubernetes.io/version: 9.5.3
spec:ingressClassName: nginxrules:- host: grafana.demo.cnhttp:paths:- backend:service:name: grafanaport:number: 3000path: /pathType: Prefixtls:- hosts:- grafana.demo.cnsecretName: grafana.demo.cn