K8S集群监控方案之Prometheus+kube-state-metrics+Grafana

序言 | Prometheus 中文文档

方案简单架构图

一、部署kube-state-metrics

1、部署文件下载

地址

kube-state-metrics/examples/standard at main · kubernetes/kube-state-metrics · GitHub

2、修改下载的文件

2.1、修改镜像

原镜像可能下载不了，这里修改deployment.yaml镜像：bitnami/kube-state-metrics:latest

2.2、修改service

这里修改为NodePort类型，可按自己需求更改

3、本案例下载更改好后的部署文件

cluster-role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:app.kubernetes.io/component: exporterapp.kubernetes.io/name: kube-state-metricsapp.kubernetes.io/version: 2.12.0name: kube-state-metrics
rules:
- apiGroups:- ""resources:- configmaps- secrets- nodes- pods- services- serviceaccounts- resourcequotas- replicationcontrollers- limitranges- persistentvolumeclaims- persistentvolumes- namespaces- endpointsverbs:- list- watch
- apiGroups:- appsresources:- statefulsets- daemonsets- deployments- replicasetsverbs:- list- watch
- apiGroups:- batchresources:- cronjobs- jobsverbs:- list- watch
- apiGroups:- autoscalingresources:- horizontalpodautoscalersverbs:- list- watch
- apiGroups:- authentication.k8s.ioresources:- tokenreviewsverbs:- create
- apiGroups:- authorization.k8s.ioresources:- subjectaccessreviewsverbs:- create
- apiGroups:- policyresources:- poddisruptionbudgetsverbs:- list- watch
- apiGroups:- certificates.k8s.ioresources:- certificatesigningrequestsverbs:- list- watch
- apiGroups:- discovery.k8s.ioresources:- endpointslicesverbs:- list- watch
- apiGroups:- storage.k8s.ioresources:- storageclasses- volumeattachmentsverbs:- list- watch
- apiGroups:- admissionregistration.k8s.ioresources:- mutatingwebhookconfigurations- validatingwebhookconfigurationsverbs:- list- watch
- apiGroups:- networking.k8s.ioresources:- networkpolicies- ingressclasses- ingressesverbs:- list- watch
- apiGroups:- coordination.k8s.ioresources:- leasesverbs:- list- watch
- apiGroups:- rbac.authorization.k8s.ioresources:- clusterrolebindings- clusterroles- rolebindings- rolesverbs:- list- watch

cluster-role-binding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:app.kubernetes.io/component: exporterapp.kubernetes.io/name: kube-state-metricsapp.kubernetes.io/version: 2.12.0name: kube-state-metrics
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: kube-state-metrics
subjects:
- kind: ServiceAccountname: kube-state-metricsnamespace: kube-system

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:labels:app.kubernetes.io/component: exporterapp.kubernetes.io/name: kube-state-metricsapp.kubernetes.io/version: 2.12.0name: kube-state-metricsnamespace: kube-system
spec:replicas: 1selector:matchLabels:app.kubernetes.io/name: kube-state-metricstemplate:metadata:labels:app.kubernetes.io/component: exporterapp.kubernetes.io/name: kube-state-metricsapp.kubernetes.io/version: 2.12.0spec:automountServiceAccountToken: truecontainers:- image: bitnami/kube-state-metrics:latestlivenessProbe:httpGet:path: /healthzport: 8080initialDelaySeconds: 5timeoutSeconds: 5name: kube-state-metricsports:- containerPort: 8080name: http-metrics- containerPort: 8081name: telemetryreadinessProbe:httpGet:path: /port: 8081initialDelaySeconds: 5timeoutSeconds: 5securityContext:allowPrivilegeEscalation: falsecapabilities:drop:- ALLreadOnlyRootFilesystem: truerunAsNonRoot: truerunAsUser: 65534seccompProfile:type: RuntimeDefaultnodeSelector:kubernetes.io/os: linuxserviceAccountName: kube-state-metrics

service-account.yaml

apiVersion: v1
automountServiceAccountToken: false
kind: ServiceAccount
metadata:labels:app.kubernetes.io/component: exporterapp.kubernetes.io/name: kube-state-metricsapp.kubernetes.io/version: 2.12.0name: kube-state-metricsnamespace: kube-system

service.yaml

apiVersion: v1
kind: Service
metadata:labels:app.kubernetes.io/component: exporterapp.kubernetes.io/name: kube-state-metricsapp.kubernetes.io/version: 2.12.0name: kube-state-metricsnamespace: kube-system
spec:type: NodePortports:- name: http-metricsport: 8080targetPort: 8080nodePort: 31666- name: telemetryport: 8081targetPort: 8081selector:app.kubernetes.io/name: kube-state-metrics

4、部署kube-state-metrics

[K8S@k8s-master kube-state-metrics]$ ll
总用量 20
-rw-rw-r-- 1 K8S K8S 418 5月 17 16:11 cluster-role-binding.yaml
-rw-rw-r-- 1 K8S K8S 1950 5月 17 16:11 cluster-role.yaml
-rw-rw-r-- 1 K8S K8S 1471 5月 17 16:11 deployment.yaml
-rw-rw-r-- 1 K8S K8S 270 5月 17 16:11 service-account.yaml
-rw-rw-r-- 1 K8S K8S 453 5月 17 16:11 service.yaml
[K8S@k8s-master kube-state-metrics]$ pwd
/home/K8S/k8s-project/monitor/kube-state-metrics
[K8S@k8s-master kube-state-metrics]$ kubectl apply -f ./

查看部署结果：

[K8S@k8s-master kube-state-metrics]$ kubectl get all -n kube-system

Running状态表示部署成功

访问Service暴露的31666端口，能成功返回

二、部署Prometheus

1、创建名称空间

[K8S@k8s-master prometheus]$ kubectl create ns monitoring

2、编写配置文件 prometheus.yml

apiVersion: v1
kind: ServiceAccount
metadata:name: prometheusnamespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: prometheus
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: cluster-admin
subjects:
- kind: ServiceAccountname: prometheusnamespace: monitoring
---
apiVersion: v1
kind: ConfigMap
metadata:name: prometheus-confignamespace: monitoring
data:prometheus.yml: |global:scrape_interval: 15s      # 数据采集时间间隔scrape_timeout: 10s       # 数据采集超时时间evaluation_interval: 1mscrape_configs:- job_name: "prometheus"static_configs:- targets: ["localhost:9090"]- job_name: "k8s-info"static_configs:- targets: ["172.19.3.240:31666"]  #配置kube-state-metrice的数据源地址---
apiVersion: apps/v1
kind: Deployment
metadata:name: prometheusnamespace: monitoring
spec:replicas: 1selector:matchLabels:app: prometheustemplate:metadata:labels:app: prometheusspec:serviceAccountName: prometheuscontainers:- name: prometheusimage: prom/prometheus:v2.31.1args:- --config.file=/etc/prometheus/prometheus.yml- --storage.tsdb.path=/prometheusports:- containerPort: 9090volumeMounts:- name: config-volumemountPath: /etc/prometheus- name: storage-volumemountPath: /prometheusvolumes:- name: config-volumeconfigMap:name: prometheus-configitems:- key: prometheus.ymlpath: prometheus.yml- name: storage-volumeemptyDir: {}
---
apiVersion: v1
kind: Service
metadata:name: prometheus-servicenamespace: monitoring
spec:type: NodePortports:- name: httpport: 9090targetPort: 9090nodePort: 30007selector:app: prometheus

创建资源

[K8S@k8s-master prometheus]$ kubectl apply -f prometheus.yml

查看状态为Running，表示正常启动

访问Service暴露的30007端口 http://172.19.3.240:30007/targets

成功读取到kube-state-metrics提供的数据

prometheus部署成功

三、部署Grafana

官网安装参考

Deploy Grafana on Kubernetes | Grafana documentation

1、编写grafana.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: grafana-pvcnamespace: monitoring
spec:accessModes:- ReadWriteOnceresources:requests:storage: 1GistorageClassName: managed-nfs-devops-storage
---
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: grafananame: grafananamespace: monitoring
spec:selector:matchLabels:app: grafanatemplate:metadata:labels:app: grafanaspec:securityContext:fsGroup: 472supplementalGroups:- 0containers:- name: grafanaimage: grafana/grafana:latestimagePullPolicy: IfNotPresentports:- containerPort: 3000name: http-grafanaprotocol: TCPreadinessProbe:failureThreshold: 3httpGet:path: /robots.txtport: 3000scheme: HTTPinitialDelaySeconds: 10periodSeconds: 30successThreshold: 1timeoutSeconds: 2livenessProbe:failureThreshold: 3initialDelaySeconds: 30periodSeconds: 10successThreshold: 1tcpSocket:port: 3000timeoutSeconds: 1resources:requests:cpu: 250mmemory: 750MivolumeMounts:- mountPath: /var/lib/grafananame: grafana-pvvolumes:- name: grafana-pvpersistentVolumeClaim:claimName: grafana-pvc
---
apiVersion: v1
kind: Service
metadata:name: grafananamespace: monitoring
spec:ports:- port: 3000protocol: TCPtargetPort: http-grafananodePort: 30008selector:app: grafanasessionAffinity: Nonetype: NodePort

2、执行命令创建资源

[K8S@k8s-master monitor]$ kubectl apply -f grafana.yaml
persistentvolumeclaim/grafana-pvc created
deployment.apps/grafana created
service/grafana created

查看资源，Running，安装成功