JVM (Micrometer)监控SpringBoot（AWS EKS版）

问题

怎样使用JVM (Micrometer)面板，监控Spring？这里不涉及Prometheus和Grafana，重点介绍与Micrometer与Springboot，k8s怎样集成。

pom.xml

引入依赖，如下：

<properties><micrometer.version>1.12.5</micrometer.version><micrometer-jvm-extras.version>0.2.2</micrometer-jvm-extras.version>
</properties>
...
<dependency><groupId>io.micrometer</groupId><artifactId>micrometer-registry-prometheus</artifactId><version>${micrometer.version}</version>
</dependency><dependency><groupId>io.github.mweirauch</groupId><artifactId>micrometer-jvm-extras</artifactId><version>${micrometer-jvm-extras.version}</version>
</dependency>

JVMConfig.java

启用micrometer-jvm-extras库监控内存指标：

import io.github.mweirauch.micrometer.jvm.extras.ProcessMemoryMetrics;
import io.micrometer.core.instrument.binder.MeterBinder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;@Configuration
public class JVMConfig {@Beanpublic MeterBinder processMemoryMetrics() {return new ProcessMemoryMetrics();}
}

application.yml

management:metrics:tags:application: ${spring.profiles.active}_${spring.application.name}endpoints:web:base-path: /actuatorexposure:include: prometheus,health
server:tomcat:mbeanregistry:enabled: true

这里主要是三件事情：

标记application名称；
限制只能查询prometheus,health两个actuator查询；
启用Tomcat指标。

prometheus.yml

配置prometheus抓取程序，如下：

global:scrape_interval: 30s
scrape_configs:
...# JVM (Micrometer)- job_name: 'kubernetes-service-endpoints'kubernetes_sd_configs:- role: endpointsrelabel_configs:- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]action: keepregex: true- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]action: replacetarget_label: __scheme__regex: (https?)- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]action: replacetarget_label: __metrics_path__regex: (.+)- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]action: replacetarget_label: __address__regex: (.+)(?::\d+);(\d+)replacement: $1:$2- action: labelmapregex: __meta_kubernetes_service_label_(.+)- source_labels: [__meta_kubernetes_namespace]action: replacetarget_label: kubernetes_namespace- source_labels: [__meta_kubernetes_service_name]action: replacetarget_label: application

这里主要就是prometheus抓取程序通过k8s自动发现endpoints，找到k8s中的Spring服务暴露的prometheus指标。需要注意__meta_kubernetes_service_annotation_开头的注解，需要与k8s的service中注解保持一致。
在实践中比较完整的prometheus抓取配置内容（上面之包含了对spring中prometheus指标抓取），参考如下：

global:scrape_interval: 30sexternal_labels:clusterArn: arn:aws:eks:us-east-1:xxx:cluster/uatcluster: uat
scrape_configs:# pod metrics- job_name: pod_exporterkubernetes_sd_configs:- role: pod# container metrics- job_name: cadvisorscheme: httpsauthorization:credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenkubernetes_sd_configs:- role: noderelabel_configs:- action: labelmapregex: __meta_kubernetes_node_label_(.+)- replacement: kubernetes.default.svc:443target_label: __address__- source_labels: [__meta_kubernetes_node_name]regex: (.+)target_label: __metrics_path__replacement: /api/v1/nodes/$1/proxy/metrics/cadvisormetric_relabel_configs:- source_labels: [instance]separator: ;regex: (.+)target_label: nodereplacement: $1action: replace# apiserver metrics- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenjob_name: kubernetes-apiserverskubernetes_sd_configs:- role: endpointsrelabel_configs:- action: keepregex: default;kubernetes;httpssource_labels:- __meta_kubernetes_namespace- __meta_kubernetes_service_name- __meta_kubernetes_endpoint_port_namescheme: https# kube proxy metrics- job_name: kube-proxyhonor_labels: truekubernetes_sd_configs:- role: podrelabel_configs:- action: keepsource_labels:- __meta_kubernetes_namespace- __meta_kubernetes_pod_nameseparator: '/'regex: 'kube-system/kube-proxy.+'- source_labels:- __address__action: replacetarget_label: __address__regex: (.+?)(\\:\\d+)?replacement: $1:10249# kube-state-metrics- job_name: kube-state-metricshonor_timestamps: truescrape_interval: 1mscrape_timeout: 1mmetrics_path: /metricsscheme: httpstatic_configs:- targets:- kube-state-metrics.kube-system.svc.cluster.local:8080# node-exporter- job_name: 'node-exporter'kubernetes_sd_configs:- role: noderelabel_configs:- action: replacesource_labels: [__address__]regex: '(.*):10250'replacement: '${1}:9100'target_label: __address__# JVM (Micrometer)- job_name: 'kubernetes-service-endpoints'kubernetes_sd_configs:- role: endpointsrelabel_configs:- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]action: keepregex: true- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]action: replacetarget_label: __scheme__regex: (https?)- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]action: replacetarget_label: __metrics_path__regex: (.+)- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]action: replacetarget_label: __address__regex: (.+)(?::\d+);(\d+)replacement: $1:$2- action: labelmapregex: __meta_kubernetes_service_label_(.+)- source_labels: [__meta_kubernetes_namespace]action: replacetarget_label: kubernetes_namespace- source_labels: [__meta_kubernetes_service_name]action: replacetarget_label: application

k8s.yaml

k8s的service部署配置，如下：

apiVersion: v1
kind: Service
metadata:labels:app: {{ .Values.services.xxxxx.name }}name: {{ .Values.services.xxxxx.name }}namespace: {{ .Release.Namespace }}annotations:alb.ingress.kubernetes.io/healthcheck-path: {{ .Values.services.xxxxx.health.path }}alb.ingress.kubernetes.io/healthcheck-port: '{{ .Values.services.xxxxx.health.port }}'prometheus.io/path: {{ .Values.services.xxxxx.prometheus.path }}prometheus.io/port: '{{ .Values.services.xxxxx.prometheus.port }}'prometheus.io/scrape: "true"
spec:ports:- name: httpport: {{ .Values.services.xxxxx.port }}targetPort: {{ .Values.services.xxxxx.port }}selector:app: {{ .Values.services.xxxxx.name }}type: ClusterIP

注意，这里的注解需要与prometheus抓取程序保持一致，如下图：
prometheus抓取成与k8s对应
这里还有一个就是k8s里面的注释不规则命名，如点，斜杆符号等，在prometheus抓取程序这边都被转化成下划线符号，反正prometheus抓取程序遇到不规则的k8s注释命名符号，都被转成下划线。
三个关键注释：

prometheus.io/path:设置springboot的暴露的prometheus指标路径，即spring actuator路径；
prometheus.io/port:设置springboot的暴露的prometheus指标端口，即spring actuator端口；
prometheus.io/scrape:是否启用prometheus抓取程序。

JVM (Micrometer)面板效果

grafana效果

总结

到这里就完成了对Spring项目添加prometheus指标过程，主要就是添加micrometer和micrometer-jvm-extras依赖，启用micrometer-jvm-extras的内存指标。放开spring actuator prometheus相关端点，启用tomcat指标，配置prometheus抓取程序，设置K8S Service的prometheus抓取程序注释配置。重新发布部署，在grafana查看效果。