Kube-Prometheus 监控Istio

推荐 Istio 多集群监控使用 Prometheus，其主要原因是基于 Prometheus 的分层联邦（Hierarchical Federation）。

通过 Istio 部署到每个集群中的 Prometheus 实例作为初始收集器，然后将数据聚合到网格层次的 Prometheus 实例上。网格层次的 Prometheus 既可以部署在网格之外（外部），也可以部署在网格内的集群中。

使用 Istio 以及 Prometheus 进行生产规模的监控时推荐的方式是使用分层联邦并且结合一组记录规则。

尽管安装 Istio 不会默认部署 Prometheus，入门指导中 Option 1: Quick Start 的部署按照 Prometheus 集成指导安装了 Prometheus。此 Prometheus 部署刻意地配置了很短的保留窗口（6 小时）。此快速入门 Prometheus 部署同时也配置为从网格上运行的每一个 Envoy 代理上收集指标，同时通过一组有关它们的源的标签（instance、pod 和 namespace）来扩充指标
在这里插入图片描述

使用负载级别的聚合指标进行联邦

为了建立 Prometheus 联邦，请修改您的 Prometheus 生产部署配置来抓取 Istio Prometheus 联邦终端的指标数据。

将以下的 Job 添加到配置中：

- job_name: 'istio-prometheus'honor_labels: truemetrics_path: '/federate'kubernetes_sd_configs:- role: podnamespaces:names: ['istio-system']metric_relabel_configs:- source_labels: [__name__]regex: 'workload:(.*)'target_label: __name__action: replaceparams:'match[]':- '{__name__=~"workload:(.*)"}'- '{__name__=~"pilot(.*)"}'

如果您使用的是 Prometheus Operator，请使用以下的配置：

我这边就是使用这种方式

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:name: istio-federationlabels:app.kubernetes.io/name: istio-prometheus
spec:namespaceSelector:matchNames:- istio-systemselector:matchLabels:app: prometheusendpoints:- interval: 30sscrapeTimeout: 30sparams:'match[]':- '{__name__=~"workload:(.*)"}'- '{__name__=~"pilot(.*)"}'path: /federatetargetPort: 9090honorLabels: truemetricRelabelings:- sourceLabels: ["__name__"]regex: 'workload:(.*)'targetLabel: "__name__"action: replace

kubectl apply -f istio-federation.yaml -n monitoring[root@bt ~]# kubectl get servicemonitors.monitoring.coreos.com -n monitoring  istio-federation 
NAME               AGE
istio-federation   44d

联邦配置的关键是首先匹配通过 Istio 部署的 Prometheus 中收集 Istio 标准指标的 Job。并且将收集到的指标重命名，方法为去除负载等级记录规则命名前缀 (workload:)。这使得现有的仪表盘以及引用能够无缝地针对生产用 Prometheus 继续工作（并且不在指向 Istio 实例）。

您可以在设置联邦时包含额外的指标（例如 envoy、go 等）。

控制面指标也被生产用 Prometheus 收集并联邦。

配置文件监控kubernetes-istio-pods

Prometheus 的抓取作业配置文件，怎么实现 Prometheus 如何抓取和处理与 Kubernetes Istio pods 相关的指标数据？

1、首先Istio内部集成安装Prometheus

2、我们可以去istio查看人家的配置怎么写

3、根据人家的配置，修改一下，集成到Prometheus上

global:evaluation_interval: 1mscrape_interval: 15sscrape_timeout: 10s
rule_files:
- /etc/config/recording_rules.yml
- /etc/config/alerting_rules.yml
- /etc/config/rules
- /etc/config/alerts
scrape_configs:
- job_name: prometheusstatic_configs:- targets:- localhost:9090
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenjob_name: kubernetes-apiserverskubernetes_sd_configs:- role: endpointsrelabel_configs:- action: keepregex: default;kubernetes;httpssource_labels:- __meta_kubernetes_namespace- __meta_kubernetes_service_name- __meta_kubernetes_endpoint_port_namescheme: httpstls_config:ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crtinsecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenjob_name: kubernetes-nodeskubernetes_sd_configs:- role: noderelabel_configs:- action: labelmapregex: __meta_kubernetes_node_label_(.+)- replacement: kubernetes.default.svc:443target_label: __address__- regex: (.+)replacement: /api/v1/nodes/$1/proxy/metricssource_labels:- __meta_kubernetes_node_nametarget_label: __metrics_path__scheme: httpstls_config:ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crtinsecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenjob_name: kubernetes-nodes-cadvisorkubernetes_sd_configs:- role: noderelabel_configs:- action: labelmapregex: __meta_kubernetes_node_label_(.+)- replacement: kubernetes.default.svc:443target_label: __address__- regex: (.+)replacement: /api/v1/nodes/$1/proxy/metrics/cadvisorsource_labels:- __meta_kubernetes_node_nametarget_label: __metrics_path__scheme: httpstls_config:ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crtinsecure_skip_verify: true
- honor_labels: truejob_name: kubernetes-service-endpointskubernetes_sd_configs:- role: endpointsrelabel_configs:- action: keepregex: truesource_labels:- __meta_kubernetes_service_annotation_prometheus_io_scrape- action: dropregex: truesource_labels:- __meta_kubernetes_service_annotation_prometheus_io_scrape_slow- action: replaceregex: (https?)source_labels:- __meta_kubernetes_service_annotation_prometheus_io_schemetarget_label: __scheme__- action: replaceregex: (.+)source_labels:- __meta_kubernetes_service_annotation_prometheus_io_pathtarget_label: __metrics_path__- action: replaceregex: (.+?)(?::\d+)?;(\d+)replacement: $1:$2source_labels:- __address__- __meta_kubernetes_service_annotation_prometheus_io_porttarget_label: __address__- action: labelmapregex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)replacement: __param_$1- action: labelmapregex: __meta_kubernetes_service_label_(.+)- action: replacesource_labels:- __meta_kubernetes_namespacetarget_label: namespace- action: replacesource_labels:- __meta_kubernetes_service_nametarget_label: service- action: replacesource_labels:- __meta_kubernetes_pod_node_nametarget_label: node
- honor_labels: truejob_name: kubernetes-service-endpoints-slowkubernetes_sd_configs:- role: endpointsrelabel_configs:- action: keepregex: truesource_labels:- __meta_kubernetes_service_annotation_prometheus_io_scrape_slow- action: replaceregex: (https?)source_labels:- __meta_kubernetes_service_annotation_prometheus_io_schemetarget_label: __scheme__- action: replaceregex: (.+)source_labels:- __meta_kubernetes_service_annotation_prometheus_io_pathtarget_label: __metrics_path__- action: replaceregex: (.+?)(?::\d+)?;(\d+)replacement: $1:$2source_labels:- __address__- __meta_kubernetes_service_annotation_prometheus_io_porttarget_label: __address__- action: labelmapregex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)replacement: __param_$1- action: labelmapregex: __meta_kubernetes_service_label_(.+)- action: replacesource_labels:- __meta_kubernetes_namespacetarget_label: namespace- action: replacesource_labels:- __meta_kubernetes_service_nametarget_label: service- action: replacesource_labels:- __meta_kubernetes_pod_node_nametarget_label: nodescrape_interval: 5mscrape_timeout: 30s
- honor_labels: truejob_name: prometheus-pushgatewaykubernetes_sd_configs:- role: servicerelabel_configs:- action: keepregex: pushgatewaysource_labels:- __meta_kubernetes_service_annotation_prometheus_io_probe
- honor_labels: truejob_name: kubernetes-serviceskubernetes_sd_configs:- role: servicemetrics_path: /probeparams:module:- http_2xxrelabel_configs:- action: keepregex: truesource_labels:- __meta_kubernetes_service_annotation_prometheus_io_probe- source_labels:- __address__target_label: __param_target- replacement: blackboxtarget_label: __address__- source_labels:- __param_targettarget_label: instance- action: labelmapregex: __meta_kubernetes_service_label_(.+)- source_labels:- __meta_kubernetes_namespacetarget_label: namespace- source_labels:- __meta_kubernetes_service_nametarget_label: service
- honor_labels: truejob_name: kubernetes-podskubernetes_sd_configs:- role: podrelabel_configs:- action: keepregex: truesource_labels:- __meta_kubernetes_pod_annotation_prometheus_io_scrape- action: dropregex: truesource_labels:- __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow- action: replaceregex: (https?)source_labels:- __meta_kubernetes_pod_annotation_prometheus_io_schemetarget_label: __scheme__- action: replaceregex: (.+)source_labels:- __meta_kubernetes_pod_annotation_prometheus_io_pathtarget_label: __metrics_path__- action: replaceregex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})replacement: '[$2]:$1'source_labels:- __meta_kubernetes_pod_annotation_prometheus_io_port- __meta_kubernetes_pod_iptarget_label: __address__- action: replaceregex: (\d+);((([0-9]+?)(\.|$)){4})replacement: $2:$1source_labels:- __meta_kubernetes_pod_annotation_prometheus_io_port- __meta_kubernetes_pod_iptarget_label: __address__- action: labelmapregex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)replacement: __param_$1- action: labelmapregex: __meta_kubernetes_pod_label_(.+)- action: replacesource_labels:- __meta_kubernetes_namespacetarget_label: namespace- action: replacesource_labels:- __meta_kubernetes_pod_nametarget_label: pod- action: dropregex: Pending|Succeeded|Failed|Completedsource_labels:- __meta_kubernetes_pod_phase
- honor_labels: truejob_name: kubernetes-pods-slowkubernetes_sd_configs:- role: podrelabel_configs:- action: keepregex: truesource_labels:- __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow- action: replaceregex: (https?)source_labels:- __meta_kubernetes_pod_annotation_prometheus_io_schemetarget_label: __scheme__- action: replaceregex: (.+)source_labels:- __meta_kubernetes_pod_annotation_prometheus_io_pathtarget_label: __metrics_path__- action: replaceregex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})replacement: '[$2]:$1'source_labels:- __meta_kubernetes_pod_annotation_prometheus_io_port- __meta_kubernetes_pod_iptarget_label: __address__- action: replaceregex: (\d+);((([0-9]+?)(\.|$)){4})replacement: $2:$1source_labels:- __meta_kubernetes_pod_annotation_prometheus_io_port- __meta_kubernetes_pod_iptarget_label: __address__- action: labelmapregex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)replacement: __param_$1- action: labelmapregex: __meta_kubernetes_pod_label_(.+)- action: replacesource_labels:- __meta_kubernetes_namespacetarget_label: namespace- action: replacesource_labels:- __meta_kubernetes_pod_nametarget_label: pod- action: dropregex: Pending|Succeeded|Failed|Completedsource_labels:- __meta_kubernetes_pod_phasescrape_interval: 5mscrape_timeout: 30s

在这里插入图片描述

上面是人家的写的配置，修改配置为下方

在prometheus-prometheus.yaml的配置文件中，官方内置了一个字段additionalScrapeConfigs用于添加自定义的抓取目标

https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#PrometheusSpec

所以只需要按照官方的要求写入配置即可。新建额外抓取的信息prometheus-additional.yaml：

prometheus-additional.yaml

- job_name: kubernetes-istio-podshonor_labels: truehonor_timestamps: truescrape_interval: 15sscrape_timeout: 10smetrics_path: /metricsscheme: httpfollow_redirects: truerelabel_configs:- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]separator: ;regex: "true"replacement: $1action: keep- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]separator: ;regex: "true"replacement: $1action: drop- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]separator: ;regex: (https?)target_label: __scheme__replacement: $1action: replace- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]separator: ;regex: (.+)target_label: __metrics_path__replacement: $1action: replace- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]separator: ;regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})target_label: __address__replacement: '[$2]:$1'action: replace- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_ip]separator: ;regex: (\d+);((([0-9]+?)(\.|$)){4})target_label: __address__replacement: $2:$1action: replace- separator: ;regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)replacement: __param_$1action: labelmap- separator: ;regex: __meta_kubernetes_pod_label_(.+)replacement: $1action: labelmap- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_phase]separator: ;regex: Pending|Succeeded|Failed|Completedreplacement: $1action: dropkubernetes_sd_configs:- role: podkubeconfig_file: ""follow_redirects: true

抓取的目标制定为targets列表，创建secret对象istio-additional-configs引用该配置

kubectl create secret generic `istio-additional-configs` --from-file=./prometheus-additional.yaml -n monitoring

kubectl edit prometheus -n monitoringspec:additionalScrapeConfigs:key: prometheus-additional.yamlname: istio-additional-configs

在这里插入图片描述
或者

在promethes-prometheus.yaml中添加additionalScrapeConfigs引用创建的secret

serviceAccountName: prometheus-k8sserviceMonitorNamespaceSelector: {}serviceMonitorSelector: {}version: v2.11.0additionalScrapeConfigs:name: ingress-nginx-additional-configskey: prometheus-additional.yaml

重建promethes配置:

$ kubectl delete -f ./prometheus-prometheus.yaml && kubectl apply -f ./prometheus-prometheus.yaml

在这里插入图片描述

Grafan监控

关于Grafan的监控面板，可以去官方进行寻找

https://grafana.com/grafana/dashboards/

Istio也提供了自身平台的监控大盘，如下：
在这里插入图片描述
可以看出Istio的默认监控大盘非常全面，该监控的都监控起来了

参考文档：https://www.cnblogs.com/justmine/p/12269587.html