k8s的部署管理以及prometheus相关监控

安装运行prometheus和grafana
编辑prometheus配置文件，增加采集prometheus自身数据的采集规则
grafana上导入prometheus相关图表的dashboard

前面三步主要根据https://blog.csdn.net/shnu_cdk/article/details/132182858?spm=1001.2014.3001.5506

用Go编写一个prometheus exporter，包含prometheus四种指标类型

这一部分直接查看官方文档使用说明。

package mainimport ("fmt""math/rand""net/http""time""github.com/prometheus/client_golang/prometheus""github.com/prometheus/client_golang/prometheus/promhttp"
)func main() {// 创建并注册指标counter := prometheus.NewCounter(prometheus.CounterOpts{Name: "my_counter",Help: "A counter metric",})prometheus.MustRegister(counter)gauge := prometheus.NewGauge(prometheus.GaugeOpts{Name: "my_gauge",Help: "A gauge metric",})prometheus.MustRegister(gauge)histogram := prometheus.NewHistogram(prometheus.HistogramOpts{Name: "my_histogram",Help: "A histogram metric",})prometheus.MustRegister(histogram)summary := prometheus.NewSummary(prometheus.SummaryOpts{Name: "my_summary",Help: "A summary metric",})prometheus.MustRegister(summary)// 定期更新指标值go func() {for {counter.Inc()gauge.Set(rand.Float64() * 100)histogram.Observe(rand.Float64() * 100)summary.Observe(rand.Float64() * 100)time.Sleep(time.Second)}}()// 启动HTTP服务，暴露指标http.Handle("/metrics", promhttp.Handler())fmt.Println("Exporter is running on http://localhost:8080/metrics")http.ListenAndServe(":8080", nil)
}

prometheus能够正常采集第四步中的exporter，并在grafana上展示
在yml配置文件上配置相对应的端口，这一部分和配置一个node差不多：

scrape_configs:# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.- job_name: "prometheus"# metrics_path defaults to '/metrics'# scheme defaults to 'http'.static_configs:- targets: ["localhost:9090"]- job_name: 'export'static_configs:- targets: ['192.168.60.177:8080']labels:instance: export

了解alertmanager，知道alertmanager的webhook推送告警方式
主要是要在prometheus上配置相应的端口

# Alertmanager configuration
alerting:alertmanagers:- static_configs:- targets: ['192.168.60.169:9093']

然后在altermanager里面配置相应的路由树

route:group_by: ['alertname']group_wait: 1sgroup_interval: 1srepeat_interval: 1hreceiver: 'web.hook'
receivers:- name: 'web.hook'webhook_configs:- url: 'http://192.168.60.188:8093/demo'
inhibit_rules:- source_match:severity: 'critical'target_match:severity: 'warning'equal: ['alertname', 'dev', 'instance']