黑盒监控

“白盒监控” 是需要把对应的Exporter程序安装到被监控的目标主机上，从而实现对主机各种资源以及状态的数据采集工作 ”黑盒监控“ 是不需要把Exporter程序部署到被监控的目标主机上，比如全球的网络质量的稳定性，通常用ping操作，对选取的节点进行icmp测试 Prometheus社区提供黑盒解决方案，用户只需要将其安装在与Prometheus和被监控目标互通的环境中，通过HTTP、HTTPS、DNS、TCP、ICMP,证书的监控等 方式对网络进行探测监控。

blackbox_exporter

Prometheus 官方提供的exporter，可以提供http dns tcp icmp的监控数据采集

创建配置文件

mkdir /data/blackbox_exporter -p
​
cat >/data/blackbox_exporter/config.yml <<'EOF'
modules:http_2xx:prober: httphttp:method: GEThttp_port_2xx:prober: httphttp:  method: POSTtcp_connect:prober: tcppop3s_banner:prober: tcptcp:query_response:- expect: "^+OK"tls: truetls_config:insecure_skip_verify: falsegrpc:prober: grpcgrpc:tls: truepreferred_ip_protocol: "ip4"grpc_plain:prober: grpctls: falseservice: "service1"ssh_banner:prober: tcptcp:query_response:- expect: "^SSH-2.0-"- send: "SSH-2.0-blackbox-ssh-check"irc_banner:prober: tcptcp:query_response:- send: "NICK prober"- send: "USER prober prober prober :prober"- expect: "PING :([^ ]+)"send: "PONG ${1}"- expect: "^:[^ ]+ 001"icmp:prober: icmpicmp_ttl5:prober: icmptimeout: 5sicmp:ttl: 5
EOF

修改后

root@prometheus220:/data/blackbox_exporter# cat /data/blackbox_exporter/config.yml 
modules:http_2xx:prober: httphttp:method: GEThttp_port_2xx:prober: httphttp:  method: POSTtcp_connect:prober: tcpicmp:prober: icmp

docker 直接运行

docker run -d --restart=always --name blackbox-exporter -p 9115:9115 \
-v /data/blackbox_exporter:/etc/blackbox_exporter prom/blackbox-exporter:v0.16.0 \
--config.file=/ect/blackbox_exporter/config.yml

docker-compose 方式

cd /data/blackbox_exporter/
cat >docker-compose.yml << 'EOF'
version: '3.3'
services:blackbox_exporter:image: prom/blackbox-exporter:v0.16.0container_name: blackbox_exporterrestart: alwaysvolumes:- /data/blackbox_exporter:/etc/blackbox_exporterports:- 9115:9115
EOF

启动

docker-compose up -d

确认启动

docker ps -a

访问测试

http://10.19.1.220:9115/

添加Prometheus的配置

cd /opt/data/docker-prometheus/prometheus
cat >> prometheus/prometheus.yml<<'EOF'
#http 配置- job_name: 'blackbox_http'metrics_path: /probeparams: module: [http_2xx]static_configs:- targets: - https://www.baidu.com- https://www.163.comrelabel_configs:- source_labels: [__address__]target_label: __param_target- source_labels: [__param_target]target_label: instance- target_label: __address__replacement: 10.19.1.220:9115
​
#tcp 配置- job_name: 'blackbox_tcp'metrics_path: /probeparams: module: [tcp_connect]static_configs:- targets: - 10.19.1.220:22- 10.19.1.206:9090relabel_configs:- source_labels: [__address__]target_label: __param_target- source_labels: [__param_target]target_label: instance- target_label: __address__replacement: 10.19.1.220:9115
​
#icmp 配置 ping- job_name: 'blackbox_icmp'metrics_path: /probeparams: module: [icmp]static_configs:- targets: - 10.19.1.206- 10.19.1.220relabel_configs:- source_labels: [__address__]target_label: __param_target- source_labels: [__param_target]target_label: instance- target_label: __address__replacement: 10.19.1.220:9115
EOF

热启动

curl -X POST http://localhost:9090/-/reload

Prometheus界面已经显示配置成功

监控项：

probe_
​
probe_success  #是否探测成功(取值1、0，分别表示成功、失败)
probe_duration_seconds #探测的耗时
#关于DNS
probe_dns_lookup_time_seconds #DNS解析的耗时
probe_ip_protocol #IP协议取值为4、6
probe_ip_addr_hash #IP地址的哈希值，用于判断IP是否变化
#关于HTTP
probe_http_status_code #HTTP响应的状态码，如果发生重定向，则取决于最后一次响应
probe_http_content_length #HTTP响应的body长度，单位bytes
probe_http_version #HTTP响应的协议版本，比如1.1
probe_http_ssl #HTTP响应是否采用SSL，取值 1、0
probe_ssl_earliest_cert_expiry #SSL证书过期时间，为Unix时间戳
​

触发器

Prometheus 配置

#报警(触发器)配置
rule_files:- "alert.yml"- "rules/*.yml"

添加blackbox_exporter触发器

cd /opt/data/docker-prometheus/

创建触发器文件

cat > prometheus/rules/blackbox_exporter.yml << 'EOF'
groups:
- name: Blackboxrules:- alert: 黑盒子探测失败报警expr: probe_success == 0for: 1mlabels:severity: criticalannotations:summary: '黑盒子探测失败 {{ $labels.instance }}'description: "黑盒子探测失败,当前值{{ $value }}"- alert: 请求慢告警expr: avg_over_time(probe_duration_seconds[1m]) > 1for: 1mlabels:severity: warningannotations:summary: '请求慢 {{ $labels.instance }}'description: "请求时间操过1秒，值{{ $value }}"- alert: http状态码检测失败expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400for: 1mlabels:severity: criticalannotations:summary: 'http状态码检测失败{{ $labels.instance }}'description: "HTTP状态码非200-399,当前状态码为{{ $value }}"- alert: ssl证书即将到期expr: probe_ssl_earliest_cert_expiry - time() < 86400 *30for: 1mlabels:severity: warningannotations:summary: '证书即将到期 {{ $labels.instance }}'description: "SSL 证书在30天后到期,值{{ $value }}"    
EOF

重新加载配置

curl -X POST http://localhost:9090/-/reload

alert中生效

Grafana增加dashboard

Blackbox Exporter Dashboard 20220412-StarsL.cn | Grafana Labs

Grafana

admin/password