alter 告警

下载解压安装

wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar -zxvf alertmanager-0.26.0.linux-amd64.tar.gz -C /usr/local
mv alertmanager-0.26.0.linux-amd64 alertmanager

配置起启动服务

cat > /lib/systemd/system/alertmanager.service <<EOF
[Unit]
Description=mysqld_exporter
Documentation=https://prometheus.io/
After=network.target[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/exporter/mysqld_exporter --config.my-cnf=/usr/local/exporter/config.my-cnf --web.config.file=/usr/local/exporter/webconf.yml
Restart=on-failure[Install]
WantedBy=multi-user.target
EOF

启动服务

systemctl daemon-reload
systemctl enable --now alertmanager
systemctl status alertmanager

prometheus配置

prometheus.yml（缺省配置打开即可）

# Alertmanager configuration
alerting:alertmanagers:- static_configs:- targets:# 打开下面这个配置- alertmanager:9093
rule_files:
# 打开下面这个配置- "first_rules.yml"

将主机名增加到增加hosts文件，或者修改为对应的主机名或者IP地址

增加rules ./rules/alarm_rule.yml

  - alert: 服务告警expr: up == 0for: 30slabels:severity: criticalannotations:instance: "{{ $labels.instance }}"decriptions: "{{ $labels.job }} 服务已关闭 "

重载配置 curl -X POST http://localhost:9090/-/reload -u admin:123456

已经可以看到告警信息

报警状态State分3种：

• inactive：没有异常。
• pending：已触发阈值，但未满足告警持续时间（即 rule 中的 for 字段）。
• firing：已触发阈值且满足条件并发送至 alertmanager。

配置邮箱通知

https://prometheus.io/docs/alerting/latest/configuration/

https://blog.51cto.com/u_14205795/4561320

https://blog.51cto.com/u_10874766/2523351

global:smtp_from: '790079600@qq.com'smtp_smarthost: 'smtp.qq.com:465'               smtp_auth_username: '790079600@qq.com'smtp_auth_password: 'xxx'smtp_require_tls: falsesmtp_hello: 'qq.com'
route:group_by: ['alertname']group_wait: 30sgroup_interval: 5mrepeat_interval: 5mreceiver: 'email'
receivers:- name: 'email'email_configs:- to: '790079600@qq.com'send_resolved: true
inhibit_rules:- source_match:severity: 'critical'target_match:severity: 'warning'equal: ['alertname', 'dev', 'instance']

配置第三方通知

webhook1

安装

wget https://github.com/Rainbowhhy/alertmanager-webhook/releases/download/v1.0/alertmanager-webhook-v1.0-linux-amd64.tar.gz
tar -zxvf alertmanager-webhook-v1.0-linux-amd64.tar.gz -C /usr/local/
mv alertmanager-webhook-v1.0-linux-amd64/ alertmanager-webhook

配置文件

cat alertmanager-webhook.yaml.orig
# 企业微信机器人key
# 使用企业微信时必须配置，不使用则留空
qywechatKey:# 飞书机器人key
# 使用飞书时必须配置，不使用则留空
feishuKey:# 钉钉机器人key
# 使用钉钉时必须配置，不使用则留
dingdingKey:# Redis配置
redisServer: 127.0.0.1  # 必须配置
redisPort:  # 可选项，为空默认为6379
redisPassword: wecloudsre@2023 # redis未设置密码则留空，如果设置了密码登陆则必须配置# 日志配置
logFileDir:   # 可选项，为空则为程序运行目录
logFilePath: alertmanager-webhook.log # 必须配置# 服务监听配置
port: 9095 # 可选项，为空则默认为9095
host: 0.0.0.0 # 可选项，为空默认监听 127.0.0.1

启动

 ./alertmanager-webhook -c alertmanager-webhook.yaml

alertmanager 修改配置

global:smtp_from: '790079600@qq.com'smtp_smarthost: 'smtp.qq.com:465'               smtp_auth_username: '790079600@qq.com'smtp_auth_password: 'xxxxxx'smtp_require_tls: falsesmtp_hello: 'qq.com'
route:group_by: ['alertname']group_wait: 10sgroup_interval: 1mrepeat_interval: 5m receiver: 'web.hook'
receivers:- name: 'web.hook'webhook_configs:- url: 'http://127.0.0.1:9095/qywechat'send_resolved: true- name: 'email'email_configs:- to: '790079600@qq.com'send_resolved: true
inhibit_rules:- source_match:severity: 'critical'target_match:severity: 'warning'equal: ['alertname', 'dev', 'instance']

webhook2 - 强烈推荐，太好用了

https://github.com/feiyu563/PrometheusAlert?tab=readme-ov-file

安装

wget https://github.com/feiyu563/PrometheusAlert/releases/download/v4.9/linux.zip
unzip linux.zip
mv linux /usr/local/prometheusalert
cd /usr/local/prometheusalert
./PrometheusAlert

飞书和企微群告警

github上有详细的配置指导

• 修改app.conf配置

  #是否开启微信告警通道,可同时开始多个通道0为关闭,1为开启
  open-weixin=1
  #默认企业微信机器人地址
  wxurl=https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=22b46592-2601-40c4-aa17-xxx
  #是否开启飞书告警通道,可同时开始多个通道0为关闭,1为开启
  open-feishu=1
  #默认飞书机器人地址
  fsurl=https://open.feishu.cn/open-apis/bot/hook/53e6cda3-b391-4f30-8d8f-xxx
  

• 修改配置alertmanager.yml

  global:resolve_timeout: 5m
  route:group_by: ['instance']group_wait: 10sgroup_interval: 10srepeat_interval: 5m receiver: 'web.hook.prometheusalert'
  receivers:- name: 'web.hook.prometheusalert'webhook_configs:- url: 'http://127.0.0.1:8080/prometheusalert?type=fs&tpl=prometheus-fs&fsurl=【app.conf中的地址】'send_resolved: true- url: 'http://127.0.0.1:8080/prometheusalert?type=wx&tpl=prometheus-wx&wxurl=【app.conf中的地址】'- url: 'http://127.0.0.1:8080/prometheusalert?type=email&tpl=prometheus-email&email=【收件人的值】'
  

• 配置服务

  cat > /lib/systemd/system/prometheusalert.service <<EOF
  [Unit]
  Description=prometheusalert[Service]
  Restart=on-failure
  ExecStart=/usr/local/prometheusalert/PrometheusAlert[Install]
  WantedBy=multi-user.target
  EOF
  

alter 告警

下载解压安装

wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar -zxvf alertmanager-0.26.0.linux-amd64.tar.gz -C /usr/local
mv alertmanager-0.26.0.linux-amd64 alertmanager

配置起启动服务

cat > /lib/systemd/system/alertmanager.service <<EOF
[Unit]
Description=mysqld_exporter
Documentation=https://prometheus.io/
After=network.target[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/exporter/mysqld_exporter --config.my-cnf=/usr/local/exporter/config.my-cnf --web.config.file=/usr/local/exporter/webconf.yml
Restart=on-failure[Install]
WantedBy=multi-user.target
EOF

启动服务

systemctl daemon-reload
systemctl enable --now alertmanager
systemctl status alertmanager

prometheus配置

prometheus.yml（缺省配置打开即可）

# Alertmanager configuration
alerting:alertmanagers:- static_configs:- targets:# 打开下面这个配置- alertmanager:9093
rule_files:
# 打开下面这个配置- "first_rules.yml"

将主机名增加到增加hosts文件，或者修改为对应的主机名或者IP地址

增加rules ./rules/alarm_rule.yml

  - alert: 服务告警expr: up == 0for: 30slabels:severity: criticalannotations:instance: "{{ $labels.instance }}"decriptions: "{{ $labels.job }} 服务已关闭 "

重载配置 curl -X POST http://localhost:9090/-/reload -u admin:123456

已经可以看到告警信息

报警状态State分3种：

• inactive：没有异常。
• pending：已触发阈值，但未满足告警持续时间（即 rule 中的 for 字段）。
• firing：已触发阈值且满足条件并发送至 alertmanager。

配置邮箱通知

https://prometheus.io/docs/alerting/latest/configuration/

https://blog.51cto.com/u_14205795/4561320

https://blog.51cto.com/u_10874766/2523351

global:smtp_from: '790079600@qq.com'smtp_smarthost: 'smtp.qq.com:465'               smtp_auth_username: '790079600@qq.com'smtp_auth_password: 'xxx'smtp_require_tls: falsesmtp_hello: 'qq.com'
route:group_by: ['alertname']group_wait: 30sgroup_interval: 5mrepeat_interval: 5mreceiver: 'email'
receivers:- name: 'email'email_configs:- to: '790079600@qq.com'send_resolved: true
inhibit_rules:- source_match:severity: 'critical'target_match:severity: 'warning'equal: ['alertname', 'dev', 'instance']

配置第三方通知

webhook1

安装

wget https://github.com/Rainbowhhy/alertmanager-webhook/releases/download/v1.0/alertmanager-webhook-v1.0-linux-amd64.tar.gz
tar -zxvf alertmanager-webhook-v1.0-linux-amd64.tar.gz -C /usr/local/
mv alertmanager-webhook-v1.0-linux-amd64/ alertmanager-webhook

配置文件

cat alertmanager-webhook.yaml.orig
# 企业微信机器人key
# 使用企业微信时必须配置，不使用则留空
qywechatKey:# 飞书机器人key
# 使用飞书时必须配置，不使用则留空
feishuKey:# 钉钉机器人key
# 使用钉钉时必须配置，不使用则留
dingdingKey:# Redis配置
redisServer: 127.0.0.1  # 必须配置
redisPort:  # 可选项，为空默认为6379
redisPassword: wecloudsre@2023 # redis未设置密码则留空，如果设置了密码登陆则必须配置# 日志配置
logFileDir:   # 可选项，为空则为程序运行目录
logFilePath: alertmanager-webhook.log # 必须配置# 服务监听配置
port: 9095 # 可选项，为空则默认为9095
host: 0.0.0.0 # 可选项，为空默认监听 127.0.0.1

启动

 ./alertmanager-webhook -c alertmanager-webhook.yaml

alertmanager 修改配置

global:smtp_from: '790079600@qq.com'smtp_smarthost: 'smtp.qq.com:465'               smtp_auth_username: '790079600@qq.com'smtp_auth_password: 'xxxxxx'smtp_require_tls: falsesmtp_hello: 'qq.com'
route:group_by: ['alertname']group_wait: 10sgroup_interval: 1mrepeat_interval: 5m receiver: 'web.hook'
receivers:- name: 'web.hook'webhook_configs:- url: 'http://127.0.0.1:9095/qywechat'send_resolved: true- name: 'email'email_configs:- to: '790079600@qq.com'send_resolved: true
inhibit_rules:- source_match:severity: 'critical'target_match:severity: 'warning'equal: ['alertname', 'dev', 'instance']

webhook2 - 强烈推荐，太好用了

https://github.com/feiyu563/PrometheusAlert?tab=readme-ov-file

安装

wget https://github.com/feiyu563/PrometheusAlert/releases/download/v4.9/linux.zip
unzip linux.zip
mv linux /usr/local/prometheusalert
cd /usr/local/prometheusalert
./PrometheusAlert

飞书和企微群告警

github上有详细的配置指导

• 修改app.conf配置

  #是否开启微信告警通道,可同时开始多个通道0为关闭,1为开启
  open-weixin=1
  #默认企业微信机器人地址
  wxurl=https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=22b46592-2601-40c4-aa17-xxx
  #是否开启飞书告警通道,可同时开始多个通道0为关闭,1为开启
  open-feishu=1
  #默认飞书机器人地址
  fsurl=https://open.feishu.cn/open-apis/bot/hook/53e6cda3-b391-4f30-8d8f-xxx
  

• 修改配置alertmanager.yml

  global:resolve_timeout: 5m
  route:group_by: ['instance']group_wait: 10sgroup_interval: 10srepeat_interval: 5m receiver: 'web.hook.prometheusalert'
  receivers:- name: 'web.hook.prometheusalert'webhook_configs:- url: 'http://127.0.0.1:8080/prometheusalert?type=fs&tpl=prometheus-fs&fsurl=【app.conf中的地址】'send_resolved: true- url: 'http://127.0.0.1:8080/prometheusalert?type=wx&tpl=prometheus-wx&wxurl=【app.conf中的地址】'- url: 'http://127.0.0.1:8080/prometheusalert?type=email&tpl=prometheus-email&email=【收件人的值】'
  

• 配置服务

  cat > /lib/systemd/system/prometheusalert.service <<EOF
  [Unit]
  Description=prometheusalert[Service]
  Restart=on-failure
  ExecStart=/usr/local/prometheusalert/PrometheusAlert[Install]
  WantedBy=multi-user.target
  EOF