方法一
修改告警规则
- alert: cpu使用率大于88%expr: instance:node_cpu_utilization:ratio * 100 > 88for: 5mlabels:severity: criticallevel: 3kind: CpuUsageannotations:summary: "cpu使用率大于85%"description: "主机 {{ $labels.hostname }} 的cpu使用率为 {{ $value | humanize }}"根据Kind区分，规则一kind1，规则二是kind2。
alertmanager配置示例
global:resolve_timeout: 5msmtp_from: from@email.comsmtp_smarthost: smtp.net:portsmtp_auth_username: from@email.comsmtp_auth_password: PASSsmtp_require_tls: false
route:receiver: 'email'group_by: ['alertname']group_wait: 10sgroup_interval: 10srepeat_interval: 10mroutes:- receiver: 'our'group_wait: 10smatch_re:severity: warning- receiver: 'other'group_wait: 10smatch_re:severity: busitemplates:- '*.html'
receivers:
- name: 'email'email_configs:- to: 'xuxd@email.com'send_resolved: falsehtml: '{{ template "default-monitor.html" . }}'headers: { Subject: "[WARN] 报警邮件" } #邮件主题
- name: 'our'webhook_configs:- url: http://127.0.0.1:8060/dingtalk/our/send
- name: 'other'webhook_configs:- url: http://127.0.0.1:8060/dingtalk/other/send- route:除了email这个全局配置的接收者外，下面的routes指定了两个特定的接收者，一个接收者叫“our”，匹配warning级别的；另一个叫“other”，匹配busi级别的，这两个级别在最前面的规则里定义，不是什么特定关键字，就是自己随便定义的一个标记
- receivers:这里指定了上面定义的接收者的配置，email指定邮件发给谁；“our”指定dingtalk的发送url，注意这个uri的末尾，send前用的"our";“other”下面指定了两个url，区别就是url末尾的send前面，一个是“our”，另一个是"other"
prometheus-webhook-dingtalk配置
## Customizable templates path
templates:- /home/user/monitor/alert/prometheus-webhook-dingtalk-1.4.0.linux-amd64/template/template.tmpl## Targets, previously was known as "profiles"
targets:our:url: https://oapi.dingtalk.com/robot/send?access_token=xxxxsecret: xxx_secretother:url: https://oapi.dingtalk.com/robot/send?access_token=xxx_othersecret: xxx_other_secrettargets下有两个，分别是"our"和"other",这里对应上面alertmanager配置的url里的"our"和"other。
这样配置，如果规则一告警，就是alertmanager的name为other的receiver来发送告警通知，发送到我们的钉钉群和业务侧钉钉群。如果是规则二告警，通过our发送，便只发送到我们的钉钉群。
vmalert配置文件value.yaml
# Default values for victoria-metrics-alert.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.serviceAccount:# Specifies whether a service account should be createdcreate: true# Annotations to add to the service accountannotations: {}# The name of the service account to use.# If not set and create is true, a name is generated using the fullname templatename:# mount API token to pod directlyautomountToken: trueimagePullSecrets: []rbac:create: truepspEnabled: truenamespaced: falseextraLabels: {}annotations: {}server:name: serverenabled: trueimage:repository: victoriametrics/vmalerttag: "" # rewrites Chart.AppVersionpullPolicy: IfNotPresentnameOverride: ""fullnameOverride: ""## See `kubectl explain poddisruptionbudget.spec` for more## ref: https://kubernetes.io/docs/tasks/run-application/configure-pdb/podDisruptionBudget:enabled: false# minAvailable: 1# maxUnavailable: 1labels: {}# -- Additional environment variables (ex.: secret tokens, flags) https://github.com/VictoriaMetrics/VictoriaMetrics#environment-variablesenv:[]# - name: VM_remoteWrite_basicAuth_password# valueFrom:# secretKeyRef:# name: auth_secret# key: passwordreplicaCount: 1# deployment strategy, set to standard k8s defaultstrategy:type: RollingUpdaterollingUpdate:maxSurge: 25%maxUnavailable: 25%# specifies the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing/terminating# 0 is the standard k8s defaultminReadySeconds: 0# vmalert reads metrics from source, next section represents its configuration. It can be any service which supports# MetricsQL or PromQL.datasource:url: "http://192.168.47.9:8481/select/0/prometheus/"basicAuth:username: ""password: ""remote:write:url: ""read:url: ""notifier:alertmanager:url: "http://x.x.x.x:9093"extraArgs:envflag.enable: "true"envflag.prefix: VM_loggerFormat: json# Additional hostPath mountsextraHostPathMounts:[]# - name: certs-dir# mountPath: /etc/kubernetes/certs# subPath: ""# hostPath: /etc/kubernetes/certs# readOnly: true# Extra Volumes for the podextraVolumes:[]#- name: example# configMap:# name: example# Extra Volume Mounts for the containerextraVolumeMounts:[]# - name: example# mountPath: /exampleextraContainers:[]#- name: config-reloader# image: reloader-imageservice:annotations: {}labels: {}clusterIP: ""## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips##externalIPs: []loadBalancerIP: ""loadBalancerSourceRanges: []servicePort: 8880type: ClusterIP# Ref: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip# externalTrafficPolicy: "local"# healthCheckNodePort: 0ingress:enabled: falseannotations: {}# kubernetes.io/ingress.class: nginx# kubernetes.io/tls-acme: 'true'extraLabels: {}hosts: []# - name: vmselect.local# path: /select# port: httptls: []# - secretName: vmselect-ingress-tls# hosts:# - vmselect.local# For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName# See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress# ingressClassName: nginx# -- pathType is only for k8s >= 1.1=pathType: PrefixpodSecurityContext: {}# fsGroup: 2000securityContext:{}# capabilities:# drop:# - ALL# readOnlyRootFilesystem: true# runAsNonRoot: true# runAsUser: 1000resources:{}# We usually recommend not to specify default resources and to leave this as a conscious# choice for the user. This also increases chances charts run on environments with little# resources, such as Minikube. If you do want to specify resources, uncomment the following# lines, adjust them as necessary, and remove the curly braces after 'resources:'.# limits:# cpu: 100m# memory: 128Mi# requests:# cpu: 100m# memory: 128Mi# Annotations to be added to the deploymentannotations: {}# labels to be added to the deploymentlabels: {}# Annotations to be added to podpodAnnotations: {}podLabels: {}nodeSelector: {}priorityClassName: ""tolerations: []affinity: {}# vmalert alert rules configuration configuration:# use existing configmap if specified# otherwise .config values will be usedconfigMap: ""config:alerts:groups:- name: 磁盘挂载错误rules:- alert: 磁盘挂载错误annotations:description: '{{$labels.job}}链{{$labels.instance}}节点磁盘挂载错误'expr: mount_error{job=~"dev|sit"} == 1for: 1mlabels:severity: criticalkind: kind1- name: 进程不存在rules:- alert: 进程不存在annotations:description: '{{$labels.job}}链{{$labels.instance}}进程不存在'expr: process_total_error{job=~"dev|sit"} == 1for: 1mlabels:severity: criticalkind: kind2serviceMonitor:enabled: falseextraLabels: {}annotations: {}
# interval: 15s
# scrapeTimeout: 5s# -- Commented. HTTP scheme to use for scraping.
# scheme: https# -- Commented. TLS configuration to use when scraping the endpoint
# tlsConfig:
# insecureSkipVerify: truealertmanager:enabled: truereplicaCount: 1podMetadata:labels: {}annotations: {}image: prom/alertmanagertag: v0.20.0retention: 120hnodeSelector: {}priorityClassName: ""resources: {}tolerations: []imagePullSecrets: []podSecurityContext: {}extraArgs: {}# key: value# external URL, that alertmanager will expose to receiversbaseURL: ""# use existing configmap if specified# otherwise .config values will be usedconfigMap: ""config:global:resolve_timeout: 5mroute:# default receiverreceiver: aldaba# tag to group bygroup_by: [alertname]# How long to initially wait to send a notification for a group of alertsgroup_wait: 30s# How long to wait before sending a notification about new alerts that are added to a groupgroup_interval: 60s# How long to wait before sending a notification again if it has already been sent successfully for an alertrepeat_interval: 1hroutes:- receiver: 'mychain'group_wait: 10smatch_re:kind: mychainreceivers:- name: aldabawebhook_configs:- url: http://192.168.208.133:8080/prometheusalert?type=dd&tpl=prometheus-dd&ddurl=https://oapi.dingtalk.com/robot/send?access_token=72a3a55795094a6878c2c2443a81a3545add1f688ddee18701c0dd753dbb3b2a&split=falsesend_resolved: true- name: mychainwebhook_configs:- url: http://192.168.208.133:8080/prometheusalert?type=dd&tpl=prometheus-dd&ddurl=https://oapi.dingtalk.com/robot/send?access_token=307270fdcd1bb0c4b0533e29005cca7cb353c27d7f988fdff0ec00e6affc6e83&split=falsesend_resolved: trueinhibit_rules:- source_match:#severity: 'warning'target_match:#severity: 'warning'#equal: ['alertname', 'job']templates: {}# alertmanager.tmpl: |-service:annotations: {}type: ClusterIPport: 9093# if you want to force a specific nodePort. Must be use with service.type=NodePort# nodePort:ingress:enabled: falseannotations:# nginx.ingress.kubernetes.io/auth-realm: Authentication Required# nginx.ingress.kubernetes.io/auth-secret: victoria-metrics/basic-auth# nginx.ingress.kubernetes.io/auth-type: basic# kubernetes.io/ingress.class: nginx# kubernetes.io/tls-acme: 'true'extraLabels: {}hosts: {}# - name: wangjuan.test.com# path: /# port: webtls: []# - secretName: alertmanager-ingress-tls# hosts:# - alertmanager.local# For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName# See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress# ingressClassName: nginx# -- pathType is only for k8s >= 1.1=pathType: PrefixpersistentVolume:# -- Create/use Persistent Volume Claim for alertmanager component. Empty dir if falseenabled: false# -- Array of access modes. Must match those of existing PV or dynamic provisioner. Ref: [http://kubernetes.io/docs/user-guide/persistent-volumes/](http://kubernetes.io/docs/user-guide/persistent-volumes/)accessModes:- ReadWriteOnce# -- Persistant volume annotationsannotations: {}# -- StorageClass to use for persistent volume. Requires alertmanager.persistentVolume.enabled: true. If defined, PVC created automaticallystorageClass: ""# -- Existing Claim name. If defined, PVC must be created manually before volume will be boundexistingClaim: ""# -- Mount path. Alertmanager data Persistent Volume mount root path.mountPath: /data# -- Mount subpathsubPath: ""# -- Size of the volume. Better to set the same as resource limit memory property.size: 50Mi方法二
根据job过滤
alertmanager配置
apiVersion: v1
data:alertmanager.yaml: |-global:resolve_timeout: 5minhibit_rules:- equal:- alertname- jobsource_match:severity: warningtarget_match:severity: warningreceivers:- name: nftwebhook_configs:- send_resolved: falseurl: http://x.x.x.x:8080/prometheusalert?type=dd&tpl=prometheus-dd&ddurl=https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxxx&split=false- name: poapwebhook_configs:- send_resolved: falseurl: http://x.x.x.x:8080/prometheusalert?type=dd&tpl=prometheus-dd&ddurl=https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxxxx&split=false- name: ipforcewebhook_configs:- send_resolved: falseurl: http://x.x.x.x:8080/prometheusalert?type=dd&tpl=prometheus-dd&ddurl=https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxxxxxx&split=falseroute:group_by:- alertnamegroup_interval: 60sgroup_wait: 30sreceiver: nftrepeat_interval: 1hroutes:- group_wait: 10smatch_re:job: test_poapreceiver: poap- group_wait: 10smatch_re:job: test_ipforcereceiver: ipforce
kind: ConfigMap
metadata:annotations:meta.helm.sh/release-name: vmalertmeta.helm.sh/release-namespace: victoria-metricscreationTimestamp: '2022-04-06T07:31:38Z'labels:app: alertmanagerapp.kubernetes.io/instance: vmalertapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: victoria-metrics-alerthelm.sh/chart: victoria-metrics-alert-0.4.33managedFields:- apiVersion: v1fieldsType: FieldsV1fieldsV1:'f:data': {}'f:metadata':'f:annotations':.: {}'f:meta.helm.sh/release-name': {}'f:meta.helm.sh/release-namespace': {}'f:labels':.: {}'f:app': {}'f:app.kubernetes.io/instance': {}'f:app.kubernetes.io/managed-by': {}'f:app.kubernetes.io/name': {}'f:helm.sh/chart': {}manager: helmoperation: Updatetime: '2022-04-06T07:31:38Z'- apiVersion: v1fieldsType: FieldsV1fieldsV1:'f:data':'f:alertmanager.yaml': {}manager: ACK-Console Apache-HttpClientoperation: Updatetime: '2023-01-05T07:52:13Z'name: vmalert-alertmanager-alertmanager-confignamespace: victoria-metricsresourceVersion: '80954053'uid: 653e4633-86e5-41ce-9a17-301f75224e9c
)