2023-03-03 Prometheus Alertmanager Webhook与钉钉结合

首先配置alertmanager.yml

global:  #全局配置
  resolve_timeout: 5m
route:    #分组路由
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 4h
  receiver: 'web.hook'
receivers:   #接收器 通过webhook发送钉钉接口
- name: 'web.hook'
  webhook_configs:
  - url: 'http://localhost:8060/dingtalk/webhook1/send'
inhibit_rules:     #告警抑制
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

其次配置alertmanager与prometheus相结合，在prometheus.yml中增加

alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - 127.0.0.1:9093

然后还是配置prometheus.yml,加入规则文件加载路径,告警的是之前配置过的job_name，这里不进行说明。

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
    - "conf/rules/*.yml"

配置规则rules.yml ,这些模板可以在网上下载，例如:

groups:
- name: YarnStatsAlert
  rules:
  - alert: hadoop_yarn_resourcemanager_apps_pending 数量过多
    expr: hadoop_yarn_resourcemanager_apps_pending > 100
    for: 5m
    labels:
      severity: P0
      alertroute: dingding|phone
      partyid: "2"
    annotations:
      summary: "Instance {{ $labels.instance }} "
      description: "hadoop_yarn_resourcemanager_apps_pending数量过多, (当前值：{{ $value }})"
  - alert: nodemanager节点不健康
    expr: hadoop_yarn_resourcemanager_nodemanager_total{serivce="YARN",status="Unhealthy"} >0
    for: 1m
    labels:
      severity: P1
      alertroute: dingding|phone
      partyid: "2"
    annotations:
      summary: "Instance {{ $labels.instance }} "
      description: "nodemanager节点不健康,请及时查看 (当前值：{{ $value }})"

最后，从网上下载prometheus-webhook-dingtalk，安装后就是之前我们在webhook中配置的http://localhost:8060/dingtalk/webhook1/send服务。
需要配置config.yml确定实际的钉钉接口地址。
template.tmpl中配置告警话术。

登录alertmanager部署主机。例如 http://localhost:9093/#/alerts
点击NEW Silence配置静默。