多渠道报警

背景

统一监控报警平台
使用Grafana汇聚了,Prometheus,ELK,CloudWatch监控信息.
简单理解 ELK,AWS均可以实现短信和电话报警

需求

1.Grafana支持短信渠道报警和电话渠道报警.
2.对不同微服务的开发团队发送报警信息.

要求

1.阿里云短信&电话服务API
2.Jenkins-plugin:Generic Webhook Trigger Plugin

逻辑

• 阿里云

  • 下载阿里云短信服务SDK
  • 创建短信服务ACCESS_KEY_ID

• Grafana

  • 产生报警信息
  • 将报警信息发送给Jenkins的webhook接口

• Jenkins

  • 创建JOB,执行python脚本
  • 设置JOB的webhook地址
  • 创建电话号码变量

上图说明

首先看下阿里云短信接口的python demo

# -*- coding: utf-8 -*-
import sys
from aliyunsdkdysmsapi.request.v20170525 import SendSmsRequest
from aliyunsdkdysmsapi.request.v20170525 import QuerySendDetailsRequest
from aliyunsdkcore.client import AcsClient
import uuid
from aliyunsdkcore.profile import region_provider
from aliyunsdkcore.http import method_type as MT
from aliyunsdkcore.http import format_type as FT
import const
import json

"""
短信业务调用接口示例，版本号：v20170525

Created on 2017-06-12

"""
try:
    reload(sys)
    sys.setdefaultencoding('utf8')
except NameError:
    pass
except Exception as err:
    raise err

# 注意：不要更改
REGION = "cn-hangzhou"
PRODUCT_NAME = "Dysmsapi"
DOMAIN = "dysmsapi.aliyuncs.com"

#Jenkins webhook 变量
PhoneNumbers= sys.argv[1].split(",")
Contents= sys.argv[2]
State= sys.argv[3]

print "PhoneNumbers:%s" % PhoneNumbers
print "Contents: %s" % Contents


acs_client = AcsClient(const.ACCESS_KEY_ID, const.ACCESS_KEY_SECRET, REGION)
region_provider.add_endpoint(PRODUCT_NAME, REGION, DOMAIN)

def send_sms(business_id, phone_numbers, sign_name, template_code, template_param=None):
    smsRequest = SendSmsRequest.SendSmsRequest()
    # 申请的短信模板编码,必填
    smsRequest.set_TemplateCode(template_code)

    # 短信模板变量参数
    if template_param is not None:
        smsRequest.set_TemplateParam(template_param)

    # 设置业务请求流水号，必填。
    smsRequest.set_OutId(business_id)

    # 短信签名
    smsRequest.set_SignName(sign_name)

    # 数据提交方式
    # smsRequest.set_method(MT.POST)

    # 数据提交格式
    # smsRequest.set_accept_format(FT.JSON)

    # 短信发送的号码列表，必填。
    smsRequest.set_PhoneNumbers(phone_numbers)

    # 调用短信发送接口，返回json
    smsResponse = acs_client.do_action_with_exception(smsRequest)

    # TODO 业务处理

    return smsResponse



if __name__ == '__main__':
    __business_id = uuid.uuid1()
    #print(__business_id)
    params = {"name":State,"contents":Contents}
    params = json.dumps(params)
    print params
    for PhoneNumber in PhoneNumbers:
        print(send_sms(__business_id, PhoneNumber, "玉符", "SMS_142145090", params))

再看下阿里云电话接口的python demo

# -*- coding: utf-8 -*-
from aliyunsdkdyvmsapi.request.v20170525 import SingleCallByTtsRequest
from aliyunsdkdyvmsapi.request.v20170525 import SingleCallByVoiceRequest
from aliyunsdkcore.client import AcsClient
import uuid
import json
import sys
import re

"""
语音业务调用接口示例，版本号：v20170525
Created on 2017-06-12

安装依赖库
wget http://ytx-sdk.oss-cn-shanghai.aliyuncs.com/dyvms_python.zip
unzip dyvms_python.zip
sudo python dyvms_python/api_demo/aliyun-python-sdk-core/setup.py install
sudo python dyvms_python/api_demo/aliyun-python-sdk-dyvmsapi/setup.py install
sudo python dyvms_python/api_sdk/aliyun-python-sdk-core/setup.py install
sudo python dyvms_python/api_sdk/aliyun-python-sdk-dyvmsapi/setup.py install


""" 
#暂时不支持多region,默认配置杭州
REGION = "cn-hangzhou"
# ACCESS_KEY_ID/ACCESS_KEY_SECRET 根据实际申请的账号信息进行替换
ACCESS_KEY_ID = "xxxxxxx"
ACCESS_KEY_SECRET = "xxxxxx"
#初始化AcsClient
acs_client = AcsClient(ACCESS_KEY_ID, ACCESS_KEY_SECRET, REGION)


def tts_call(business_id, called_number, called_show_number, tts_code, tts_param=None):
    ttsRequest = SingleCallByTtsRequest.SingleCallByTtsRequest()
    # 申请的语音通知tts模板编码,必填
    ttsRequest.set_TtsCode(tts_code)
    # 设置业务请求流水号，必填。后端服务基于此标识区分是否重复请求的判断
    ttsRequest.set_OutId(business_id)
    # 语音通知的被叫号码，必填。
    ttsRequest.set_CalledNumber(called_number)
    # 语音通知显示号码，必填。
    ttsRequest.set_CalledShowNumber(called_show_number)
    # tts模板变量参数
    if tts_param is not None:
        ttsRequest.set_TtsParam(tts_param)
    ttsResponse = acs_client.do_action_with_exception(ttsRequest)
    return ttsResponse


__business_id = uuid.uuid1()
print __business_id
#模板中不存在变量的情况下为{}


PhoneNumbers = sys.argv[1].split(",")
Contents = re.sub('[\[\]]', '', sys.argv[2])
State = sys.argv[3]
Appname = sys.argv[4].split("-")[1]
Level = "高级别"
Value = "-"

params = {"level": Level, "Appname": Appname, "Contents": Contents}
params = json.dumps(params)

print Appname, Contents


print "State:%s" % State
if State == "alerting":
    for PhoneNumber in PhoneNumbers:
        print "正在呼叫%s" % PhoneNumber
        print tts_call(__business_id, PhoneNumber,
                       "073182705718", "TTS_148381535", params)

安装Jenkins插件

访问jenkins页面 https://jenkins.$demo.com/pluginManager/available
搜索并安装插件 Generic Webhook Trigger Plugin
创建Jenkins Job (下图是几处关键设置)

image.png

image.png

短信

image.png

电话

image.png

Jenkins Job创建完成

创建Grafana Notification channels

Url: https://jenkins.$dome.com/generic-webhook-trigger/invoke?token=sms (此处的sms是jenkins中的token)

image.png

完成

在Grafana触发报警后会通过这个Channel 发送一个POST请求到Jenkins.
Jenkins会执行对应的JOB
job执行结果

image.png