本项目是通过采集Nginx的日志，将日志存储到ES中，使用EA（ElasticAlert）调用ES的接口抓取数据过滤，对于触发EA规则的调用钉钉的接口发送告警
安装过程；
1、Nginx安装（略）
2、安装ES（略）
3、安装python 3
4、安装elastalert
1、数据流：
（1）、用户访问nginx，nginx产生访问日志（这里可以将Nginx与modsecurity整合做成WAF）；
（2）、filebeat读取nginx的日志发给logstash；
（3）、logstash接收到日志，存储到ES中；
（4）、在EA中定义规则，并抽取ES数据过滤，对于触发规则的数据调用钉钉；
（5）、钉钉收到告警信息显示给用户。
2、安装python 3.x
由于vim\yum\firewall-cmd命令都是依赖python 2.x，安装python 3.x后会导致这些命令不可用
yum -y install wget openssl openssl-devel gcc gcc-c++
wget https://www.python.org/ftp/python/3.6.9/Python-3.6.9.tgz
tar xf Python-3.6.9.tgz
cd Python-3.6.9
./configure --prefix=/usr/local/python --with-openssl
make && make install
mv /usr/bin/python /usr/bin/python_old
ln -s /usr/local/python/bin/python3 /usr/bin/python
ln -s /usr/local/python/bin/pip3 /usr/bin/pip
pip install --upgrade pip
sed -i '1s/python/python2.7/g' /usr/bin/yum
sed -i '1s/python/python2.7/g' /usr/libexec/urlgrabber-ext-down
python -V
显示为3.6.9
3、安装elasticalert
可以直接安装
pip install elastalert
也可以拉取源码
git clone https://github.com/Yelp/elastalert.git //建议使用源码安装
cd alastalert/
pip install setuptools-rust
python setup.py install
pip install -r requirements.txt
如有报错
elastalert 0.2.4 requires jira>=2.0.0, but you'll have jira 1.0.14 which is incompatible.
可执行
pip install jira==2.0.0 //执行后不用再执行pip install -r requirements.txt
4、创建索引
# cd /usr/local/python3.6/bin/
# ./elastalert-create-index
root@ea bin]# ./elastalert-create-index
Enter Elasticsearch host: 192.168.25.160
Enter Elasticsearch port: 9200
Use SSL? t/f: f
Enter optional basic-auth username (or leave blank):
Enter optional basic-auth password (or leave blank):
Enter optional Elasticsearch URL prefix (prepends a string to the URL of every request):
New index name? (Default elastalert_status)
New alias name? (Default elastalert_alerts)
Name of existing index to copy? (Default None)
Elastic Version: 7.4.0
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index elastalert_status created
Done!
或
0. curl -XPUT http:///elastalert-create-index
1. curl -XPUT http:///elastalert_status_status
2. curl -XPUT http:///elastalert_status_error
3. curl -XPUT http:///elastalert_status_silence
4. curl -XPUT http:///elastalert_status_past
6、申请钉钉机器人
在钉钉中创建群聊，在群聊的群设置中的智能群助手中添加机器人
在本步骤中获取到webhook地址，并将自己运行EA的服务器的公网IP加入到机器人安全设置的IP地址段中。
7、安装钉钉告警插件
mkdir /home/dingding
cd /home/dingding
wget https://github.com/xuyaoqiang/elastalert-dingtalk-plugin/archive/master.zip
unzip master.zip
cd elastalert-dingtalk-plugin-master/
pip install pyOpenSSL==16.2.0
pip install setuptools==46.1.3
设置config文件
#vim config.yaml //修改es的IP和端口，如果ES设置了认证需要设置用户名和密码
es_host: 192.168.25.160
es_port: 9200
设置rule
以下是查询数据，发现用户访问有404即告警
ame: API error (staus >= 400)
type: frequency
index: nginx-prod-*
num_events: 1
timeframe:
minutes: 5
filter:
- term:
response: "404"
#只需要的字段 https://elastalert.readthedocs.io/en/latest/ruletypes.html#include
include: ["method", "url_path", "url_args", "response", "request_time"]
alert:
- "elastalert_modules.dingtalk_alert.DingTalkAlerter"
dingtalk_webhook: "https://oapi.dingtalk.com/robot/send?access_token=****************"
dingtalk_msgtype: "text"
启动插件
cd /home/dingding/elastalert-dingtalk-plugin-master
python -m elastalert.elastalert --config ./config.yaml --rule ./rules/api_error.yaml --verbose
8、测试
访问nginx上一个不存在地址
以下是告警内容
./rules/api_error
At least 1 events occurred between 2021-09-30 22:13 CST and 2021-09-30 22:18 CST
@timestamp: 2021-09-30T14:18:36Z
_id: 2VoSN3wB-MARRt7DZxkg
_index: nginx-prod-2021.09.30
_type: _doc
num_hits: 49
num_matches: 8
response: 404
#将告警中的时间戳修改成本地时间，在rule后添加，添加后由于日期格式不一致需要在ES中删除elastalert_status索引，该索引会自动重建
match_enhancements:
- "elastalert.enhancements.TimeEnhancement"
elastalert重启
#cd /root/media/dingding/elastalert-dingtalk-plugin-master
# python3 -m elastalert.elastalert --config ./config.yaml --rule ./rules/api_error.yaml
#nohup python3 -m elastalert.elastalert --verbose > elastalert.log 2>&1 &
钉钉接口测试python脚本
# filename: dingtalk.py
import requests
import json
import sys
def gaojing(data):
# 将消息提交给钉钉机器人
headers = {'Content-Type': 'application/json;charset=utf-8'}
# 注意替换钉钉群的机器人webhook
webhook = "https://oapi.dingtalk.com/robot/send?access_token=fc716b8d501a66345970599e7833c943cb2f66beea96a404d630dcc6945b3930&sign=SEC5b5fc4ddc31ded61bbd07b18e831b960f7be3562ca07e56f6b97e3b36b8ae8d8"
requests.post(url=webhook,data=json.dumps(data),headers=headers)
def get_data(text_content):
# 返回钉钉机器人所需的文本格式
text = {
"msgtype": "text",
"text": {
"content": text_content
},
}
# print(json.dumps(text))
return text
if __name__ == "__main__":
# 命令行第一个参数为告警内容
text_content = sys.argv[1]
data = get_data(text_content)
gaojing(data)
该脚本需要在命令行中运行：python dingtalk hello
安装参考：
https://blog.csdn.net/hljczm/article/details/109510750
https://github.com/Yelp/elastalert/issues/1675
创建elastalert索引：
https://elastalert.readthedocs.io/en/latest/running_elastalert.html#setting-up-elasticsearch
告警功能扩展：
