filebeat+logstash收集错误日志发送邮件提醒

20181109110339174.png

因为只收集错误日志并且数据量并不是非常大所以简化流程

使用filebeat+logstash发送异常日志

一、filebeat配置

1、将安装包解压到指定目录

drwxr-xr-x  3 root root     4096 Oct 28 09:46 ./
drwx------ 13 root root     4096 Nov  1 13:30 ../
drwxr-xr-x  7 root root     4096 Oct 31 16:30 filebeat-8.4.3-linux-x86_64/
-rw-r--r--  1 root root 60723429 Oct 28 09:46 filebeat-8.4.3-linux-x86_64.tar.gz

2、找到filebeat.yml文件

root@e:/# cd filebeat-8.4.3-linux-x86_64/
root@e:/# filebeat-8.4.3-linux-x86_64# ls
@  data  fields.yml  filebeat  filebeat.reference.yml  filebeat.yml  kibana  LICENSE.txt  logs  module  modules.d  nohup.out  NOTICE.txt  README.md

3、配置采集

参考文档：https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html

# ============================== Filebeat inputs ===============================
max_procs: 1 # 配置cpu核数 减少资源占用
queue.mem.events: 2048                  # 存储于内存队列的事件数，排队发送 (默认4096)
queue.mem.flush.min_events: 1536        # 小于 queue.mem.events ,增加此值可提高吞吐量 (默认值2048)

filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream  # 指定数据的输入类型
  ignore_older: 48h # 忽略这个时间之前的文件(根据文件改变时间)
  max_bytes: 20480  # *单条日志的大小限制,将其从默认10M降低到20k，按照公式计算 20k * 4096 ～= 80M
  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id
  # Change to true to enable this input configuration.
  enabled: true # 启用

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/logs/aaa/debug.log           # 日志路径
    #- c:\programdata\elasticsearch\logs\*
  tail_files: true   # 从日志最后一行开始读取
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  exclude_lines: ['DEBUG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  include_lines: ['ERROR','Exception']  #只处理包含ERROR 和 Exception日志
  parsers:
   - multiline:              #合并多行日志 
       pattern: '^\d{4}-\d{2}-\d{2}'       # 匹配以 YYYY-MM-DD HH:mm:ss 开头的行 
       negate: true                        # 是否匹配pattern的情况
       match: after                        # 将其追加到上一行之后 pattern + negate + match 组合成一条语意为: 如果匹配 YYYY-MM-DD HH:mm:ss 开头的行，则将其合并到当前行的上一行
       max_lines: 20                      # 最多匹配多少行，如果超出最大行数，则丢弃多余的行(默认500)
       timeout: 2s                         # 超时时间后,即使还未匹配到下一个行日志(下一个多行事件)，也将此次匹配的事件刷出 (默认5s)
  
# ------------------------------ Logstash Output -------------------------------
#指定输出的logstash地址
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.1.12:5044"]

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: info  #filebeat日志级别 调试时可以设为debug

4、启动

/filebeat-8.4.3-linux-x86_64# ./filebeat -e -c filebeat.yml    前台启动
/filebeat-8.4.3-linux-x86_64# nohup ./filebeat -e -c filebeat.yml &  后台启动

注：正常运行一段时间后filebeat自动关闭
解决：
1、使用service方式启动
2、在原来的命令之前加上disown参数，这个参数将会使启动的nohup进程从当前shell的作业列表中清除，从而避免nohup进程在关闭这个shell时接收到SIGHUP信号

nohup ./filebeat -e -c filebeat.yml & disown

二、logstash配置

1、将安装包解压到指定目录

drwxr-xr-x  4 root root      4096 Oct 31 14:13 ./
drwxr-xr-x 25 root root      4096 Oct 28 11:12 ../
drwxr-xr-x 14 root root      4096 Oct 31 16:29 logstash-8.5.3/
-rw-r--r--  1 root root 330116325 Oct 28 09:25 logstash-8.5.3-linux-x86_64.tar.gz

2、找到logstash-sample.conf 文件

root@e:/logstash-8.5.3/config# ls
jvm.options  log4j2.properties  logstash-sample.conf  logstash.yml  pipelines.yml  startup.options

3、配置

参考文档：https://www.elastic.co/guide/en/logstash/current/output-plugins.html

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
  beats {
    host => '192.168.1.12'  #本机ip
    port => 5044
  }
}

filter {
   # 匹配message中的时间格式化到logdate
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:logdate}"]
    }  
   # 排除 IdempotentException 异常（不处理该异常）
    if [message] =~ "IdempotentException" {
      drop { }
    }
}

output {
  # 输出到文件
  #file{
  # path => "/test/test.log"
  # codec => line {format => "%{message}"}
  # }
  
  # 接收参数输出到控制台
  #  stdout { codec => rubydebug }
  
  #输出到邮件 腾讯企业邮箱为例
    email {
        port => 587
        address => 'smtp.exmail.qq.com'
        username => '***@***.com'
        password => '******'      #授权码
        authentication => 'plain'
        from => '***@***.com'
        subject => '错误告警：IP:%{[host][ip][0]}, 时间:%{logdate}'
        to => '***@***.com'
        use_tls => true
        domain => 'smtp.exmail.qq.com'
        body => '[服务器信息]IP：%{[host][ip][0]}\n[错误信息]：%{[log][file][path]}\n%{message}'
    }
}

4、启动

/logstash-8.5.3# ./bin/logstash -f ./config/logstash-sample.conf     前台启动
/logstash-8.5.3# nohup ./bin/logstash -f ./config/logstash-sample.conf  后台启动

5、 logstash配置调优 (logstash.yml、jvm.options）

lostash.yml: 
pipeline.workers: 1 (不配置的情况下，默认是系统核数，控制output或filter插件的工作线程数（只能设置正整数），当发现事件正在备份或CPU没有饱和，则可以增加工作线程，以提高性能。) 

#内存大小 根据服务器的性能进行配置
jvm.options: 
-Xms1g 
-Xmx1g

6、邮件内容

d0aa7d04d1015dc48272f2828085bf73.png