filebeat安装配置

elasticsearch logstash kibana filebeat 最好版本一致 我使用的是7.6.2版本的

下载rpm包并安装

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.2-x86_64.rpm
sudo rpm -vi filebeat-7.6.2-x86_64.rpm

修改配置文件

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /data/docker/containers/*/*.log
  fields:
    log_source: bymex.dev
  fields_under_root: true
  multiline.pattern: ^\d{4}-\d{1,2}-\d{1,2}
  multiline.negate: true
  multiline.match: after
  scan_frequency: 5s
  close_inactive: 1h
  ignore_older: 24h
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
output.logstash:
  hosts: ["a192.168.1.2:5044"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: 
      host: "unix:///var/run/docker.sock"                                               
      match_source: true                                                
      match_source_index: 3 
  - add_kubernetes_metadata: ~
logging.level: info

修改logstash配置

在/etc/logstash/conf.d下添加文件docker.conf

input {
       beats {
               port => 5044
       }
}

output {
       if [log_source] == "tmex.dev" {
               elasticsearch {
                       hosts => ["es01.loc:9200","es02.loc:9200","es03.loc:9200"]
                       index => "tmex.dev-%{+YYYY.MM.dd}"
               }
       }
       if [log_source] == "tmex.pro" {
               elasticsearch {
                       hosts => ["es01.loc:9200","es02.loc:9200","es03.loc:9200"]
                       index => "tmex.pro-%{+YYYY.MM.dd}"
               }
       }
       if [log_source] == "bymex.dev" {
               elasticsearch {
                       hosts => ["es01.loc:9200","es02.loc:9200","es03.loc:9200"]
                       index => "bymex.dev-%{+YYYY.MM.dd}"
               }
       }
}

说明

在filebeat收集docker日志时，通过fields的log_source标记日志来源，我这里是bymex.dev，说明对日志来自bymex.dev环境。配置日志输出到logstash，填写logstash的IP和端口。在logstash的docker.conf配置文件中，监听来源于5044端口的beat日志，输出到elasticsearch存储的时候，通过判断log_source来设置不同的索引存储到es库，这样就通过不同的索引查找区分不同来源的日志。