filebeat+logstash+ES+kibana收集nginx日志

1. filebeat

• filebeat安装

    rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
  

  添加repo源

    cd /etc/yum.repos.d/
    vim elastic.repo
    '''
      [elastic-7.x]
      name=Elastic repository for 7.x packages
      baseurl=https://artifacts.elastic.co/packages/7.x/yum
      gpgcheck=1
      gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
      enabled=1
      autorefresh=1
      type=rpm-md
    '''
  

  yum 安装

    yum install filebeat
    systemctl restart filebeat
  

• filebeat配置

  /etc/filebeat/filebeat.yml

    # nginx input 开启
    - type: log
  
      # Change to true to enable this input configuration.
      enabled: true
  
      # Paths that should be crawled and fetched. Glob based paths.
      paths:
          - /var/log/nginx/access_spark.log
          #- c:\programdata\elasticsearch\logs\*
    ......
    # logstatsh output开启
    output.logstash:
      # The Logstash hosts
      hosts: ["172.17.1.65:5044"]
  

• filebeat 调试技巧

  filebeat开启后，不会主动输出日志，需要手动运行filebeat，打开调试模式

    filebeat -e -d "*"
  

2.nginx

• 修改nginx配置

  /etc/nginx/

    log_format spark '$remote_addr $time_local $request_method $uri $status';
    access_log /var/log/nginx/access_spark.log spark;
  

3.logstash + ES + Kibana

• 安装

  直接docker安装，有elk的docker镜像

    docker pull sebp/elk
    
    docker run -d -p 5044:5044 -p 5601:5601 -p 9200:9200 -p 9300:9300 -v /var/data/elk:/var/lib/elasticsearch --mount type=bind,source=/var/log/nginx,target=/var/log/nginx  --name=elk sebp/elk
  
    docker exec -it elk  /bin/bash
  

• 修改配置

  修改logstash配置，接收filebeat的input, 输出到ES。
  /etc/logstash/conf.d

    input {
      beats {
        port => 5044
        ssl => false
        #ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
        #ssl_key => "/etc/pki/tls/private/logstash-beats.key"
      }
    }
  
  filter {
    grok {
      match => {"message" => "%{IPORHOST:remote_addr} %{HTTPDATE:time_local} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:http_version}"}
    }
    if "/static" in [request] or "/favicon.ico" in [request] {
      drop{}
    }
    if !("/login" in [request] or "/admin/" in [request] or "/msp/" in [request] or "/company/" in [request] or "/sso_login" in [request]){
      drop {}
    }
  }
  
  output {
  elasticsearch {
      hosts => ["localhost:9200"]
      index => "logstash-nginx-%{+YYYY.MM.dd}"
    }
  }
  

  运行logstash，貌似修改配置后service logstatsh restart 报错，手动运行

    nohup /opt/logstash/bin/logstash -f /etc/logstash/conf.d/beats-access.conf &
  

小结