背景:
在网上看过filebeat的很多文档,可能因为时间原因或者filebeat使用者较少吧,总之不能满足我的需求,所以记录下,希望能帮到有需要的人吧.奥力给!
一共分为两部分 第一部分为基础说明,第二部分为真实演示流程,第三部分为碰到过的问题收集
一 基础说明:
1 下载安装filebeat,可以看之前的记录
2 默认配置文件为filebeat.reference.yml,正式使用配置文件为filebeat.yml
3 配置文件基础解读:
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
#每收集一个日志,需要配置一个type
- type: log
enabled: true
paths:
- /home/work/dsp/log/webserver/access_log #日志路径
fields:
type: "php-nginx-access" #自定义字段
close_renamed: true #日志替换名字时,停止采集
close_removed: true #日志移动时.停止采集
scan_frequency: 10s #频率 10s去检测日志更新
#整块注释如上#
- type: log
enabled: true
paths:
- /home/work/dsp/log/service/service.log
fields:
type: "service-log"
close_renamed: true
close_removed: true
scan_frequency: 10s
#整块注释如上#
#- type: log
# enabled: true
# paths:
# - /home/work/dsp/log/webserver/error_log.2019111217
# fields:
# type: "nginx-error"
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
#setup.template.settings:
# index.number_of_shards: 3
#index.number_of_replicas: 2
# index:
# number_of_shards: 3
# #codec: best_compression
# #number_of_routing_shards: 30
#setup.template.settings:
#index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
host: "10.19.145.2:5601"
username: "YouName"
password: "YourPassword"
# Kibana Host
#-------------------------- Elasticsearch output ------------------------------
setup.ilm.enabled: false
setup.template.name: "php-nginx" # 给咱的模板起个名字,随便喊
setup.template.pattern: "php-nginx-*" #调取的正则名称
setup.template.settings: # 配置生成索引的分片与副本数
index.number_of_shards: 3
index.number_of_replicas: 1
setup.template.overwrite: true
setup.template.enabled: true
setup.template.name: "php-service"
setup.template.pattern: "php-service-*"
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 1
setup.template.overwrite: true
setup.template.enabled: true
#es 集群配置
output.elasticsearch:
hosts: ["ip1:9200", "ip2.15:9200", "ip3:9200"]
username: "youName"
password: "YouPassword"
index: "php-nginx-%{+yyyy.MM.dd}"
indices:
- index: "php-nginx-%{+yyyy.MM.dd}"
when.equals:
fields.type: "php-nginx-access"
- index: "php-service-%{+yyyy.MM.dd}"
when.equals:
fields.type: "service-log"
pipelines:
- pipeline: "php-nginx-access"
when.equals:
fields.type: "php-nginx-access"
- pipeline: "service-log"
when.equals:
fields.type: "service-log"
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
二 真实演示流程:
完整流程举例: 添加nginx接入层日志采集
1 在filebeat.yml 添加type
- type: log
enabled: true
paths:
- /home/work/nginx/logs/access_log
fields:
type: "nginx-access"
close_renamed: true
close_removed: true
scan_frequency: 10s
exclude_lines: ['\/static\/img\/', '\/static\/js\/', '\/static\/css\/', '\/static\/fonts\/'] #目的是为了过滤css,js等你想过滤的东西
2 在filebeat.yml 添加template
setup.template.overwrite: true
setup.template.name: "nginx-flow"
setup.template.pattern: "nginx-flow-*"
setup.template.fields: ${path.config}/nginx-template.yml
setup.template.enabled: false
3 构建es管道
PUT _ingest/pipeline/nginx-flow
参数为:
{
"nginx-access" : {
"description" : "nginx-flow",
"processors" : [
{
"grok" : {
"ignore_failure" : true,
"field" : "message",
"patterns" : [
"""%{IPV4:remote_addr_ip} - (%{USERNAME:user_name}|-) \[%{HTTPDATE:log_timestamp:date}\] \"(?<method>[A-Z]+) (?<request_uri>[\s\S]*) (?<proto>[A-Za-z]+([A-Za-z0-9+\-./]+)+)\" (?<status>\d+) (?<body_size>\d+) \"(?<refer>(.*?))\" \"(?<cookie>(.*?))\" \"(?<user_agent_info>(.*?)*)\" %{NUMBER:cost_time} %{IPV4:remote_addr} %{IPV4:server_addr} (?<sock_path>[a-z0-9\.]+:(/?[a-z0-9\-]+)+(\.sock)?|-) (?<service_name>[a-z\d\.]+) \"(?<forwarded_foo>(.*?))\" (?<log_id>[\w\-]*) (?<log_id>[\w\-]*) %{NUMBER:time} %{NUMBER:time}"""
]
}
},
{
"date" : {
"ignore_failure" : true,
"field" : "log_timestamp",
"formats" : [
"dd/MMM/yyyy:HH:mm:ss Z"
],
"timezone" : "Asia/Shanghai"
}
}
]
}
}
4 在filebeat.yml 添加索引与es管道配置
output.elasticsearch:
hosts: ["ip1:9200", "ip2:9200", "ip3:9200"]
username: "YouName"
password: "YouPassword"
index: "php-nginx-%{+yyyy.MM.dd}"
indices:
- index: "php-nginx-%{+yyyy.MM.dd}"
when.equals:
fields.type: "php-nginx-access"
- index: "php-service-%{+yyyy.MM.dd}"
when.equals:
fields.type: "service-log"
#在原有配置上的新增行1 start ###
- index: "nginx-flow-%{+yyyy.MM.dd}"
when.equals:
fields.type: "nginx-access"
#在原有配置上的新增行1 end ###
pipelines:
- pipeline: "php-nginx-access"
when.equals:
fields.type: "php-nginx-access"
- pipeline: "service-log"
when.equals:
fields.type: "service-log"
#在原有配置上的新增行2 start ###
- pipeline: "nginx-access"
when.equals:
fields.type: "nginx-access"
#在原有配置上的新增行2 end ###
5 重启filebeat服务
supervisorctl restart filebeat
# 或者
./filebeat -c filebeat.yml -e -d '*'
三 碰到过的问题:
1 es 集群 cpu负载过高,集群内每个节点的cpu负载都接近100%
解决手段: 1 在es管道配置中新增如下图1中标红参数.
2 调整es的jvm gc参数,如下图2中
1.png
2.png
2 当采集日志数量>=2时,filebeat设置es索引分片与副本数失败
解决手段: 采集日志数量>=2时,es的分片与副本数量收集需要在不同的模板下分别设置,否则会使设置参数失效.如下图所示:
3.png
