一、部署忽略
参考
https://www.jianshu.com/p/a6084c9450c6
二、开启system模块
filebeat modules enable system
三、配置filebeat
filebeat.inputs:
- type: log
enabled: false
paths:
- /var/log/*.log
- type: filestream
enabled: false
paths:
- /var/log/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
host: "192.168.18.13:5601"
setup.template.overwrite: true
setup.template.enabled: true
setup.ilm.enabled: false
output.kafka:
hosts: ["192.168.18.15:9092","192.168.18.16:9092"]
topic: "linux"
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
四、配置logsatsh
1、input.conf
input {
kafka {
bootstrap_servers => "192.168.18.15:9092,192.168.18.16:9092"
topics => ["linux"]
add_field => { type_name => "linux_system" }
consumer_threads => 5
codec => json
}
}
2、output.conf
output {
elasticsearch {
hosts => ["192.168.18.13:9200"]
index => "test-os-linux-%{+YYYY.MM.dd}"
}
}
五、在kibana中查看索引和数据
image.png
image.png
日志时间为logstash收到的时间，后面优化
六、在kafka中查看消息的命令
/etc/kafka/kafka_2.12-2.2.2/bin/kafka-console-consumer.sh --bootstrap-server 192.168.18.16:9092 --topic linux --from-beginning
