filebeat
官网
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html
下载
https://www.elastic.co/cn/downloads/beats/filebeat
这里我们选择匹配的版本：Linux aarch64
filebeat-8.6.2-linux-arm64.tar.gz
安装
tar -zxvf filebeat-8.6.2-linux-arm64.tar.gz filebeat-8.6.2-linux-arm64
启动
./filebeat -e -c filebeat.yaml
配置项
输入项 Inputs
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html
输出项 Output
https://www.elastic.co/guide/en/beats/filebeat/current/configuring-output.html
部分配置项
通用设置
https://www.elastic.co/guide/en/beats/filebeat/8.7/filtering-and-enhancing-data.html
processors: # 处理器 必须在配置的顶层
# 在这里可以设置要去除的字段
- drop_fields:
# when: 可以设置去除的条件
# condition
fields: ["log","host","input","agent","ecs"]
ignore_missing: false #如果当指定的字段不存在时 处理器不会返回错误
name: "my-beat" #beat名称，无设置则取hostname agent.name
tags: ["my-service","hardware","test"]
实例
- 用于监听
pd_cd_server.log文件的的数据，将发送到控制台console
- 定义配置文件
filebeta.yaml
filebeat.inputs: #定义输入配置
- type: log #文件类型为log
paths: #文件路径
- /opt/cd/log/pd_cd_server.log
- /var/log/supervisor/*.log
multiline:
type: pattern #定义要使用的聚合方法。正则表达式
pattern: '^\[' #正则表达式
negate: true #默认是false，匹配pattern的行合并到上一行
match: after #指定 Filebeat 如何将匹配的行合并到事件中把多行合并成一个事件
output.console: #定义输出
pretty: true
enable: true
- 启动该配置文件
./filebeat -e -c filebeat.yaml
- 返回打印控制台
{
"@timestamp": "2023-04-06T03:18:28.411Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.6.2"
},
"log": {
"file": {
"path": "/opt/cd/log/pd_cd_server.log"
},
"offset": 6374
},
"message": "INFO: 192.168.0.181:10173 - \"GET /docs HTTP/1.1\" 200 OK",
"input": {
"type": "log"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "Test_007"
},
"agent": {
"name": "Test_007",
"type": "filebeat",
"version": "8.6.2",
"ephemeral_id": "e9e3e7f9-dfea-4d15-9abf-fd5e2d93e0cb",
"id": "05f3a3cf-de63-447d-aa38-985a81983e26"
}
}
logstash
官网
https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html
tar -zxvf logstash-7.14.0-linux-x86_64.tar.gz
运行
./bin/logstash -f config/logstash-sample1.conf
配置
#接收beat数据，标准输出控制台
input { #从filebeat取数据，端口与filebeat配置文件一致
beats {
port => 9022
}
}
filter {
if [filetype] == "log_pd_cd"{
json {
source => "message"
remove_field => ["log","offset","tags","instance_id"] #移除字段，不需要采集
}
}
}
output {
stdout {}
}
output {
elasticsearch {
hosts => [ "192.168.0.3:9200" ]
index => "first-9an--%{+YYYY.MM.dd}"
}
}
