一、配置Logstash
input {
s3 {
aws_credentials_file => "/app/aws-credentials.yaml"
region => "us-west-1"
bucket => "aws-test-qa"
prefix => "QA_APILOG_2021-04/ip-10-0-1-10/2021-04-14"
sincedb_path => "/app/logstash/.qa-kong-logs-s3-sincedb"
type => "qa-kong-logs-s3"
}
}
filter {
if [type] == "qa-kong-logs-s3" {
grok {
match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA:msg_content}" }
}
json {
source => "msg_content"
remove_field => ["message","msg_content"]
}
date {
match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
target => "@timestamp"
remove_field => [ "timestamp" ]
}
if "_grokparsefailure" in [tags] {
drop { }
}
}
}
output {
if [type] == "qa-kong-logs-s3" {
elasticsearch {
hosts => ["localhost:9200"]
user => "qa"
password => "xxxxx"
index => "qa-kong-logs-s3-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
}
二、测试配置
sudo /usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/qa-kong-logs-s3.conf
几秒钟后，如果没有语法错误，它应该会显示Configuration OK。否则，尝试读取错误输出，查看Logstash配置有什么问题
三、重启Logstash
sudo systemctl restart logstash
sudo systemctl enable logstash
四、检查Elasticsearch中的索引
curl http://localhost:9200/_cat/indices/qa-kong-logs-s3*
