- 首先安装Java环境：
apt-get install openjdk-8-jdk - 在官网下载Logstash：
https://www.elastic.co/cn/downloads/logstash - 在conf目录新建一个配置文件：
input {
#这里可以同时监控多个文件
file {
path => ["/usr/local/nginx/logs/error.log"]
start_position => "beginning"
type => "error"
}
file {
path => ["/usr/local/nginx/logs/www.xxx.com.access.log"]
start_position => "beginning"
type => "access"
}
}
filter {
#每种文件需要配置自己的grok插件语法来搜集需要的数据
if [type] == "access"{
grok {
match => {
#这里的须发需要自定义配置
"message" => "^%{IPV4:remote_addr} \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{INT:status} %{INT:body_bytes_sent} \"%{NOTSPACE:http_referer}\" %{NUMBER:request_time} \"%{IPV4:upstream_addr}:%{POSINT:upstream_port}\" %{NUMBER:upstream_response_time} \"%{DATA:http_user_agent}\" \"%{NOTSPACE:http_x_forwarded_for}\""
}
}
#配置GeoIP的数据库解析ip
geoip {
source => "remote_addr"
}
}
}
output {
# 不满足筛选条件的就不写入
if "_grokparsefailure" not in [tags] {
#数据输出到elasticsearch
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logstash-nginx-%{type}-%{+YYYY-MM}"
}
}
#调试
stdout{codec => rubydebug}
}
NginxAccess日志格式：
log_format main '$remote_addr [$time_local] "$request" $status $body_bytes_sent "$http_referer" $request_time "$upstream_addr" $upstream_response_time "$http_user_agent" "$http_x_forwarded_for"';
日志示例：
69.126.145.85 [25/Jun/2018:07:31:27 +0000] "POST /api/userInfoRongCloud HTTP/1.1" 200 197 "-" 0.191 "18.191.5.101:9000" 0.191 "dating/1.0.5 (iPhone; iOS 12.0; Scale/3.00)" "-"
这一步的grok语法需要自己调试，在线调试地址：
https://grokdebug.herokuapp.com/
这里可以使用很多已经写好的模式变量，参见：
https://github.com/elastic/logstash/blob/1.4/patterns/grok-patterns
- 使用
-f参数指定配置文件测试解析结果是否正确 - 添加到守护进程中执行(正式环境需要把配置文件里面的调试输出去掉，不然会报错)：
[program:logstash-worker]
process_name=%(program_name)s_%(process_num)02d
command=/usr/local/logstash-6.3.0/bin/logstash -f /usr/local/logstash-6.3.0/config/nginx-access.conf
autostart=true
autorestart=true
user=root
numprocs=1
redirect_stderr=true
stdout_logfile=/var/log/logstash-worker.log
supervisorctl reread
supervisorctl update
