logstash + influxdb监控nginx日志
成果
1.采用logstash收集/过滤数据
1.1 具体安装请看https://www.elastic.co/官方文档
1.2 logstash配置
1.2.1 从nginx收集数据，首先配置一下nginx的日志格式
log_format main '$remote_addr - $http_host $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" $request_time $upstream_addr';
1.2.2 我们需要通过grok-pattern正则来处理nginx的日志
在logstash/patterns下创建一个名为nginx文件，内容为：
需要注意的是：NGINXACCESS需要和nginx日志内容一一对应
WZ ([^ ]*)
URIPARM1 [A-Za-z0-9$.+!*'|(){},~@#%&/=:;^\\_<>`?\-\[\]]*
URIPATH1 (?:/[\\A-Za-z0-9$.+!*'(){},~:;=@#% \[\]_<>^\-&?]*)+
HOSTNAME1 \b(?:[0-9A-Za-z_\-][0-9A-Za-z-_\-]{0,62})(?:\.(?:[0-9A-Za-z_\-][0-9A-Za-z-:\-_]{0,62}))*(\.?|\b)
STATUS ([0-9.]{0,3}[, ]{0,2})+
HOSTPORT1 (%{IPV4}:%{POSINT}[, ]{0,2})+
FORWORD (?:%{IPV4}[,]?[ ]?)+|%{WORD}
URI1 (%{URIPROTO}://)?(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?
NGINXACCESS %{IPV4:remote_addr} - (%{HOSTNAME1:http_host}|-) (%{USERNAME:remote_user}|-) \[%{HTTPDATE:time_local}\] "%{WORD:method} %{WZ:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:body_bytes_sent} \"(?:%{GREEDYDATA:http_referer}|-)\" \"(%{GREEDYDATA:http_user_agent}|-)\" \"(%{FORWORD:http_x_forword_for}|-)\" (%{STATUS:request_time}|-) (?:%{HOSTPORT1:upstream_addr}|-)
1.2.3 在logstash/config下创个配置文件，这里是influxdb.conf
#输入
input {
file {
#我们从nginx的日志文件收集数据
path => ["/usr/local/openresty/nginx/xxx.log"]
type => "xxx"
start_position => "beginning"
}
}
#过滤
filter {
grok {
#指定路径下的正则
patterns_dir => "/***/logstash/patterns"
#通过NGINXACCESS的正则处理
match => ["message", '%{NGINXACCESS}']
}
}
#输出
output {
influxdb {
db => "metrics"
host => "127.0.0.1"
port => "8086"
user => "user"
password => "123456"
measurement => "apiTotal"
coerce_values => {
"costTime" => "float"
}
data_points => {
"costTime" => "%{request_time}"
"uri" => "%{request}"
#"status" => "%{status}"
#"referer"=>"%{referer}"
#"agent"=>"%{agent}"
#"method"=>"%{method}"
#"remote_ip"=>"%{remote_ip}"
#"bytes"=>"%{bytes}"
#"host"=>"%{host}"
#"timestamp"=>"%{timestamp}"
#"path"=>"%{path}"
}
#data_points里面若有同名的，则可以拿来做tag
send_as_tags => ["uri"]
}
stdout { codec => rubydebug }
}
2.用influxdb做数据的收集
2.1 logstash中influxdb插件的安装
#查看目前是否有influxdb的插件
bin/logstash-plugin list --group output
#如果没有的话，运行此命令安装
bin/logstash-plugin install logstash-output-influxdb
若出现相关报错
Unable to download data from https://ruby.taobao.org/ - certificate verify failed (https://ruby.taobao.org/latest_specs.4.8.gz)
ERROR: Installation aborted, verification failed for logstash-output-influxdb
更改logstash/Gemfile文件，使用我们国内的RubyGems镜像
image.png
2.2 logstash运行
#选中指定的配置运行，改了配置能够自动重启
bin/logstash -f /xxx/logstash/config/influxdb.conf --config.reload.automatic
