实现思路:
1.硬件端日志访问nginx附带json格式日志
2.后通过logstash实时读取nginx日志存储于mysql中
1.nginx安装和配置
1.1 point-record为自定义nginx服务名
docker run -p 80:80 --name=point-record -v /opt/point-record/conf.d:/etc/nginx/conf.d -v /opt/point-record/web:/etc/nginx/web -v /opt/nginx-log/point-record/:/var/log/nginx/ -v /etc/timezone:/etc/timezone -v /etc/localtime:/etc/localtime -d nginx
1.2 在/opt/point-record/conf.d创建配置文件point.conf
cd /opt/point-record/conf.d
nano point.conf
point.conf:
log_format json escape=json '{"timestamp":"$time_iso8601",'
'"version":"1",'
'"client":"$remote_addr",'
'"url":"$uri",'
'"status":"$status",'
'"domain":"$host",'
'"host":"$server_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"referer":"$http_referer",'
#消息体打印
'"body":$request_body,'
'"ua": "$http_user_agent"'
'}';
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /etc/nginx/web;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_redirect off;
#必须经过一次转发,否则nginx不会读取消息体,打印的request_body为空
proxy_pass $scheme://127.0.0.1:$server_port/success;
access_log /var/log/nginx/access_json.log json;
}
location /success {
return 200;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
2.logstash安装与配置
2.1 官网下载 logstash-7.5.1.tar.gz 上传服务器并解压
tar -xzvf logstash-7.5.1.tar.gz -C /usr/local/
cd /usr/local/
#修改文件夹名称 ("logstash7" 是为了区分版本,可以同时安装多个版本)
mv logstash-7.5.1 logstash7
cd logstash7
目录下文件:
image.png
2.2 logstash原生版本只有input中自带jdbc插件,这意味着只支持jdbc导入不支持jdbc导出,需要额外安装网友的自制插件:
2.2.1 logstash配置mysql-connector-java包
MySQL Connector/J是MySQL官方JDBC驱动程序，JDBC（Java Data Base Connectivity,java数据库连接）是一种用于执行SQL语句的Java API，可以为多种关系数据库提供统一访问，它由一组用Java语言编写的类和接口组成。
官方下载地址：https://dev.mysql.com/downloads/connector/
下载地址：https://dev.mysql.com/downloads/connector/j/
mkdir -p /usr/local/logstash7/jdbc
cd /usr/local/logstash7/jdbc
# 上传mysql-connector-java包
rz
tar -xzvf mysql-connector-java-8.0.19.tar.gz
2.2.2 安装 logstash-output-jdbc插件
更改gem源：
国外的gem源由于网络原因，从国内访问太慢而且不稳定，还经常安装不成功
cd /usr/local/logstash7/
在安装 目录下找到Gemfile文件
image.png
编辑该文件将source改为 https://gems.ruby-china.com
image.png
修改完成后运行安装命令
/usr/local/logstash7/bin/logstash-plugin install logstash-output-jdbc
运行后输出Installation successful表示成功
image.png
2.3 配置config文件
在 /usr/local/logstash7/config/目录下创建nginx.conf文件
input {
file {
path => ["/opt/nginx-log/point-record/access_json.log"]
type => "nginx"
}
}
filter {
mutate {
#先替换掉字符串当中的特殊字符在做json转换
gsub =>["message", "\\n", ""]
gsub =>["message", "\\r", ""]
gsub =>["message", "\\t", ""]
gsub =>["message", "[\\]", ""]
}
json {
source => "message"
}
mutate {
gsub =>["timestamp", "T", " "]
gsub =>["timestamp", "\+08:00", ""]
add_field => {"pass_flag"=>"0"}
add_field =>{"appId"=>"%{[body][appID]}"}
add_field =>{"appType"=>"%{[body][appType]}"}
add_field =>{"appVer"=>"%{[body][appVer]}"}
add_field =>{"userID"=>"%{[body][userID]}"}
add_field =>{"sessionID"=>"%{[body][sessionID]}"}
add_field =>{"deviceID"=>"%{[body][deviceID]}"}
add_field =>{"deviceType"=>"%{[body][deviceType]}"}
add_field =>{"OSVer"=>"%{[body][OSVer]}"}
add_field =>{"lang"=>"%{[body][lang]}"}
add_field =>{"deviceInfo_devRes"=>"%{[body][deviceInfo][devRes]}"}
add_field =>{"deviceInfo_auxRes"=>"%{[body][deviceInfo][auxRes]}"}
add_field =>{"sign"=>"%{[body][sign]}"}
add_field =>{"ntime"=>"%{[body][ntime]}"}
add_field =>{"eventId"=>"%{[body][eventId]}"}
add_field =>{"arg1"=>"%{[body][arg1]}"}
add_field =>{"arg2"=>"%{[body][arg2]}"}
add_field =>{"arg3"=>"%{[body][arg3]}"}
add_field =>{"arg4"=>"%{[body][arg4]}"}
add_field =>{"arg5"=>"%{[body][arg5]}"}
add_field =>{"arg6"=>"%{[body][arg6]}"}
add_field =>{"arg7"=>"%{[body][arg7]}"}
add_field =>{"arg8"=>"%{[body][arg8]}"}
add_field =>{"arg9"=>"%{[body][arg9]}"}
add_field =>{"arg10"=>"%{[body][arg10]}"}
}
ruby {
code =>"
salt='XXXXX.com@0723'
appid = event.get('appId')
appType =event.get('appType')
appVer=event.get('appVer')
userID=event.get('userID')
sessionID=event.get('sessionID')
deviceID = event.get('deviceID')
target = '' << appType << appid << appVer << userID << salt << sessionID << deviceID
event.set('md5_value',target)
require 'digest/md5';
md5id = Digest::MD5.hexdigest(target)
event.set('md5id',md5id)
if md5id== event.get('sign')
event.set('pass_flag','1')
end
"
}
mutate {
remove_field =>["body","@timestamp","message"]
}
}
output {
stdout {codec => "rubydebug"}
if [pass_flag] == "1" {
jdbc {
driver_jar_path => "/usr/local/logstash7/jdbc/mysql-connector-java-8.0.19/mysql-connector-java-8.0.19.jar"
driver_class => "com.mysql.jdbc.Driver"
#数据库连接
connection_string => "jdbc:mysql://rm-bp131frjzi5x1h304.mysql.rds.aliyuncs.com:3306/edu-big-class-test?user=root&password=sairobo@mxxz3rzzz"
#insert语句
statement => [ "insert into point_record (`client_ip`,`svrtime`,`app_type`,`app_id`,`app_ver`,`user_id`,`session_id`,`device_id`,`device_type`,`os_ver`,`lang`,`device_dev_res`,`device_aux_res`,`ntime`,`event_id`,`arg1`,`arg2`,`arg3`,`arg4`,`arg5`,`arg6`,`arg7`,`arg8`,`arg9`,`arg10`) value(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
"client","timestamp","appType","appId","appVer","userID","sessionID","deviceID","deviceType","OSVer","lang","deviceInfo_devRes","deviceInfo_auxRes","ntime","eventId","arg1","arg2","arg3","arg4","arg5","arg6","arg7","arg8","arg9","arg10"]
}
}
}
3 启动
前台启动命令如下,
/usr/local/logstash7/bin/logstash -f /usr/local/logstash7/config/nginx.conf
结束: Ctrl+c
后台启动
nohup /usr/local/logstash7/bin/logstash -f /usr/local/logstash7/config/nginx.conf >/dev/null &
结束:
jobs -l
image.png
获取到pid为28978
kill -9 28978
使用postman模拟测试
image.png
每次send请求模拟一次硬件端的日志请求
