更新时间：
yum install ntpdate -y
ntpdate time1.aliyun.com
*/5 * * * * /usr/sbin/ntpdate ntp3.aliyun.com >/dev/null 2>&1
image.png
1.ELK介绍：
E:elasticsearch ##存储+搜索
F:filebeat ##日志收集
L:logstash ##过滤筛选
K:kibana ##图形化展示
1.1日志介绍：
/var/log/messages ##系统日志
/var/log/secure ##安全日志
web层面日志：
nginx /var/log/nginx/access.log ##访问日志
nginx /var/log/nginx/error.log ##错误日志
tomcat
php
apache
数据库层：
MySQL
redis
mongoDB
elasticsearch
2.收集nginx日志
准备环境
下载nginx
yum install nginx -y
启动nginx
systemctl start nginx
3.上传filebeat软件
rpm -ivh filebeat-6.6.0-x86_64.rpm
4.编辑filebeat配置文件
先做个备份需要配置再添加
cp /etc/filebeat/filebeat.yml{,.bak}
vim /etc/filebeat/filebeat.yml
filebeat.inputs: ### 输入
- type: log ### 输入得类型
enabled: true
paths:
- /var/log/nginx/access.log ### 收集得日志路径
json.keys_under_root: true ### 解析成json格式
json.overwrite_keys: true
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings: ### 模板设计
index.number_of_shards: 3 ### 默认分片
setup.kibana:
output.elasticsearch: ### 输出到哪里
hosts: ["10.0.0.41:9200"] ### 输出到ES
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
启动filebeat
systemctl start filebeat
filebeat收集日志原理：
1.类似于tail -f
2.每三十秒检测一次日志有没有发生变化
停掉filedeat，再启动，中间得数据如何处理：
从断开得地方开始读取。
以上不完善得地方：
1.不能单独统计展示访问网站得某项内容
2.访问网站所用得终端
nginx的原日志格式：
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
10.0.0.1 - - [10/Jul/2019:18:02:58 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Saf
ari/537.36" "-"
期望的格式：
$remote_addr：10.0.0.1
$remote_user：-
[$time_local]：[10/Jul/2019:17:59:52 +0800]
$request：GET/db01.html HTTP/1.1"
$status ：404
$body_bytes_sent：3650
$http_referer：-
$http_user_agent：Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36
$http_x_forwarded_for：-
Json格式的修改
（所有nginx节点都需操作）
log_format json '{ "time_local": "$time_local", '
'"remote_addr": "$remote_addr", '
'"referer": "$http_referer", '
'"request": "$request", '
'"status": $status, '
'"bytes": $body_bytes_sent, '
'"agent": "$http_user_agent", '
'"x_forwarded": "$http_x_forwarded_for", '
'"up_addr": "$upstream_addr",'
'"up_host": "$upstream_http_host",'
'"upstream_time": "$upstream_response_time",'
'"request_time": "$request_time"'
' }';
access_log /var/log/nginx/access.log json;
现在的日志格式：
{ "time_local": "10/Jul/2019:22:29:20 +0800", "remote_addr": "10.0.0.1", "referer": "-", "request": "GET / HTTP/1.1", "status": 304, "bytes": 0, "agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36", "x_forwarded": "-", "up_addr": "-","up_host": "-","upstream_time": "-","request_time": "0.000" }
5.上传kibana
rpm -ihv kibana-6.6.0-x86_64.rpm
6.修改kibana配置文件
[root@lixin03 ~]# grep -Ev "#|^$" /etc/kibana/kibana.yml
server.port: 5601
server.host: "10.0.0.5"
server.name: "lixin03"
elasticsearch.hosts: ["http://10.0.0.41:9200"]
systemctl restart kibana
