日志收集平台 ELK stack方案学习笔记

ELK Stack解决方案

Elastic Search + LogStash + Kibana =ELK Stack

https://elastic.co
https://www.elastic.co/guide/index.html

Elasticsearch 概念

https://baike.baidu.com/item/elasticsearch/3411206?fr=aladdin

Elasticsearch安装方法

https://www.elastic.co/guide/en/elasticsearch/reference/5.5/rpm.html

yum install java -y
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/elasticsearch.repo << EOF
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

yum install -y elasticsearch

[root@jump elasticsearch]# grep ^[a-z] elasticsearch.yml 
cluster.name: fbo
node.name: linux-node-1
path.data: /data/es-data
path.logs: /var/log/elasticsearch/
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
transport.host: localhost
transport.tcp.port: 9300

Elasticsearch查询

支持 java api，RESTful API

RESTful API 查询

# curl -i -XGET 'http://192.168.57.100:9200/_count?pretty' -d'{ 
"query":{"match_all":{}}}'
HTTP/1.1 200 OK
Warning: 299 Elasticsearch-5.5.1-19c13d0 "Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header." "Tue, 08 Aug 2017 03:57:56 GMT"
content-type: application/json; charset=UTF-8
content-length: 95

{
  "count" : 0,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "failed" : 0
  }
}

安装插件工具Marvel插件

cd /usr/share/elasticsearch
sudo bin/plugin install license
sudo bin/plugin install marvel-agent
sudo bin/plugin install mobz/elasticsearch

这个插件5.0之后的版本包括在x-pack下，安装方法如下：
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/installing-xpack-es.html

安装开源插件elasticsearch-head
https://github.com/mobz/elasticsearch-head#running-with-built-in-server

安装开源插件bigdesk
https://github.com/lukas-vlcek/bigdesk
安装开源插件kopf
https://github.com/lmenezes/elasticsearch-kopf

logstash

安装logstash

https://www.elastic.co/guide/en/logstash/5.5/installing-logstash.html

bin/logstash -e 'input { stdin { } } output { stdout {} }'
bin/logstash -e 'input {stdin{}} output {elasticsearch{hosts => ["192.168.57.100:9200"]} stdout{codec => rubydebug}}'

配置logstash

https://www.elastic.co/guide/en/logstash/5.5/configuration.html

input 插件

https://www.elastic.co/guide/en/logstash/5.5/input-plugins.html

codec插件

https://www.elastic.co/guide/en/logstash/5.5/codec-plugins.html

file java nginx syslog tcp

json格式收集nginx日志

nc 192.168.57.100 6666 < /etc/resolv.conf
伪终端
echo "oldboy" > /dev/tcp/192.168.57.100/6666

filter插件

https://www.elastic.co/guide/en/logstash/5.5/filter-plugins.html
grok插件
https://www.elastic.co/guide/en/logstash/5.5/plugins-filters-grok.html
https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

ELK架构设计

解耦之消息队列

image.png

经典设计架构