部署方案

根据houyi平台规划，日志归集相关的技术组件都是基于k8s进行部署。
相关部署文件列表如图所示：

image.png

对应各个技术组件的部署以最简单的单点方式部署，暂时不考虑高可用性及性能弹性伸缩等非功能需求。

elasticsearch部署

Elasticsearch Deployment部署

该部署比较简单，就是对外提供9200，9300两个端口提供服务，也没有考虑数据存储到容器外。

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: elasticsearch-deployment
  namespace: t-paas
  labels:
    layer: t-paas
    env: all
    app: elasticsearch
    name: elasticsearch-server
spec:
  replicas: 1
  template:
    metadata:
      labels:
        layer: t-paas
        env: all
        app: elasticsearch
        name: elasticsearch-server
    spec:
      containers:
      - name: elasticsearch
        image: elasticsearch:6.6.2
        ports:
          - containerPort: 9200
          - containerPort: 9300
        env:
          - name: discovery.type
            value: "single-node"
          - name: http.cors.enabled
            value: "true"
          - name: http.cors.allow-origin
            value: "*"

Elasticsearch Service部署

创建一个ClusterService, K8S内服务可以通过服务名称elasticsearch-service或一个固定IP访问es。

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-service
  namespace: t-paas
spec:
  ports:
  - name: p9200
    port: 9200 
    targetPort: 9200
  - name: p9300
    port: 9300 
    targetPort: 9300
  selector:
    layer: t-paas
    env: all
    app: elasticsearch
    name: elasticsearch-server

Logstash 部署

Logstash Config

自定义logstash logstash.conf配置文件及logstash.yml文件，配置如下所示：
logstash.conf主要配置为：

输入数据由filebeat通过5044端口发送过来。
如果输入数据的标签为api-gateway，则将数据存入到elasticsearch中（通过es的9200数据端口传输）。
并且在es中根据日期创建api-gateway索引，并将数据存入到该索引中。

logstash.yml的配置主要是实现与es之间的心跳。（通过监控日志及名称猜测是这个作用，并没有核实，如理解有误望指点。）

kind: ConfigMap
apiVersion: v1
metadata:
  name: logstash-conf
  namespace: t-paas
data:
  logstash.conf: |-
    input {
     beats {
      port => 5044
     }
    }

    output {
      if "api-gateway" in [tags] {
      elasticsearch {
       hosts => ["elasticsearch-service:9200"]
       index => "api-gateway-%{+YYYY.MM.dd}"
     }

    }
    }
  logstash.yml: |-    
    http.host: "0.0.0.0"
    xpack.monitoring.elasticsearch.url: http://elasticsearch-service:9200

Logstash Deployment

logstash 容器启动logstash，并指定配置文件，打开5044端口。

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: logstash
  namespace: t-paas
spec:
  template:
    metadata:
      labels:
        type: logstash
    spec:
      containers:
      - image: docker.elastic.co/logstash/logstash:6.6.2
        name: logstash
        ports:
        - containerPort: 5044
        command:
        - logstash
        - '-f'
        - '/etc/logstash_c/logstash.conf'
        volumeMounts:
        - name: config-volume
          mountPath: /etc/logstash_c/
        - name: config-logstash-yml
          mountPath: /usr/share/logstash/config/logstash.yml
          subPath: logstash.yml
      volumes:
      - name: config-volume
        configMap:
          name: logstash-conf
          items:
          - key: logstash.conf
            path: logstash.conf
      - name: config-logstash-yml
        configMap:
          name: logstash-conf
          items:
          - key: logstash.yml
            path: logstash.yml

logstash-service

apiVersion: v1
kind: Service
metadata:
  name: logstash-svc
  namespace: t-paas
spec:
  ports:
  - name: p5044
    port: 5044 
    targetPort: 5044
  selector:
    type: logstash

Filebeat部署

filebeat config

通过filebeat config定义filebeat配置文件filebeat.yml：

配置关注/app-logs/api-gateway/路径下的日志变化。
将以"["开头的行合并或识别为一条记录。
为记录添加标签"api-gateway"。
记录级别为info
将记录通过logstash的service名称及端口发送给logstash。

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: t-paas
data:
  filebeat.yml: |-
    filebeat.prospectors:
    - type: log
      paths:
        - /app-logs/api-gateway/*.log
      multiline:
        pattern: '^\['
        negate:  true
        match:   after        
      tags: ["api-gateway"]
      fields_under_root: true
      fields:
        level: info

    output.logstash:
      hosts: ['logstash-svc:5044']

filebeat deployment

根据filebeat 配置，读取指定路径下的增量日志信息，

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: t-paas
  labels:
    logs: filebeat
spec:
  template:
    metadata:
      labels:
        logs: filebeat
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:6.6.2
        args: [
          "-c", "/usr/share/filebeat.yml",
          "-e",
        ]
        volumeMounts:
        - name: config
          mountPath: /usr/share/filebeat.yml
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: app-logs
          mountPath: /app-logs
      volumes:
      - name: config
        configMap:
          name: filebeat-config
      - name: data
        emptyDir: {}
      - name: app-logs
        hostPath:
          path: /d/home/k8s/containers/houyi-platform/application

微服务治理-日志归集-环境搭建(开发及测试）