Prometheus+Grafana+Loki监控容器服务/收集日志

概述

Grafana：以图表的形式展示Prometheus数据，和查看loki的日志数据,实现邮箱，钉钉告警

Loki：接收日志，使用promtail收集发送日志到Loki

Prometheus：监控服务，可以配置各种监控服务，例如监控docker的cadvisor，监控主机node-exporter.

展示效果图

容器监控

image

日志查看

image

部署服务结构图

image

服务端部署

部署采用docker-compose方式部署

|-- docker-compose.yaml
`-- prometheus
    |-- config
    |   `-- prometheus.yml        
    |-- data                             #保存监控数据目录

docker-compose文件如下

version: '3.4'
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    network_mode: host
    user: root
    hostname: prometheus
    restart: unless-stopped
    cpus: '0.5'
    mem_limit: 500m
    volumes:
      - /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime:ro
      - ./prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/data:/prometheus
    command:
      - '--web.enable-lifecycle'
      - '--web.enable-admin-api'
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention=30d'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    logging:
      driver: json-file
      options:
        max-file: '3'
        max-size: 100m
  loki:
    image: grafana/loki:2.0.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"

prometheus.yml配置文件

global:
  scrape_interval:     15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: 'prometheus-service'
    file_sd_configs:
     - files: ['/etc/prometheus/discovery/prometheus.yml']
       refresh_interval: 5s
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['192.168.1.5:8080']  #填写cadvisor所在的服务器ip端口，多个用 , 隔开

配置完即可以启动监控主机上的服务
docker-compose up -d

客户端部署服务

客户端的主机上需要安装cadvisor监控容器服务，和promtail收集日志
机构目录

|-- docker-compose.yml
`-- promtail
    `-- config.yaml

docker-compose文件

version: "3.4"
services:
  promtail:
    image: grafana/promtail:2.0.0
    container_name: promtail
    cpus: '0.5'
    mem_limit: 500m
    command: -config.file=/etc/promtail/config.yml
    volumes:
      - ./promtail/config.yaml:/etc/promtail/config.yml
      - ../logs:/opt/logs     #需要收集日志的目录映射进来
  cadvisor:
    image: google/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    cpus: '0.5'
    mem_limit: 500m
    ports:
      - '8080:8080'
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime:ro

容器化部署，需要把要收集的日志的目录映射到 promtail容器里面去，再发送到loki
promtail配置文件

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push  #输入上面loki服务对应的ip+端口

scrape_configs:
- job_name: app_server    #查看日志时可以根据name和labels来选择对应的日志
  static_configs:
  - targets:
      - localhost
    labels:
      job: app_server
      __path__: /opt/logs   #日志文件映射进来之后的目录

启动服务
docker-compose up -d
服务启动后可以登录prometheus来查看对应的job是不是正常工作