prometheus-operator 之监测外部机器，自定义监控项，新增告警

1.1.1.5 内网服务器 k8s master
1.1.1.6 内网服务器 k8s node
1.1.1.7 内网服务器 k8s node
1.1.1.8 内网服务器 k8s node
1.1.1.1 外部机器

1.node_exporter制作(1.1.1.1)

1.1镜像制作

此步骤是为了开启 自定义监控项

cd /home/jenkins/docker/node_exporter
touch Dockerfile

cat Dockerfile

FROM 1.1.1.1:5000/node-exporter:v1.3.1
ENTRYPOINT ["/bin/node_exporter", "--collector.textfile.directory=/textfile"]

docker build -t  1.1.1.1:5000/node-exporter-client:v1.0.0 .

1.2镜像启动

/home/jenkins/prometheus/textfile即为自定义监控项目录，其下配置文件中符合规律监控项都会被自动发现

cd /home/jenkins/docker/node_exporter
touch node_exporter_start.sh

cat node_exporter_start.sh

docker run -d --net="host" --name node_exporter --restart=unless-stopped -p 9100:9100 \
-v "/proc:/host/proc:ro" \
-v "/sys:/host/sys:ro" \
-v "/:/rootfs:ro" \
-v "/home/jenkins/prometheus/textfile:/textfile:ro" \
1.1.1.1:5000/node-exporter-client:v1.0.0

sh node_exporter_start.sh

例：cat /home/jenkins/prometheus/textfile/port.prom

port_8080 1

2.prometheus-operator配置(1.1.1.5)

mkdir -p /home/kubernetes/k8s/app/external_machine_monitor
touch external_machine_monitor.yaml

cat external_machine_monitor.yaml

kind: Service
apiVersion: v1
metadata:
 name: external-machines
 namespace: kube-system
 labels:
   k8s-app: external-machines
spec:
 type: ClusterIP
 clusterIP: None
 ports:
 - name: metrics
   port: 9100

---
kind: Endpoints
apiVersion: v1
metadata:
 name: external-machines
 namespace: kube-system
 labels:
   k8s-app: external-machines
subsets:
 - addresses:
     - ip: 1.1.1.1
   ports:
       - name: metrics
         port: 9100

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: external-machines
  name: external-machines
  namespace: monitoring
spec:
  jobLabel: k8s-app
  endpoints:
  - interval: 30s
    port: metrics
  selector:
    matchLabels:
      k8s-app: external-machines
  namespaceSelector:
    matchNames:
    - kube-system

---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: external-machines-rules
  name: external-machines-rules
  namespace: monitoring
spec:
  groups:
    - name: external-machines
      rules:
        - alert: portUnavailable
          annotations:
            summary: port Unavailable
            description: If port Unavailable
          expr: |
            port_8080 < 1
          for: 1m
          labels:
            severity: critical

kubectl apply -f external_machine_monitor.yaml