Pod自动扩容/缩容
HPA介绍
001 Horizontal Pod Autoscaler（HPA，Pod水平自动伸缩）
002 根据资源利用率或者自定义指标自动调整Deployment的Pod副本数量，提供应用并发。
003 HPA不适于无法缩放的对象，例如DaemonSet。
1639534110940.png
HPA基本工作原理
001 Kubernetes 中的 Metrics Server 持续采集所有 Pod 副本的指标数据。
002 HPA 控制器通过 Metrics Server 的 API（聚合 API）获取这些数据，
003 基于用户定义的扩缩容规则进行计算，得到目标 Pod 副本数量。
004 当目标 Pod 副本数量与当前副本数量不同时，HPA 控制器就向 Pod 的Deployment控制器发起scale 操作，调整 Pod 的副本数量，完成扩缩容操作
1639534522974.png
使用HPA前提条件
001 启用Kubernetes API聚合层
002 相应的API已注册
• 对于资源指标（例如CPU、内存），将使用metrics.k8s.io API，一般由metrics-server提供
• 对于自定义指标（例如QPS），将使用custom.metrics.k8s.io API，由相关适配器（Adapter）服务提供
API聚合层
001 在 Kubernetes 1.7 版本引入了聚合层，允许第三方应用程序通过将自己注册到kube-apiserver上，仍然通过 API Server 的 HTTP URL 对新的 API 进行访问和操作。
002 为了实现这个机制，Kubernetes 在 kube-apiserver 服务中引入了一个API 聚合层（API Aggregation Layer），用于将扩展 API 的访问请求转发到用户服务的功能
1639535446518.png
启用聚合层
#用kubeadm 默认已开启
#二进制方式
[root@k8s-m1 ~]# vi /opt/kubernetes/cfg/kube-apiserver.conf
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \
--proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \
--proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \
--requestheader-allowed-names=kubernetes \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--enable-aggregator-routing=true \
基于资源指标
Metrics Server部署
[root@k8s-m1 hpa]# kubectl apply -f metrics-server.yaml
[root@k8s-m1 hpa]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
......
metrics-server-b66888848-jqqww 1/1 Running 0 4m31s
查看指标并验证
[root@k8s-m1 hpa]# kubectl get apiservices
...
v1beta1.metrics.k8s.io kube-system/metrics-server True 5m7s
[root@k8s-m1 hpa]# kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes
{...}
[root@k8s-m1 hpa]# kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods
{...}
#查看Node资源消耗
[root@k8s-m1 hpa]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-m1 309m 7% 1044Mi 28%
k8s-node1 142m 3% 377Mi 10%
#查看Pod资源消耗
[root@k8s-m1 hpa]# kubectl top pod -n kube-system
NAME CPU(cores) MEMORY(bytes)
calico-kube-controllers-97769f7c7-wfvvd 2m 14Mi
calico-node-76zz7 69m 71Mi
calico-node-lfjsk 83m 74Mi
coredns-6cc56c94bd-2zsgn 5m 13Mi
metrics-server-b66888848-jqqww 5m 21Mi
#如果能正常显示资源消耗说明Metrics Server服务工作正常
应用部署
部署应用
[root@k8s-m1 hpa]# kubectl create deployment web --image=nginx --dry-run=client -o yaml > deployment.yaml
[root@k8s-m1 hpa]# vi deployment.yaml
spec:
containers:
- image: nginx
name: nginx
resources:
requests:
cpu: 0.5
[root@k8s-m1 hpa]# kubectl apply -f deployment.yaml
deployment.apps/web created
[root@k8s-m1 hpa]# kubectl expose deployment web --port=80 --target-port=80
service/web exposed
[root@k8s-m1 hpa]# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/web-67bbbf48fc-cgflp 1/1 Running 0 5m42s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 2d21h
service/web ClusterIP 10.0.0.129 <none> 80/TCP 5m8s
创建HPA
[root@k8s-m1 hpa]# kubectl autoscale deployment web --min=2 --max=10 --cpu-percent=80
[root@k8s-m1 hpa]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web Deployment/web <unknown>/80% 2 10 1 17s
说明：为名为web的deployment创建一个HPA对象，目标CPU使用率为80%，副本数量配置为2到10之间
压测
yum install httpd-tools
# 总20w请求，并发1000
[root@k8s-m1 hpa]# ab -n 200000 -c 1000 http://10.0.0.129/index.html
观察扩容状态
#3分钟后扩容
[root@k8s-m1 ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web Deployment/web 0%/80% 2 10 4 11m
[root@k8s-m1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
web-67bbbf48fc-5gkf8 1/1 Running 0 11m
web-67bbbf48fc-6xvdk 1/1 Running 0 4m14s
web-67bbbf48fc-cgflp 1/1 Running 0 18m
web-67bbbf48fc-ftvct 1/1 Running 0 4m14s
#5分钟后冷却
[root@k8s-m1 ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web Deployment/web 0%/80% 2 10 2 17m
[root@k8s-m1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
web-67bbbf48fc-5gkf8 1/1 Running 0 18m
web-67bbbf48fc-cgflp 1/1 Running 0 25m
基于自定义指标
概述
001 为满足更多的需求，HPA也支持自定义指标，例如QPS、5xx错误状态码等
002 实现自定义指标由autoscaling/v2版本提供
003 对于自定义指标（例如QPS），将使用custom.metrics.k8s.io API，由相关适配器（Adapter）服务提供
1639547057253.png
基于qps实现pod动态扩容缩容案例
假设我们有一个网站，想基于每秒接收到的HTTP请求对其Pod进行
自动缩放，实现HPA大概步骤：
001 部署Prometheus
002 对应用暴露指标，部署应用，并让Prometheus采集暴露的指标
003 部署Prometheus Adapter
004 为指定HPA配置Prometheus Adapter
005 创建HPA
006 压测、验证
部署Prometheus
[root@k8s-m1 prometheus]# kubectl apply -f .
[root@k8s-m1 prometheus]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
...
prometheus-8474f8559d-pff4r 2/2 Running 0 4m43s
1639548227041.png
对应用暴露指标，部署应用，并让Prometheus采集暴露的指标
apiVersion: apps/v1
kind: Deployment
metadata:
name: metrics-flask-app
spec:
replicas: 3
selector:
matchLabels:
app: flask-app
template:
metadata:
labels:
app: flask-app
# 声明Prometheus采集
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "80"
prometheus.io/path: "/metrics"
spec:
containers:
- image: lizhenliang/metrics-flask-app
name: web
......
----------------------------------------------------------------------------------
[root@k8s-m1 metrics-app]# kubectl get pod
NAME READY STATUS RESTARTS AGE
metrics-flask-app-66c9995b58-b2pf5 1/1 Running 0 4m57s
metrics-flask-app-66c9995b58-mqthk 1/1 Running 0 4m57s
metrics-flask-app-66c9995b58-slgzm 1/1 Running 0 4m57s
1639548836296.png
1639548983916.png
部署Prometheus Adapter
[root@k8s-m1 adapter]# ls
prometheus-adapter-configmap.yaml prometheus-adapter.yaml
[root@k8s-m1 adapter]# kubectl apply -f .
[root@k8s-m1 adapter]# kubectl get pod -n kube-system
......
NAME READY STATUS RESTARTS AGE
prometheus-adapter-7f94cc997d-gdl9k 1/1 Running 0 6m34s
[root@k8s-m1 adapter]# kubectl get apiservices
v1beta1.custom.metrics.k8s.io kube-system/prometheus-adapter True
[root@k8s-m1 adapter]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"
{...}
为指定HPA配置Prometheus Adapter
[root@k8s-m1 adapter]# vi prometheus-adapter.yaml
#加入自定义指标
data:
config.yaml: |
rules:
- seriesQuery: 'request_count_total{app="flask-app"}'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
name:
matches: "request_count_total"
as: "qps"
metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
----------------------------------------------------------------------------
部署
[root@k8s-m1 adapter]# kubectl apply -f prometheus-adapter.yaml
[root@k8s-m1 adapter]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/qps"
{"kind":"MetricValueList","apiVersion":"custom.metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/qps"},"items":[{"describedObject":{"kind":"Pod","namespace":"default","name":"metrics-flask-app-66c9995b58-b2pf5","apiVersion":"/v1"},"metricName":"qps","timestamp":"2021-12-15T07:48:11Z","value":"16m","selector":null},{"describedObject":{"kind":"Pod","namespace":"default","name":"metrics-flask-app-66c9995b58-mqthk","apiVersion":"/v1"},"metricName":"qps","timestamp":"2021-12-15T07:48:11Z","value":"16m","selector":null},{"describedObject":{"kind":"Pod","namespace":"default","name":"metrics-flask-app-66c9995b58-slgzm","apiVersion":"/v1"},"metricName":"qps","timestamp":"2021-12-15T07:48:11Z","value":"16m","selector":null}]}
----------------------------------------------------------------------------
配置描述：
• seriesQuery：Prometheus查询语句，查询应用系列指标。
• resources：Kubernetes资源标签映射到Prometheus标签。
• name：将Prometheus指标名称在自定义指标API中重命名，matches正则匹配，as指定新名称。
• metricsQuery：一个Go模板，对调用自定义指标API转换为Prometheus查询语句。
Adapter向Prometheus查询语句最终是：
sum(rate(request_count_total{app="flask-app", kubernetes_namespace="default"}[2m])) by (kubernetes_pod_name)
1639550395349.png
创建HPA
[root@k8s-m1 hpa]# vi hpa-v2-qps.yaml
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: metrics-flask-app
namespace: default
spec:
minReplicas: 1
maxReplicas: 10
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: metrics-flask-app
metrics:
- type: Pods
pods:
metric:
name: qps
target:
type: AverageValue
averageValue: 10000m
-------------------------------------------------------------------------
# 所有Pod平均值为10000m触发扩容，即每秒10个请求
[root@k8s-m1 hpa]# kubectl apply -f hpa-v2-qps.yaml
[root@k8s-m1 hpa]# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/metrics-flask-app-66c9995b58-b2pf5 1/1 Running 2 109m
pod/metrics-flask-app-66c9995b58-mqthk 1/1 Running 2 109m
pod/metrics-flask-app-66c9995b58-slgzm 1/1 Running 2 109m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/metrics-flask-app ClusterIP 10.0.0.119 <none> 80/TCP 109m
压力测试
[root@k8s-m1 hpa]# ab -n 300000 -c 1000 http://10.0.0.119/
[root@k8s-m1 hpa]# kubectl get pod
NAME READY STATUS RESTARTS AGE
metrics-flask-app-66c9995b58-5gb77 1/1 Running 0 4m4s
metrics-flask-app-66c9995b58-b2pf5 1/1 Running 2 127m
metrics-flask-app-66c9995b58-jv2dr 1/1 Running 0 4m19s
metrics-flask-app-66c9995b58-lq8s4 1/1 Running 0 4m4s
metrics-flask-app-66c9995b58-mqthk 1/1 Running 2 127m
metrics-flask-app-66c9995b58-rq8zs 1/1 Running 0 4m19s
metrics-flask-app-66c9995b58-sgtkw 1/1 Running 0 4m4s
metrics-flask-app-66c9995b58-slgzm 1/1 Running 2 127m
metrics-flask-app-66c9995b58-vzzgt 1/1 Running 0 4m19s
metrics-flask-app-66c9995b58-xh9bb 1/1 Running 0 4m4s
[root@k8s-m1 hpa]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
metrics-flask-app Deployment/metrics-flask-app 258837m/10 1 10 10
#5分钟后冷却
[root@k8s-m1 hpa]# kubectl get pod
NAME READY STATUS RESTARTS AGE
metrics-flask-app-66c9995b58-b2pf5 1/1 Running 2 133m
[root@k8s-m1 hpa]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
metrics-flask-app Deployment/metrics-flask-app 16m/10 1 10 1
