istio-1.0.6 流量管理熔断

环境：

istio版本：istio-1.0.6
apiVersion: networking.istio.io/v1alpha3
kubernetes版本: v1.13.3

前提条件

正确安装kubernetes集群
正确部署istio

部署示例应用

部署httpbin：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-v1
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - image: docker.io/citizenstig/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
  selector:
    app: httpbin

注意：
上面的pod用annotations的方式自动注入了sidecar（istio-proxy）容器，如果不想自动注入可以使用一下方法:

kubectl apply -f <(istioctl kube-inject -f httpbin-deployment.yaml)

断路器

官方的教程。

1.创建一个目标规则,针对httpbin服务设置断路器：

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - httpbin.default.svc.cluster.local
  http:
  - route:
    - destination:
        host: httpbin.default.svc.cluster.local
        subset: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin.default.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
    trafficPolicy:
      connectionPool:
        tcp:
          maxConnections: 1
        http:
          http1MaxPendingRequests: 1
          maxRequestsPerConnection: 1
      outlierDetection:
        consecutiveErrors: 1
        interval: 1s
        baseEjectionTime: 3m
        maxEjectionPercent: 100
  - name: v2
    labels:
      version: v1

virtualservice的内容就是选择目标规则子规则为v1的内容。

destination规则对v1子规则设置了断路器，v2版本没有设置断路器。

v1版本的熔断配置：
spec.trafficPolicy.connectionPool.tcp.maxConnections：最大连接数是1
spec.trafficPolicy.connectionPool.http. http1MaxPendingRequests:http最大请求等待数，默认1024
spec.trafficPolicy.connectionPool.http.maxRequestsPerConnection:每个连接到后端的最大请求数。将此参数设置为1将禁用保持活动状态。
spec.trafficPolicy.outlierDetection.baseEjectionTime:Minimum ejection duration. A host will remain ejected for a period equal to the product of minimum ejection duration and the number of times the host has been ejected. This technique allows the system to automatically increase the ejection period for unhealthy upstream servers. format: 1h/1m/1s/1ms. MUST BE >=1ms. Default is 30s.
spec.trafficPolicy.outlierDetection.maxEjectionPercent：可以被提出负载均衡池的上有主机的百分比，默认10%。

详细的字段说明请参考官方说明

应用规则:

kubectl apply -f service-rule.yaml

2.检查目标规则，确定已经正确创建：

istioctl get destinationrule httpbin -o yaml

输出结果：

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"httpbin","namespace":"default"},"spec":{"host":"httpbin.default.svc.cluster.local","subsets":[{"labels":{"version":"v1"},"name":"v1","trafficPolicy":{"connectionPool":{"http":{"http1MaxPendingRequests":1,"maxRequestsPerConnection":1},"tcp":{"maxConnections":1}},"outlierDetection":{"baseEjectionTime":"3m","consecutiveErrors":1,"interval":"1s","maxEjectionPercent":100}}},{"labels":{"version":"v1"},"name":"v2"}]}}
  creationTimestamp: null
  name: httpbin
  namespace: default
  resourceVersion: "3643883"
spec:
  host: httpbin.default.svc.cluster.local
  subsets:
  - labels:
      version: v1
    name: v1
    trafficPolicy:
      connectionPool:
        http:
          http1MaxPendingRequests: 1
          maxRequestsPerConnection: 1
        tcp:
          maxConnections: 1
      outlierDetection:
        baseEjectionTime: 180.000s
        consecutiveErrors: 1
        interval: 1.000s
        maxEjectionPercent: 100
  - labels:
      version: v1
    name: v2

部署客户端

现在我们已经设置了调用httpbin服务的规则，接下来创建一个客户端，用来向后端服务发送请求，观察是否出发熔断策略。这里要使用一个简单的负载测试客户端，名字叫fortio。这个客户端可以控制连接数、并发数以及发送HTTP请求的延迟。这里我们会把客户端进行Sidecar的注入，一次保证istio对网络交互的控制：

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: fortio-deploy
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app: fortio
    spec:
      containers:
      - name: fortio
        #image: fortio/fortio:latest_release
        image: istio/fortio:latest_release
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: http-fortio
        - containerPort: 8079
          name: grpc-ping

手动注入sidecar，可以通过下面方式：

kubectl apply -f <(istioctl kube-inject -f fortio.yaml)

接下来可以登入客户端pod并使用Ofrtio工具调用httpbin。-curl参数表示只调用一次:

FORTIO_POD=$(kubectl get pod | grep fortio | awk '{ print $1 }')
kubectl exec -it $FORTIO_POD  -c fortio /usr/local/bin/fortio -- load -curl  http://httpbin:8000/get

HTTP/1.1 200 OK
server: envoy
date: Tue, 12 Mar 2019 08:35:48 GMT
content-type: application/json
access-control-allow-origin: *
access-control-allow-credentials: true
content-length: 365
x-envoy-upstream-service-time: 17

{
  "args": {},
  "headers": {
    "Content-Length": "0",
    "Host": "httpbin:8000",
    "User-Agent": "istio/fortio-1.0.1",
    "X-B3-Sampled": "0",
    "X-B3-Spanid": "85e1c4d59ca02731",
    "X-B3-Traceid": "85e1c4d59ca02731",
    "X-Request-Id": "f9e6ee91-64f7-45c4-a5d9-2385e6bbbf9a"
  },
  "origin": "127.0.0.1",
  "url": "http://httpbin:8000/get"
}

调用成功，接下来做一些改变。

触发熔断机制

在上面的熔断设置中指定了maxConnections: 1以及http1MaxPendingRequests: 1。这意味着如果超过了一个连接同时发起请求，Istio就会熔断，阻止后续的请求或连接。接下来尝试一下两个并发(-c 2),发送20请求(-n 20):

# kubectl exec -it $FORTIO_POD  -c fortio /usr/local/bin/fortio -- load -c 2 -qps 0 -n 20 -loglevel Warning http://httpbin:8000/get

08:42:08 I logger.go:97> Log level is now 3 Warning (was 2 Info)
Fortio 1.0.1 running at 0 queries per second, 1->1 procs, for 20 calls: http://httpbin:8000/get
Starting at max qps with 2 thread(s) [gomax 1] for exactly 20 calls (10 per thread + 0)
08:42:08 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
Ended after 38.338193ms : 20 calls. qps=521.67
Aggregated Function Time : count 20 avg 0.0037474649 +/- 0.0009945 min 0.001801204 max 0.006098462 sum 0.074949299
# range, mid point, percentile, count
>= 0.0018012 <= 0.002 , 0.0019006 , 5.00, 1
> 0.003 <= 0.004 , 0.0035 , 70.00, 13
> 0.004 <= 0.005 , 0.0045 , 85.00, 3
> 0.005 <= 0.006 , 0.0055 , 95.00, 2
> 0.006 <= 0.00609846 , 0.00604923 , 100.00, 1
# target 50% 0.00369231
# target 75% 0.00433333
# target 90% 0.0055
# target 99% 0.00607877
# target 99.9% 0.00609649
Sockets used: 3 (for perfect keepalive, would be 2)
Code 200 : 19 (95.0 %)
Code 503 : 1 (5.0 %)
Response Header Sizes : count 20 avg 218.5 +/- 50.13 min 0 max 230 sum 4370
Response Body/Total Sizes : count 20 avg 576.1 +/- 82.38 min 217 max 595 sum 11522
All done 20 calls (plus 0 warmup) 3.747 ms avg, 521.7 qps

这里可以看到，几乎所有请求都通过了。

Code 200 : 19 (95.0 %)
Code 503 : 1 (5.0 %)

istio-proxy 允许一些余地。接下来吧并发连接数量提到3:

# kubectl exec -it $FORTIO_POD  -c fortio /usr/local/bin/fortio -- load -c 3 -qps 0 -n 20 -loglevel Warning http://httpbin:8000/get
08:44:16 I logger.go:97> Log level is now 3 Warning (was 2 Info)
Fortio 1.0.1 running at 0 queries per second, 1->1 procs, for 20 calls: http://httpbin:8000/get
Starting at max qps with 3 thread(s) [gomax 1] for exactly 20 calls (6 per thread + 2)
08:44:16 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
08:44:16 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
08:44:16 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
08:44:16 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
08:44:16 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
08:44:16 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
08:44:16 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
Ended after 30.193184ms : 20 calls. qps=662.4
Aggregated Function Time : count 20 avg 0.0029523684 +/- 0.002059 min 0.000273708 max 0.007145017 sum 0.059047367
# range, mid point, percentile, count
>= 0.000273708 <= 0.001 , 0.000636854 , 25.00, 5
> 0.001 <= 0.002 , 0.0015 , 35.00, 2
> 0.002 <= 0.003 , 0.0025 , 40.00, 1
> 0.003 <= 0.004 , 0.0035 , 75.00, 7
> 0.004 <= 0.005 , 0.0045 , 80.00, 1
> 0.005 <= 0.006 , 0.0055 , 85.00, 1
> 0.006 <= 0.007 , 0.0065 , 95.00, 2
> 0.007 <= 0.00714502 , 0.00707251 , 100.00, 1
# target 50% 0.00328571
# target 75% 0.004
# target 90% 0.0065
# target 99% 0.00711601
# target 99.9% 0.00714212
Sockets used: 9 (for perfect keepalive, would be 3)
Code 200 : 13 (65.0 %)
Code 503 : 7 (35.0 %)
Response Header Sizes : count 20 avg 149.5 +/- 109.7 min 0 max 230 sum 2990
Response Body/Total Sizes : count 20 avg 462.7 +/- 180.3 min 217 max 595 sum 9254
All done 20 calls (plus 0 warmup) 2.952 ms avg, 662.4 qps

这时会观察到，熔断行为按照之前的设计生效了：

Code 200 : 13 (65.0 %)
Code 503 : 7 (35.0 %)

只有65%的请求获得通过，剩余的请求被断路器拦截了，我们可以查询istio-proxy的状态，获取更多信息：

# kubectl exec -it $FORTIO_POD  -c istio-proxy  -- sh -c 'curl localhost:15000/stats' | grep httpbin | grep pending
cluster.outbound|8000|v1|httpbin.default.svc.cluster.local.upstream_rq_pending_active: 0
cluster.outbound|8000|v1|httpbin.default.svc.cluster.local.upstream_rq_pending_failure_eject: 0
cluster.outbound|8000|v1|httpbin.default.svc.cluster.local.upstream_rq_pending_overflow: 8
cluster.outbound|8000|v1|httpbin.default.svc.cluster.local.upstream_rq_pending_total: 32

upstream_rq_pending_overflow的值是8，说明8次调用被标记为熔断。

切换到v2版本

修改httpbin的VirtualService配置如下：

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - httpbin.default.svc.cluster.local
  http:
  - route:
    - destination:
        host: httpbin.default.svc.cluster.local
        subset: v2
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin.default.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
    trafficPolicy:
      connectionPool:
        tcp:
          maxConnections: 1
        http:
          http1MaxPendingRequests: 1
          maxRequestsPerConnection: 1
      outlierDetection:
        consecutiveErrors: 1
        interval: 1s
        baseEjectionTime: 3m
        maxEjectionPercent: 100
  - name: v2
    labels:
      version: v1

应用配置

kubectl apply -f httpbin-rule.yaml

重复上面部署完客户端后，使用fortio操作测试，不会出现熔断。

istio-1.0.6 流量管理 熔断

环境：