准备

首先创建redis deployment, 副本数为3，同时创建一个Cluster Service

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name:  redis-deploy-demo
  namespace: default
  labels:
    app:  redis-deploy-demo
spec:
  selector:
    matchLabels:
      app: redis-deploy-demo
  replicas: 3
  template:
    metadata:
      labels:
        app:  redis-deploy-demo
    spec:
      containers:
      - name:  redis-deploy-demo
        image:  redis:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort:  6379
          name:  redis
      restartPolicy: Always
---

apiVersion: v1
kind: Service
metadata:
  name: redis-demo-svc
  namespace: default
spec:
  selector:
    app: redis-deploy-demo
  type: ClusterIP
  ports:
  - name: redis-demo-svc
    protocol: TCP
    port: 6379
    targetPort: 6379
---
EOF

我的CNI的环境为flannel，其中

host 地址为：192.168.205.10
Pod CIDR网段为 172.16.0.0/16
Service CIDR网段为：10.68.0.0/16

查看Pod创建情况

# watch kubectl get pods -o wide
Every 2.0s: kubectl get pod -o wide                                                                                                                                                 k8s1: Thu Nov 26 09:27:55 2020

NAME                                READY   STATUS    RESTARTS   AGE     IP            NODE             NOMINATED NODE   READINESS GATES
redis-deploy-demo-7d6cf8d7b-8g8h4   1/1     Running   0          2m59s   172.16.0.10   192.168.205.10   <none>           <none>
redis-deploy-demo-7d6cf8d7b-ldw6p   1/1     Running   0          2m59s   172.16.0.11   192.168.205.10   <none>           <none>
redis-deploy-demo-7d6cf8d7b-txbx2   1/1     Running   0          2m59s   172.16.0.12   192.168.205.10   <none>           <none>

查看service创建情况

➜  ~ kubectl get svc -o wide
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE     SELECTOR
kubernetes                  ClusterIP   10.68.0.1       <none>        443/TCP    32h     <none>
redis-demo-svc              ClusterIP   10.68.87.193    <none>        6379/TCP   5m16s   app=redis-deploy-demo

探秘iptables

本节仅关注Service和endpoints如何生成iptables的，因此只关注KUBE-SERVICES这个Chain。

kube-proxy启动时会创建 KUBE-SERVICES 这个Chain，该Chain创建到多个Chain中：

Chain：FORWARD，Table：filter
Chain：OUTPUT，Table：filter
Chain：INPUT，Table：filter
Chain：OUTPUT，Table：nat
Chain：PREROUTING，Table：nat

查看filter table的KUBE-SERVICE Chain

➜  ~ iptables -L -n -t filter
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */

查看nat table的KUBE-SERVICE Chain

➜  ~ iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

如何设置Service的iptables

Kubernetes中 Service的流量进入KUBE-SERVICES链
KUBE-SERVICES链根据目标IP:Port匹配并跳转到相应的 KUBE-SVC-* 链
KUBE-SVC-*链相当于一个负载均衡器，它会将数据包平均分发到KUBE-SEP-链。
每个KUBE-SVC-链后面的KUBE-SEP-链都和Service的后端Pod数量一样；
KUBE-SEP-链通过DNAT将连接的目的地址和端口从Service的IP:port替换为后端Pod的IP:port，从而将流量转发到相应的Pod

➜  ~ iptables -L -n -t nat | grep redis-demo-svc
KUBE-MARK-MASQ  tcp  -- !172.16.0.0/16        10.68.87.193         /* default/redis-demo-svc:redis-demo-svc cluster IP */ tcp dpt:6379
KUBE-SVC-VE7MVEEUKDB7C47U  tcp  --  0.0.0.0/0            10.68.87.193         /* default/redis-demo-svc:redis-demo-svc cluster IP */ tcp dpt:6379

➜  ~ iptables -S -t nat | grep KUBE-SVC-VE7MVEEUKDB7C47U
-N KUBE-SVC-VE7MVEEUKDB7C47U
-A KUBE-SERVICES -d 10.68.87.193/32 -p tcp -m comment --comment "default/redis-demo-svc:redis-demo-svc cluster IP" -m tcp --dport 6379 -j KUBE-SVC-VE7MVEEUKDB7C47U
-A KUBE-SVC-VE7MVEEUKDB7C47U -m comment --comment "default/redis-demo-svc:redis-demo-svc" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-FICB4PLPMKAVYTRH
-A KUBE-SVC-VE7MVEEUKDB7C47U -m comment --comment "default/redis-demo-svc:redis-demo-svc" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-H4Y7KXPHOKLDT3V2
-A KUBE-SVC-VE7MVEEUKDB7C47U -m comment --comment "default/redis-demo-svc:redis-demo-svc" -j KUBE-SEP-DMEF5W3N2PX5ARY5

对于Service的流量路径，首先进入的是KUBE-SERVICES链
其中 -A KUBE-SERVICES -d 10.68.87.193/32 -p tcp -m comment --comment "default/redis-demo-svc:redis-demo-svc cluster IP" -m tcp --dport 6379 -j KUBE-SVC-VE7MVEEUKDB7C47U 表示对于目的地址为 10.68.87.193/32 ，目的端口为 6379 的流量JUMP到 KUBE-SVC-VE7MVEEUKDB7C47U

KUBE-SVC-VE7MVEEUKDB7C47U 链设置为

-A KUBE-SVC-VE7MVEEUKDB7C47U -m comment --comment "default/redis-demo-svc:redis-demo-svc" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-FICB4PLPMKAVYTRH
-A KUBE-SVC-VE7MVEEUKDB7C47U -m comment --comment "default/redis-demo-svc:redis-demo-svc" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-H4Y7KXPHOKLDT3V2
-A KUBE-SVC-VE7MVEEUKDB7C47U -m comment --comment "default/redis-demo-svc:redis-demo-svc" -j KUBE-SEP-DMEF5W3N2PX5ARY5

未完待续......

深入解读 Kubernetes 如何实现Service （一）

深入解读 Kubernetes 如何实现Service （一）

准备

探秘iptables

如何设置Service的iptables