istio流量劫持总结

envoy 启动流程

数据面组件启动流程

  1. initContainer执行初始化脚本，为Pod添加iptables规则
  2. Pilot-agent根据启动参数和K8S API Server中的配置信息生成Envoy的初始配置文件envoy-rev0.json，该文件告诉Envoy从xDS server中获取动态配置信息，并配置了xDS server的地址信息，即控制面的Pilot。
  3. Pilot-agent使用envoy-rev0.json启动Envoy进程。
  4. Envoy根据初始配置获得Pilot地址，采用xDS接口从Pilot获取到Listener，Cluster，Route等d动态配置信息。
  5. Envoy根据获取到的动态配置启动Listener，并根据Listener的配置，结合Route和Cluster对拦截到的流量进行处理。

istio劫持流量总流程

进流量：
downstream -> iptables -> envoy Inbound -> app
出流量：
app -> iptables -> envoy Outbound -> upstream

image.png

1. IPTABLES 劫持转发流量的过程

  1. 启动流程中有讲到第一步是envoy执行初始化脚本，添加iptables规则
    参考文章
    istio sidecar中创建iptables流程

2. envoy Inbound 和 Outbound 流程

istio 的监听端口

$ nsenter -t 20533 -n netstat -nl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:15090           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:15000         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:15001           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:15006           0.0.0.0:*               LISTEN
tcp6       0      0 :::15020                :::*                    LISTEN
tcp6       0      0 :::8889                 :::*                    LISTEN

由上图可以看到envoy有多个端口在监听
15000 端口: 管理端口
15001 端口：Outbound流量重定向端口
15006 端口：Inbound流量重定向端口
15020 端口：健康检查端口
15090 端口: 普罗米修斯端口

应用端口：8899

envoy Inbound处理流量过程

Inbound 流程是将 iptables 拦截到的 downstream 的流量转交给 localhost，与 Pod 内的应用程序容器建立连接。

下面以进入calculator的流量为例子，来分析一下Inbound流程

calculator Inbound流程图

calculator

Inbound配置

  1. listener
  • virtual listener
{
    "version_info": "2020-02-11T01:52:11Z/5",
    "listener": {
        "name": "virtualInbound",
        "address": {
            "socket_address": {
                "address": "0.0.0.0",
                "port_value": 15006
            }
        },
        "filter_chains": [{
            "filter_chain_match": {
                "prefix_ranges": [{
                    "address_prefix": "::0",
                    "prefix_len": 0
                }]
            },
            "filters": [{
                "name": "envoy.tcp_proxy",
                "typed_config": {
                    "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                    "stat_prefix": "InboundPassthroughClusterIpv6",
                    "cluster": "InboundPassthroughClusterIpv6"
                }
            }],
            "filter_chain_match": {
                "prefix_ranges": [{
                    "address_prefix": "192.xxx.xx.163",
                    "prefix_len": 32
                }],
                "destination_port": 8888
            },
            "filters": [{
                "name": "envoy.http_connection_manager",
                "typed_config": {
                    "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
                    "stat_prefix": "inbound_192.xxx.xx.163_8888",
                    "http_filters": [{
                            "name": "istio_authn"
                        },
                        {
                            "name": "mixer"
                        },
                        {
                            "name": "envoy.router"
                        }
                    ],
                    "route_config": {
                        "name": "inbound|8888|grpc|calculator.vm.svc.cluster.local",
                        "virtual_hosts": [{
                            "name": "inbound|http|8888",
                            "domains": [
                                "*"
                            ],
                            "routes": [{
                                "match": {
                                    "prefix": "/"
                                },
                                "decorator": {
                                    "operation": "calculator.vm.svc.cluster.local:8888/*"
                                },
                                "name": "default",
                                "route": {
                                    "timeout": "0s",
                                    "max_grpc_timeout": "0s",
                                    "cluster": "inbound|8888|grpc|calculator.vm.svc.cluster.local"
                                }
                            }]
                        }],
                        "validate_clusters": false
                    }
                }
            }]
        }]
    }
}
  • 该Listener中第三个filterchain用于处理Calculator服务的入向请求。
  • 该filterchain的匹配条件为Calculator服务的 IP和8888端口，配置了一个http_connection_manager filter
  • http_connection_manager 中又嵌入了istio_auth，Mixer，envoy.router等http filter，经过这些filter进行处理后，请求最终将被转发给""cluster": "inbound|8888|grpc|calculator.vm.svc.cluster.local""

接下来我们看下inbound cluster的配置

  1. cluster
  • inbound cluster
    {
     "version_info": "2020-02-10T08:16:25Z/3",
     "cluster": {
      "name": "inbound|8888|grpc|calculator.vm.svc.cluster.local",
      "type": "STATIC",
      "connect_timeout": "10s",
      "circuit_breakers": {
       "thresholds": [
        {
         "max_connections": 4294967295,
         "max_pending_requests": 4294967295,
         "max_requests": 4294967295,
         "max_retries": 4294967295
        }
       ]
      },
      "http2_protocol_options": {
       "max_concurrent_streams": 1073741824
      },
      "load_assignment": {
       "cluster_name": "inbound|8888|grpc|calculator.vm.svc.cluster.local",
       "endpoints": [
        {
         "lb_endpoints": [
          {
           "endpoint": {
            "address": {
             "socket_address": {
              "address": "127.0.0.1",
              "port_value": 8888
             }
            }
           }
          }
         ]
        }
       ]
      }
     }
    }
  • 这个Inbound Cluster，由于该Inbound Cluster中配置的Upstream为127.0.0.1:8888(?如何查看其upstream)
  • 由于iptable设置中127.0.0.1不会被拦截,该请求将发送到Calculator服务的8888端口上进行业务处理。

Inbound流程代码

//todo

envoy Outbound处理流量过程

webapp outbound流程图

webapp

outbound配置

  • virtual outbound istener
{
    "listener": {
        "name": "virtualOutbound",
        "address": {
            "socket_address": {
                "address": "0.0.0.0",
                "port_value": 15001
            }
        },
        "filter_chains": [{
                "filter_chain_match": {
                    "prefix_ranges": [{
                        "address_prefix": "10.244.2.73",
                        "prefix_len": 32
                    }]
                },
                "filters": [{
                    "name": "envoy.tcp_proxy",
                    "typed_config": {
                        "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                        "stat_prefix": "BlackHoleCluster",
                        "cluster": "BlackHoleCluster"
                    }
                }]
            },
            {
                "filters": [{
                        "name": "mixer"
                    },
                    {
                        "name": "envoy.tcp_proxy",
                        "typed_config": {
                            "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                            "stat_prefix": "PassthroughCluster",
                            "cluster": "PassthroughCluster"
                        }
                    }
                ]
            }
        ],
        "use_original_dst": true
    }
}

改listener的最后一行有个配置 "use_original_dst": true，表示用原有地址，不进行路由，这样就会转发到0.0.0.0_8888这个监听器上

下面看一下0.0.0.0_8888这个监听器的配置

  • outbound listener
{
    "listener": {
        "name": "0.0.0.0_8888",
        "address": {
            "socket_address": {
                "address": "0.0.0.0",
                "port_value": 8888
            }
        },
        "filter_chains": [{
                "filter_chain_match": {
                    "prefix_ranges": [{
                        "address_prefix": "10.244.2.73",
                        "prefix_len": 32
                    }]
                },
                "filters": [{
                    "name": "envoy.tcp_proxy",
                    "typed_config": {
                        "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                        "stat_prefix": "BlackHoleCluster",
                        "cluster": "BlackHoleCluster"
                    }
                }]
            },
            {
                "filters": [{
                    "name": "envoy.http_connection_manager",
                    "typed_config": {
                        "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
                        "stat_prefix": "outbound_0.0.0.0_8888",
                        "http_filters": [{
                                "name": "mixer"
                            },
                            {
                                "name": "envoy.cors"
                            },
                            {
                                "name": "envoy.fault"
                            },
                            {
                                "name": "envoy.router"
                            }
                        ],
                        "use_remote_address": false,
                        "generate_request_id": true,
                        "upgrade_configs": [{
                            "upgrade_type": "websocket"
                        }],
                        "stream_idle_timeout": "0s",
                        "normalize_path": true,
                        "rds": {
                            "config_source": {
                                "ads": {}
                            },
                            "route_config_name": "8888"
                        }
                    }
                }]
            }
        ],
        "deprecated_v1": {
            "bind_to_port": false
        },
        "listener_filters_timeout": "0.100s",
        "traffic_direction": "OUTBOUND",
        "continue_on_listener_filters_timeout": true
    }
}

同理最后有一个envoy.router filter，这个filter中route_config_name:"8888"，所以我们可以找到名为"8888"的Route配置

  • outbound Route
{
    "route_config": {
        "name": "8888",
        "virtual_hosts": [{
                "name": "calculator.vm.svc.cluster.local:8888",
                "domains": [
                    "calculator.vm.svc.cluster.local",
                    "calculator.vm.svc.cluster.local:8888",
                    "calculator.vm",
                    "calculator.vm:8888",
                    "calculator.vm.svc.cluster",
                    "calculator.vm.svc.cluster:8888",
                    "calculator.vm.svc",
                    "calculator.vm.svc:8888",
                    "10.106.45.142",
                    "10.106.45.142:8888"
                ],
                "routes": [{
                    "match": {
                        "prefix": "/"
                    },
                    "route": {
                        "weighted_clusters": {
                            "clusters": [{
                                    "name": "outbound|8888|v1|calculator.vm.svc.cluster.local",
                                    "weight": 50
                                },
                                {
                                    "name": "outbound|8888|v2|calculator.vm.svc.cluster.local",
                                    "weight": 50
                                }
                            ]
                        }
                    },
                    "decorator": {
                        "operation": "calculator:8888/*"
                    }
                }]
            },
            {
                "name": "allow_any",
                "domains": [
                    "*"
                ],
                "routes": [{
                    "match": {
                        "prefix": "/"
                    },
                    "route": {
                        "cluster": "PassthroughCluster"
                    }
                }]
            }
        ],
        "validate_clusters": false
    }
}

根据Route 8888的配置可以找到outbound cluster

  • outbound cluster
    {
     "version_info": "2020-02-07T10:38:17Z/1847",
     "cluster": {
      "name": "outbound|8888|v2|calculator.vm.svc.cluster.local",
      "type": "EDS",
      "eds_cluster_config": {
       "eds_config": {
        "ads": {}
       },
       "service_name": "outbound|8888|v2|calculator.vm.svc.cluster.local"
      },
      "connect_timeout": "10s",
      "circuit_breakers": {
       "thresholds": [
        {
         "max_connections": 4294967295,
         "max_pending_requests": 4294967295,
         "max_requests": 4294967295,
         "max_retries": 4294967295
        }
       ]
      },
      "http2_protocol_options": {
       "max_concurrent_streams": 1073741824
      },
      "metadata": {
       "filter_metadata": {
        "istio": {
         "config": "/apis/networking/v1alpha3/namespaces/vm/destination-rule/calculator"
        }
       }
      }
     },
     "last_updated": "2020-02-07T10:38:17.893Z"
    }

我们会发现这个cluster没有直接的endpoint,因为这是动态资源，
可以通过Pilot的调试接口获取该Cluster的endpoint：

$ curl http://xxx.xxx.xxx.xxx.108:15014/debug/edsz > pilot_eds_dump

最后发现endpoint为192.xxx.xxx.163:8888
然后就把流量转发到calculator服务了

  • BlackHoleCluster
    这是一个特殊的Cluster，并没有配置后端处理请求的Host。如其名字所暗示的一样，请求进入后将被直接丢弃掉。如果一个请求没有找到其对的目的服务，则被发到cluste。
{
    "cluster": {
        "name": "BlackHoleCluster",
        "type": "STATIC",
        "connect_timeout": "1s"
    }
}

*PassthroughCluster
和BlackHoleCluter相反，发向PassthroughCluster的请求会被直接发送到其请求中要求的原始目地的，Envoy不会对请求进行重新路由。

{
    "cluster": {
        "name": "PassthroughCluster",
        "type": "ORIGINAL_DST",
        "connect_timeout": "1s",
        "lb_policy": "CLUSTER_PROVIDED",
        "circuit_breakers": {
            "thresholds": [
                {
                    "max_connections": 4294967295,
                    "max_pending_requests": 4294967295,
                    "max_requests": 4294967295,
                    "max_retries": 4294967295
                }
            ]
        }
    }
}

outbound流程代码

//todo

©著作权归作者所有,转载或内容合作请联系作者
平台声明：文章内容（如有图片或视频亦包括在内）由作者上传并发布，文章内容仅代表作者本人观点，简书系信息发布平台，仅提供信息存储服务。

推荐阅读更多精彩内容