1. 前言
转载请说明原文出处, 尊重他人劳动成果!
源码位置: https://github.com/nicktming/istio
分支: tming-v1.3.6 (基于1.3.6版本)
本文承接上文 [istio源码分析] istio源码开发调试版简单安装 进行流量的分析.
本文需要对envoy有一个基本的认识.
2. ingress-gateway -> productpage
因为是访问
ingress-gateway的内部nodeport端口, 并且nodeport 31380:80, 所以kube-proxy会通过iptables转到ingress-gateway这个pod的podIP:80上.
[root@master ~]# kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-ingressgateway-768778694-sz9kw 1/1 Running 0 7h17m
istio-sidecar-injector-84d5c488d9-jqnx9 1/1 Running 0 7h29m
2.1 查看listener
[root@master analysis]# istioctl -n istio-system proxy-config listener istio-ingressgateway-768778694-sz9kw
ADDRESS PORT TYPE
0.0.0.0 80 HTTP
0.0.0.0 15090 HTTP
[root@master analysis]# istioctl -n istio-system proxy-config listener istio-ingressgateway-768778694-sz9kw --port 80 -o json
[
{
"name": "0.0.0.0_80",
"address": {
...
},
"filterChains": [
{
"filters": [
{
"name": "envoy.http_connection_manager",
"typedConfig": {
...
"routeConfigName": "http.80"
},
...
}
}
]
}
],
"trafficDirection": "OUTBOUND"
}
]
[root@master analysis]#
可以看到当访问
ingress-gateway的80端口时要根据路由(route)http.80进行转发.
2.2 查看route
[root@master analysis]# istioctl -n istio-system proxy-config route --name=http.80 istio-ingressgateway-768778694-sz9kw
NOTE: This output only contains routes loaded via RDS.
NAME VIRTUAL HOSTS
http.80 1
[root@master analysis]#
[root@master analysis]# istioctl -n istio-system proxy-config route --name=http.80 istio-ingressgateway-768778694-sz9kw -o json
[
{
"name": "http.80",
"virtualHosts": [
{
"name": "*:80",
"domains": [
"*",
"*:80"
],
"routes": [
{
"match": {
"path": "/productpage",
"caseSensitive": true
},
"route": {
"cluster": "outbound|9080||productpage.default.svc.cluster.local",
...
},
...
},
...
]
}
],
"validateClusters": false
}
]
[root@master analysis]#
可以看到访问
ingress-gateway的80端口并且匹配到/productpage的时候, 请求会交给名字为outbound|9080||productpage.default.svc.cluster.local的cluster处理.
2.3 查看clusters
[root@master analysis]# istioctl -n istio-system proxy-config cluster istio-ingressgateway-768778694-sz9kw --fqdn productpage.default.svc.cluster.local -o json
[
{
"name": "outbound_.9080_._.productpage.default.svc.cluster.local",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "0s"
},
"serviceName": "outbound_.9080_._.productpage.default.svc.cluster.local"
},
"connectTimeout": "10s",
"circuitBreakers": {
"thresholds": [
{
"maxRetries": 1024
}
]
}
},
{
"name": "outbound|9080||productpage.default.svc.cluster.local",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "0s"
},
"serviceName": "outbound|9080||productpage.default.svc.cluster.local"
},
"connectTimeout": "10s",
"circuitBreakers": {
"thresholds": [
{
"maxRetries": 1024
}
]
}
}
]
[root@master analysis]#
可以看到名为
outbound|9080||productpage.default.svc.cluster.local的cluster会转到serviceName, 所以从该serviceName中转到最终的endpoints.
2.4 查看endpoint
[root@master istio]# istioctl -n istio-system proxy-config endpoint istio-ingressgateway-768778694-sz9kw | grep productpage
10.0.15.34:9080 HEALTHY OK outbound_.9080_._.productpage.default.svc.cluster.local
10.0.15.34:9080 HEALTHY OK outbound|9080||productpage.default.svc.cluster.local
可以看到名为
outbound|9080||productpage.default.svc.cluster.local的cluster有一个endpoint就是10.0.15.34:9080.
2.5 总结
所以访问流程如下:
traffic-flow.png
3. proxy_init程序
在每个需要注入
sidecar的pod中, 都会有一个init程序, 该程序其实就是执行一些iptables规则, 用于拦截进出该pod的网络流量.
[root@master ~]# docker ps | grep productpage
fddd22d5f896 5cb4b1355aa0 "/usr/local/bin/pi..." 24 hours ago Up 24 hours k8s_istio-proxy_productpage-v1-8554d58bff-d7j8d_default_fddfd3b3-1338-4e89-979c-33775b9d91be_0
6742b3852dd3 8e754b2df1fe "/bin/sh -c 'tail ..." 24 hours ago Up 24 hours k8s_productpage_productpage-v1-8554d58bff-d7j8d_default_fddfd3b3-1338-4e89-979c-33775b9d91be_0
461d07931e67 k8s.gcr.io/pause:3.1 "/pause" 24 hours ago Up 24 hours k8s_POD_productpage-v1-8554d58bff-d7j8d_default_fddfd3b3-1338-4e89-979c-33775b9d91be_0
[root@master ~]# docker exec -it -u root --privileged=true fddd22d5f896 sh
# bash
root@productpage-v1-8554d58bff-d7j8d:/# iptables -t nat -vnL
Chain PREROUTING (policy ACCEPT 43439 packets, 2606K bytes)
pkts bytes target prot opt in out source destination
43441 2606K ISTIO_INBOUND tcp -- * * 0.0.0.0/0 0.0.0.0/0
Chain INPUT (policy ACCEPT 43441 packets, 2606K bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 6683 packets, 438K bytes)
pkts bytes target prot opt in out source destination
5453 327K ISTIO_OUTPUT tcp -- * * 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT 6683 packets, 438K bytes)
pkts bytes target prot opt in out source destination
Chain ISTIO_INBOUND (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
43439 2606K RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:15020
2 120 ISTIO_IN_REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0
Chain ISTIO_IN_REDIRECT (2 references)
pkts bytes target prot opt in out source destination
2 120 REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 redir ports 15006
Chain ISTIO_OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * lo 127.0.0.6 0.0.0.0/0
0 0 ISTIO_IN_REDIRECT all -- * lo 0.0.0.0/0 !127.0.0.1
5453 327K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 owner UID match 1337
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 owner GID match 1337
0 0 RETURN all -- * * 0.0.0.0/0 127.0.0.1
0 0 ISTIO_REDIRECT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ISTIO_REDIRECT (1 references)
pkts bytes target prot opt in out source destination
0 0 REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 redir ports 15001
root@productpage-v1-8554d58bff-d7j8d:/#
关于此部分可以参考 https://blog.csdn.net/luo15242208310/article/details/99290541 和 https://jimmysong.io/posts/envoy-sidecar-routing-of-istio-service-mesh-deep-dive/.
3.1 podIp -> localhost
3.1.1 查看listener
由于每个
istio pod都会有iptables进行拦截, 从上面可以知道进入的流量会被15006拦截.
[root@master analysis]# istioctl proxy-config listener productpage-v1-8554d58bff-d7j8d --port 15006 -o json
[
{
"name": "virtualInbound",
"address": {
"socketAddress": {
"address": "0.0.0.0",
"portValue": 15006
}
},
"filterChains": [
...
{
...
"filters": [
{
"name": "envoy.http_connection_manager",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
"statPrefix": "10.0.15.34_9080",
"routeConfig": {
"name": "inbound|9080|http|productpage.default.svc.cluster.local",
"virtualHosts": [
{
"name": "inbound|http|9080",
"domains": [
"*"
],
"routes": [
{
"name": "default",
"match": {
"prefix": "/"
},
"route": {
"cluster": "inbound|9080|http|productpage.default.svc.cluster.local",
"timeout": "0s",
"maxGrpcTimeout": "0s"
},
"decorator": {
"operation": "productpage.default.svc.cluster.local:9080/*"
},
...
}
]
}
],
"validateClusters": false
},
...
}
}
],
...
}
],
...
}
]
[root@master analysis]#
3.1.2 查看listener
同样的方法查看
listener
{
"name": "inbound|9080|http|productpage.default.svc.cluster.local",
"virtualHosts": [
{
"name": "inbound|http|9080",
"domains": [
"*"
],
"routes": [
{
"name": "default",
"match": {
"prefix": "/"
},
"route": {
"cluster": "inbound|9080|http|productpage.default.svc.cluster.local",
"timeout": "0s",
"maxGrpcTimeout": "0s"
},
"decorator": {
"operation": "productpage.default.svc.cluster.local:9080/*"
},
"typedPerFilterConfig": {
"mixer": {
"@type": "type.googleapis.com/istio.mixer.v1.config.client.ServiceConfig",
"disableCheckCalls": true,
"mixerAttributes": {
"attributes": {
"destination.service.host": {
"stringValue": "productpage.default.svc.cluster.local"
},
"destination.service.name": {
"stringValue": "productpage"
},
"destination.service.namespace": {
"stringValue": "default"
},
"destination.service.uid": {
"stringValue": "istio://default/services/productpage"
}
}
}
}
}
}
]
}
],
"validateClusters": false
},
流量将被转到
inbound|9080|http|productpage.default.svc.cluster.local这个cluster.
3.1.3 查看cluster
[root@master analysis]# istioctl proxy-config cluster productpage-v1-8554d58bff-d7j8d --fqdn productpage.default.svc.cluster.local
SERVICE FQDN PORT SUBSET DIRECTION TYPE
productpage.default.svc.cluster.local 9080 - outbound EDS
productpage.default.svc.cluster.local 9080 http inbound STATIC
[root@master analysis]#
[root@master analysis]# istioctl proxy-config cluster productpage-v1-8554d58bff-d7j8d --fqdn productpage.default.svc.cluster.local -o json
[
{
"name": "outbound|9080||productpage.default.svc.cluster.local",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "0s"
},
"serviceName": "outbound|9080||productpage.default.svc.cluster.local"
},
"connectTimeout": "10s",
"circuitBreakers": {
"thresholds": [
{
"maxRetries": 1024
}
]
}
},
{
"name": "inbound|9080|http|productpage.default.svc.cluster.local",
"type": "STATIC",
"connectTimeout": "10s",
"loadAssignment": {
"clusterName": "inbound|9080|http|productpage.default.svc.cluster.local",
"endpoints": [
{
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "127.0.0.1",
"portValue": 9080
}
}
}
}
]
}
]
},
"circuitBreakers": {
"thresholds": [
{}
]
}
}
]
[root@master analysis]#
可以看到将会该
network namespace的9080端口进程服务.
4. productpage -> reviews
因为
productpage服务需要访问reviews和details服务, 此时在该network namespace下会发起对reviews:9080和details:9080的访问, 由于原理一样, 就以访问reviews:9080为例子来进行分析.
4.1 查看listener
因为出口流量会被
15001拦截
[root@master analysis]# istioctl proxy-config listener productpage-v1-8554d58bff-d7j8d --port 15001 -o json
[
{
"name": "virtualOutbound",
"address": {
"socketAddress": {
"address": "0.0.0.0",
"portValue": 15001
}
},
"filterChains": [
{
"filterChainMatch": {
"prefixRanges": [
{
"addressPrefix": "10.0.15.34",
"prefixLen": 32
}
]
},
"filters": [
{
"name": "envoy.tcp_proxy",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"statPrefix": "BlackHoleCluster",
"cluster": "BlackHoleCluster"
}
}
]
},
{
"filters": [
{
"name": "mixer",
"typedConfig": {
"@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
"transport": {
"networkFailPolicy": {
"policy": "FAIL_CLOSE",
"baseRetryWait": "0.080s",
"maxRetryWait": "1s"
},
"checkCluster": "outbound|9091||istio-policy.istio-system.svc.cluster.local",
"reportCluster": "outbound|9091||istio-telemetry.istio-system.svc.cluster.local",
"reportBatchMaxEntries": 100,
"reportBatchMaxTime": "1s"
},
"mixerAttributes": {
"attributes": {
"context.proxy_version": {
"stringValue": "1.3.0"
},
"context.reporter.kind": {
"stringValue": "outbound"
},
"context.reporter.uid": {
"stringValue": "kubernetes://productpage-v1-8554d58bff-d7j8d.default"
},
"destination.service.host": {
"stringValue": "PassthroughCluster"
},
"destination.service.name": {
"stringValue": "PassthroughCluster"
},
"source.namespace": {
"stringValue": "default"
},
"source.uid": {
"stringValue": "kubernetes://productpage-v1-8554d58bff-d7j8d.default"
}
}
},
"disableCheckCalls": true
}
},
{
"name": "envoy.tcp_proxy",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"statPrefix": "PassthroughCluster",
"cluster": "PassthroughCluster"
}
}
]
}
],
"useOriginalDst": true
}
]
[root@master analysis]#
"use_original_dst": true的意思是将请求转发给和原始目的IP:Port匹配的listener. 参考 https://zhaohuabing.com/post/2018-09-25-istio-traffic-management-impl-intro/ 所以该请求将被转到名字为0.0.0.0_9080的listener.
查看端口是
9080的listener.
[root@master analysis]# istioctl proxy-config listener productpage-v1-8554d58bff-d7j8d --port 9080
ADDRESS PORT TYPE
10.0.15.34 9080 HTTP // 很明显这个是处理inbound, ingress-gateway->pod就是走这个listener
0.0.0.0 9080 TCP //处理出口的
查看详细信息
{
"name": "0.0.0.0_9080",
"address": {
"socketAddress": {
"address": "0.0.0.0",
"portValue": 9080
}
},
"filterChains": [
{
"filterChainMatch": {
"prefixRanges": [
{
"addressPrefix": "10.0.15.34",
"prefixLen": 32
}
]
},
...
},
{
"filters": [
{
"name": "envoy.http_connection_manager",
"typedConfig": {
"@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
"statPrefix": "0.0.0.0_9080",
"rds": {
"configSource": {
"ads": {},
"initialFetchTimeout": "0s"
},
"routeConfigName": "9080"
},
...
}
]
}
],
...
}
可以看到该
listener将请求按照名为9080的route来处理, 也就是说istio将请求按照端口来进行划分处理.
4.2 查看route
[root@master analysis]# istioctl proxy-config route productpage-v1-8554d58bff-d7j8d --name 9080 -o json
[
{
"name": "9080",
"virtualHosts": [
...
{
"name": "reviews.default.svc.cluster.local:9080",
"domains": [
"reviews.default.svc.cluster.local",
"reviews.default.svc.cluster.local:9080",
"reviews",
"reviews:9080",
"reviews.default.svc.cluster",
"reviews.default.svc.cluster:9080",
"reviews.default.svc",
"reviews.default.svc:9080",
"reviews.default",
"reviews.default:9080",
"169.169.236.122",
"169.169.236.122:9080"
],
"routes": [
{
"name": "default",
"match": {
"prefix": "/"
},
"route": {
"cluster": "outbound|9080||reviews.default.svc.cluster.local",
"timeout": "0s",
"retryPolicy": {
"retryOn": "connect-failure,refused-stream,unavailable,cancelled,resource-exhausted,retriable-status-codes",
"numRetries": 2,
"retryHostPredicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts"
}
],
"hostSelectionRetryMaxAttempts": "5",
"retriableStatusCodes": [
503
]
},
"maxGrpcTimeout": "0s"
},
...
}
]
},
...
],
"validateClusters": false
}
]
[root@master analysis]#
因为是访问
reviews:9080, 可以看到名为reviews.default.svc.cluster.local:9080的cluster中的domains含有reviews, 所以可以match到, 所以请求将由outbound|9080||reviews.default.svc.cluster.local的cluster来找到具体的endpoint.
4.3 查看cluster 和 endpoint
[root@master analysis]# istioctl proxy-config cluster productpage-v1-8554d58bff-d7j8d --fqdn reviews.default.svc.cluster.local -o json
[
{
"name": "outbound|9080||reviews.default.svc.cluster.local",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "0s"
},
"serviceName": "outbound|9080||reviews.default.svc.cluster.local"
},
"connectTimeout": "10s",
"circuitBreakers": {
"thresholds": [
{
"maxRetries": 1024
}
]
}
}
]
root@master analysis]# istioctl proxy-config endpoint productpage-v1-8554d58bff-d7j8d | grep reviews.default.svc.cluster.local
10.0.15.30:9080 HEALTHY OK outbound|9080||reviews.default.svc.cluster.local
10.0.15.31:9080 HEALTHY OK outbound|9080||reviews.default.svc.cluster.local
10.0.15.32:9080 HEALTHY OK outbound|9080||reviews.default.svc.cluster.local
可以看到该
cluster有三个endpoint可以访问, 这个时候会根据规则访问其中某一个endpoint.
{
"name": "outbound|9080||reviews.default.svc.cluster.local",
"addedViaApi": true,
"hostStatuses": [
{
"address": {
"socketAddress": {
"address": "10.0.15.30",
"portValue": 9080
}
...
},
{
"address": {
"socketAddress": {
"address": "10.0.15.31",
"portValue": 9080
}
},
...
},
{
"address": {
"socketAddress": {
"address": "10.0.15.32",
"portValue": 9080
}
},
...
}
]
}
关于
reviews的pod如何接受该请求, 这个需要分析reviews的envoy配置, 原理和ingress-gateway发请求访问productpage是一样的.
图片.png
5. 总结
这里大致分析了流量是如何转发和管理的, 可以看到都是一些配置信息, 那这些配置信息是如何来的呢, 很明显是从
k8s的service,pod以及自定义资源gateway,virtualService等等中得到的, 那pilot就是做这些事情的, 把这些资源转成envoy可以识别的配置信息. 那galley做什么呢, 负责给pilot发送gateway,virtualService等资源.
