前言
这篇文章主要解释进入docker进程的数据包是如何经过iptables的4表5链最终到达Docker容器内部，以及docker是如何配置iptables实现路由以及其他的一些功能(如访问控制等)的。
本文假设你已经知道docker的使用和iptables相关知识，这里不再赘述。
先说结论
由于数据包进入netfilter内核有2个入口(分别是主机外进入和主机内进程发起，参考最经典的iptables processing flowchart图，下面有)，所以我们对从这2个入口进来的数据包分别说明。
主机外进入：
| 顺序 | 经过表 | 经过链 | 作用 |
|---|---|---|---|
| 1 | nat | prerouting | DNAT,目的地址端口转换为容器ip和端口 |
| 2 | filter | forward | 自定义DOCKER-USER，DOCKER-ISOLATION-STAGE，DOCKER这3条链 |
| 3 | nat | postrouting | SNAT,利用MASQUERADE转换源地址 |
主机内进入：
| 顺序 | 经过表 | 经过链 | 作用 |
|---|---|---|---|
| 2 | nat | output | DNAT,目的地址端口转换为容器ip和端口 |
| 3 | nat | postrouting | SNAT,利用MASQUERADE转换源地址 |
其中
DOCKER-USER链为Docker定义给用户用来添加自定义规则来限制访问策略，这里官网有说明
DOCKER-ISOLATION-STAGE链用来实现Docker多network中的容器互相隔离，不能进行互通
DOCKER链用来实现用户配置的端口映射策略，如192.168.1.2:80映射到容器80端口
以上数据包流转过程也解释了为什么如果按往常使用iptables在filter表input链配置docker应用的访问控制的话会不起作用，因为访问docker应用的根本不会走filter[nat]，而且2个入口都不会走！
可以看到docker做的无非就是设置转换地址之类的规则，因为Docker在这之前已经为每个容器网络建立了独立的网桥，配置了路由等等，不过这里篇幅有限就不讨论这些了。下面我们就进行实验，看如何得出这些结论的。
开始实验
实验准备工作
- 我们在一台新的机器(centos)上关闭防护墙，安装好docker，并启动一个nginx容器，映射为80端口。
systemctl stop firewalld
systemctl start docker
docker run -d -p 80:80 nginx
现在我们查看iptables的nat和filter表，有如下输出：
[root@localhost ~]# iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 2 packets, 289 bytes)
pkts bytes target prot opt in out source destination
7 508 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 2 packets, 289 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 34 packets, 2399 bytes)
pkts bytes target prot opt in out source destination
4 240 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 42 packets, 2879 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
0 0 MASQUERADE tcp -- * * 172.17.0.2 172.17.0.2 tcp dpt:80
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
8 480 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:172.17.0.2:80
[root@localhost ~]# iptables -nvL -t filter
Chain INPUT (policy ACCEPT 4553 packets, 902K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
44 4498 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
44 4498 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
20 2105 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
4 240 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
20 2153 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 6081 packets, 6094K bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
4 240 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:80
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
20 2153 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
44 4498 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
20 2153 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
44 4498 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
可以看到，docker分别在nat表的prerouting output postrouting这3个链和filter表的forward链添加了内容，后面的实验我们只需要关注这几条链即可。
- 要想看到数据包流经了哪些表和链，还需要启用
ipt_LOG内核模块来追踪数据包并打印日志，使用以下命令来跟踪到达本机80端口的数据包
modprobe nf_log_ipv4
sysctl net.netfilter.nf_log.2=nf_log_ipv4
iptables -t raw -A PREROUTING -p tcp --dport 80 -j TRACE
iptables -t raw -A OUTPUT -p tcp --dport 80 -j TRACE
-
同时打开最经典的iptables数据流图参考
netfilter数据流图
再打开一个终端查看日志tail -n0 -f /var/log/messages
嗯，至此准备工作就完成了。
分析数据包
还是分2种情况，数据包从主机外部进入和数据包从主机内进程发出(当然都是发给容器的)
直接在本机之外的机器打一个请求过来
curl http://192.168.233.147/
立即查看刚刚打开终端的日志/var/log/messages
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: nat:PREROUTING:rule:1 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: nat:DOCKER:rule:2 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:4 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: nat:POSTROUTING:policy:3 IN= OUT=docker0 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=130 TOS=0x00 PREC=0x00 TTL=64 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
有很多重复，应该是一个http请求通过tcp连接传输多个包，每个包都需要走一次iptables，我们只看前几个不重复的即可。
可以看到，数据包经过的顺序为
raw:PREROUTING -> nat:PREROUTING -> filter:FORWARD -> security:FORWARD -> nat:POSTROUTING
其余带Docker字样的链为各个链中的自定义链，看到这里在对比数据流图，才发现这张图真的诚不欺我，数据包流向和图中一模一样(终于实践验证理论)。
图上大概长这样，可以看到是走转发这条路，并不是直接到达本地进程，这也是为什么安装Docker时，要启用内核数据包转发！
而且一般在filter:INPUT链添加安全策略的做法对docker也是不生效的，因为访问容器的数据包根本不走filter:INPUT。
image.png
那本机进程发出的数据包走向如何呢？
直接在本机打一个请求过来测试
curl http://192.168.233.147/
查看日志
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: nat:OUTPUT:rule:1 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: nat:DOCKER:rule:2 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: nat:POSTROUTING:policy:3 IN= OUT=docker0 SRC=192.168.233.147 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49914 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49914 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49914 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=49915 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK PSH URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=49915 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK PSH URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=49915 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK PSH URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49916 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741273973 WINDOW=350 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49916 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741273973 WINDOW=350 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49916 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741273973 WINDOW=350 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49917 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49917 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49917 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49918 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK FIN URGP=0 OPT (0101080A016E02D4016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49918 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK FIN URGP=0 OPT (0101080A016E02D4016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49918 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK FIN URGP=0 OPT (0101080A016E02D4016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49919 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932986 ACK=2741274589 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D4016E02D4)
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49919 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932986 ACK=2741274589 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D4016E02D4)
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49919 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932986 ACK=2741274589 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D4016E02D4)
可以看到，数据包经过的顺序为
raw:OUTPUT -> nat:OUTPUT -> filter:OUTPUT -> security:OUTPUT
图上大概长这样
image.png
在
nat:OUTPUT链做了DNAT和nat:POSTROUTING链做了SNAT至此，数据包流向就分析完了。
做完实验收获最大的应该还是如何跟踪数据包走向，毕竟有了这项技能，网络不通也可以debug找原因了。
感谢互联网的前辈们，他们早已把坑踩平，我们只需要找到他们的脚印。
参考
https://backreference.org/2010/06/11/iptables-debugging/
https://docs.docker.com/network/packet-filtering-firewalls/#add-iptables-policies-before-dockers-rules
