情景再现
写了一个前端的小程序，然后需要依托于Nginx才能在服务端部署，为了部署的便利性直接打成了Docker的镜像，这个镜像部署到服务器上后发现Nginx中的 $remote_addr这个获取到的IP一直是Docker网桥的IP。
配置文件差不多就是正常配置
server {
listen 32100;
gzip on;
gzip_min_length 1k;
gzip_comp_level 9;
gzip_types text/plain text/css text/javascript application/json application/javascript ;
gzip_vary on;
gzip_disable "MSIE [1-6]\.";
client_body_buffer_size 200m;
client_max_body_size 200m;
proxy_connect_timeout 800s;
proxy_send_timeout 800s;
proxy_read_timeout 800s;
root /app;
location ~.*\.(html|htm)$ {
add_header Cache-Control no-cache;
}
location ^~/static/ {
add_header Cache-Control no-cache;
}
location / {
try_files $uri $uri/ /index.html;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header REMOTE-HOST $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
请求后发现日志全是网关的的IP 158.40.0.1 但是我的IP为 192.168.1.142
158.40.0.1 - - [03/Aug/2023:11:42:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
解决过程
排查这个问题有点麻烦，先是换了一台新的服务器去试，还是有问题，开始怀疑是Nginx的配置写的有问题，网上各种找，后来在WSL中调试的时候发现WSL中是好的，才从服务器中下手。网上的那些乱七八糟的解决方案都试完了，又是Host，又是给docker0加入内部网络，又是加各种些的协议，乱七八糟的方案，除了Host那个是有效的，其他均没屌用。
后来认为是服务器内核太老了执行了 yum update -y 你猜怎么滴，还是没屌用。直到我在Github中花了4个小时的仔细研究，在一些Issues中找到了一丝线索。最后定位到就是防火墙的问题(主要是NAT模块的问题)。
然后将这个玩意开启之后一切正常
firewall-cmd --zone=public --add-masquerade --permanent
firewall-cmd --reload
开启之后效果如下
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
192.168.1.142 - - [03/Aug/2023:11:53:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
192.168.1.142 - - [03/Aug/2023:11:53:49 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
192.168.1.142 - - [03/Aug/2023:11:53:49 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
192.168.1.142 - - [03/Aug/2023:11:53:49 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
我以为我已经搞定了，然后依葫芦画瓢，在另外一台服务器上面进行如此操作，你猜怎么着，屌用没有。
继续对比两台服务器，发现了问题所在。
[root@local app]# firewall-cmd --version
0.5.3
防火墙版本两边没对上，一个是执行了yum update -y的0.6.3,一个是默认的0.5.3
直接升级防火墙版本解决
yum update firewalld -y
总结
这种情况一般出现在Centos7上面，或者各种Centos7的变体上面(各种国产麒麟系统)。
如下两步，搞定
yum update firewalld -y #升级防火墙版本到0.6.3
firewall-cmd --zone=public --add-masquerade --permanent # 防火墙添加 masquerade 规则
firewall-cmd --reload # 重启防火墙
我多测试了一种情况，如果Docker配置文件中的"iptables": false 也是会造成这个问题的。
{
"log-driver": "json-file",
"log-opts": {
"max-size": "100m",
"max-file": "50"
},
"bip": "158.40.0.1/24",
"iptables": false, // 这玩意是false也会出现，修改为true就对了
"data-root": "/data/docker"
}
