Linux虚拟化-网络虚拟化测试

1. 参考文档

Linux bridge原理参考：https://www.cnblogs.com/still-smile/p/14932131.html
Linux虚拟网络接口原理参考：https://thiscute.world/posts/linux-virtual-network-interfaces/
Linux网卡子接口（别名）原理参考：https://www.jianshu.com/p/301243b2b250

2. veth pair测试

添加一对veth接口，命名为veth01和veth01-peer

$ sudo ip link add veth01 type veth peer name veth01-peer

为veth01和veth01-peer设置IP，并启用

$ sudo ip addr add 172.16.10.10/24 dev veth01
$ sudo ip addr add 172.16.10.11/24 dev veth01-peer 
$ sudo ip link set veth01 up
$ sudo ip link set veth01-peer up

查看设置后的结果

8: veth01-peer@veth01: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 66:e5:c6:28:03:3e brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.11/24 scope global veth01-peer
       valid_lft forever preferred_lft forever
    inet6 fe80::64e5:c6ff:fe28:33e/64 scope link 
       valid_lft forever preferred_lft forever
9: veth01@veth01-peer: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 62:3f:6d:30:11:27 brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.10/24 scope global veth01
       valid_lft forever preferred_lft forever
    inet6 fe80::603f:6dff:fe30:1127/64 scope link 
       valid_lft forever preferred_lft forever

2.1 veth01&veth01-peer ping测试

两端互ping不通

$ ping -I veth01-peer 172.16.10.10
PING 172.16.10.10 (172.16.10.10) from 172.16.10.11 veth01-peer: 56(84) bytes of data.

--- 172.16.10.10 ping statistics ---
378 packets transmitted, 0 received, 100% packet loss, time 386047ms

从两端抓包来看，veth01-peer发出的icmp请求包有到达veth01，但没有icmp reply包.

root# tcpdump -nnt host 172.16.10.10 -i veth01-peer
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on veth01-peer, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 18, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 19, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 20, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 21, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 22, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 23, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 24, length 64

root# tcpdump -nnt host 172.16.10.10 -i veth01
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on veth01, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 57, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 58, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 59, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 60, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 61, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 62, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 772, seq 63, length 64

抓包lo，veth01收到的icmp request包，最后进入了本地网络协议栈的loopback

root# tcpdump -nnt host 172.16.10.10 -i lo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 172.16.10.10 > 172.16.10.11: ICMP echo reply, id 772, seq 272, length 64
IP 172.16.10.10 > 172.16.10.11: ICMP echo reply, id 772, seq 273, length 64
IP 172.16.10.10 > 172.16.10.11: ICMP echo reply, id 772, seq 274, length 64
IP 172.16.10.10 > 172.16.10.11: ICMP echo reply, id 772, seq 275, length 64
IP 172.16.10.10 > 172.16.10.11: ICMP echo reply, id 772, seq 276, length 64
IP 172.16.10.10 > 172.16.10.11: ICMP echo reply, id 772, seq 277, length 64

2.2 将veth01-peer放入NAMESPACE ns01中

新建NAMESPACE ns01

ip netns add ns01

veth01-peer放入ns01，设置ip并启用

ip netns exec ns01 ip addr add 172.16.10.11/24 dev veth01-peer
ip netns exec ns01 ip link set veth01-peer up

查看ns01中网络设置情况

root# ip netns exec ns01 ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
8: veth01-peer@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 66:e5:c6:28:03:3e brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.16.10.11/24 scope global veth01-peer
       valid_lft forever preferred_lft forever
    inet6 fe80::64e5:c6ff:fe28:33e/64 scope link 
       valid_lft forever preferred_lft forever

veth01&veth01-peer ping测试

veth01 ping veth01-peer结果及抓包

root# ping -I veth01 172.16.10.11
PING 172.16.10.11 (172.16.10.11) from 172.16.10.10 veth01: 56(84) bytes of data.
64 bytes from 172.16.10.11: icmp_seq=1 ttl=64 time=0.101 ms
64 bytes from 172.16.10.11: icmp_seq=2 ttl=64 time=0.078 ms
64 bytes from 172.16.10.11: icmp_seq=3 ttl=64 time=0.068 ms

抓包veth01-peer
root# ip netns exec ns01 tcpdump -nnt -i veth01-peer
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on veth01-peer, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 172.16.10.10 > 172.16.10.11: ICMP echo request, id 1026, seq 6, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo reply, id 1026, seq 6, length 64
IP 172.16.10.10 > 172.16.10.11: ICMP echo request, id 1026, seq 7, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo reply, id 1026, seq 7, length 64

veth01-peer ping veth01结果及抓包

root@# ip netns exec ns01 ping 172.16.10.10
PING 172.16.10.10 (172.16.10.10) 56(84) bytes of data.

--- 172.16.10.10 ping statistics ---
136 packets transmitted, 0 received, 100% packet loss, time 138251ms

#veth01及lo抓包
root# tcpdump -nnt -i veth01
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on veth01, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 24428, seq 30, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 24428, seq 31, length 64
IP 172.16.10.11 > 172.16.10.10: ICMP echo request, id 24428, seq 32, length 64

root# tcpdump -nnt host 172.16.10.10 -i lo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 172.16.10.10 > 172.16.10.10: ICMP host 172.16.10.11 unreachable, length 92
IP 172.16.10.10 > 172.16.10.10: ICMP host 172.16.10.11 unreachable, length 92
IP 172.16.10.10 > 172.16.10.10: ICMP host 172.16.10.11 unreachable, length 92

2.3 veth01、veth01-peer都放入NAMESPACE中

新建NAMESPACE ns02

ip netns add ns02

veth01放入ns02，设置IP并启用

ip link set veth01 netns ns02
ip netns exec ns02 ip addr add 172.16.10.10/24 dev veth01
ip netns exec ns02 ip link set veth01 up

查看veth01、veth01-peer设置

root# ip netns exec ns01 ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
8: veth01-peer@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 66:e5:c6:28:03:3e brd ff:ff:ff:ff:ff:ff link-netns ns02
    inet 172.16.10.11/24 scope global veth01-peer
       valid_lft forever preferred_lft forever
    inet6 fe80::64e5:c6ff:fe28:33e/64 scope link 
       valid_lft forever preferred_lft forever

root# ip netns exec ns02 ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
9: veth01@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 62:3f:6d:30:11:27 brd ff:ff:ff:ff:ff:ff link-netns ns01
    inet 172.16.10.10/24 scope global veth01
       valid_lft forever preferred_lft forever
    inet6 fe80::603f:6dff:fe30:1127/64 scope link 
       valid_lft forever preferred_lft forever

veth01、veth01-peer ping测试，可以互通

veth01 ping veth01-peer
root# ip netns exec ns02 ping 172.16.10.11
PING 172.16.10.11 (172.16.10.11) 56(84) bytes of data.
64 bytes from 172.16.10.11: icmp_seq=1 ttl=64 time=0.092 ms
64 bytes from 172.16.10.11: icmp_seq=2 ttl=64 time=0.047 ms

veth01-peer ping veth01
root# ip netns exec ns01 ping 172.16.10.10
PING 172.16.10.10 (172.16.10.10) 56(84) bytes of data.
64 bytes from 172.16.10.10: icmp_seq=1 ttl=64 time=0.066 ms
64 bytes from 172.16.10.10: icmp_seq=2 ttl=64 time=0.059 ms

小结
2.1、2.2环境下，由于数据包最后都进入了宿主机网络的loopback接口中，导致两端不能互通；2.3中，我们将两端都放入NAMESPACE中，相当于放入两个隔离的容器，两端的网络协议栈和宿主机隔离开，两端就可以互通了. veth-pair直连可以实现两个NAMESPACE之间的互通，但想要访问外网的话，还需要借助其他虚拟接口来实现，如bridge.

2.4 veth-pair 和 bridge结合使用

测试环境架构

测试环境搭建

ip netns add ns01
ip netns add ns02
ip link add bridge01 type bridge
ip link add veth01 type veth peer name veth01-br
ip link add veth02 type veth peer name veth02-br
ip link set veth01 netns ns01
ip link set veth02 netns ns02
ip link set veth01-br master bridge01
ip link set veth02-br master bridge01
ip link set bridge01 up
ip link set veth01-br up
ip link set veth02-br up
ip netns exec ns01 ip link set veth01 up
ip netns exec ns02 ip link set veth02 up
ip netns exec ns01 ip addr add 172.16.100.10/24 dev veth01
ip netns exec ns02 ip addr add 172.16.100.11/24 dev veth02

配置完结果查询

root@wan:/# brctl show bridge01
bridge name bridge id       STP enabled interfaces
bridge01        8000.d65aacd47b77   no      veth01-br
                            veth02-br
root@wan:/# ip netns exec ns01 ip a 
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
12: veth01@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b2:2a:82:bf:e8:ac brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.16.100.10/24 scope global veth01
       valid_lft forever preferred_lft forever
    inet6 fe80::b02a:82ff:febf:e8ac/64 scope link 
       valid_lft forever preferred_lft forever
root@wan:/# ip netns exec ns02 ip a 
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
14: veth02@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fa:74:99:72:91:68 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.16.100.11/24 scope global veth02
       valid_lft forever preferred_lft forever
    inet6 fe80::f874:99ff:fe72:9168/64 scope link 
       valid_lft forever preferred_lft forever

ns01 ns02 ping测试，可以互通.

root@wan:/# ip netns exec ns02 ping 172.16.100.10
PING 172.16.100.10 (172.16.100.10) 56(84) bytes of data.
64 bytes from 172.16.100.10: icmp_seq=1 ttl=64 time=0.105 ms
64 bytes from 172.16.100.10: icmp_seq=2 ttl=64 time=0.074 ms
64 bytes from 172.16.100.10: icmp_seq=3 ttl=64 time=0.076 ms
64 bytes from 172.16.100.10: icmp_seq=4 ttl=64 time=0.073 ms

root@wan:/# ip netns exec ns01 ping 172.16.100.11
PING 172.16.100.11 (172.16.100.11) 56(84) bytes of data.
64 bytes from 172.16.100.11: icmp_seq=1 ttl=64 time=0.076 ms
64 bytes from 172.16.100.11: icmp_seq=2 ttl=64 time=0.069 ms
64 bytes from 172.16.100.11: icmp_seq=3 ttl=64 time=0.084 ms
64 bytes from 172.16.100.11: icmp_seq=4 ttl=64 time=0.070 ms

ns ping 宿主机或外网，不通

---ns ping 宿主机本地网卡
root@wan:/# ip netns exec ns01 ping 192.168.2.8
ping: connect: 网络不可达
---ns ping 外网
root@wan:/# ip netns exec ns01 ping 114.114.114.114
PING 114.114.114.114 (114.114.114.114) 56(84) bytes of data.

--- 114.114.114.114 ping statistics ---
2406 packets transmitted, 0 received, 100% packet loss, time 2462728ms

2.4.1 ns ping通宿主机方法

给bridge01配置ip 172.16.100.1，作为ns的默认网关

root@wan:/# ip addr add 172.16.100.1/24 dev bridge01
root@wan:/# ip netns exec ns01 ip route add default via 172.16.100.1

ns ping 宿主机

root@wan:/# ip netns exec ns01 ping 192.168.2.8
PING 192.168.2.8 (192.168.2.8) 56(84) bytes of data.
64 bytes from 192.168.2.8: icmp_seq=1 ttl=64 time=0.167 ms
64 bytes from 192.168.2.8: icmp_seq=2 ttl=64 time=0.108 ms
64 bytes from 192.168.2.8: icmp_seq=3 ttl=64 time=0.121 ms

ns ping 外网，仍不通，从宿主机外网卡抓包看，ns ping包有被转发，但源ip是内部ip，回包无法返回，猜测配置nat 可以实现连接外网.

root@wan:/# tcpdump -nnt host 114.114.114.114  -i wlp0s20f3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wlp0s20f3, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 172.16.100.10 > 114.114.114.114: ICMP echo request, id 55296, seq 26, length 64
IP 172.16.100.10 > 114.114.114.114: ICMP echo request, id 55296, seq 27, length 64
IP 172.16.100.10 > 114.114.114.114: ICMP echo request, id 55296, seq 28, length 64
IP 172.16.100.10 > 114.114.114.114: ICMP echo request, id 55296, seq 29, length 64

2.4.2 ns ping通外网方法

配置nat（前提条件ip_forward已打开）

root@wan:# iptables -t nat -A  POSTROUTING -s 172.16.100.0/24 -o wlp0s20f3 -j MASQUERADE

查看iptables nat，多了如下一条规则
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination                   
MASQUERADE  all  --  172.16.100.0/24      anywhere

ns ping 外网测试

root@wan:/# ip netns exec ns01 ping 114.114.114.114
PING 114.114.114.114 (114.114.114.114) 56(84) bytes of data.
64 bytes from 114.114.114.114: icmp_seq=1 ttl=66 time=16.2 ms
64 bytes from 114.114.114.114: icmp_seq=2 ttl=71 time=16.5 ms
64 bytes from 114.114.114.114: icmp_seq=3 ttl=86 time=16.9 ms

外网卡抓包，可以看到src ip已替换
root@wan:/# tcpdump -nnt host 114.114.114.114  -i wlp0s20f3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wlp0s20f3, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 192.168.2.8 > 114.114.114.114: ICMP echo request, id 18685, seq 21, length 64
IP 114.114.114.114 > 192.168.2.8: ICMP echo reply, id 18685, seq 21, length 64
IP 192.168.2.8 > 114.114.114.114: ICMP echo request, id 18685, seq 22, length 64
IP 114.114.114.114 > 192.168.2.8: ICMP echo reply, id 18685, seq 22, length 64

3. tap/tun测试

3.1 基于tap/tun的 qemu-kvm虚拟机网络

qemu-kvm通过对/dev/net/tun 设备的读写，来模拟出虚拟机需要的网卡，同时在宿主机上生成对应的tap/tun虚拟接口；所有虚拟接口，连接到qemu-kvm生成的bridge上，实现虚拟机之间及对外的网络通信（对外通信通过nat实现）.


qemu-kvm虚拟机网络架构

启动一个虚拟机

root@wan:/# virsh list
 Id   名称        状态
---------------------------
 1    centos7.0   running

----qemu-kvm网桥
6: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:7b:e7:b5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

---虚拟机虚拟网卡
9: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:d9:2c:cf brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fed9:2ccf/64 scope link 
       valid_lft forever preferred_lft forever

从虚拟机内ping 外网，抓包vnet0

---虚拟机内
[root@192 ~]# ping www.baidu.com
PING www.a.shifen.com (14.215.177.38) 56(84) bytes of data.
64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=1 ttl=53 time=27.9 ms
64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=2 ttl=53 time=29.4 ms
64 bytes from 14.215.177.38 (14.215.177.38): icmp_seq=3 ttl=53 time=29.3 ms

---宿主机vnet0抓包
root@wan:/# tcpdump -nnt host 14.215.177.38 -i vnet0 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vnet0, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 192.168.122.82 > 14.215.177.38: ICMP echo request, id 6308, seq 73, length 64
IP 14.215.177.38 > 192.168.122.82: ICMP echo reply, id 6308, seq 73, length 64
IP 192.168.122.82 > 14.215.177.38: ICMP echo request, id 6308, seq 74, length 64

3.2 基于tap/tun的vpn网络

使用vpn软件连接vpn网络，查看vpn执行了那些动作
a. 监听了/dev/net/tun设备

root@wan:/# lsof /dev/net/tun
COMMAND     PID         USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sv_websrv  1128         root   11u   CHR 10,200    0t203  141 /dev/net/tun

b. 生成了tap/tun虚拟接口

11: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1300 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 12.0.118.24 peer 12.0.118.23/16 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::840d:f364:3125:4c17/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

c. 通过主机外网接口建立一个加密连接

sv_websrv 1128 root    8u  IPv4 856294      0t0     TCP 192.168.2.8:56778->118.26.139.138:https (ESTABLISHED)

d. 生成了一些特定的路由，使部分地址的访问通过tun0出去

root@wan:/# route -n
内核 IP 路由表
目标            网关            子网掩码        标志  跃点   引用  使用 接口
0.0.0.0         192.168.2.1     0.0.0.0         UG    600    0        0 wlp0s20f3
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 tun0
10.1.0.0        0.0.0.0         255.255.0.0     U     0      0        0 tun0
10.1.10.0       0.0.0.0         255.255.255.0   U     0      0        0 tun0
10.1.10.160     0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.1.10.167     0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.1.10.201     0.0.0.0         255.255.255.255 UH    0      0        0 tun0

访问vpn网络地址10.1.80.199，同时抓包tun0 和 物理外网网卡.

---tun0抓包结果
1649858598.838416 IP 10.1.80.199.18486 > 12.0.118.24.51520: Flags [FP.], seq 9255:9439, ack 536, win 235, options [nop,nop,TS val 2394276572 ecr 418097227], length 184
1649858598.838445 IP 10.1.80.199.18486 > 12.0.118.24.51526: Flags [P.], seq 1:18, ack 472, win 235, options [nop,nop,TS val 2394276578 ecr 418097229], length 17
1649858598.838456 IP 12.0.118.24.51526 > 10.1.80.199.18486: Flags [.], ack 18, win 512, options [nop,nop,TS val 418097331 ecr 2394276578], length 0
1649858598.838483 IP 10.1.80.199.18486 > 12.0.118.24.51526: Flags [P.], seq 18:619, ack 472, win 235, options [nop,nop,TS val 2394276578 ecr 418097229], length 601
1649858598.838489 IP 12.0.118.24.51526 > 10.1.80.199.18486: Flags [.], ack 619, win 508, options [nop,nop,TS val 418097331 ecr 2394276578], length 0
1649858598.838501 IP 10.1.80.199.18486 > 12.0.118.24.51526: Flags [F.], seq 619, ack 472, win 235, options [nop,nop,TS val 2394276578 ecr 418097229], length 0
1649858598.838532 IP 12.0.118.24.51520 > 10.1.80.199.18486: Flags [F.], seq 536, ack 9440, win 503, options [nop,nop,TS val 418097331 ecr 2394276572], length 0
1649858598.839196 IP 12.0.118.24.51526 > 10.1.80.199.18486: Flags [F.], seq 472, ack 620, win 508, options [nop,nop,TS val 418097332 ecr 2394276578], length 0

---物理外网网卡抓包
1649858598.832584 IP 118.26.139.138.443 > 192.168.2.8.56778: Flags [P.], seq 25196:25576, ack 11226, win 201, length 380
1649858598.832594 IP 192.168.2.8.56778 > 118.26.139.138.443: Flags [.], ack 25576, win 36875, length 0
1649858598.838310 IP 118.26.139.138.443 > 192.168.2.8.56778: Flags [P.], seq 25576:26796, ack 12192, win 206, length 1220
1649858598.838360 IP 192.168.2.8.56778 > 118.26.139.138.443: Flags [P.], seq 13572:13848, ack 26796, win 37565, length 276
1649858598.840435 IP 118.26.139.138.443 > 192.168.2.8.56778: Flags [P.], seq 26796:26897, ack 12192, win 206, length 101
1649858598.866899 IP 118.26.139.138.443 > 192.168.2.8.56778: Flags [P.], seq 26897:26998, ack 13572, win 212, length 101
1649858598.866994 IP 192.168.2.8.56778 > 118.26.139.138.443: Flags [.], ack 26998, win 37565, length 0
1649858598.882027 IP 118.26.139.138.443 > 192.168.2.8.56778: Flags [P.], seq 26998:27099, ack 13848, win 217, length 101
1649858598.882064 IP 192.168.2.8.56778 > 118.26.139.138.443: Flags [P.], seq 13848:14400, ack 27099, win 37565, length 552
1649858598.915774 IP 118.26.139.138.443 > 192.168.2.8.56778: Flags [P.], seq 27099:27200, ack 14400, win 223, length 101

可以看到，vpn加密数据是通过vpn软件通过外网卡建立的连接（TCP 192.168.2.8:56778->118.26.139.138:https）进行传输的，但vpn数据的接收和发送，应该都有tun0的参与，之间数据转换的流程分析如下:
假设通过浏览器去访问vpn地址
a. 数据发送流程
浏览器访问vpn地址 → 根据路由表，数据发给tun0 → vpn进程通过/dev/net/tun读取到数据 →vpn进程对数据进行处理，修改数据包的ip源地址和目的地地址 → vpn将数据包从外网卡发出去
b. 数据接收流程
外网卡接收到数据，提交给上层应用 → vpn进程监听连接（TCP 192.168.2.8:56778 >118.26.139.138:https），读取数据 →vpn进程对数据进行处理，修改数据包的ip源地址和目的地地址 →vpn进程通过/dev/net/tun写入tun0 →tun0接收数据，提交给浏览器

4. bridge + vlan

4.1 基于虚拟子接口的vlan实现

搭建测试环境

ip netns add ns01
ip netns add ns02
ip link add bridge01 type bridge
ip link add bridge02 type bridge
ip link add veth01 type veth peer name veth01-br
ip link add veth02 type veth peer name veth02-br
ip link set veth01 netns ns01
ip link set veth02 netns ns02
ip link set veth01-br master bridge01
ip link set veth02-br master bridge02
ip link set bridge01 up
ip link set bridge02 up
ip link set veth01-br up
ip link set veth02-br up
ip netns exec ns01 ip link set veth01 up
ip netns exec ns02 ip link set veth02 up
ip netns exec ns01 ip addr add 192.168.2.100/24 dev veth01
ip netns exec ns02 ip addr add 192.168.2.110/24 dev veth02
ip link add link wlp0s20f3 name wlp0s20f3.01 type vlan id 100
ip link add link wlp0s20f3 name wlp0s20f3.02 type vlan id 200
ip link set wlp0s20f3.01 up
ip link set wlp0s20f3.02 up
ip link set wlp0s20f3.01 master bridge01
ip link set wlp0s20f3.02 master bridge02

测试环境架构

原理：
物理网卡wlp0s20f3添加vlan子接口后，wlp0s20f3相当于一个trunk口，从wlp0s20f3.01过来的数据包会添加上vlan tag.

从ns01 ping外网，查看抓包.

---抓包bridge01
root@wan:/# tcpdump -nnt -i bridge01 -e -v
tcpdump: listening on bridge01, link-type EN10MB (Ethernet), capture size 262144 bytes
ca:bb:93:b9:8e:25 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.1 tell 192.168.2.100, length 28
ca:bb:93:b9:8e:25 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.1 tell 192.168.2.100, length 28
ca:bb:93:b9:8e:25 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.1 tell 192.168.2.100, length 28
ca:bb:93:b9:8e:25 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.1 tell 192.168.2.100, length 28

---抓包wlp0s20f3.01
root@wan:/# tcpdump -nnt -i wlp0s20f3.01 -e -v
tcpdump: listening on wlp0s20f3.01, link-type EN10MB (Ethernet), capture size 262144 bytes
ca:bb:93:b9:8e:25 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.1 tell 192.168.2.100, length 28
ca:bb:93:b9:8e:25 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.1 tell 192.168.2.100, length 28
ca:bb:93:b9:8e:25 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.1 tell 192.168.2.100, length 28
ca:bb:93:b9:8e:25 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.1 tell 192.168.2.100, length 28

---抓包wlp0s20f3，可以看到加上了vlan tag
root@wan:/# tcpdump -nnt -i wlp0s20f3 -e -v
tcpdump: listening on wlp0s20f3, link-type EN10MB (Ethernet), capture size 262144 bytes
aa:c8:a5:e0:ad:2a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.1 tell 192.168.2.110, length 28
c0:94:ad:84:1e:b7 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.8 tell 192.168.2.1, length 28
aa:c8:a5:e0:ad:2a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.1 tell 192.168.2.110, length 28
c0:94:ad:84:1e:b7 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.8 tell 192.168.2.1, length 28
aa:c8:a5:e0:ad:2a > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 200, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.1 tell 192.168.2.110, length 28
c0:94:ad:84:1e:b7 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.8 tell 192.168.2.1, length 28

4.1 基于bridge vlan filtering 的vlan实现

测试环境搭建

ip netns add ns01
ip netns add ns02
ip netns add ns03
ip link add bridge01 type bridge
ip link add veth01 type veth peer name veth01-br
ip link add veth02 type veth peer name veth02-br
ip link add veth03 type veth peer name veth03-br
ip link set veth01 netns ns01
ip link set veth02 netns ns02
ip link set veth03 netns ns03
ip link set veth01-br master bridge01
ip link set veth02-br master bridge01
ip link set veth03-br master bridge01
ip link set bridge01 up
ip link set veth01-br up
ip link set veth02-br up
ip link set veth03-br up
ip netns exec ns01 ip link set veth01 up
ip netns exec ns02 ip link set veth02 up
ip netns exec ns03 ip link set veth03 up
ip netns exec ns01 ip addr add 192.168.2.100/24 dev veth01
ip netns exec ns02 ip addr add 192.168.2.110/24 dev veth02
ip netns exec ns03 ip addr add 192.168.2.120/24 dev veth03
ip  link set bridge01 type bridge vlan_filtering 1
bridge vlan add dev veth01-br vid 100 pvid untagged  master
bridge vlan add dev veth02-br vid 200 pvid untagged  master
bridge vlan add dev veth03-br vid 100 master
bridge vlan add dev veth03-br vid 200 untagged master
测试环境架构

从ns01 ping ns03，查看抓包.

---抓包bridge01，可以看到从ns01出来的包，加上了vlan tag 100
root@wan:/# tcpdump -nnt -i bridge01 -e -v
tcpdump: listening on bridge01, link-type EN10MB (Ethernet), capture size 262144 bytes
5e:cf:6b:0d:97:3c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 100, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.120 tell 192.168.2.100, length 28
5e:cf:6b:0d:97:3c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 100, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.120 tell 192.168.2.100, length 28
5e:cf:6b:0d:97:3c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 100, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.120 tell 192.168.2.100, length 28
5e:cf:6b:0d:97:3c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 100, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.120 tell 192.168.2.100, length 28
5e:cf:6b:0d:97:3c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 100, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.120 tell 192.168.2.100, length 28

---ns01的ping包，通过veth03-br转发到ns03，仍然带vlan tag 100，所以ping不通
root@wan:/# ip netns exec ns03 tcpdump -nnt -i veth03 -e -v
tcpdump: listening on veth03, link-type EN10MB (Ethernet), capture size 262144 bytes
5e:cf:6b:0d:97:3c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 100, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.120 tell 192.168.2.100, length 28
5e:cf:6b:0d:97:3c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 100, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.120 tell 192.168.2.100, length 28
5e:cf:6b:0d:97:3c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 100, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.120 tell 192.168.2.100, length 28
5e:cf:6b:0d:97:3c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 100, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.120 tell 192.168.2.100, length 28
5e:cf:6b:0d:97:3c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 100, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.120 tell 192.168.2.100, length 28

从ns01 ping ns03，查看抓包

root@wan:/# tcpdump -nnt -i bridge01 -e -v
tcpdump: listening on bridge01, link-type EN10MB (Ethernet), capture size 262144 bytes
f6:3d:b4:e1:57:af > a6:2d:7b:a6:8e:bf, ethertype 802.1Q (0x8100), length 102: vlan 200, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25077, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.2.110 > 192.168.2.120: ICMP echo request, id 564, seq 14, length 64
a6:2d:7b:a6:8e:bf > f6:3d:b4:e1:57:af, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 23603, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.2.120 > 192.168.2.110: ICMP echo reply, id 564, seq 14, length 64
f6:3d:b4:e1:57:af > a6:2d:7b:a6:8e:bf, ethertype 802.1Q (0x8100), length 102: vlan 200, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25161, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.2.110 > 192.168.2.120: ICMP echo request, id 564, seq 15, length 64
a6:2d:7b:a6:8e:bf > f6:3d:b4:e1:57:af, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 23644, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.2.120 > 192.168.2.110: ICMP echo reply, id 564, seq 15, length 64
f6:3d:b4:e1:57:af > a6:2d:7b:a6:8e:bf, ethertype 802.1Q (0x8100), length 102: vlan 200, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 25340, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.2.110 > 192.168.2.120: ICMP echo request, id 564, seq 16, length 64
a6:2d:7b:a6:8e:bf > f6:3d:b4:e1:57:af, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 23760, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.2.120 > 192.168.2.110: ICMP echo reply, id 564, seq 16, length 64

---抓包ns03，veth03-br vlan200因配置了untagged属性，vlan200 的ping 包，经veth03-br转发后剥去了vlan tag，所以可以ping 通.
root@wan:/# ip netns exec ns03 tcpdump -nnt -i veth03 -e -v
tcpdump: listening on veth03, link-type EN10MB (Ethernet), capture size 262144 bytes
f6:3d:b4:e1:57:af > a6:2d:7b:a6:8e:bf, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 37617, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.2.110 > 192.168.2.120: ICMP echo request, id 564, seq 101, length 64
a6:2d:7b:a6:8e:bf > f6:3d:b4:e1:57:af, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 33987, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.2.120 > 192.168.2.110: ICMP echo reply, id 564, seq 101, length 64
f6:3d:b4:e1:57:af > a6:2d:7b:a6:8e:bf, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 37861, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.2.110 > 192.168.2.120: ICMP echo request, id 564, seq 102, length 64
a6:2d:7b:a6:8e:bf > f6:3d:b4:e1:57:af, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 34055, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.2.120 > 192.168.2.110: ICMP echo reply, id 564, seq 102, length 64
f6:3d:b4:e1:57:af > a6:2d:7b:a6:8e:bf, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 38084, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.2.110 > 192.168.2.120: ICMP echo request, id 564, seq 103, length 64
a6:2d:7b:a6:8e:bf > f6:3d:b4:e1:57:af, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 34254, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.2.120 > 192.168.2.110: ICMP echo reply, id 564, seq 103, length 64

vlan属性说明：
untagged：带vlan的数据包，经过后会剥离vlan tag
pvid：数据包出去时，默认会打上vlan tag

5 macvlan&ipvlan

5.1 macvlan

环境搭建：
在物理网卡enx00e19900eaa0上创建macvlan子接口，提供给ns使用，ns通过macvlan接口能相互通信并直接访问外网.

root@wan:/# ip link add link enx00e19900eaa0 macvlan01 type macvlan mode bridge
root@wan:/# ip link add link enx00e19900eaa0 macvlan02 type macvlan mode bridge
root@wan:/# ip netns add ns01
root@wan:/# ip netns add ns02
root@wan:/# ip link set macvlan01 netns ns01
root@wan:/# ip link set macvlan02 netns ns02
root@wan:/# ip netns exec ns01 dhclient macvlan01
root@wan:/# ip netns exec ns02 dhclient macvlan02

查看ns01 ns02网络情况

root@wan:/# ip netns exec ns02 ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
21: macvlan02@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ee:29:c3:33:99:0e brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.10.15.242/16 brd 10.10.255.255 scope global dynamic macvlan02
       valid_lft 3598sec preferred_lft 3598sec
    inet6 fe80::ec29:c3ff:fe33:990e/64 scope link 
       valid_lft forever preferred_lft forever
root@wan:/# ip netns exec ns01 ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
20: macvlan01@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:ac:0a:50:18:4b brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.10.15.243/16 brd 10.10.255.255 scope global dynamic macvlan01
       valid_lft 3586sec preferred_lft 3586sec
    inet6 fe80::50ac:aff:fe50:184b/64 scope link 
       valid_lft forever preferred_lft forever

ns ping测试

---ns01 ping ns02
root@wan:/# ip netns exec ns01 ping 10.10.15.242
PING 10.10.15.242 (10.10.15.242) 56(84) bytes of data.
64 bytes from 10.10.15.242: icmp_seq=1 ttl=64 time=0.161 ms
64 bytes from 10.10.15.242: icmp_seq=2 ttl=64 time=0.065 ms
64 bytes from 10.10.15.242: icmp_seq=3 ttl=64 time=0.063 ms


---ns01 ping 外网
root@wan:/# ip netns exec ns01 ping 114.114.114.114
PING 114.114.114.114 (114.114.114.114) 56(84) bytes of data.
64 bytes from 114.114.114.114: icmp_seq=1 ttl=68 time=763 ms
64 bytes from 114.114.114.114: icmp_seq=2 ttl=66 time=904 ms
64 bytes from 114.114.114.114: icmp_seq=3 ttl=83 time=629 ms

5.2 ipvlan

ipvlan类似于macvlan，环境搭建

root@wan:/# ip netns add ns01
root@wan:/# ip netns add ns02
root@wan:/# ip link add link wlp0s20f3 ipvlan01 type ipvlan mode l2
root@wan:/# ip link add link wlp0s20f3 ipvlan02 type ipvlan mode l2
root@wan:/# ip link set ipvlan01 netns ns01
root@wan:/# ip link set ipvlan02 netns ns02
root@wan:/# ip netns exec ns01 ip addr add 10.0.0.110/16 dev ipvlan01
root@wan:/# ip netns exec ns02 ip addr add 10.0.0.120/16 dev ipvlan02
root@wan:/# ip netns exec ns01 ip link set ipvlan01 up
root@wan:/# ip netns exec ns02 ip link set ipvlan02 up

ns 相互ping和ping 网关测试

root@wan:/# ip netns exec ns01 ping 10.0.0.120
PING 10.0.0.120 (10.0.0.120) 56(84) bytes of data.
64 bytes from 10.0.0.120: icmp_seq=1 ttl=64 time=0.093 ms
64 bytes from 10.0.0.120: icmp_seq=2 ttl=64 time=0.034 ms
64 bytes from 10.0.0.120: icmp_seq=3 ttl=64 time=0.031 ms

root@wan:/# ip netns exec ns01 ping 10.0.0.254
PING 10.0.0.254 (10.0.0.254) 56(84) bytes of data.
64 bytes from 10.0.0.254: icmp_seq=2 ttl=128 time=1.29 ms
64 bytes from 10.0.0.254: icmp_seq=3 ttl=128 time=2.64 ms
64 bytes from 10.0.0.254: icmp_seq=4 ttl=128 time=2.56 ms
最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 217,084评论 6赞 503
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 92,623评论 3赞 392
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 163,450评论 0赞 353
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 58,322评论 1赞 293
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 67,370评论 6赞 390
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 51,274评论 1赞 300
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 40,126评论 3赞 418
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 38,980评论 0赞 275
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 45,414评论 1赞 313
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 37,599评论 3赞 334
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 39,773评论 1赞 348
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 35,470评论 5赞 344
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 41,080评论 3赞 327
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 31,713评论 0赞 22
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 32,852评论 1赞 269
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 47,865评论 2赞 370
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 44,689评论 2赞 354

推荐阅读更多精彩内容