一、前提
1 nginx编译安装
假设已经安装好了nginx服务器，切能正常启动访问，个人版本为 nginx version: nginx/1.14.0，使用strace/pstack进行调试（或者gdb，没调试过）。strace 和ltrace 可以查看一个应用程序在运行过程中所发起的系统调用
2 strace命令
strace - trace system calls and signals 跟踪系统调用和信号，最重要的参数 -p，
strace -p pid
Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to many processes in addition to command (which is optional if at least one -p option is given). -p "pidof PROG" syntax is supported
详细参数，自行查看 man strace（yum install strace）
3 nginx配置
nginx采用master-worker模式，且只有一个worker进程，方便调试
worker_processes 1;
events {
worker_connections 1024;
}
二、gstack 调试
ps aux | grep nginx | grep -v grep
www 13366 0.0 0.1 21784 2164 ? S Mar23 0:00 nginx: worker process
root 15221 0.0 0.1 21784 2132 ? ts Mar13 0:00 nginx: master process /usr/local/nginx/sbin/nginx
[@bx16-77-246 /usr/local/nginx]# gstack 13366
#0 0x00007fe98da830f3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1 0x00000000004320b8 in ngx_epoll_process_events (cycle=0xd7e1b0, timer=18446744073709551615, flags=1) at src/event/modules/ngx_epoll_module.c:800
#2 0x0000000000429c48 in ngx_process_events_and_timers (cycle=cycle@entry=0xd7e1b0) at src/event/ngx_event.c:242
#3 0x0000000000430644 in ngx_worker_process_cycle (cycle=0xd7e1b0, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:750
#4 0x000000000042ed72 in ngx_spawn_process (cycle=cycle@entry=0xd7e1b0, proc=proc@entry=0x4305d3 <ngx_worker_process_cycle>, data=data@entry=0x0, name=name@entry=0x480ead "worker process", respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:199
#5 0x000000000042f90b in ngx_start_worker_processes (cycle=cycle@entry=0xd7e1b0, n=1, type=type@entry=-4) at src/os/unix/ngx_process_cycle.c:359
#6 0x0000000000431358 in ngx_master_process_cycle (cycle=cycle@entry=0xd7e1b0) at src/os/unix/ngx_process_cycle.c:244
#7 0x000000000040c84b in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:382
从main()函数到epoll_wait()函数的调用关系一目了然，说明：（调用栈从下自上）
1 入口是nginx.c文件里的main函数，解析命令行参数，初始化
2 根据ngx_process变量，进入分支ngx_master_process_cycle（另一个分支是ngx_single_process_cycle）
3 进入master后，就需要启动worker进程，调用ngx_start_worker_processes
4 调用ngx_spawn_process启动worker进行，并执行回调ngx_worker_process_cycle
5 进入worker主循环，其中最重要的逻辑是ngx_process_events_and_timers
6 调用ngx_worker_process_init，执行各个模块的init钩子，并设置action为ngx_epoll_process_events
7 epoll开始进入事件等待 __epoll_wait_nocancel
三、strace 调试
#参数1：9是epoll_create创建文件描述符，等待事件触发，1代表rdlist就绪队列有事件要处理
epoll_wait(9, [{EPOLLIN, {u32=14344048, u64=14344048}}], 512, -1) = 1
#接收client请求，3是socket的文件描述符
accept4(17, {sa_family=AF_INET, sin_port=htons(57695), sin_addr=inet_addr("10.2.82.135")}, [16], SOCK_NONBLOCK) = 3
#将3 （accept4 产生）加入到epoll的事件监听机制
epoll_ctl(9, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=14344720, u64=14344720}}) = 0
#等待就绪事件，1表示有事件要处理
epoll_wait(9, [{EPOLLIN, {u32=14344720, u64=14344720}}], 512, 60000) = 1
#从client接收数据，长度484字节
recvfrom(3, "GET /ae/test HTTP/1.1\r\nHost: tem"..., 16384, 0, NULL, NULL) = 484
#判断_not_exists_是否存在，-1表示不存在
stat("/usr/local/nginx/html/_not_exists_", 0x7fff9e42c1c0) = -1 ENOENT (No such file or directory)
#更新事件注册
epoll_ctl(9, EPOLL_CTL_MOD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=14344720, u64=14344720}}) = 0
#初始化新的socket 4
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 4
#对设备的I/O通道进行管理，0表示成功
ioctl(4, FIONBIO, [1]) = 0
#将socket 4 加入epoll的事件监听机制
epoll_ctl(9, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=14344944, u64=14344944}}) = 0
# socket 4 建立连接
connect(4, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
#rdlist就绪队列里有两个事件待处理
epoll_wait(9, [{EPOLLOUT, {u32=14344720, u64=14344720}}, {EPOLLOUT, {u32=14344944, u64=14344944}}], 512, 60000) = 2
#获取套接字的选项信息
# 第一个参数：socket套接字
# 第二个参数：套接字层次
# 第三个参数：要获取的信息项
getsockopt(4, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
#响应头509字节内容写入socket4缓冲区
writev(4, [{"GET /ae/test HTTP/1.0\r\nX-Forward"..., 509}], 1) = 509
#rdlist就绪队列里有一个事件待处理，即socket 3
epoll_wait(9, [{EPOLLIN|EPOLLOUT|EPOLLRDHUP, {u32=14344944, u64=14344944}}], 512, 60000) = 1
#从socket 4 读取响应头及内容
recvfrom(4, "HTTP/1.0 200 OK\r\nContent-Length:"..., 4096, 0, NULL, NULL) = 4096
readv(4, [{"m' => 'input',\n "..., 4096}], 1) = 4096
readv(4, [{" 'file_size' => 0,\n "..., 4096}], 1) = 4096
readv(4, [{"put',\r\n 'rule' "..., 4096}], 1) = 4096
readv(4, [{"#39;2' => array(\r\n "..., 4096}], 1) = 4096
readv(4, [{" 'file_size' => arr"..., 4096}], 1) = 4096
readv(4, [{"\r\n 'file_size' =&"..., 4096}], 1) = 4096
readv(4, [{" 'predownload'"..., 4096}], 1) = 4096
readv(4, [{"y (\n 'label' => "..., 4096}], 1) = 4096
#向socket 3 写入数据
writev(3, [{"HTTP/1.1 200 OK\r\nServer: nginx/1"..., 165}, {"<!DOCTYPE html>\n<html>\n <head>\n"..., 3949}, {"m' => 'input',\n "..., 4096}], 3) = 8210
writev(3, [{" 'file_size' => 0,\n "..., 4096}, {"put',\r\n 'rule' "..., 4096}], 2) = 8192
writev(3, [{"#39;2' => array(\r\n "..., 4096}, {" 'file_size' => arr"..., 4096}], 2) = 8192
writev(3, [{"\r\n 'file_size' =&"..., 4096}, {" 'predownload'"..., 4096}], 2) = 8192
writev(3, [{"y (\n 'label' => "..., 4096}], 1) = 4096
readv(4, [{" array(\n "..., 4096}, {"9;file_ext' => 'mp4,g"..., 4096}, {"\r\n <div class=\"col-md-1\">9</d"..., 4096}, {"=> array(\r\n 'gotourl&#"..., 4096}, {"tle\" id=\"exampleModalScrollableT"..., 4096}, {"", 4096}, {"", 4096}, {"", 4096}, {"", 4096}], 9) = 17810
readv(4, [{"", 2670}, {"", 4096}, {"", 4096}, {"", 4096}, {"", 4096}], 5) = 0
writev(3, [{" array(\n "..., 4096}, {"9;file_ext' => 'mp4,g"..., 4096}, {"\r\n <div class=\"col-md-1\">9</d"..., 4096}, {"=> array(\r\n 'gotourl&#"..., 4096}, {"tle\" id=\"exampleModalScrollableT"..., 1426}], 5) = 17810
#关闭socket 4
close(4) = 0
write(28, "10.2.82.135 - - [24/Mar/2020:12:"..., 220) = 220
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
#查看epoll的rdlist就绪队列是否有事件，0表示无
epoll_wait(9, [], 512, 65000) = 0
#关闭socket 3
close(3) = 0
#继续等待事件触发
epoll_wait(9,
整个过程，你应该清晰的知道epoll对象上的rdlist就绪队列和rbtree的变化情况
延伸阅读
参考文档
1 epoll_create
2 strace工具
3 nginx 源码注释
