记一次XXL_JOB线上问题排查

服务器日志出现大量以下错误:

ullorder-9000][2020-03-17 11:00:09.636][l-79965715][ERROR][c.x.r.r.n.i.n.s.NettyHttpServerHandler][OPR:|MAT:|MAN:|TID:|SKU:][] - xxl-rpc request data empty.
com.xxl.rpc.util.XxlRpcException: xxl-rpc request data empty.
        at com.xxl.rpc.remoting.net.impl.netty_http.server.NettyHttpServerHandler.process(NettyHttpServerHandler.java:78)
        at com.xxl.rpc.remoting.net.impl.netty_http.server.NettyHttpServerHandler.access$000(NettyHttpServerHandler.java:27)
        at com.xxl.rpc.remoting.net.impl.netty_http.server.NettyHttpServerHandler$1.run(NettyHttpServerHandler.java:51)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[pullorder-9000][2020-03-17 11:00:09.638][1530585378][ERROR][c.x.r.r.n.i.n.s.NettyHttpServerHandler][OPR:|MAT:|MAN:|TID:|SKU:][] - xxl-rpc request data empty.
com.xxl.rpc.util.XxlRpcException: xxl-rpc request data empty.
        at com.xxl.rpc.remoting.net.impl.netty_http.server.NettyHttpServerHandler.process(NettyHttpServerHandler.java:78)
        at com.xxl.rpc.remoting.net.impl.netty_http.server.NettyHttpServerHandler.access$000(NettyHttpServerHandler.java:27)
        at com.xxl.rpc.remoting.net.impl.netty_http.server.NettyHttpServerHandler$1.run(NettyHttpServerHandler.java:51)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[pullorder-9000][2020-03-17 11:00:09.639][pGroup-5-1][ERROR][c.x.r.r.n.i.n.s.NettyHttpServerHandler][OPR:|MAT:|MAN:|TID:|SKU:][] - >>>>>>>>>>> xxl-rpc provider netty server caught exception
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1125)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

上十万的错误日志,着实不妥。先排查了这个错误不影响正常的业务操作,但是大量错误日志会导致日志文件非常大。网上找了一下原因,有博客说换了XXL_JOB的版本号就没事,旧版本的XXL_JOB是使用@JobHander注解,新版本的XXL_JOB已经改成了@Xxl_job注解....觉得此方案不可行，线上环境,暂不采用此方案。
然后就采用抓包方式对线上环境进行抓包，抓包步骤：

1.ip addr可以查询到网卡名称

[root@centos64 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:8b:1f:3d brd ff:ff:ff:ff:ff:ff
    inet 192.168.67.xxx/24 brd 192.168.67.xxx scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::75c3:3b6e:7a05:a954/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

上图中ens33就是网卡名称

2.执行抓包命令：sudo tcpdump -i 网卡名 -s 0 -w 文件名.pcap

l注意,需要使用管理员账号才有权限执行此命令，执行后会生成一个文件名.pcap文件，需要从服务器中拉到本地进行分析。

3.使用wireshark分析

Wireshark（前称Ethereal）是一个网络数据包分析软件。使用教程可参考https://www.cnblogs.com/koushuige/p/9212033.html博主的。
把文件拉到本地,用wireshark这个软件就可以排查网络问题。

网络分析.png