- 背景
- 最近在使用netty作为一个文件上传与查询的服务器，用于文件上传分析，期间踩了不少的坑，为此记录两次踩坑的经历与大家分享，由于公司的源码无法公布，这里就通过netty源码进行分析给出解决问题的方向与思路
一.关于netty的请求参数分析
- 在某天清晨兴高采烈的去上班，在上班的路途上同事，打电话告诉我请求端口无法正常使用，返回的异常信息为
java.io.IOException: No space left on device，当时还在车上的我，一看这个异常，哈哈，小儿科，肯定是服务器内存满了，待会也就小清理一下日志啊，磁盘堆积已久的文件啊之类的，so easy，结果到了公司，打开电脑连上服务器,输入以下命令:
df -h
1534013192532.png
当时就看傻了，不对啊，项目的磁盘空是足够的，没有问题，当时就陷入了一个迷茫的排查当中，服务器明明报的是java.io.IOException: No space left on device，很明显告诉我磁盘的空间不足，但是明显磁盘是足够的，此时应有搜索引擎，立马搜索了一下度娘跟谷歌爸爸，最终查到一个原因，目录的inode满了(具体是什么自己百度，这里不是重点)，这是什么鬼，奇了怪了，顺腾摸瓜的查询了所在分区的inode,结果发现没毛病，此时应有源码观察分析:
1. 由于在整个项目之中,netty充当的是https文件上传服务器，使用的是HttpPostRequestDecoder获取请求的相关参数信息，并且抛出的异常也在HttpPostRequestDecoder解析请求参数前后，此时就跟进代码:
public HttpPostRequestDecoder(HttpDataFactory factory, HttpRequest request, Charset charset) {
if (factory == null) {
throw new NullPointerException("factory");
}
if (request == null) {
throw new NullPointerException("request");
}
if (charset == null) {
throw new NullPointerException("charset");
}
// Fill default values
if (isMultipart(request)) {
decoder = new HttpPostMultipartRequestDecoder(factory, request, charset);
} else {
decoder = new HttpPostStandardRequestDecoder(factory, request, charset);
}
}
在此跟进入到decoder = new HttpPostMultipartRequestDecoder(factory, request, charset);当中:
public HttpPostMultipartRequestDecoder(HttpDataFactory factory, HttpRequest request, Charset charset) {
if (factory == null) {
throw new NullPointerException("factory");
}
if (request == null) {
throw new NullPointerException("request");
}
if (charset == null) {
throw new NullPointerException("charset");
}
this.request = request;
this.charset = charset;
this.factory = factory;
// Fill default values
setMultipart(this.request.headers().getAndConvert(HttpHeaderNames.CONTENT_TYPE));
if (request instanceof HttpContent) {
// Offer automatically if the given request is als type of HttpContent
// See #1089
offer((HttpContent) request);
} else {
undecodedChunk = buffer();
parseBody();
}
}
对于以上代码，我注意到parseBody();，继续跟进:
private void parseBody() {
if (currentStatus == MultiPartStatus.PREEPILOGUE || currentStatus == MultiPartStatus.EPILOGUE) {
if (isLastChunk) {
currentStatus = MultiPartStatus.EPILOGUE;
}
return;
}
parseBodyMultipart();
}
在此处会发现一段代码，parseBodyMultipart()，这里就是解析请求参数的核心方法：
private void parseBodyMultipart() {
if (undecodedChunk == null || undecodedChunk.readableBytes() == 0) {
// nothing to decode
return;
}
InterfaceHttpData data = decodeMultipart(currentStatus);
while (data != null) {
addHttpData(data);
if (currentStatus == MultiPartStatus.PREEPILOGUE || currentStatus == MultiPartStatus.EPILOGUE) {
break;
}
data = decodeMultipart(currentStatus);
}
}
这么一段代码，跟进decodeMultipart()中:
private InterfaceHttpData decodeMultipart(MultiPartStatus state) {
switch (state) {
case NOTSTARTED:
throw new ErrorDataDecoderException("Should not be called with the current getStatus");
case PREAMBLE:
// Content-type: multipart/form-data, boundary=AaB03x
throw new ErrorDataDecoderException("Should not be called with the current getStatus");
case HEADERDELIMITER: {
// --AaB03x or --AaB03x--
return findMultipartDelimiter(multipartDataBoundary, MultiPartStatus.DISPOSITION,
MultiPartStatus.PREEPILOGUE);
}
case DISPOSITION: {
// content-disposition: form-data; name="field1"
// content-disposition: form-data; name="pics"; filename="file1.txt"
// and other immediate values like
// Content-type: image/gif
// Content-Type: text/plain
// Content-Type: text/plain; charset=ISO-8859-1
// Content-Transfer-Encoding: binary
// The following line implies a change of mode (mixed mode)
// Content-type: multipart/mixed, boundary=BbC04y
return findMultipartDisposition();
}
case FIELD: {
// Now get value according to Content-Type and Charset
Charset localCharset = null;
Attribute charsetAttribute = currentFieldAttributes.get(HttpHeaderValues.CHARSET);
if (charsetAttribute != null) {
try {
localCharset = Charset.forName(charsetAttribute.getValue());
} catch (IOException e) {
throw new ErrorDataDecoderException(e);
}
}
Attribute nameAttribute = currentFieldAttributes.get(HttpHeaderValues.NAME);
if (currentAttribute == null) {
try {
currentAttribute = factory.createAttribute(request,
cleanString(nameAttribute.getValue()));
} catch (NullPointerException e) {
throw new ErrorDataDecoderException(e);
} catch (IllegalArgumentException e) {
throw new ErrorDataDecoderException(e);
} catch (IOException e) {
throw new ErrorDataDecoderException(e);
}
if (localCharset != null) {
currentAttribute.setCharset(localCharset);
}
}
// load data
try {
loadFieldMultipart(multipartDataBoundary);
} catch (NotEnoughDataDecoderException ignored) {
return null;
}
Attribute finalAttribute = currentAttribute;
currentAttribute = null;
currentFieldAttributes = null;
// ready to load the next one
currentStatus = MultiPartStatus.HEADERDELIMITER;
return finalAttribute;
}
case FILEUPLOAD: {
// eventually restart from existing FileUpload
return getFileUpload(multipartDataBoundary);
}
case MIXEDDELIMITER: {
// --AaB03x or --AaB03x--
// Note that currentFieldAttributes exists
return findMultipartDelimiter(multipartMixedBoundary, MultiPartStatus.MIXEDDISPOSITION,
MultiPartStatus.HEADERDELIMITER);
}
case MIXEDDISPOSITION: {
return findMultipartDisposition();
}
case MIXEDFILEUPLOAD: {
// eventually restart from existing FileUpload
return getFileUpload(multipartMixedBoundary);
}
case PREEPILOGUE:
return null;
case EPILOGUE:
return null;
default:
throw new ErrorDataDecoderException("Shouldn't reach here.");
}
}
在这当中，看到那么一段代码
// Is it a FileUpload
Attribute filenameAttribute = currentFieldAttributes.get(HttpHeaderValues.FILENAME);
if (currentStatus == MultiPartStatus.DISPOSITION) {
if (filenameAttribute != null) {
// FileUpload
currentStatus = MultiPartStatus.FILEUPLOAD;
// do not change the buffer position
return decodeMultipart(MultiPartStatus.FILEUPLOAD);
} else {
// Field
currentStatus = MultiPartStatus.FIELD;
// do not change the buffer position
return decodeMultipart(MultiPartStatus.FIELD);
}
} else {
if (filenameAttribute != null) {
// FileUpload
currentStatus = MultiPartStatus.MIXEDFILEUPLOAD;
// do not change the buffer position
return decodeMultipart(MultiPartStatus.MIXEDFILEUPLOAD);
} else {
// Field is not supported in MIXED mode
throw new ErrorDataDecoderException("Filename not found");
}
}
在这段代码中有那么一个工厂类，会生成一个attrbute对象，factory.createAttribute(request,cleanString(nameAttribute.getValue()));，跟进到方法当中
在实现的方法中，我在初始化decode的时候用了一下那么一段代码(此处有坑，埋得相当的深):
HttpDataFactory factory = new DefaultHttpDataFactory(DefaultHttpDataFactory.MAXSIZE);
HttpPostRequestDecoder decoder=new HttpPostRequestDecoder(factory, httpRequest, Charset.forName("UTF-8"));
HttpDataFactory factory = new DefaultHttpDataFactory(DefaultHttpDataFactory.MAXSIZE);在此处代码中，由于对netty不熟悉，在此处买下了一个深坑，这个后续道来，由前面的代码可以看到
public class DefaultHttpDataFactory implements HttpDataFactory {
...
/**
* Proposed default MINSIZE as 16 KB.
*/
public static final long MINSIZE = 0x4000;
/**
* Proposed default MAXSIZE = -1 as UNLIMITED
*/
public static final long MAXSIZE = -1;
private final boolean useDisk;
private final boolean checkSize;
private long minSize;
private long maxSize = MAXSIZE;
private Charset charset = HttpConstants.DEFAULT_CHARSET;
/**
* Keep all HttpDatas until cleanAllHttpData() is called.
*/
private final Map<HttpRequest, List<HttpData>> requestFileDeleteMap = PlatformDependent.newConcurrentHashMap();
/**
* HttpData will be in memory if less than default size (16KB).
* The type will be Mixed.
*/
public DefaultHttpDataFactory() {
useDisk = false;
checkSize = true;
minSize = MINSIZE;
}
public DefaultHttpDataFactory(Charset charset) {
this();
this.charset = charset;
}
/**
* HttpData will be always on Disk if useDisk is True, else always in Memory if False
*/
public DefaultHttpDataFactory(boolean useDisk) {
this.useDisk = useDisk;
checkSize = false;
}
public DefaultHttpDataFactory(boolean useDisk, Charset charset) {
this(useDisk);
this.charset = charset;
}
/**
* HttpData will be on Disk if the size of the file is greater than minSize, else it
* will be in memory. The type will be Mixed.
*/
public DefaultHttpDataFactory(long minSize) {
useDisk = false;
checkSize = true;
this.minSize = minSize;
}
public DefaultHttpDataFactory(long minSize, Charset charset) {
this(minSize);
this.charset = charset;
}
@Override
public Attribute createAttribute(HttpRequest request, String name, long definedSize) {
if (useDisk) {
Attribute attribute = new DiskAttribute(name, definedSize, charset);
attribute.setMaxSize(maxSize);
List<HttpData> list = getList(request);
list.add(attribute);
return attribute;
}
if (checkSize) {
Attribute attribute = new MixedAttribute(name, definedSize, minSize, charset);
attribute.setMaxSize(maxSize);
List<HttpData> list = getList(request);
list.add(attribute);
return attribute;
}
MemoryAttribute attribute = new MemoryAttribute(name, definedSize);
attribute.setMaxSize(maxSize);
return attribute;
}
...
}
DefaultHttpDataFactory是在解析请求的时候生成一个Attribute对象的，在此处要关注一个变量useDisk,从DefaultHttpDataFactory的构造方法中可以找到，这个值是在初始化时候有创建方进行赋值，如果为true则使用DiskAttribute这个对象进行变量的存储解析，DiskAttribute对象主要是用于在临时文件夹生成一个临时文件来存储属性，如果为false则进入下一步，判断是否要进行长度校验。为此，useDisk在对象初始化的时候我并没有对其赋值，则为false，程序继续往下执行，在前面的方法中，我使用的是public DefaultHttpDataFactory(long minSize)构造函数，则checkSize = true;,由此可以得出结论，产生MixedAttribute对象，该对象的注释Mixed implementation using both in Memory and in File with a limit of size,大致可以知道，如果长度超过限制，则会生成一个File，也就是DiskAttribute对象来存请求的参数，如果请求参数长度小于长度的限制，则使用MemoryAttribute，见文知意，这个对象是将请求参数存放在内存当中。按照正常逻辑思维来看，一个请求参数不会太大，通过以下代码可以看到
@Override
public void setContent(ByteBuf buffer) throws IOException {
checkSize(buffer.readableBytes());
if (buffer.readableBytes() > limitSize) {
if (attribute instanceof MemoryAttribute) {
// change to Disk
attribute = new DiskAttribute(attribute.getName());
attribute.setMaxSize(maxSize);
}
}
attribute.setContent(buffer);
}
netty是用过limitSize这个长度来限制的，而默认的长度由之前的代码可以看到minSize = MINSIZE;限制在16K，应该是通过MemoryAttribute来存储请求参数，那么除了文件上传，其余的属性就不会产生任何的问题。为此，我们回顾之前的初始化代码new DefaultHttpDataFactory(DefaultHttpDataFactory.MAXSIZE);,通过这里看到，这个长度，在初始化的时候，传入了DefaultHttpDataFactory.MAXSIZE这么一个值，但是这个值由之前的源代码可以看到public static final long MAXSIZE = -1;,为此，这个坑就很明显了，当buffer.readableBytes() > limitSize时，由于limitSize的值为-1，导致了change to Disk的发生，产生了DiskAttribute对象
二.对于文件磁盘空间不足问题解决
public class DiskAttribute extends AbstractDiskHttpData implements Attribute {
public static String baseDirectory;
public static boolean deleteOnExitTemporaryFile = true;
public static final String prefix = "Attr_";
public static final String postfix = ".att";
/**
* Constructor used for huge Attribute
*/
public DiskAttribute(String name) {
this(name, HttpConstants.DEFAULT_CHARSET);
}
public DiskAttribute(String name, Charset charset) {
super(name, charset, 0);
}
public DiskAttribute(String name, String value) throws IOException {
this(name, value, HttpConstants.DEFAULT_CHARSET);
}
public DiskAttribute(String name, String value, Charset charset) throws IOException {
super(name, charset, 0); // Attribute have no default size
setValue(value);
}
}
//其父类的两个方法
public void setContent(ByteBuf buffer) throws IOException {
if (buffer == null) {
throw new NullPointerException("buffer");
}
try {
size = buffer.readableBytes();
checkSize(size);
if (definedSize > 0 && definedSize < size) {
throw new IOException("Out of size: " + size + " > " + definedSize);
}
if (file == null) {
file = tempFile();
}
if (buffer.readableBytes() == 0) {
// empty file
if (!file.createNewFile()) {
throw new IOException("file exists already: " + file);
}
return;
}
FileOutputStream outputStream = new FileOutputStream(file);
try {
FileChannel localfileChannel = outputStream.getChannel();
ByteBuffer byteBuffer = buffer.nioBuffer();
int written = 0;
while (written < size) {
written += localfileChannel.write(byteBuffer);
}
buffer.readerIndex(buffer.readerIndex() + written);
localfileChannel.force(false);
} finally {
outputStream.close();
}
setCompleted();
} finally {
// Release the buffer as it was retained before and we not need a reference to it at all
// See https://github.com/netty/netty/issues/1516
buffer.release();
}
}
private File tempFile() throws IOException {
String newpostfix;
String diskFilename = getDiskFilename();
if (diskFilename != null) {
newpostfix = '_' + diskFilename;
} else {
newpostfix = getPostfix();
}
File tmpFile;
if (getBaseDirectory() == null) {
// create a temporary file
tmpFile = File.createTempFile(getPrefix(), newpostfix);
} else {
tmpFile = File.createTempFile(getPrefix(), newpostfix, new File(
getBaseDirectory()));
}
if (deleteOnExit()) {
tmpFile.deleteOnExit();
}
return tmpFile;
}
- 在这一部分可以看到，
DiskAttribute通过tempFile创建文件保存参数，由于项目是跑在tomcat中的，先不说代码的编写问题，由于上文的那个坑，导致了当有post请求的时候，就会在tomcat的temp的目录中产生
1534617419885.png
类似的临时文件，项目在服务器跑了两个月后，由于在netty请求过后没有主动的去清理这一类文件，导致了临时文件一致在无限的堆积。最终导致了temp目录中产生了上千万的文件，撑爆了temp目录的indode(具体自己百度)，导致了文件无法往内写入，为此导致了java.io.IOException: No space left on device的异常的抛出。随后，在每当请求结束的时候，通过调用decoder的cleanfiler直接取清理临时文件。
三.对于内存溢出问题的分析
-
在上一个问题件解决后，服务器频繁的出现了内存溢出的问题，即使jvm最大内存配置到了8G，也无法避免溢出的问题，为此，在tomcat的参数中加上了这么一个配置
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/apache-tomcat-7.0.88/bin/配置之后，根据内存溢出后自动会在bin目录下产生一个jvm溢出时候的dump，通过mat进行内存溢出的分析，得到的异常报告如下:
1534619622657.png为此，从内存堆积着一堆无法GC的
LinkedHashMap,而这个Map来自于java.io.DeleteOnExitHook这个对象。为此，从源码中可以看到:
private File tempFile() throws IOException {
String newpostfix;
String diskFilename = getDiskFilename();
if (diskFilename != null) {
newpostfix = '_' + diskFilename;
} else {
newpostfix = getPostfix();
}
File tmpFile;
if (getBaseDirectory() == null) {
// create a temporary file
tmpFile = File.createTempFile(getPrefix(), newpostfix);
} else {
tmpFile = File.createTempFile(getPrefix(), newpostfix, new File(
getBaseDirectory()));
}
if (deleteOnExit()) {
tmpFile.deleteOnExit();
}
return tmpFile;
}
在新建temp文件的时候会调用tmpFile.deleteOnExit();这么一句话，跟进源码：
public void deleteOnExit() {
SecurityManager security = System.getSecurityManager();
if (security != null) {
security.checkDelete(path);
}
if (isInvalid()) {
return;
}
DeleteOnExitHook.add(path);
}
package java.io;
import java.util.*;
import java.io.File;
/**
* This class holds a set of filenames to be deleted on VM exit through a shutdown hook.
* A set is used both to prevent double-insertion of the same file as well as offer
* quick removal.
*/
class DeleteOnExitHook {
private static LinkedHashSet<String> files = new LinkedHashSet<>();
static {
// DeleteOnExitHook must be the last shutdown hook to be invoked.
// Application shutdown hooks may add the first file to the
// delete on exit list and cause the DeleteOnExitHook to be
// registered during shutdown in progress. So set the
// registerShutdownInProgress parameter to true.
sun.misc.SharedSecrets.getJavaLangAccess()
.registerShutdownHook(2 /* Shutdown hook invocation order */,
true /* register even if shutdown in progress */,
new Runnable() {
public void run() {
runHooks();
}
}
);
}
private DeleteOnExitHook() {}
static synchronized void add(String file) {
if(files == null) {
// DeleteOnExitHook is running. Too late to add a file
throw new IllegalStateException("Shutdown in progress");
}
files.add(file);
}
static void runHooks() {
LinkedHashSet<String> theFiles;
synchronized (DeleteOnExitHook.class) {
theFiles = files;
files = null;
}
ArrayList<String> toBeDeleted = new ArrayList<>(theFiles);
// reverse the list to maintain previous jdk deletion order.
// Last in first deleted.
Collections.reverse(toBeDeleted);
for (String filename : toBeDeleted) {
(new File(filename)).delete();
}
}
}
- DeleteOnExitHook主要用于当
File对象调用deleteOnExit方法时，会在DeleteOnExitHook的LinkedHashMap存放相关的文件信息，用于在JVM正常退出的时候进行文件的清理 - 为此，由于DeleteOnExitHook中的
LinkedHashMap中存在着tomcat的temp目录下的大量引用信息，在decoder.cleanFiles();中，虽然清除了临时文件，但是，文件已经正常了清理，检查代码发现没有什么异常问题，最终调用的是file.delete(),但是file.delete()并没有对LinkedHashMap中的索引进行删除，导致了文件的堆积。
@Override
public void cleanRequestHttpData(HttpRequest request) {
List<HttpData> fileToDelete = requestFileDeleteMap.remove(request);
if (fileToDelete != null) {
for (HttpData data: fileToDelete) {
data.delete();
}
fileToDelete.clear();
}
}
@Override
public void delete() {
if (fileChannel != null) {
try {
fileChannel.force(false);
fileChannel.close();
} catch (IOException e) {
logger.warn("Failed to close a file.", e);
}
fileChannel = null;
}
if (! isRenamed) {
if (file != null && file.exists()) {
if (!file.delete()) {
logger.warn("Failed to delete: {}", file);
}
}
file = null;
}
}
四.总结
- 对于以上问题，最终根源在于
DefaultHttpDataFactory(DefaultHttpDataFactory.MAXSIZE)，为此绕了一个很大的弯子，只需要将其改为DefaultHttpDataFactory(DefaultHttpDataFactory.MINSIZE)即可，为此记录主要是对于netty对请求参数的解析以及相关的坑的留意。 - 在此次的问题中在于自己对netty的使用上还不熟悉，但是在这个解决问题过程中，自己学会通过jmap进行jvm分析，通过jstack进行线程堆栈的分析。
