多线程栈OOM-jstack

failed: BoundedExecutor is in a failed state

 public void execute(Runnable task)
    {
        checkState(!failed.get(), "BoundedExecutor is in a failed state");

        queue.add(task);

        int size = queueSize.incrementAndGet();
        if (size <= maxThreads) {
            // If able to grab a permit (aka size <= maxThreads), then we are short exactly one draining thread
            try {
                coreExecutor.execute(this::drainQueue);
            }
            catch (Throwable e) {
                failed.set(true);
                log.error("BoundedExecutor state corrupted due to underlying executor failure");
                throw e;
            }
        }
    }

出现了线程OOM问题。上面一旦failed跑出来true就再也不行了。
2017-01-10T12:00:19.869+0800 ERROR Query-20170110_040019_00058_5rdfv-5997 io.airlift.concurrent.BoundedExecutor BoundedExecutor state corrupted due to underlying executor failure
2017-01-10T12:00:19.870+0800 ERROR hive-hive-2874 io.airlift.concurrent.BoundedExecutor BoundedExecutor state corrupted due to underlying executor failure
2017-01-10T12:00:19.870+0800 ERROR Query-20170110_040019_00057_5rdfv-5996 io.airlift.concurrent.BoundedExecutor BoundedExecutor state corrupted due to underlying executor failure
2017-01-10T12:00:19.872+0800 ERROR hive-hive-2856 io.airlift.concurrent.BoundedExecutor BoundedExecutor state corrupted due to underlying executor failure
2017-01-10T12:00:19.872+0800 WARN hive-hive-2874 com.facebook.presto.hive.util.ResumableTasks ResumableTask completed exceptionally
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
at io.airlift.concurrent.BoundedExecutor.execute(BoundedExecutor.java:62)
at com.facebook.presto.hive.HiveSplitManager$ErrorCodedExecutor.execute(HiveSplitManager.java:317)
at com.facebook.presto.hive.util.AsyncQueue.completeAsync(AsyncQueue.java:140)
at com.facebook.presto.hive.util.AsyncQueue.offer(AsyncQueue.java:94)
at com.facebook.presto.hive.HiveSplitSource.addToQueue(HiveSplitSource.java:67)
at com.facebook.presto.hive.HiveSplitSource.addToQueue(HiveSplitSource.java:59)
at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:247)
at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:78)
at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:179)
at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:45)
at com.facebook.presto.hive.util.ResumableTasks.lambda$submit$1(ResumableTasks.java:33)
at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

cat /proc/15676/status #查看进程状态
Threads: 325

java -XX:+PrintFlagsFinal -version | grep ThreadStackSize
intx CompilerThreadStackSize = 0
intx ThreadStackSize = 1024
intx VMThreadStackSize = 1024

cat /proc/sys/kernel/threads-max #允许最大的线程数
1030511

cat /proc/sys/kernel/pid_max #允许最大创建进程数
32768

pstree -p | wc -l #当前线程数
3283

pstree -p 15676 | wc -l #pid线程数
326

cat /proc/sys/vm/max_map_count #每一个进程最大VMA数量
65530

文件包含限制一个进程可以拥有的VMA(虚拟内存区域)的数量。虚拟内存区域是一个连续的虚拟地址空间区域。在进程的生命周期中,每当程序尝试在内存中映射文件,链接到共享内存段,或者分配堆空间的时候,这些区域将被创建。调优这个值将限制进程可拥有VMA的数量。限制一个进程拥有VMA的总数可能导致应用程序出错,因为当进程达到了VMA上线但又只能释放少量的内存给其他的内核进程使用时,操作系统会抛出内存不足的错误。如果你的操作系统在NORMAL区域仅占用少量的内存,那么调低这个值可以帮助释放内存给内核用。

pmap -x 15676 #查看单进程map的虚拟内存情况

sar -q #查看线程数(plist-sz)

http://blog.csdn.net/odailidong/article/details/50561257

vi /etc/security/limits.d/90-nproc.conf

内容如下:
* soft nproc 1024
root soft nproc unlimited

[hdfs@xxxx]$ ps h -Led -o user | sort | uniq -c | sort -n
806 hdfs
2786 root

ps -o nlwp,pid,lwp,args -u hdfs | sort -n
这里根据用户hdfs来查看第一列就是我们的线程数

最后编辑于 :
©著作权归作者所有,转载或内容合作请联系作者
【社区内容提示】社区部分内容疑似由AI辅助生成,浏览时请结合常识与多方信息审慎甄别。
平台声明:文章内容(如有图片或视频亦包括在内)由作者上传并发布,文章内容仅代表作者本人观点,简书系信息发布平台,仅提供信息存储服务。

相关阅读更多精彩内容

  • /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home...
    DeepThinkAI2026阅读 4,253评论 2赞 8
  • 背景 10月2号凌晨12:08收到报警,所有请求失败,处于完全不可用状态 应用服务器共四台resin,resin之...
    AGIHunt阅读 22,559评论 6赞 16
  • Android 自定义View的各种姿势1 Activity的显示之ViewRootImpl详解 Activity...
    passiontim阅读 181,011评论 25赞 708
  • 1、运行环境 主机IP 主机名 2、配置主机名(分别在五台机器上执行) hostname +主机名例如: h...
    献给记性不好的自己阅读 3,934评论 0赞 6
  • 结婚前一直梦想在鸟鸣山幽的地方有一处我的蜗居。在那里,槛外竹影摇曳生姿,门前溪流琮琮作响,窗前山风徐徐扑面。偶尔还...
    晓刀阅读 281评论 0赞 1

友情链接更多精彩内容