logback 死锁问题

遇到生产环境锁死的问题,nginx 出现大量的 502。以前也偶尔出现,都是快速重启恢复服务。今天又出现了,抓了下日志,发现居然是 logback 锁死的,坑爹。


[root@host ~]# grep 0x000000008683cf68 catalina.out.bak
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - locked <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        - waiting to lock <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)

实际锁死线程:


"http-nio-8080-exec-188" #671375 daemon prio=5 os_prio=0 tid=0x00007ff239512000 nid=0x7519 runnable [0x00007ff18ceae000]
   java.lang.Thread.State: RUNNABLE
        at java.util.Arrays.copyOf(Arrays.java:3181)
        at java.text.DateFormatSymbols.copyMembers(DateFormatSymbols.java:850)
        at java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:758)
        at java.text.DateFormatSymbols.<init>(DateFormatSymbols.java:145)
        at sun.util.locale.provider.DateFormatSymbolsProviderImpl.getInstance(DateFormatSymbolsProviderImpl.java:85)
        at java.text.DateFormatSymbols.getProviderInstance(DateFormatSymbols.java:364)
        at java.text.DateFormatSymbols.getInstance(DateFormatSymbols.java:340)
        at java.util.Calendar.getDisplayName(Calendar.java:2110)
        at java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:1125)
        at java.text.SimpleDateFormat.format(SimpleDateFormat.java:966)
        at java.text.SimpleDateFormat.format(SimpleDateFormat.java:936)
        at java.text.DateFormat.format(DateFormat.java:345)
        at ch.qos.logback.core.util.CachingDateFormatter.format(CachingDateFormatter.java:48)
        - locked <0x000000008683cf68> (a ch.qos.logback.core.util.CachingDateFormatter)
        at ch.qos.logback.classic.pattern.DateConverter.convert(DateConverter.java:61)
        at ch.qos.logback.classic.pattern.DateConverter.convert(DateConverter.java:23)
        at ch.qos.logback.core.pattern.FormattingConverter.write(FormattingConverter.java:36)
        at ch.qos.logback.core.pattern.PatternLayoutBase.writeLoopOnConverters(PatternLayoutBase.java:115)
        at ch.qos.logback.classic.PatternLayout.doLayout(PatternLayout.java:141)
        at ch.qos.logback.classic.PatternLayout.doLayout(PatternLayout.java:39)
        at ch.qos.logback.core.encoder.LayoutWrappingEncoder.encode(LayoutWrappingEncoder.java:115)
        at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:230)
        at ch.qos.logback.core.rolling.RollingFileAppender.subAppend(RollingFileAppender.java:235)
        at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
        at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
        at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
        at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
        at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
        at ch.qos.logback.classic.Logger.filterAndLog_2(Logger.java:414)
        at ch.qos.logback.classic.Logger.warn(Logger.java:700)
        at com.xxx.common.cache.Cache.cancel(Cache.java:140)
        at com.xxx.common.cache.Cache.schedule(Cache.java:84)
        at com.xxx.common.cache.Cache.expireIn(Cache.java:238)
        at com.xxx.common.cache.Cache.setObject(Cache.java:216)
        at com.xxx.platform.cache.CacheManager.setObject(CacheManager.java:66)
        at com.xxx.platform.session.SessionUtil.setUserInfo(SessionUtil.java:415)
        at com.xxx.platform.session.SessionUtil.setUserInfo(SessionUtil.java:356)
        at com.xxx.platform.session.SessionUtil.setUserInfo(SessionUtil.java:340)
        at com.xxx.platform.interceptor.ServletInterceptor.increaseActivities(ServletInterceptor.java:128)
        at com.xxx.platform.interceptor.ServletInterceptor.before(ServletInterceptor.java:57)
        at com.xxx.web.ActionServlet.__do_service__(ActionServlet.java:321)
        at com.xxx.web.ActionServlet.service(ActionServlet.java:287)

logback作者说1.2.x已经解决死锁的问题了,然而我们用的是1.2.3,貌似并没有解决,照样锁死。具体原因是什么,暂时不知道,先记录一下问题。

©著作权归作者所有,转载或内容合作请联系作者
平台声明:文章内容(如有图片或视频亦包括在内)由作者上传并发布,文章内容仅代表作者本人观点,简书系信息发布平台,仅提供信息存储服务。

推荐阅读更多精彩内容

  • 在Java多线程开发中死锁问题并不少见,当线程间相互等待资源,而又不释放自身的资源时就会导致无穷无尽的等待。 举一...
    猫清扬阅读 469评论 0 0
  • 记得以前看过一个笑话:面试官问死锁怎么回事儿?我想了想,然后回答面试官说你先发offer给我,我再回答你。这就是现...
    From64KB阅读 338评论 0 0
  • 多线程开发避不开锁,而锁又避不开死锁问题,所以弄清楚死锁问题才能开发出好的多线程程序。 死锁出现原因与解决方法 在...
    IT乐知阅读 186评论 0 0
  • 现象 某天晚上,同事正在发布,突然线上大量报警,很多是关于数据库死锁的,报警提示信息如下: 通过报警,我们基本可以...
    小陈阿飞阅读 1,915评论 0 3
  • 如果遇到线上问题,需要从几个方面排查问题。接口成功率,响应时间,CPU、内存占用率,IO性能①如果是接口成功率为低...
    会跳的八爪鱼阅读 940评论 0 3