起因
更新测试环境下的服务器端，更新完后发现一个hs_err_pid*.log文件，一看文件最后修改日期是刚刚的
环境
- CentOS 7
- Java 8
日志分析
日志的概述
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f948b84b40d, pid=9068, tid=0x00007f94614b8700
#
# JRE version: OpenJDK Runtime Environment (8.0_262-b10) (build 1.8.0_262-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.262-b10 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libc.so.6+0x15640d] __memcpy_ssse3_back+0x19cd
#
# Core dump written. Default location: /home/fun/waw/uuo_java_waw/core or core.9068
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
根据概述中的以下内容可以看出是由native code引起的线程崩溃
# Problematic frame:
# C [libc.so.6+0x15640d] __memcpy_ssse3_back+0x19cd
- C: Native C frame
- j: Interpreted Java frame
- V: VMframe
- v: VMgenerated stub frame
- J: Other frame types, including compiled Java frames
# The crash happened outside the Java Virtual Machine in native code.
引起崩溃的线程
线程概述
--------------- T H R E A D ---------------
Current thread (0x00007f9448003000): JavaThread "pool-1-thread-23468" [_thread_in_native, id=23762, stack(0x00007f94613b8000,0x00007f94614b9000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x00007f9488027000
线程类型:
- JavaThread java 线程
- VMThread：jvm 的内部线程
- CompilerThread：用来调用 JITing，实时编译装卸 class
- GCTaskThread：执行 gc 的线程
- WatcherThread：jvm 周期性任务调度的线程
- ConcurrentMarkSweepThread：jvm 在进行 CMS GC 的时候，会创建一个该线程去进行 GC
线程状态:
- _thread_uninitialized：线程还没有创建，它只在内存原因崩溃的时候才出现
- _thread_new：线程已经被创建，但是还没有启动
- _thread_in_native：线程正在执行本地代码
- _thread_in_vm：线程正在执行虚拟机代码
- _thread_in_Java：线程正在执行解释或者编译后的 Java 代码
- _thread_blocked：线程处于阻塞状态
- *_trans：以_trans 结尾，线程正处于要切换到其它状态的中间状态
这里可以看出是 Java 线程JavaThread 在执行本地代码_thread_in_native时进程崩溃的
崩溃前执行的代码
Stack: [0x00007f94613b8000,0x00007f94614b9000], sp=0x00007f94614b5298, free space=1012k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0x15640d] __memcpy_ssse3_back+0x19cd
C [libzip.so+0x60b4] ZIP_GetEntry2+0xf4
C [libzip.so+0x3c9d] Java_java_util_zip_ZipFile_getEntry+0xfd
J 135 java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007f947515060e [0x00007f9475150540+0xce]
J 1509 C2 sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (85 bytes) @ 0x00007f94751f05ac [0x00007f94751effe0+0x5cc]
J 530 C1 sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (83 bytes) @ 0x00007f947521c8ac [0x00007f947521c740+0x16c]
J 507 C1 java.net.URLClassLoader$1.run()Ljava/lang/Class; (63 bytes) @ 0x00007f947520f39c [0x00007f947520f1a0+0x1fc]
J 506 C1 java.net.URLClassLoader$1.run()Ljava/lang/Object; (5 bytes) @ 0x00007f947520b4bc [0x00007f947520b440+0x7c]
v ~StubRoutines::call_stub
V [libjvm.so+0x69c03e] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0xf5e
···略
V [libjvm.so+0xafe902] JavaThread::thread_main_inner()+0x212
V [libjvm.so+0x93a382] java_start(Thread*)+0xf2
C [libpthread.so.0+0x7ea5] start_thread+0xc5
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 135 java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007f9475150598 [0x00007f9475150540+0x58]
J 1509 C2 sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (85 bytes) @ 0x00007f94751f05ac [0x00007f94751effe0+0x5cc]
···略
j sun.net.httpserver.ServerImpl$Exchange.run()V+903
J 1672 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f947525bad4 [0x00007f947525b540+0x594]
J 1718 C2 java.lang.Thread.run()V (17 bytes) @ 0x00007f94752a14b4 [0x00007f94752a1460+0x54]
v ~StubRoutines::call_stub
根据以上信息问题在C [libzip.so+0x60b4] ZIP_GetEntry2+0xf4。在JDK BUG SYSTEM和Java Bug Datebase上搜ZIP_GetEntry 找出一篇文章翻译成中文如下:
在大多数情况下，当正在访问的 jar 文件在 JVM 实例运行时被修改/覆盖时，ZIP_GetEntry 发生崩溃。出于性能原因，HotSpot JVM 内存使用 mmap 映射每个 Jar 文件的中央目录结构。这样做是为了避免每次需要从 Jar 文件中读取条目时从磁盘读取中央目录结构数据。当修改或覆盖磁盘上的 Jar 文件时，先前读取的数据的 JVM 副本与磁盘上的 jar 文件不一致，并且任何尝试从修改后的 jar 中读取和加载条目都可能导致应用程序崩溃。从 1.6.0_23 开始，可以使用一个属性来禁用 Jar 文件的中央目录结构的内存映射：
-Dsun.zip.disableMemoryMapping = true
请注意，启用此属性会对应用程序产生一些性能影响，因为 JVM 需要在每次读取 Jar 文件条目时一次又一次从磁盘上的 Jar 文件中读取中央目录结构。因此，最好确保在 JVM 加载了映像文件时，不会修改或覆盖 jar 文件。在 Java 9 中，此 ZIP 崩溃已通过以下增强功能得以解决： JDK-8142508：将 juzZipFile 的本机实现引入 Java 中，以消除昂贵的 jni 成本和 mmap 崩溃风险
到这里到就基本确定了问题的原因
当前所有线程信息
Java Threads: ( => current thread )
=>0x00007f9448003000 JavaThread "pool-1-thread-23468" [_thread_in_native, id=23762, stack(0x00007f94613b8000,0x00007f94614b9000)]
0x00007f948404b800 JavaThread "DestroyJavaVM" [_thread_blocked, id=9071, stack(0x00007f948c41b000,0x00007f948c51b000)]
···略
0x00007f9484116800 JavaThread "C1 CompilerThread1" daemon [_thread_blocked, id=9085, stack(0x00007f9474dff000,0x00007f9474eff000)]
0x00007f94840da800 JavaThread "Finalizer" daemon [_thread_blocked, id=9077, stack(0x00007f9488128000,0x00007f9488229000)]
0x00007f94840d5800 JavaThread "Reference Handler" daemon [_thread_blocked, id=9076, stack(0x00007f9488229000,0x00007f948832a000)]
Other Threads:
0x00007f94840cc000 VMThread [stack: 0x00007f948832b000,0x00007f948842b000] [id=9075]
0x00007f948411c800 WatcherThread [stack: 0x00007f9474bfd000,0x00007f9474cfd000] [id=9087]
虚拟机状态
VM state:not at safepoint (normal execution)
虚拟机状态:
- not at safepoint:正常运行
- at safepoint：所有线程都因为虚拟机等待状态而阻塞，等待一个虚拟机操作完成
- synchronizing：一个特殊的虚拟机操作，要求虚拟机内的其它线程保持等待状态
虚拟机锁
VM Mutex/Monitor currently owned by a thread: None
Mutex（互斥锁）虚拟机内部的锁 Monitor（监视器／管程）跨进程的
堆信息
Heap:
def new generation total 8640K, used 3933K [0x00000000e4e00000, 0x00000000e5750000, 0x00000000edea0000)
eden space 7744K, 50% used [0x00000000e4e00000, 0x00000000e51d4710, 0x00000000e5590000)
from space 896K, 1% used [0x00000000e5670000, 0x00000000e5672d10, 0x00000000e5750000)
to space 896K, 0% used [0x00000000e5590000, 0x00000000e5590000, 0x00000000e5670000)
tenured generation total 19136K, used 5582K [0x00000000edea0000, 0x00000000ef150000, 0x0000000100000000)
the space 19136K, 29% used [0x00000000edea0000, 0x00000000ee413b70, 0x00000000ee413c00, 0x00000000ef150000)
Metaspace used 13696K, capacity 13864K, committed 14080K, reserved 1062912K
class space used 1416K, capacity 1468K, committed 1536K, reserved 1048576K
Card table
Card table byte_map: [0x00007f9488f43000,0x00007f948901d000] byte_map_base: 0x00007f948881c000
Card table表示一种卡表，是 jvm 维护的一种数据结构，用于记录更改对象时的引用，以便 gc 时遍历更少的 table 和 root。
本地代码缓存
CodeCache: size=245760Kb used=4617Kb max_used=4660Kb free=241143Kb
bounds [0x00007f9475000000, 0x00007f94754b0000, 0x00007f9484000000]
total_blobs=1747 nmethods=1327 adapters=333
compilation: enabled
用于编译和保存native code的内存。
编译事件
Compilation events (250 events):
Event: 95753.631 Thread 0x00007f9484116800 1632 3 java.lang.Thread::setDaemon (25 bytes)
Event: 95753.631 Thread 0x00007f9484116800 nmethod 1632 0x00007f94753dc150 code [0x00007f94753dc300, 0x00007f94753dc828]
···略
一共编译了 250 次
gc 历史
GC Heap History (250 events):
Event: 169542.176 GC heap before
{Heap before GC invocations=10756 (full 0):
def new generation total 8640K, used 7904K [0x00000000e4e00000, 0x00000000e5750000, 0x00000000edea0000)
eden space 7744K, 100% used [0x00000000e4e00000, 0x00000000e5590000, 0x00000000e5590000)
from space 896K, 17% used [0x00000000e5590000, 0x00000000e55b8098, 0x00000000e5670000)
to space 896K, 0% used [0x00000000e5670000, 0x00000000e5670000, 0x00000000e5750000)
tenured generation total 19136K, used 5431K [0x00000000edea0000, 0x00000000ef150000, 0x0000000100000000)
the space 19136K, 28% used [0x00000000edea0000, 0x00000000ee3edf38, 0x00000000ee3ee000, 0x00000000ef150000)
Metaspace used 13677K, capacity 13864K, committed 14080K, reserved 1062912K
class space used 1415K, capacity 1468K, committed 1536K, reserved 1048576K
Event: 169542.178 GC heap after
Heap after GC invocations=10757 (full 0):
def new generation total 8640K, used 160K [0x00000000e4e00000, 0x00000000e5750000, 0x00000000edea0000)
eden space 7744K, 0% used [0x00000000e4e00000, 0x00000000e4e00000, 0x00000000e5590000)
from space 896K, 17% used [0x00000000e5670000, 0x00000000e5698190, 0x00000000e5750000)
to space 896K, 0% used [0x00000000e5590000, 0x00000000e5590000, 0x00000000e5670000)
tenured generation total 19136K, used 5431K [0x00000000edea0000, 0x00000000ef150000, 0x0000000100000000)
the space 19136K, 28% used [0x00000000edea0000, 0x00000000ee3edf38, 0x00000000ee3ee000, 0x00000000ef150000)
Metaspace used 13677K, capacity 13864K, committed 14080K, reserved 1062912K
class space used 1415K, capacity 1468K, committed 1536K, reserved 1048576K
}
···略
gc 次数（full gc），这里有每次 gc 前后的内存信息
Deoptimization events (250 events)
Classes redefined (0 events)
Internal exceptions (250 events)
异常信息
Events (250 events)
Dynamic libraries
jvm 内存映射,这些信息是虚拟机崩溃时的虚拟内存列表区域。它可以告诉你崩溃原因时哪些类库正在被使用，位置在哪里，还有堆栈和守护页信息。
VM Arguments
Environment Variables
Signal Handlers
操作系统信息
- 操作信息信息
- 内存
- cpu
