线上服务假死排查实战

2023-11-09 03:04:13

大家好，我是烤鸭：

最近线上问题有点多啊，分享一个服务假死排查过程。

问题

9点10分，收到进程无响应报警（一共6台机器，有1台出问题）。

排查步骤

登录服务器，查看进程状态

[root@server ~]# ps -ef | grep java
java  20973 20855  0 08:08:58 ?        00:00:02 /usr/java/jdk1.8.0_251/bin/java -server -Xmx1024m -Xms1024m -Xss256k -XX:+HeapDumpOnOutOfMemoryError -jar xxx.jar

可以看出，进程还在运行，但状态为“D”（不可中断的睡眠）。

使用gdb工具调试进程

[root@server ~]# gdb -p 20973
(gdb) thread apply all bt
#0  0x00007f437622c55d in poll () from /lib64/libc.so.6
#1  0x00007f437622c875 in poll () from /lib64/libc.so.6
#2  0x00007f437622cb0a in epoll_wait () from /lib64/libc.so.6
#3  0x00000000005061b9 in ?? () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#4  0x00000000005061f4 in ?? () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#5  0x000000000050624c in ?? () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#6  0x0000000000506293 in ?? () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#7  0x00000000005062e5 in ?? () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#8  0x0000000000506322 in ?? () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#9  0x0000000000506358 in ?? () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#10 0x0000000000506394 in ?? () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#11 0x00000000004008da in sun.nio.ch.EPollArrayWrapper.poll(int, int[])V () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#12 0x00000000004009c1 in sun.nio.ch.EPollSelectorImpl.doSelect(int)I () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#13 0x0000000000400a52 in sun.nio.ch.SelectorImpl.lockAndDoSelect(int)I () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#14 0x0000000000400b72 in sun.nio.ch.SelectorImpl.select(int)I () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#15 0x0000000000400ebd in sun.nio.ch.SelectorImpl.select()I () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#16 0x000000000040115e in sun.nio.ch.EPollSelectorImpl.select(int)I () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#17 0x000000000040904e in sun.nio.ch.ServerSocketChannelImpl.accept0(java.nio.channels.SocketChannel)Ljava.nio.channels.SocketChannel; () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#18 0x0000000000409450 in sun.nio.ch.ServerSocketChannelImpl.accept()Ljava.nio.channels.SocketChannel; () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#19 0x000000000040a107 in sun.nio.ch.ServerSocketChannelImpl.accept()Ljava.net.Socket; () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar
#20 0x00000000005d5d09 in org.apache.tomcat.util.net.NioEndpoint.accept(int)Ljava.net.Socket; () from /usr/local/tomcat/lib/tomcat-coyote.jar
#21 0x00000000005d5ba3 in org.apache.tomcat.util.net.NioEndpoint$Acceptor.run()V () from /usr/local/tomcat/lib/tomcat-coyote.jar
#22 0x00000000005d6038 in java.lang.Thread.run()V () from /usr/java/jdk1.8.0_251/jre/lib/rt.jar

从gdb的输出可以看出，进程被阻塞在epoll_wait()函数上。

查看日志文件

在日志文件中发现，在9点09分的时候，出现了一条“java.lang.OutOfMemoryError: Java heap space”的错误。

使用jstack工具查看线程栈信息

[root@server ~]# jstack 20973 > thread.dump

在thread.dump文件中，发现有一个线程一直在等待获取锁。

使用内存分析工具查看内存泄漏情况

[root@server ~]# jmap -dump:format=b,file=heapdump.bin 20973

使用jhat工具分析heapdump.bin文件，发现有一个对象占用了大量的内存，导致了内存泄漏。

解决方案

增加JVM堆内存大小

[root@server ~]# vim /etc/sysctl.conf
vm.max_map_count=655360

修复内存泄漏问题

public class Test {
    private static List<Object> list = new ArrayList<>();

    public static void main(String[] args) {
        while (true) {
            list.add(new Object());
        }
    }
}