-
基本信息
客户名称：xxx
产品名称：
版本号：
问题分类：编码问题
环境类型：实体机
-
问题现象
保险公司测试环境爆内存，机器挂掉。总共64g的内存，在没有开始测试tomcat部署war包前内存使用率为25%左右。开始测试tomcat后，内存使用率一直上涨，内存使用率从12-08下午起就维持在98%的水平，知道最后机器down掉，才发现存在的问题。
-
问题原因
tomcat执行shutdown后，虽然端口占用被释放，tomcat也已经停止。但tomcat的进程并未被杀死，仍旧占用着一开始分配给他的xmx的jvm内存空间。
Tomcat是一个Servlet容器，用于部署Serlvet程序（我们通常写的各种Java Web应用本质上就是一个Servlet程序）。也就说，在停止Tomcat时不仅仅需要释放Tomcat进程本身所占用的资源，还需要释放Serlvet程序所占用的资源。而出现“停止Tomcat之后进程依然存在”这种现象的主要原因就是：我们自己写的Java Web应用在Tomcat容器停止时没有正常释放所占用的系统资源，比如：线程池未关闭，输入输出流未关闭等等。这些系统资源没有被完全释放，导致tomcat进程一直存在，且无法被正常回收。
下面是tomcat的shutdown执行的catalina命令，可以看到是程序是先调用Bootstrap类的方法释放Tomcat进程所占用的资源。再使用kill命令停止Tomcat进程：kill -15 <tomcat_process_id>。到在这里使用kill命令发送的信号为SIGTERM（15），kill -15在进程未释放系统资源时不能停止Tomcat进程（如：程序的线程池未关闭）。
eval ""$_RUNJAVA"" $LOGGING_MANAGER $JAVA_OPTS-Djava.endorsed.dirs=""$JAVA_ENDORSED_DIRS"" -classpath ""$CLASSPATH""-Dcatalina.base=""$CATALINA_BASE""-Dcatalina.home=""$CATALINA_HOME""-Djava.io.tmpdir=""$CATALINA_TMPDIR""org.apache.catalina.startup.Bootstrap "$@" stop
# stop failed. Shutdown port disabled? Try a normal kill.if [ $? != 0 ]; thenif [ ! -z "$CATALINA_PID" ]; thenecho "The stop command failed. Attempting to signal the process to stop through OS signal."kill -15 `cat "$CATALINA_PID"` >/dev/null 2>&1fifi-
排查过程
察觉到问题发生时，机器已经down掉了。服务器管理员已经执行了重启的操作，时间点为10:29。因此无法直接得知机器当时的内存占用的情况。只能通过一些命令获取重启前机器的部分信息
1、查看系统日志，查看宕机时内存和cpu的情况
1.1、查看是否有杀内存的情况发生：cat /var/log/messages |grep -i "kill"
可以看到在宕机时，发生了杀内存的情况，证明在机器出问题之前，内存已经相当吃紧。基本上可以确认问题就出在机器内存满了。
1.2、查看历史cpu占用情况：sar -u -f /var/log/sa/sa09
cup看起来没什么异常
1.3、查看历史内存：sar -r -f /var/log/sa/sa09
可以看到内存占用一直处在99.45%的超高状态，用户占有率高达181%
1.4、再向前看一天 sar -r -f /var/log/sa/sa09
可以观察到8时50分和9时00分，内存激增10g左右。这个有明显的异常，且内存激增的时间点和tomcat开始测试的时间点吻合。因此怀疑是tomcat导致的问题。
2、本地测试，尝试还原问题原因
2.1、启动tomcat容器和容器中的luna-project-hub服务，查看内存占用
netstat -tnlp | grep 8455（port）
ps -ef | grep 2730（pid）
2.2、关闭tomcat
执行shutdown命令后，项目已经关闭。
重新执行ps -ef | grep 2730，发现进程居然仍旧存在，没有被杀死。
2.3、jstack，查看tomcat关闭后，进程中仍旧存在的线程
发现存在的都是一些ScheduledExecutorService类创建的线程。这些线程没有无法关闭，导致了这次的问题。
[root@localhost bin]# jstack -l 2730
2022-12-09 17:16:57
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.301-b09 mixed mode):"Attach Listener" #215 daemon prio=9 os_prio=0 tid=0x00007f3608001800 nid=0xc4e waiting on condition [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- None"DestroyJavaVM" #214 prio=5 os_prio=0 tid=0x00007f364c5fe800 nid=0xb27 waiting on condition [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- None"CACHE_LOAD_POOL1" #184 prio=5 os_prio=0 tid=0x00007f35d829e800 nid=0xbe1 waiting on condition [0x00007f359bfdf000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000b0a627b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"com.alibaba.nacos.naming.beat.sender" #183 daemon prio=5 os_prio=0 tid=0x00007f35b8008000 nid=0xbe0 waiting on condition [0x00007f359c0e0000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000ab99ee30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"DictionaryCacheRefreshThreadPool1" #181 prio=5 os_prio=0 tid=0x00007f35d8296000 nid=0xbde waiting on condition [0x00007f359c1e1000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000b0a8f210> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"Curator-Framework-0" #178 daemon prio=5 os_prio=0 tid=0x00007f35a899f800 nid=0xbdb waiting on condition [0x00007f359c4e2000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000b16411d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:938)at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- <0x00000000b16413d8> (a java.util.concurrent.ThreadPoolExecutor$Worker)"localhost-startStop-1-EventThread" #177 daemon prio=5 os_prio=0 tid=0x00007f35a899d000 nid=0xbda waiting on condition [0x00007f359c5e3000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000b1647918> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:504)Locked ownable synchronizers:- None"localhost-startStop-1-SendThread(10.60.44.187:2181)" #176 daemon prio=5 os_prio=0 tid=0x00007f35a899b800 nid=0xbd9 runnable [0x00007f359c6e4000]java.lang.Thread.State: RUNNABLEat sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)- locked <0x00000000b1618ba8> (a sun.nio.ch.Util$3)- locked <0x00000000b1618b98> (a java.util.Collections$UnmodifiableSet)- locked <0x00000000b16186f0> (a sun.nio.ch.EPollSelectorImpl)at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)Locked ownable synchronizers:- None"Curator-ConnectionStateManager-0" #175 daemon prio=5 os_prio=0 tid=0x00007f35a892e800 nid=0xbd8 waiting on condition [0x00007f359c7e5000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000b1641938> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:272)at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:44)at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:133)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- <0x00000000b1641b30> (a java.util.concurrent.ThreadPoolExecutor$Worker)"TaskCommandCacheRefreshThreadPool1" #173 prio=5 os_prio=0 tid=0x00007f35a81ff000 nid=0xbd6 waiting on condition [0x00007f359c8e6000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000aedc4b70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"com.alibaba.nacos.naming.beat.sender" #172 daemon prio=5 os_prio=0 tid=0x00007f35a81fd000 nid=0xbd5 waiting on condition [0x00007f359c9e7000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000ab99ee30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"THREAD_POOL_ALARM_POOL-1" #169 prio=5 os_prio=0 tid=0x00007f35a80fc800 nid=0xbd3 waiting on condition [0x00007f359d7e9000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000b0e8b0e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"Timer-3" #162 daemon prio=5 os_prio=0 tid=0x00007f35a8b0b000 nid=0xbcc in Object.wait() [0x00007f359e6ee000]java.lang.Thread.State: WAITING (on object monitor)at java.lang.Object.wait(Native Method)at java.lang.Object.wait(Object.java:502)at java.util.TimerThread.mainLoop(Timer.java:526)- locked <0x00000000ad465348> (a java.util.TaskQueue)at java.util.TimerThread.run(Timer.java:505)Locked ownable synchronizers:- None"com.alibaba.nacos.naming.push.receiver" #127 daemon prio=5 os_prio=0 tid=0x00007f35a80a7800 nid=0xba9 runnable [0x00007f35a3a19000]java.lang.Thread.State: RUNNABLEat java.net.PlainDatagramSocketImpl.receive0(Native Method)- locked <0x00000000ab99e588> (a java.net.PlainDatagramSocketImpl)at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:149)- locked <0x00000000ab99e588> (a java.net.PlainDatagramSocketImpl)at java.net.DatagramSocket.receive(DatagramSocket.java:812)- locked <0x00000000abe96300> (a java.net.DatagramPacket)- locked <0x00000000ab99e548> (a java.net.DatagramSocket)at com.alibaba.nacos.client.naming.core.PushReceiver.run(PushReceiver.java:83)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- <0x00000000ab99e2d8> (a java.util.concurrent.ThreadPoolExecutor$Worker)"com.alibaba.nacos.naming.failover" #126 daemon prio=5 os_prio=0 tid=0x00007f35a80a5000 nid=0xba8 waiting on condition [0x00007f35a3b1a000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000ab9b4980> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"com.alibaba.nacos.client.naming.updater" #125 daemon prio=5 os_prio=0 tid=0x00007f35a80a3000 nid=0xba7 waiting on condition [0x00007f35a3c1b000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000ab9b3c60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"com.alibaba.nacos.client.naming.updater" #124 daemon prio=5 os_prio=0 tid=0x00007f35a80a1000 nid=0xba6 waiting on condition [0x00007f35a3d1c000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000ab9b3c60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"com.alibaba.nacos.naming.client.listener" #123 daemon prio=5 os_prio=0 tid=0x00007f35a809e800 nid=0xba5 waiting on condition [0x00007f35a3e1d000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000ab99eb90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)at com.alibaba.nacos.client.naming.core.EventDispatcher$Notifier.run(EventDispatcher.java:163)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- <0x00000000ab99e918> (a java.util.concurrent.ThreadPoolExecutor$Worker)"AsyncAppender-Worker-ConsoleAppender" #117 daemon prio=5 os_prio=0 tid=0x00007f35e5959000 nid=0xb9f waiting on condition [0x00007f35a5722000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000aa244020> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)Locked ownable synchronizers:- None"com.alibaba.nacos.client.Worker.fixed-10.60.52.109_8848-insurance-ats-upgrade-urtrust" #114 daemon prio=5 os_prio=0 tid=0x00007f35e59d2000 nid=0xb9c runnable [0x00007f35a501f000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000aa250eb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"com.alibaba.nacos.client.config.security.updater" #113 daemon prio=5 os_prio=0 tid=0x00007f35e59d0000 nid=0xb9b waiting on condition [0x00007f35a5120000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000aa2506b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"Timer-2" #112 daemon prio=5 os_prio=0 tid=0x00007f35e59c9800 nid=0xb9a in Object.wait() [0x00007f35a5421000]java.lang.Thread.State: TIMED_WAITING (on object monitor)at java.lang.Object.wait(Native Method)at java.util.TimerThread.mainLoop(Timer.java:552)- locked <0x00000000aa251358> (a java.util.TaskQueue)at java.util.TimerThread.run(Timer.java:505)Locked ownable synchronizers:- None"AsyncAppender-Worker-ConsoleAppender" #108 daemon prio=5 os_prio=0 tid=0x00007f35e5945800 nid=0xb96 waiting on condition [0x00007f35a5b24000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000aa0d74c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)Locked ownable synchronizers:- None"CACHE_LOAD_POOL1" #100 prio=5 os_prio=0 tid=0x00007f3618022000 nid=0xb8c waiting on condition [0x00007f35a6a27000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a771aee0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"com.alibaba.nacos.naming.beat.sender" #99 daemon prio=5 os_prio=0 tid=0x00007f35ec004000 nid=0xb8b waiting on condition [0x00007f35a6d28000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a3300ea8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"DictionaryCacheRefreshThreadPool1" #96 prio=5 os_prio=0 tid=0x00007f3618003000 nid=0xb88 waiting on condition [0x00007f35a6e29000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a771abb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"Curator-Framework-0" #93 daemon prio=5 os_prio=0 tid=0x00007f35e7630000 nid=0xb85 waiting on condition [0x00007f35a7ffe000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a83f4b90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:938)at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- <0x00000000a83f4d90> (a java.util.concurrent.ThreadPoolExecutor$Worker)"localhost-startStop-1-EventThread" #92 daemon prio=5 os_prio=0 tid=0x00007f35e70f6000 nid=0xb84 waiting on condition [0x00007f35ac550000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a83f4490> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:504)Locked ownable synchronizers:- None"localhost-startStop-1-SendThread(10.60.44.187:2181)" #91 daemon prio=5 os_prio=0 tid=0x00007f35e70f4800 nid=0xb83 runnable [0x00007f35ac651000]java.lang.Thread.State: RUNNABLEat sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)- locked <0x00000000a83e2b00> (a sun.nio.ch.Util$3)- locked <0x00000000a83e2af0> (a java.util.Collections$UnmodifiableSet)- locked <0x00000000a83e2648> (a sun.nio.ch.EPollSelectorImpl)at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)Locked ownable synchronizers:- None"Curator-ConnectionStateManager-0" #90 daemon prio=5 os_prio=0 tid=0x00007f35e749c800 nid=0xb82 waiting on condition [0x00007f35ac952000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a83f52d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:272)at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:44)at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:133)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- <0x00000000a83f54c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)"TaskCommandCacheRefreshThreadPool1" #88 prio=5 os_prio=0 tid=0x00007f35e6b33800 nid=0xb80 waiting on condition [0x00007f35aca53000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a48584c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"com.alibaba.nacos.naming.beat.sender" #86 daemon prio=5 os_prio=0 tid=0x00007f35e6b2f800 nid=0xb7e waiting on condition [0x00007f35acc55000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a3300ea8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"THREAD_POOL_ALARM_POOL-1" #83 prio=5 os_prio=0 tid=0x00007f35e7185800 nid=0xb7c waiting on condition [0x00007f35ad257000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a7f8c2e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"Java2D Disposer" #81 daemon prio=10 os_prio=0 tid=0x00007f35e76f4800 nid=0xb7b in Object.wait() [0x00007f3630892000]java.lang.Thread.State: WAITING (on object monitor)at java.lang.Object.wait(Native Method)- waiting on <0x00000000a7a92958> (a java.lang.ref.ReferenceQueue$Lock)at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)- locked <0x00000000a7a92958> (a java.lang.ref.ReferenceQueue$Lock)at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)at sun.java2d.Disposer.run(Disposer.java:148)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"Timer-1" #74 daemon prio=5 os_prio=0 tid=0x00007f35e7ab2000 nid=0xb74 in Object.wait() [0x00007f35af408000]java.lang.Thread.State: WAITING (on object monitor)at java.lang.Object.wait(Native Method)- waiting on <0x00000000a420e9f0> (a java.util.TaskQueue)at java.lang.Object.wait(Object.java:502)at java.util.TimerThread.mainLoop(Timer.java:526)- locked <0x00000000a420e9f0> (a java.util.TaskQueue)at java.util.TimerThread.run(Timer.java:505)Locked ownable synchronizers:- None"com.alibaba.nacos.naming.push.receiver" #39 daemon prio=5 os_prio=0 tid=0x00007f35e6d09800 nid=0xb51 runnable [0x00007f362adfa000]java.lang.Thread.State: RUNNABLEat java.net.PlainDatagramSocketImpl.receive0(Native Method)- locked <0x00000000a3300670> (a java.net.PlainDatagramSocketImpl)at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:149)- locked <0x00000000a3300670> (a java.net.PlainDatagramSocketImpl)at java.net.DatagramSocket.receive(DatagramSocket.java:812)- locked <0x00000000a34f2448> (a java.net.DatagramPacket)- locked <0x00000000a3300630> (a java.net.DatagramSocket)at com.alibaba.nacos.client.naming.core.PushReceiver.run(PushReceiver.java:83)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- <0x00000000a33001e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)"com.alibaba.nacos.naming.failover" #38 daemon prio=5 os_prio=0 tid=0x00007f35e6d04800 nid=0xb50 waiting on condition [0x00007f362b0fb000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a33029e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"com.alibaba.nacos.client.naming.updater" #37 daemon prio=5 os_prio=0 tid=0x00007f35e6d01000 nid=0xb4f waiting on condition [0x00007f362b1fc000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a3301ed8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"com.alibaba.nacos.client.naming.updater" #36 daemon prio=5 os_prio=0 tid=0x00007f35e6cff000 nid=0xb4e waiting on condition [0x00007f362b2fd000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a3301ed8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"com.alibaba.nacos.naming.client.listener" #35 daemon prio=5 os_prio=0 tid=0x00007f35e6cf8800 nid=0xb4d waiting on condition [0x00007f362b3fe000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a3300c08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)at com.alibaba.nacos.client.naming.core.EventDispatcher$Notifier.run(EventDispatcher.java:163)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- <0x00000000a3300918> (a java.util.concurrent.ThreadPoolExecutor$Worker)"AsyncAppender-Worker-ConsoleAppender" #29 daemon prio=5 os_prio=0 tid=0x00007f35e6059800 nid=0xb47 waiting on condition [0x00007f3631f2d000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a20439f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)Locked ownable synchronizers:- None"com.alibaba.nacos.client.Worker.fixed-10.60.52.109_8848-insurance-ats-upgrade-urtrust" #26 daemon prio=5 os_prio=0 tid=0x00007f35e6387000 nid=0xb44 runnable [0x00007f3630690000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a20a3b98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"com.alibaba.nacos.client.config.security.updater" #25 daemon prio=5 os_prio=0 tid=0x00007f35e6380800 nid=0xb43 waiting on condition [0x00007f3630791000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a20a3090> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- None"Timer-0" #23 daemon prio=5 os_prio=0 tid=0x00007f35e634c800 nid=0xb41 in Object.wait() [0x00007f3630b93000]java.lang.Thread.State: TIMED_WAITING (on object monitor)at java.lang.Object.wait(Native Method)at java.util.TimerThread.mainLoop(Timer.java:552)- locked <0x00000000a2856cf0> (a java.util.TaskQueue)at java.util.TimerThread.run(Timer.java:505)Locked ownable synchronizers:- None"AsyncAppender-Worker-ConsoleAppender" #19 daemon prio=5 os_prio=0 tid=0x00007f35e61a3800 nid=0xb3d waiting on condition [0x00007f3631296000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a1e7a940> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)Locked ownable synchronizers:- None"GC Daemon" #12 daemon prio=2 os_prio=0 tid=0x00007f364c523000 nid=0xb36 in Object.wait() [0x00007f36381c9000]java.lang.Thread.State: TIMED_WAITING (on object monitor)at java.lang.Object.wait(Native Method)- waiting on <0x00000000a0f796d8> (a sun.misc.GC$LatencyLock)at sun.misc.GC$Daemon.run(GC.java:117)- locked <0x00000000a0f796d8> (a sun.misc.GC$LatencyLock)Locked ownable synchronizers:- None"AsyncFileHandlerWriter-1259475182" #11 daemon prio=5 os_prio=0 tid=0x00007f364c1b6000 nid=0xb35 waiting on condition [0x00007f3638520000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for <0x00000000a0f796e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)at org.apache.juli.AsyncFileHandler$LoggerThread.run(AsyncFileHandler.java:160)Locked ownable synchronizers:- None"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f364c0cd000 nid=0xb33 runnable [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- None"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f364c0c2000 nid=0xb32 waiting on condition [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- None"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f364c0c0000 nid=0xb31 waiting on condition [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- None"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f364c0bd000 nid=0xb30 waiting on condition [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- None"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f364c0bb800 nid=0xb2f runnable [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- None"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f364c088800 nid=0xb2e in Object.wait() [0x00007f3639351000]java.lang.Thread.State: WAITING (on object monitor)at java.lang.Object.wait(Native Method)at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)- locked <0x00000000a0f79d30> (a java.lang.ref.ReferenceQueue$Lock)at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)Locked ownable synchronizers:- None"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f364c084000 nid=0xb2d in Object.wait() [0x00007f3639452000]java.lang.Thread.State: WAITING (on object monitor)at java.lang.Object.wait(Native Method)at java.lang.Object.wait(Object.java:502)at java.lang.ref.Reference.tryHandlePending(Reference.java:191)- locked <0x00000000a0f79d70> (a java.lang.ref.Reference$Lock)at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)Locked ownable synchronizers:- None"VM Thread" os_prio=0 tid=0x00007f364c07a000 nid=0xb2c runnable"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f364c020800 nid=0xb28 runnable"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f364c022800 nid=0xb29 runnable"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f364c024800 nid=0xb2a runnable"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f364c026000 nid=0xb2b runnable"VM Periodic Task Thread" os_prio=0 tid=0x00007f364c0d2000 nid=0xb34 waiting on conditionJNI global references: 35412.4、tomcat关闭时的报错信息
tomcat关闭时，其实有线程无法关闭的报错信息。但当时认为端口已经释放，tomcat程序已经关闭，就没有意识到这个问题。
-
解决方案
手动关停无法关闭的tomcat的进程：
./shutdown.sh
ps -ef |grep tomcat
kill -9 pid
-
排查内存问题时可能用到的其他命令
查看当前内存占用高低排序：ps aux --sort -%mem查看内存占用最高的10个：ps aux | head -1;ps aux |grep -v PID |sort -rn -k +4 | head -10查看当前内存占用：free参考文档
排查思路：https://www.cnblogs.com/zcyy/p/16033677.html
tomcat关闭时，nacos线程报错关闭的原因：tomcat无法正常关闭问题分析及解决 - 码农教程
tomcat无法正常关闭的原因：记一次关于tomcat关闭时，清理线程时的警告日志问题排查。_org.apache.catalina.loader.webappclassloaderbase:1-CSDN博客
![[BUUCTF NewStarCTF 2023 公开赛道] week4 crypto/pwn](https://img-blog.csdnimg.cn/2434140bc5bf4cb1a22ce7b8e95f37dd.png)
