大家好，我是烤鸭：

    今天分享一次线上tomcat宕机原因。

 

1 .  最近两次的宕机时间部分日志

  1.1

2019-04-04 13:22:27.187 [Thread-10] INFO  o.s.context.support.DefaultLifecycleProcessor - Stopping beans in phase 0
2019-04-04 13:22:27.188 [Thread-10] INFO  o.s.b.actuate.endpoint.jmx.EndpointMBeanExporter - Unregistering JMX-exposed beans on shutdown
2019-04-04 13:22:27.188 [Thread-10] INFO  o.s.b.actuate.endpoint.jmx.EndpointMBeanExporter - Unregistering JMX-exposed beans
2019-04-04 13:22:27.189 [Thread-10] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Unregistering JMX-exposed beans on shutdown
2019-04-04 13:22:27.189 [Thread-10] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Unregistering JMX-exposed beans
04-Apr-2019 13:22:28.364 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.alibaba.druid.proxy.DruidDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
04-Apr-2019 13:22:28.364 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
04-Apr-2019 13:22:28.366 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:[java.lang.Object.wait(Native Method)java.lang.Thread.join(Thread.java:1252)java.lang.Thread.join(Thread.java:1326)java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)java.lang.Shutdown.runHooks(Shutdown.java:123)java.lang.Shutdown.sequence(Shutdown.java:167)java.lang.Shutdown.exit(Shutdown.java:212)java.lang.Runtime.exit(Runtime.java:109)java.lang.System.exit(System.java:971)com.ctfo.customer.util.FtpUtils.connect(FtpUtils.java:73)com.ctfo.customer.service.impl.certification.CertificationServiceImpl.judgeCertifiedStatus(CertificationServiceImpl.java:429)sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)java.lang.reflect.Method.invoke(Method.java:498)org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)com.sun.proxy.$Proxy95.judgeCertifiedStatus(Unknown Source)com.ctfo.customer.controller.CustomerController.findCustomerByCertifiedStatus(CustomerController.java:66)sun.reflect.GeneratedMethodAccessor174.invoke(Unknown Source)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)java.lang.reflect.Method.invoke(Method.java:498)org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)javax.servlet.http.HttpServlet.service(HttpServlet.java:635)org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)javax.servlet.http.HttpServlet.service(HttpServlet.java:742)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1697)org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)java.lang.Thread.run(Thread.java:748)]
04-Apr-2019 13:22:28.366 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboRegistryFailedRetryTimer-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)

1.2

2019-04-06 22:22:52.150 [http-nio2-80-exec-1041] INFO  c.c.customer.service.impl.login.LoginServiceImpl - 通过手机号查询游客数据:不存在
2019-04-06 22:22:52.226 [Thread-10] INFO  o.s.context.support.DefaultLifecycleProcessor - Stopping beans in phase 0
2019-04-06 22:22:52.229 [Thread-10] INFO  o.s.b.actuate.endpoint.jmx.EndpointMBeanExporter - Unregistering JMX-exposed beans on shutdown
2019-04-06 22:22:52.229 [Thread-10] INFO  o.s.b.actuate.endpoint.jmx.EndpointMBeanExporter - Unregistering JMX-exposed beans
2019-04-06 22:22:52.230 [Thread-10] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Unregistering JMX-exposed beans on shutdown
2019-04-06 22:22:52.230 [Thread-10] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Unregistering JMX-exposed beans
2019-04-06 22:22:52.551 [http-nio2-80-exec-996] INFO  c.c.customer.service.impl.login.LoginServiceImpl - 
2019-04-06 22:22:52.552 [http-nio2-80-exec-996] ERROR c.c.customer.service.impl.login.LoginServiceImpl - 登录失败，系统异常！
org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at Sat Apr 06 22:22:52 CST 2019
### The error may exist in file [/opt/web_app/tomcat_8.0/apache-tomcat-8.5.34/webapps/ROOT/WEB-INF/classes/mapper/dictionary/EtcDictionaryMapper.xml]
### The error may involve com.ctfo.customer.dao.dictionary.EtcDictionaryMapper.selectOneByObject
### The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at Sat Apr 06 22:22:52 CST 2019at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:79)at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:447)at com.sun.proxy.$Proxy86.selectOne(Unknown Source)at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:167)at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:75)at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:53)at com.sun.proxy.$Proxy96.selectOneByObject(Unknown Source)at com.ctfo.customer.service.impl.login.LoginServiceImpl.getUserInfo(LoginServiceImpl.java:228)at com.ctfo.customer.controller.LoginController.saveCustomerInfo(LoginController.java:68)at sun.reflect.GeneratedMethodAccessor157.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1697)at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1050)at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$4.completed(Nio2Endpoint.java:630)at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$4.completed(Nio2Endpoint.java:608)at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)at sun.nio.ch.Invoker$2.run(Invoker.java:218)at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at Sat Apr 06 22:22:52 CST 2019
### The error may exist in file [/opt/web_app/tomcat_8.0/apache-tomcat-8.5.34/webapps/ROOT/WEB-INF/classes/mapper/dictionary/EtcDictionaryMapper.xml]
### The error may involve com.ctfo.customer.dao.dictionary.EtcDictionaryMapper.selectOneByObject
### The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at Sat Apr 06 22:22:52 CST 2019at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:150)at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141)at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:77)at sun.reflect.GeneratedMethodAccessor150.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:434)... 74 more
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at Sat Apr 06 22:22:52 CST 2019at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:84)at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:70)at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:337)at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:84)at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62)at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83)at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148)... 80 more
Caused by: com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at Sat Apr 06 22:22:52 CST 2019at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1057)at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:960)at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4544)at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:661)at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4540)at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:938)at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:930)at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:102)at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)... 90 more
2019-04-06 22:22:53.047 [http-nio2-80-exec-1016] ERROR c.c.customer.service.impl.login.LoginServiceImpl - 登录失败，系统异常！
org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at Sat Apr 06 22:22:52 CST 2019
### The error may exist in file [/opt/web_app/tomcat_8.0/apache-tomcat-8.5.34/webapps/ROOT/WEB-INF/classes/mapper/dictionary/EtcDictionaryMapper.xml]
### The error may involve com.ctfo.customer.dao.dictionary.EtcDictionaryMapper.selectOneByObject
### The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at Sat Apr 06 22:22:52 CST 2019at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:79)at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:447)at com.sun.proxy.$Proxy86.selectOne(Unknown Source)at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:167)at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:75)at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:53)at com.sun.proxy.$Proxy96.selectOneByObject(Unknown Source)at com.ctfo.customer.service.impl.login.LoginServiceImpl.getUserInfo(LoginServiceImpl.java:228)at com.ctfo.customer.controller.LoginController.saveCustomerInfo(LoginController.java:68)at sun.reflect.GeneratedMethodAccessor157.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1697)at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1050)at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$4.completed(Nio2Endpoint.java:630)at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$4.completed(Nio2Endpoint.java:608)at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)at sun.nio.ch.Invoker$2.run(Invoker.java:218)at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at Sat Apr 06 22:22:52 CST 2019
### The error may exist in file [/opt/web_app/tomcat_8.0/apache-tomcat-8.5.34/webapps/ROOT/WEB-INF/classes/mapper/dictionary/EtcDictionaryMapper.xml]
### The error may involve com.ctfo.customer.dao.dictionary.EtcDictionaryMapper.selectOneByObject
### The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at Sat Apr 06 22:22:52 CST 2019at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:150)at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141)at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:77)at sun.reflect.GeneratedMethodAccessor150.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:434)... 74 more
....中间省略一些
06-Apr-2019 22:22:53.404 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.alibaba.druid.proxy.DruidDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
06-Apr-2019 22:22:53.404 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
06-Apr-2019 22:22:53.406 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboRegistryFailedRetryTimer-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.406 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboSaveRegistryCache-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.407 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboClientReconnectTimer-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.408 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [NettyClientWorker-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.409 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [dubbo-remoting-client-heartbeat-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.410 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboClientReconnectTimer-thread-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.411 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [NettyClientWorker-thread-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.412 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [dubbo-remoting-client-heartbeat-thread-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.413 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [NettyClientWorker-thread-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.414 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [NettyClientWorker-thread-5] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.415 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.416 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboResponseTimeoutScanTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.lang.Thread.sleep(Native Method)com.alibaba.dubbo.remoting.exchange.support.DefaultFuture$RemotingInvocationTimeoutScan.run(DefaultFuture.java:300)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.416 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.lang.Object.wait(Native Method)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
06-Apr-2019 22:22:53.417 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.418 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.419 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.420 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-5] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.421 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.422 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.423 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-8] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.424 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-9] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.425 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.425 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-11] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.426 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-12] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.427 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-13] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.428 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-14] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.429 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-15] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.430 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-16] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.431 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [DubboServerHandler-20.0.22.39:20882-thread-17] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.432 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [NettyClientWorker-thread-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)java.lang.Thread.run(Thread.java:748)
06-Apr-2019 22:22:53.434 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:[java.lang.Object.wait(Native Method)java.lang.Thread.join(Thread.java:1252)java.lang.Thread.join(Thread.java:1326)java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)java.lang.Shutdown.runHooks(Shutdown.java:123)java.lang.Shutdown.sequence(Shutdown.java:167)java.lang.Shutdown.exit(Shutdown.java:212)java.lang.Runtime.exit(Runtime.java:109)java.lang.System.exit(System.java:971)com.ctfo.customer.util.FtpUtils.connect(FtpUtils.java:73)com.ctfo.customer.service.impl.certification.CertificationServiceImpl.judgeCertifiedStatus(CertificationServiceImpl.java:412)sun.reflect.GeneratedMethodAccessor159.invoke(Unknown Source)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)java.lang.reflect.Method.invoke(Method.java:498)org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)com.sun.proxy.$Proxy95.judgeCertifiedStatus(Unknown Source)com.ctfo.customer.controller.CustomerController.findCustomerByCertifiedStatus(CustomerController.java:66)sun.reflect.GeneratedMethodAccessor158.invoke(Unknown Source)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)java.lang.reflect.Method.invoke(Method.java:498)org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)javax.servlet.http.HttpServlet.service(HttpServlet.java:635)org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)javax.servlet.http.HttpServlet.service(HttpServlet.java:742)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1697)org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)java.lang.Thread.run(Thread.java:748)]

 

2.  原因猜测

     由于只有这些日志，只能针对报错部分的代码寻找原因。

     

java.lang.Object.wait(Native Method)java.lang.Thread.join(Thread.java:1252)java.lang.Thread.join(Thread.java:1326)java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)java.lang.Shutdown.runHooks(Shutdown.java:123)java.lang.Shutdown.sequence(Shutdown.java:167)java.lang.Shutdown.exit(Shutdown.java:212)java.lang.Runtime.exit(Runtime.java:109)java.lang.System.exit(System.java:971)com.ctfo.customer.util.FtpUtils.connect(FtpUtils.java:73)

     原因已找到，是由于代码中出现了  System.exit(1);  导致项目直接宕机了。

     4号发现已经改过了，为什么6号又出现了呢。因为没有打新包。。。

当时看日志的时候，自以为是的找了一些关键字 

Stopping beans in phase 0.

but has failed to stop it. This is very likely to create a memory leak.

这些都不是根本原因啊....不过还是查到了一些东西...

最开始以为是OOM导致的宕机，各种看tomcat的锅。以下是查到的资料。

Tomcat 8.5.38   
https://tomcat.apache.org/tomcat-8.5-doc/changelog.html
 Update the packaged version of the Tomcat Native Library to 1.2.21 to pick up the memory leak fixes when using NIO/NIO2 with OpenSSL. (markt)

修复了由于使用 NIO 连接 openSSL 时产生的内存泄漏。

而这个原因  but has failed to stop it. This is very likely to create a memory leak.

是由于项目停止时，连接池的连接没有关闭。 忽略就好了，这个不是主要原因。