记一次应用配置的数据库连接被打满问题

线上应用，配置的数据库连接数为50，正常情况是已经够用了，但是有天发现50个连接全部被占满，并且长时间无法恢复，重启服务后会好一段时间。

1、问题现象

Druid获取MySQL数据库连接超时，超时时间设置的为60s

2019-09-23 15:38:24.627-|XNIO-3 task-80-|ERROR-|7fba6a1c917f42878b43f937df37e575-|10.123.18.241-|-|/api/v1/qimao/admin/distributor/add-|-|Java/1.8.0_181-|c.q.k.q.web.controller.admin.DistributorController-|method:addDistributor
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 10, maxActive 10, creating 0at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:289)at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:447)at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277)at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)at com.qiyi.kpp.qimao.web.service.impl.DistributorServiceImpl$$EnhancerBySpringCGLIB$$e95f0879.addDistributor(<generated>)at com.qiyi.kpp.qimao.web.controller.admin.DistributorController.addDistributor(DistributorController.java:174)at com.qiyi.kpp.qimao.web.controller.admin.DistributorController$$FastClassBySpringCGLIB$$9c955533.invoke(<generated>)at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)at com.qiyi.kpp.qimao.web.controller.admin.DistributorController$$EnhancerBySpringCGLIB$$100c45af.addDistributor(<generated>)at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source).............at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)at io.undertow.server.Connectors.executeRootHandler(Connectors.java:211)at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:809)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 10, maxActive 10, creating 0at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1512)at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1255)at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5007)at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:680)at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5003)at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1233)at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1225)at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:90)at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246)... 100 common frames omitted

2、解决办法

添加配置，当连接超过30秒钟后会强制进行回收

<!-- 超过时间限制是否回收 -->  
druid.removeAbandoned=true
<!-- 超时时间；单位为秒。 -->  
druid.removeAbandonedTimeout=30
<!-- 关闭abanded连接时输出错误日志 -->
druid.logAbandoned=true

最终重现了这个问题，打印日志如下

2019-09-27 16:07:21.369-|AsyncResolver-bootstrap-executor-0-|INFO-|-|-|-|-|-|-|c.n.d.shared.resolver.aws.ConfigClusterResolver-|Resolving eureka endpoints via configuration
2019-09-27 16:07:28.954-|Druid-ConnectionPool-Destroy-786728464-|ERROR-|-|-|-|-|-|-|com.alibaba.druid.pool.DruidDataSource-|abandon connection, owner thread: XNIO-3 task-862, connected at : 1569571561703, open stackTraceat java.lang.Thread.getStackTrace(Thread.java:1552)at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1313)at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5007)at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:680)at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5003)at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1233)at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1225)at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:90)at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246)at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:447)at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277)at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)at com.qiyi.kpp.qimao.web.service.impl.DistributorServiceImpl$$EnhancerBySpringCGLIB$$ec30275b.addDistributor(<generated>)at com.qiyi.kpp.qimao.web.controller.admin.DistributorController.addDistributor(DistributorController.java:174)at com.qiyi.kpp.qimao.web.controller.admin.DistributorController$$FastClassBySpringCGLIB$$9c955533.invoke(<generated>)at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)at com.qiyi.kpp.qimao.web.controller.admin.DistributorController$$EnhancerBySpringCGLIB$$34df12db.addDistributor(<generated>)