拿不到连接

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:

【概述】 我在做压力测试。有五台压测机,不停的插入数据。集群有三台TIDB,通过tidb-loadbalance做负载均衡。每个压测机起300个线程,300个数据库连接测试。压测一段时间后,就会报不能获得连接异常。已經造了大概30亿數據后,开始出现这个问题的,之前一直正常
java.util.concurrent.CompletionException: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:309)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:601)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:385)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:702)
at com.example.demo.service.MakeDataService$$SpringCGLIB$$0.makeData()
at com.example.demo.service.MyService.lambda$start$10(MyService.java:146)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
… 3 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:265)
… 14 common frames omitted
Caused by: java.sql.SQLException: can not get connection
at com.tidb.jdbc.LoadBalancingDriver.connect(LoadBalancingDriver.java:276)
at com.tidb.jdbc.LoadBalancingDriver.connect(LoadBalancingDriver.java:286)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:733)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
… 3 common frames omitted
报异常后,只能重启压测程序。这个问题是什么原因导致的,如何解决,谢谢

【应用框架及开发适配业务逻辑】

【背景】 做过哪些操作

【现象】 业务和数据库现象

【问题】 当前遇到的问题

【业务影响】

【TiDB 版本】

【附件】 相关日志及监控

看看tidb-server有没有OOM

java报的连接池不足引起的,修改一下连接池配置

没OOM,再起测试程序,又可以开始运行了

压测工具的连接池不足

tidb的连接相关监控可以看看

为什么在30亿条记录之前没这个问题?之后就开始频繁报错

应用的连接池满了。

1、连接池是不是满了 2、数据库是不是异常了