今天早上突然发现数据库连接不上了,查看tidb-server提示Got too many pings from the client, closing the connection.

【 TiDB 使用环境】生产环境
【 TiDB 版本】6.3.0
【复现路径】做过哪些操作出现的问题
没有动,突然遇到的
【遇到的问题:问题现象及影响】
这个是客户端连接时候报的错误


这个是java程序报的错误

### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 20, creating 0, createErrorCount 272
        at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:153)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:145)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:427)
        ... 51 common frames omitted
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 20, creating 0, createErrorCount 272
        at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:83)
        at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:80)
        at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:67)
        at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:337)
        at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:86)
        at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62)
        at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)
        at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
        at com.baomidou.mybatisplus.extension.plugins.MybatisPlusInterceptor.intercept(MybatisPlusInterceptor.java:81)
        at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:62)
        at jdk.proxy2/jdk.proxy2.$Proxy385.query(Unknown Source)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:151)
        ... 58 common frames omitted
Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 20, creating 0, createErrorCount 272
        at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1775)
        at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1429)
        at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5059)
        at com.alibaba.druid.filter.FilterAdapter.dataSource_getConnection(FilterAdapter.java:2756)
        at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5055)
        at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:726)
        at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5055)
        at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1407)
        at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1399)
        at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:100)
        at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:159)
        at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:117)
        at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
        ... 69 common frames omitted
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
        at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
        at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
        at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
        at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
        at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198)
        at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:156)
        at com.alibaba.druid.filter.FilterAdapter.connection_connect(FilterAdapter.java:787)
        at com.alibaba.druid.filter.FilterEventAdapter.connection_connect(FilterEventAdapter.java:38)
        at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:150)
        at com.alibaba.druid.filter.stat.StatFilter.connection_connect(StatFilter.java:251)
        at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:150)
        at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1669)
        at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1733)
        at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2840)
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at jdk.internal.reflect.GeneratedConstructorAccessor85.newInstance(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
        at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
        at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:544)
        at com.mysql.cj.protocol.a.NativeProtocol.readServerCapabilities(NativeProtocol.java:496)
        at com.mysql.cj.protocol.a.NativeProtocol.beforeHandshake(NativeProtocol.java:383)
        at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1346)
        at com.mysql.cj.NativeSession.connect(NativeSession.java:157)
        at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:956)
        at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:826)
        ... 12 common frames omitted
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
        at com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:67)
        at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:63)
        at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:45)
        at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:538)
        ... 18 common frames omitted

【资源配置】
【附件:截图/日志/监控】




这个是错误的日志数据

[2023/01/10 08:44:39.402 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: transport: Connection closing"] [system=grpc] [grpc_log=true]
[2023/01/10 08:45:39.401 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: Got too many pings from the client, closing the connection."] [system=grpc] [grpc_log=true]
[2023/01/10 08:45:39.401 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: transport: Connection closing"] [system=grpc] [grpc_log=true]
[2023/01/10 08:46:49.400 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: Got too many pings from the client, closing the connection."] [system=grpc] [grpc_log=true]
[2023/01/10 08:46:49.401 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: transport: Connection closing"] [system=grpc] [grpc_log=true]
[2023/01/10 08:47:39.401 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: Got too many pings from the client, closing the connection."] [system=grpc] [grpc_log=true]
[2023/01/10 08:47:39.402 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: transport: Connection closing"] [system=grpc] [grpc_log=true]
[2023/01/10 08:48:39.405 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: Got too many pings from the client, closing the connection."] [system=grpc] [grpc_log=true]
[2023/01/10 08:48:39.410 +08:00] [Error] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: transport: Connection closing"] [system=grpc] [grpc_log=true]

这是服务器
image
image
image

这个是集群的检查


这个问题感觉毫无头绪,不管那么多了,我现在正在执行升级操作,先升级到6.5版本再说

参考下?

1 个赞

提供一下tidb的Overview监控 面板看看

1 个赞

这个问题目前已经解决,似乎不是tidb的问题,闹乌龙了 :sweat_smile:

升级完成后,我刚刚试过内网是可以连接的

因为我在前面用apisix做了tcp的负载均衡,昨天我的确是更新了一下apisix的配置,可能是这个问题导致的

我将继续排查apisix问题,感谢各位的帮助

我升级完后暂时没有看到任何ERROR日志,这感觉非常好

我爱tidb

我一般遇到navicat客户端抛错的时候,会尝试直接在服务器用shell访问试试。

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。