【 TiDB 使用环境】 测试/ Poc
【 TiDB 版本】5.4.3
【复现路径】压力测试，putKey
【遇到的问题：问题现象及影响】
TiKV client偶尔跑出来错误，服务端没有任何的错误。
org.tikv.common.exception.TiKVException: TimeOut Exceeded for current operation.
at org.tikv.common.util.ClientUtils.getTasks(ClientUtils.java:181)
at org.tikv.raw.RawKVClient.doSendBatchPut(RawKVClient.java:712)
at org.tikv.raw.RawKVClient.batchPut(RawKVClient.java:222)
at org.tikv.raw.RawKVClient.batchPut(RawKVClient.java:206)
2022-12-16 03:59:03,868 WARN SlowLogImpl - A request spent 6001 ms. start=03:58:57.852, end=03:59:03.853, SlowLog:{“trace_id”:14895108676224612315,“error”:“TimeOut Exceeded for current operation.”,“spans”:[{“event”:“batchPut”,“begin”:“03:58:57.852”,“duration_ms”:6001,“properties”:{“keySize”:“2”}},{“event”:“getRegionByKey”,“begin”:“03:58:57.852”,“duration_ms”:0},{“event”:“getRegionByKey”,“begin”:“03:58:57.852”,“duration_ms”:0},{“event”:“callWithRetry”,“begin”:“03:58:57.852”,“duration_ms”:1,“properties”:{“method”:“tikvpb.Tikv/RawBatchPut”}},{“event”:“gRPC”,“begin”:“03:58:57.852”,“duration_ms”:1,“properties”:{“method”:“tikvpb.Tikv/RawBatchPut”}}],“pd_addresses”:[“http://127.0.0.1:2379”],“cluster_id”:“7177588669324898328”}
PD log：
[2022/12/16 03:50:51.372 +00:00] [ERROR] [grpclog.go:75] [“transport: Got too many pings from the client, closing the connection.”]
[2022/12/16 03:50:51.372 +00:00] [ERROR] [grpclog.go:75] [“transport: loopyWriter.run returning. Err: transport: Connection closing”]
[2022/12/16 03:50:51.373 +00:00] [WARN] [grpclog.go:60] [“transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:58442: use of closed network connection”]
[2022/12/16 03:50:51.391 +00:00] [ERROR] [grpclog.go:75] [“transport: Got too many pings from the client, closing the connection.”]
[2022/12/16 03:50:51.391 +00:00] [ERROR] [grpclog.go:75] [“transport: loopyWriter.run returning. Err: transport: Connection closing”]
[2022/12/16 03:50:51.391 +00:00] [WARN] [grpclog.go:60] [“transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:58448: use of closed network connection”]
[2022/12/16 03:50:51.406 +00:00] [ERROR] [grpclog.go:75] [“transport: Got too many pings from the client, closing the connection.”]
[2022/12/16 03:50:51.406 +00:00] [ERROR] [grpclog.go:75] [“transport: loopyWriter.run returning. Err: transport: Connection closing”]
[2022/12/16 03:50:51.406 +00:00] [WARN] [grpclog.go:60] [“transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:2379->127.0.0.1:58454: use of closed network connection”]
Local 磁盘的能力：
[ec2-user@ip-172-31-4-185 tikvdata]$ dd if=/dev/zero of=/mnt/tikvdata/smalletst bs=1024K count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 14.7687 s, 727 MB/s
【资源配置】
128G内存
【附件：截图/日志/监控】