RawKV 进行多线程 client.batchPut 时,大量出现 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[28422] ConfVer[77] Version[33] Store[1001] KeyRange[Leo_user,#r,#60922708]:[Leo_user,#r,#61199499]}]

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:
【 TiDB 使用环境】
裸kv集群5.1.0
tikv-client-Java-3.1.0
k8s1.21
tidb-operator1.1.12
【概述】 场景 + 问题概述
使用maven打包tikv-client写的写入数据代码,每对kv大小达到3.5kb,大量出现下面日志

21:35:23,530 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:23,538 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:23,550 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:23,630 INFO org.tikv.shade.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - [id: 0x8dd33058, L:/192.168.101.41:44180 - R:lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc/192.168.201.188:20160] ignoring HEADERS frame for stream RST_STREAM sent.
21:35:23,696 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:23,766 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:23,836 INFO org.tikv.shade.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - [id: 0x8dd33058, L:/192.168.101.41:44180 - R:lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc/192.168.201.188:20160] ignoring HEADERS frame for stream RST_STREAM sent.
21:35:23,968 INFO org.tikv.shade.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - [id: 0x8dd33058, L:/192.168.101.41:44180 - R:lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc/192.168.201.188:20160] ignoring HEADERS frame for stream RST_STREAM sent.
21:35:23,976 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,029 INFO org.tikv.shade.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - [id: 0x8dd33058, L:/192.168.101.41:44180 - R:lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc/192.168.201.188:20160] ignoring HEADERS frame for stream RST_STREAM sent.
21:35:24,029 INFO org.tikv.shade.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - [id: 0x8dd33058, L:/192.168.101.41:44180 - R:lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc/192.168.201.188:20160] ignoring HEADERS frame for stream RST_STREAM sent.
21:35:24,116 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,123 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,135 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,152 INFO org.tikv.shade.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - [id: 0x8dd33058, L:/192.168.101.41:44180 - R:lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc/192.168.201.188:20160] ignoring HEADERS frame for stream RST_STREAM sent.
21:35:24,267 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,271 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,287 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,408 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,682 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,166 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,178 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,244 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,369 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,371 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,372 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,377 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,422 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,426 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,449 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,481 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,688 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,711 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,803 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,862 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,872 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:26,321 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:26,336 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:26,354 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:26,840 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,100 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,230 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,346 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,347 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,466 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,543 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,612 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,756 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,770 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,806 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,819 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,924 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,928 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,952 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,979 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,980 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
21:35:28,234 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,280 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,405 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,415 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,502 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,546 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,590 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,595 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,614 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,616 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,623 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,624 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,662 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,696 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,734 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,808 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,890 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,895 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,899 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,942 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,999 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,018 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,026 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,049 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,049 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,059 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,059 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,079 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,079 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,148 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,148 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,202 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,202 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,204 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,205 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,241 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,241 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,435 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,435 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,435 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,435 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,435 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,435 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,440 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,447 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,493 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,501 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,503 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,532 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,542 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,572 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,589 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,609 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,620 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,642 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:36:09,699 WARN org.tikv.common.operation.KVErrorHandler - Stale Epoch encountered for region [{Region[20758] ConfVer[240] Version[27] Store[8] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]

其中 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]最多。

导致写数据很慢
监控同时出现grpc-msg-fail




【背景】 做过哪些操作

【现象】 业务和数据库现象

【问题】 当前遇到的问题

【业务影响】

【TiDB 版本】
裸kv集群5.1.0
【应用软件及版本】

【附件】 相关日志及配置信息

  • TiUP Cluster Display 信息
  • TiUP CLuster Edit config 信息

监控(https://metricstool.pingcap.com/)

  • TiDB-Overview Grafana监控
  • TiDB Grafana 监控
  • TiKV Grafana 监控
  • PD Grafana 监控
  • 对应模块日志(包含问题前后 1 小时日志)

若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

Hi ~ 麻烦反馈一下单线程和低线程并发 batchput 有没有影响 ?另外提供一下 TiKV 的完整的 log 和 TiKV-details 的监控,可以通过 metrictools 工具做一下导出 。

请问

  1. 导入的数据是否正确?(符合预期)
  2. 能否用 v3.1.2 版本试一下 https://github.com/tikv/client-java/tree/v3.1.2
  3. 能否试一下开启 titan https://docs.pingcap.com/zh/tidb/stable/titan-overview

已经打开了 enable titan,每批次导入100对,开线程池100,10都出现了grpc-msg-fail,线程池开到1,2的时候没有grpc-msg-fail,但是依然出现写慢,观察是99% gRPC messge duration增大,gRPC message count降低的原因,
链接:https://pan.baidu.com/s/1h3wDTiLawjK13FLdJ4KDdA
提取码:ypqj
日志比北京时间慢了8小时

没什么变化,
跟3.1.0差不多

能否提供一下客户端的代码?好像除了 put 操作外,还有 get 操作

CreateUserDataByTiKV.java (13.8 KB) 写数据,其中线程池改成了1,base设置的每批次大小改成了100.
pom.xml (10.3 KB) tikv-client改了3.1.2
test.java (8.6 KB) scan,可能中间有scan过

可以试一下每个线程使用自己独立的 TiSession 和 RawKVClient

TiSession session = TiSession.create(conf);
RawKVClient client = session.createRawClient();

好的我试一下
补充一下,刚刚tikv的日志里只有昨天晚上到今天上午的,所以没有导入时的日志,下面是刚刚测试的
链接:https://pan.baidu.com/s/1h3wDTiLawjK13FLdJ4KDdA
提取码:ypqj

CreateItemDataByTiKV.java (19.8 KB)

Aug 31, 2021 9:33:32 PM org.tikv.shade.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: ~~~ Channel ManagedChannelImpl{logId=394603, target=lastest-tikv-4.lastest-tikv-peer.tidb-lastest.svc:20160} was not shutdown properly!!! ~~~
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at org.tikv.shade.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:94)
at org.tikv.shade.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:52)
at org.tikv.shade.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:43)
at org.tikv.shade.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:512)
at org.tikv.common.util.ChannelFactory.lambda$getChannel$0(ChannelFactory.java:56)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.tikv.common.util.ChannelFactory.getChannel(ChannelFactory.java:35)
at org.tikv.common.region.StoreHealthyChecker.checkStoreHealth(StoreHealthyChecker.java:66)
at org.tikv.common.region.StoreHealthyChecker.run(StoreHealthyChecker.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Aug 31, 2021 9:54:32 PM org.tikv.shade.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: ~~~ Channel ManagedChannelImpl{logId=851457, target=lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc:20160} was not shutdown properly!!! ~~~
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at org.tikv.shade.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:94)
at org.tikv.shade.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:52)
at org.tikv.shade.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:43)
at org.tikv.shade.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:512)
at org.tikv.common.util.ChannelFactory.lambda$getChannel$0(ChannelFactory.java:56)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.tikv.common.util.ChannelFactory.getChannel(ChannelFactory.java:35)
at org.tikv.common.region.StoreHealthyChecker.checkStoreHealth(StoreHealthyChecker.java:66)
at org.tikv.common.region.StoreHealthyChecker.run(StoreHealthyChecker.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

改了以后出现这个问题
监控如下

这个错是监控检查,可以忽略
请问还有其他问题吗?

  1. 数据是否正确?
  2. 导入速度是否符合你们的要求?

数据扫描一小部分出来,应该是没问题
然后速度上还是一样出现,偶尔grpc-msg-count下降,从而导致写速度下降


同时发现变慢的时候都是tikv-1的qps在执行,只是速度很慢,只有其他kv的一半

请用这个工具(https://metricstool.pingcap.com/) 导出一下 grafana 上的图

  • TiDB-Overview Grafana监控
  • TiDB Grafana 监控
  • TiKV Grafana 监控 包括 summary 和 detail
  • PD Grafana 监控

链接:https://pan.baidu.com/s/1h3wDTiLawjK13FLdJ4KDdA
提取码:ypqj
没有tidb,所以只有这四个

  • TiDB-Overview Grafana监控
  • TiKV Grafana 监控 包括 summary 和 detail
  • PD Grafana 监控


这是 key 读写的 QPS,这应该远远没到 tikv 的性能瓶颈

请问你客户端能否在多台机器上同时运行?我怀疑可能是客户端的瓶颈导致的

这是啥意思,数据分批同时在不同的机器上导入数据吗?

是的,多台机器同时导入如果能把 QPS 拉高,说明瓶颈在客户端

另外现在的代码逻辑里面为什么既有读又有写?能解释一下吗?
如果可以的话可以单独测一下写入的速度,排除一下读取导致的慢

代码里没有读吧,只有batchput,我不知道为啥监控里会有读
代码:
CreateItemDataByTiKV.java (19.8 KB)

batch put 在遇到一批数据不在一个 reigon 的时候,会拆开成多个请求发到 TiKV,也就是说可以先把导入的数据进行排序,再进行导入,这样同一个 batch 的大部分数据都会是一个 region 的,这样会更快。可以试试这个方法。

另外可以试一下用多台机器同时导入。