RawKV 进行多线程 client.batchPut 时，大量出现 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[28422] ConfVer[77] Version[33] Store[1001] KeyRange[Leo_user,#r,#60922708]:[Leo_user,#r,#61199499]}]

Hacker_CYAz1WiB · 2021 年8 月 30 日 02:54

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：
【 TiDB 使用环境】
裸kv集群5.1.0
tikv-client-Java-3.1.0
k8s1.21
tidb-operator1.1.12
【概述】场景 + 问题概述
使用maven打包tikv-client写的写入数据代码，每对kv大小达到3.5kb，大量出现下面日志

21:35:23,530 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:23,538 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:23,550 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:23,630 INFO org.tikv.shade.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - [id: 0x8dd33058, L:/192.168.101.41:44180 - R:lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc/192.168.201.188:20160] ignoring HEADERS frame for stream RST_STREAM sent.
21:35:23,696 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:23,766 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:23,836 INFO org.tikv.shade.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - [id: 0x8dd33058, L:/192.168.101.41:44180 - R:lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc/192.168.201.188:20160] ignoring HEADERS frame for stream RST_STREAM sent.
21:35:23,968 INFO org.tikv.shade.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - [id: 0x8dd33058, L:/192.168.101.41:44180 - R:lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc/192.168.201.188:20160] ignoring HEADERS frame for stream RST_STREAM sent.
21:35:23,976 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,029 INFO org.tikv.shade.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - [id: 0x8dd33058, L:/192.168.101.41:44180 - R:lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc/192.168.201.188:20160] ignoring HEADERS frame for stream RST_STREAM sent.
21:35:24,029 INFO org.tikv.shade.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - [id: 0x8dd33058, L:/192.168.101.41:44180 - R:lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc/192.168.201.188:20160] ignoring HEADERS frame for stream RST_STREAM sent.
21:35:24,116 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,123 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,135 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,152 INFO org.tikv.shade.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - [id: 0x8dd33058, L:/192.168.101.41:44180 - R:lastest-tikv-1.lastest-tikv-peer.tidb-lastest.svc/192.168.201.188:20160] ignoring HEADERS frame for stream RST_STREAM sent.
21:35:24,267 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,271 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,287 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,408 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:24,682 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,166 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,178 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,244 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,369 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,371 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,372 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,377 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,422 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,426 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,449 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,481 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,688 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,711 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,803 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,862 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:25,872 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:26,321 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:26,336 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:26,354 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:26,840 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,100 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,230 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,346 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,347 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,466 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,543 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,612 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,756 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,770 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,806 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,819 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,924 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,928 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,952 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,979 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:27,980 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
21:35:28,234 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,280 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,405 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,415 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,502 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,546 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,590 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,595 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,614 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,616 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,623 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,624 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,662 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,696 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,734 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,808 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,890 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,895 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,899 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,942 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:28,999 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,018 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,026 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “Raft raft: proposal dropped”
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,049 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,049 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,059 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,059 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,079 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,079 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,148 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,148 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,202 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,202 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,204 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,205 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,241 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,241 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,435 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,435 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,435 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,435 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,435 WARN org.tikv.common.operation.KVErrorHandler - NotLeader Error with region id 20758 and store id 6, new store id 8
21:35:29,435 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,440 WARN org.tikv.common.region.RegionManager - Cannot find peer when updating leader (20758,8)
21:35:29,447 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,493 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,501 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,503 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,532 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,542 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,572 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,589 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,609 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,620 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:35:29,642 WARN org.tikv.common.operation.KVErrorHandler - Unknown error message: “region 20758 not found”
region_not_found {
region_id: 20758
}
for region [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]
21:36:09,699 WARN org.tikv.common.operation.KVErrorHandler - Stale Epoch encountered for region [{Region[20758] ConfVer[240] Version[27] Store[8] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]

其中 WARN org.tikv.common.operation.KVErrorHandler - Request Failed with unknown reason for [{Region[20758] ConfVer[240] Version[27] Store[6] KeyRange[Leo_item,#r,#3894801]:[Leo_item,#r,#3912453]}]最多。

导致写数据很慢
监控同时出现grpc-msg-fail

【背景】做过哪些操作

【现象】业务和数据库现象

【问题】当前遇到的问题

【业务影响】

【TiDB 版本】
裸kv集群5.1.0
【应用软件及版本】

【附件】相关日志及配置信息

TiUP Cluster Display 信息
TiUP CLuster Edit config 信息

监控（https://metricstool.pingcap.com/)

TiDB-Overview Grafana监控
TiDB Grafana 监控
TiKV Grafana 监控
PD Grafana 监控
对应模块日志（包含问题前后 1 小时日志）

若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出的打印结果，请务必全选并复制粘贴上传。

Lucien · 2021 年8 月 30 日 09:52

Hi ～麻烦反馈一下单线程和低线程并发 batchput 有没有影响？另外提供一下 TiKV 的完整的 log 和 TiKV-details 的监控，可以通过 metrictools 工具做一下导出。

marsishandsome · 2021 年8 月 31 日 03:03

请问

导入的数据是否正确？（符合预期）
能否用 v3.1.2 版本试一下 https://github.com/tikv/client-java/tree/v3.1.2
能否试一下开启 titan https://docs.pingcap.com/zh/tidb/stable/titan-overview

Hacker_CYAz1WiB · 2021 年8 月 31 日 03:04

已经打开了 enable titan,每批次导入100对，开线程池100，10都出现了grpc-msg-fail,线程池开到1，2的时候没有grpc-msg-fail，但是依然出现写慢，观察是99% gRPC messge duration增大，gRPC message count降低的原因，
链接：https://pan.baidu.com/s/1h3wDTiLawjK13FLdJ4KDdA
提取码：ypqj
日志比北京时间慢了8小时

Hacker_CYAz1WiB · 2021 年8 月 31 日 06:33

没什么变化，
跟3.1.0差不多

marsishandsome · 2021 年8 月 31 日 06:41

能否提供一下客户端的代码？好像除了 put 操作外，还有 get 操作

marsishandsome · 2021 年8 月 31 日 07:09

可以试一下每个线程使用自己独立的 TiSession 和 RawKVClient

TiSession session = TiSession.create(conf);
RawKVClient client = session.createRawClient();

Hacker_CYAz1WiB · 2021 年8 月 31 日 07:36

好的我试一下
补充一下，刚刚tikv的日志里只有昨天晚上到今天上午的，所以没有导入时的日志，下面是刚刚测试的
链接：https://pan.baidu.com/s/1h3wDTiLawjK13FLdJ4KDdA
提取码：ypqj

marsishandsome · 2021 年9 月 1 日 02:18

这个错是监控检查，可以忽略
请问还有其他问题吗？

数据是否正确？
导入速度是否符合你们的要求？

Hacker_CYAz1WiB · 2021 年9 月 2 日 06:22

数据扫描一小部分出来，应该是没问题
然后速度上还是一样出现，偶尔grpc-msg-count下降，从而导致写速度下降

同时发现变慢的时候都是tikv-1的qps在执行，只是速度很慢，只有其他kv的一半

marsishandsome · 2021 年9 月 2 日 06:27

请用这个工具（https://metricstool.pingcap.com/) 导出一下 grafana 上的图

TiDB-Overview Grafana监控
TiDB Grafana 监控
TiKV Grafana 监控包括 summary 和 detail
PD Grafana 监控

Hacker_CYAz1WiB · 2021 年9 月 2 日 07:00

链接：https://pan.baidu.com/s/1h3wDTiLawjK13FLdJ4KDdA
提取码：ypqj
没有tidb，所以只有这四个

TiDB-Overview Grafana监控
TiKV Grafana 监控包括 summary 和 detail
PD Grafana 监控

marsishandsome · 2021 年9 月 2 日 09:37

这是 key 读写的 QPS，这应该远远没到 tikv 的性能瓶颈

请问你客户端能否在多台机器上同时运行？我怀疑可能是客户端的瓶颈导致的

Hacker_CYAz1WiB · 2021 年9 月 2 日 11:23

这是啥意思，数据分批同时在不同的机器上导入数据吗？

marsishandsome · 2021 年9 月 3 日 01:55

是的，多台机器同时导入如果能把 QPS 拉高，说明瓶颈在客户端

marsishandsome · 2021 年9 月 3 日 01:56

另外现在的代码逻辑里面为什么既有读又有写？能解释一下吗？
如果可以的话可以单独测一下写入的速度，排除一下读取导致的慢

Hacker_CYAz1WiB · 2021 年9 月 3 日 05:40

代码里没有读吧.

marsishandsome · 2021 年9 月 6 日 06:04

batch put 在遇到一批数据不在一个 reigon 的时候，会拆开成多个请求发到 TiKV，也就是说可以先把导入的数据进行排序，再进行导入，这样同一个 batch 的大部分数据都会是一个 region 的，这样会更快。可以试试这个方法。

另外可以试一下用多台机器同时导入。

Hacker_CYAz1WiB · 2021 年9 月 6 日 06:08

好的我尝试先排序一下，再增加到6台机器试一下

system · 2022 年10 月 31 日 19:07

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。