TiDB日志中大量"switch region leader to specific leader due to kv return NotLeader"信息

Culbr · 2020 年11 月 25 日 14:53

为提高效率，提问时请提供以下信息，问题描述清晰可优先响应。

【TiDB 版本】：4.0.8 w/ DM 2.0.0
【问题描述】：

当前我们正在利用DM从测试库中向TiDB同步50张表的数据，实际QPS很低（大约20~40之间）。在DM Task写入的那个TiDB实例日志上发现大量的switch region leader to specific leader due to kv return NotLeader信息，且有周期性，每隔30s就会产生一批。如下：

[2020/11/25 20:01:53.508 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=2] [leaderStoreID=2]
[2020/11/25 20:01:53.508 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=2] [leaderStoreID=2]
[2020/11/25 20:01:53.508 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=2] [leaderStoreID=2]
[2020/11/25 20:01:53.508 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=0] [leaderStoreID=8]
[2020/11/25 20:01:53.508 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=0] [leaderStoreID=8]
[2020/11/25 20:01:53.508 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=0] [leaderStoreID=8]
[2020/11/25 20:01:53.510 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=0] [leaderStoreID=28]
[2020/11/25 20:01:53.510 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=0] [leaderStoreID=28]
[2020/11/25 20:01:53.510 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=0] [leaderStoreID=28]
[2020/11/25 20:01:53.510 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=0] [leaderStoreID=28]
[2020/11/25 20:01:53.779 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=0] [leaderStoreID=14]
[2020/11/25 20:01:53.779 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=0] [leaderStoreID=14]
[2020/11/25 20:01:53.779 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=0] [leaderStoreID=14]
[2020/11/25 20:01:53.779 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=0] [leaderStoreID=14]
[2020/11/25 20:01:53.779 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=0] [leaderStoreID=14]
[2020/11/25 20:01:53.779 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=0] [leaderStoreID=14]
[2020/11/25 20:02:23.511 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=0] [leaderStoreID=25]
[2020/11/25 20:02:23.511 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=0] [leaderStoreID=25]
[2020/11/25 20:02:23.511 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=0] [leaderStoreID=25]
[2020/11/25 20:02:23.512 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=2] [leaderStoreID=27]
[2020/11/25 20:02:23.512 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=2] [leaderStoreID=27]
[2020/11/25 20:02:23.512 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=2] [leaderStoreID=27]
[2020/11/25 20:02:23.514 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=1] [leaderStoreID=15]
[2020/11/25 20:02:23.514 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=1] [leaderStoreID=15]
[2020/11/25 20:02:23.514 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=1] [leaderStoreID=15]
[2020/11/25 20:02:23.514 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=1] [leaderStoreID=15]
[2020/11/25 20:02:53.483 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=1] [leaderStoreID=2]
[2020/11/25 20:02:53.483 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=1] [leaderStoreID=2]
[2020/11/25 20:02:53.484 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=1] [leaderStoreID=2]
[2020/11/25 20:02:53.484 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=0] [leaderStoreID=8]
[2020/11/25 20:02:53.484 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=0] [leaderStoreID=8]
[2020/11/25 20:02:53.484 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=0] [leaderStoreID=8]
[2020/11/25 20:02:53.486 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=0] [leaderStoreID=28]
[2020/11/25 20:02:53.486 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=0] [leaderStoreID=28]
[2020/11/25 20:02:53.486 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=0] [leaderStoreID=28]
[2020/11/25 20:02:53.486 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=0] [leaderStoreID=28]
[2020/11/25 20:02:53.624 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=2] [leaderStoreID=12]
[2020/11/25 20:02:53.624 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=2] [leaderStoreID=12]
[2020/11/25 20:02:53.624 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=2] [leaderStoreID=12]
[2020/11/25 20:02:53.624 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=2] [leaderStoreID=12]
[2020/11/25 20:02:53.624 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=2] [leaderStoreID=12]
[2020/11/25 20:03:07.257 +08:00] [INFO] [client_batch.go:655] ["recycle idle connection"] [target=10.0.46.173:20162]
[2020/11/25 20:03:07.257 +08:00] [INFO] [client_batch.go:655] ["recycle idle connection"] [target=10.0.46.178:20162]
[2020/11/25 20:03:23.501 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=0] [leaderStoreID=25]
[2020/11/25 20:03:23.501 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=0] [leaderStoreID=25]
[2020/11/25 20:03:23.501 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=0] [leaderStoreID=25]
[2020/11/25 20:03:23.502 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=2] [leaderStoreID=27]
[2020/11/25 20:03:23.502 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=2] [leaderStoreID=27]
[2020/11/25 20:03:23.502 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=2] [leaderStoreID=27]
[2020/11/25 20:03:23.504 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=1] [leaderStoreID=15]
[2020/11/25 20:03:23.504 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=1] [leaderStoreID=15]
[2020/11/25 20:03:23.504 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=1] [leaderStoreID=15]
[2020/11/25 20:03:23.504 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=1] [leaderStoreID=15]
[2020/11/25 20:03:23.612 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=0] [leaderStoreID=23]
[2020/11/25 20:03:23.612 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=0] [leaderStoreID=23]
[2020/11/25 20:03:53.486 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=1] [leaderStoreID=2]
[2020/11/25 20:03:53.486 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=1] [leaderStoreID=2]
[2020/11/25 20:03:53.486 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22624] [currIdx=1] [leaderStoreID=2]
[2020/11/25 20:03:53.486 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=0] [leaderStoreID=30]
[2020/11/25 20:03:53.487 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=0] [leaderStoreID=30]
[2020/11/25 20:03:53.487 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=0] [leaderStoreID=30]
[2020/11/25 20:03:53.487 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=0] [leaderStoreID=30]
[2020/11/25 20:03:53.487 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22593] [currIdx=0] [leaderStoreID=30]
[2020/11/25 20:03:53.488 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=0] [leaderStoreID=28]
[2020/11/25 20:03:53.488 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=0] [leaderStoreID=28]
[2020/11/25 20:03:53.488 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=0] [leaderStoreID=28]
[2020/11/25 20:03:53.488 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=22552] [currIdx=0] [leaderStoreID=28]
[2020/11/25 20:03:53.630 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=1] [leaderStoreID=14]
[2020/11/25 20:03:53.630 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=1] [leaderStoreID=14]
[2020/11/25 20:03:53.630 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=1] [leaderStoreID=14]
[2020/11/25 20:03:53.630 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=1] [leaderStoreID=14]
[2020/11/25 20:03:53.630 +08:00] [INFO] [region_cache.go:839] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=23436] [currIdx=1] [leaderStoreID=14]

停掉DM Task之后，此信息就不再输出。集群中有3个TiDB实例和18（6 * 3）个TiKV实例。

请问是什么原因？感谢~

spc_monkey · 2020 年11 月 26 日 03:10

1、少量的该错误并不影响使用，主要意思是：发送的读写请求到指定的 region 后，发现该 region 已经不是 leader，数据库会自动重试，所以并不会影响使用，但如果数量较多，可能会影响执行效率
2、目前怀疑集群有较多的调度，等集群稳定后，情况会变好，这个属于正常调度，暂时无法确定是否异常

Culbr · 2020 年11 月 26 日 03:49

好的，目前确实没有影响到读写，我们再观察一段时间，感谢~

spc_monkey · 2020 年11 月 26 日 06:05

有问题可以随时开帖

system · 2022 年10 月 31 日 19:08

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。