down-peer-region-count和pending-peer-region-count很多如何排查处理

[2021/12/21 09:08:36.316 +08:00] [INFO] [operator_controller.go:424] [“add operator”] [region-id=33132] [operator="“remove-orphan-peer {rm peer: store [3]} (kind:region, region:33132(418,23), createAt:2021-12-21 09:08:36.31680021 +0800 CST m=+10882612.174309628, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, steps:[remove peer on store 3])”"] [“additional info”=]
[2021/12/21 09:08:36.316 +08:00] [INFO] [operator_controller.go:620] [“send schedule command”] [region-id=33132] [step=“remove peer on store 3”] [source=create]
[2021/12/21 09:08:36.318 +08:00] [INFO] [cluster.go:551] [“region ConfVer changed”] [region-id=33132] [detail=“Remove peer:{id:1595158 store_id:3 is_learner:true }”] [old-confver=23] [new-confver=24]
[2021/12/21 09:08:36.319 +08:00] [INFO] [operator_controller.go:537] [“operator finish”] [region-id=33132] [takes=2.191023ms] [operator="“remove-orphan-peer {rm peer: store [3]} (kind:region, region:33132(418,23), createAt:2021-12-21 09:08:36.31680021 +0800 CST m=+10882612.174309628, startAt:2021-12-21 09:08:36.316831504 +0800 CST m=+10882612.174340925, currentStep:1, steps:[remove peer on store 3]) finished”"] [“additional info”=]
[2021/12/21 09:40:06.167 +08:00] [INFO] [operator_controller.go:537] [“operator finish”] [region-id=1405573] [takes=1.300592ms] [operator="“remove-orphan-peer {rm peer: store [1]} (kind:region, region:1405573(40465,137), createAt:2021-12-21 09:40:06.165774805 +0800 CST m=+10884502.023284223, startAt:2021-12-21 09:40:06.165803872 +0800 CST m=+10884502.023313295, currentStep:1, steps:[remove peer on store 1]) finished”"] [“additional info”=]
[2021/12/21 10:47:59.130 +08:00] [INFO] [operator_controller.go:424] [“add operator”] [region-id=33132] [operator="“replace-rule-offline-peer {mv peer: store [2] to [3]} (kind:region,replica, region:33132(418,24), createAt:2021-12-21 10:47:59.13004767 +0800 CST m=+10888574.987557092, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, steps:[add learner peer 1596345 on store 3, promote learner peer 1596345 on store 3 to voter, remove peer on store 2])”"] [“additional info”=]
[2021/12/21 10:47:59.130 +08:00] [INFO] [operator_controller.go:620] [“send schedule command”] [region-id=33132] [step=“add learner peer 1596345 on store 3”] [source=create]
[2021/12/21 10:47:59.131 +08:00] [INFO] [cluster.go:551] [“region ConfVer changed”] [region-id=33132] [detail=“Add peer:{id:1596345 store_id:3 is_learner:true }”] [old-confver=24] [new-confver=25]
[2021/12/21 10:47:59.131 +08:00] [INFO] [operator_controller.go:620] [“send schedule command”] [region-id=33132] [step=“add learner peer 1596345 on store 3”] [source=heartbeat]

[2021/12/21 10:57:54.926 +08:00] [INFO] [operator_controller.go:620] [“send schedule command”] [region-id=33132] [step=“add learner peer 1596345 on store 3”] [source=heartbeat]
[2021/12/21 10:58:00.242 +08:00] [INFO] [operator_controller.go:560] [“operator timeout”] [region-id=33132] [takes=10m1.112458294s] [operator="“replace-rule-offline-peer {mv peer: store [2] to [3]} (kind:region,replica, region:33132(418,24), createAt:2021-12-21 10:47:59.13004767 +0800 CST m=+10888574.987557092, startAt:2021-12-21 10:47:59.130113642 +0800 CST m=+10888574.987623063, currentStep:0, steps:[add learner peer 1596345 on store 3, promote learner peer 1596345 on store 3 to voter, remove peer on store 2]) timeout”"]

pd上有很多迁移peer超时的日志,像上面这种