down-peer-region-count和pending-peer-region-count很多如何排查处理

[spc_monkey (carry@pingcap.com)]

pd 需要根据 标签调度，来保证数据分布在不同的 lable 下，来保证高可用，负载均衡之类

[haizi]

Starting component ctl: /home/tidb/.tiup/components/ctl/v4.0.9/ctl pd -u http://ip:2379 operator show
[
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:314588(1332,42), createAt:2021-12-23 17:41:55.15718673 +0800 CST m=+11086211.014696148, startAt:2021-12-23 17:41:55.157274595 +0800 CST m=+11086211.014784017, currentStep:0, steps:[add learner peer 1635959 on store 1, promote learner peer 1635959 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447761(41447,35), createAt:2021-12-23 17:44:56.774911176 +0800 CST m=+11086392.632420594, startAt:2021-12-23 17:44:56.774945754 +0800 CST m=+11086392.632455176, currentStep:0, steps:[add learner peer 1636044 on store 1498967, promote learner peer 1636044 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:105552(1460,54), createAt:2021-12-23 17:45:19.887022972 +0800 CST m=+11086415.744532390, startAt:2021-12-23 17:45:19.887117661 +0800 CST m=+11086415.744627088, currentStep:0, steps:[add learner peer 1636070 on store 1, promote learner peer 1636070 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447957(41441,31), createAt:2021-12-23 17:44:56.772827605 +0800 CST m=+11086392.630337024, startAt:2021-12-23 17:44:56.772858146 +0800 CST m=+11086392.630367570, currentStep:0, steps:[add learner peer 1636021 on store 1498967, promote learner peer 1636021 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447653(41438,33), createAt:2021-12-23 17:44:56.773181065 +0800 CST m=+11086392.630690487, startAt:2021-12-23 17:44:56.773209538 +0800 CST m=+11086392.630718960, currentStep:0, steps:[add learner peer 1636025 on store 1498967, promote learner peer 1636025 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448145(41461,41), createAt:2021-12-23 17:44:56.771974043 +0800 CST m=+11086392.629483465, startAt:2021-12-23 17:44:56.772011761 +0800 CST m=+11086392.629521185, currentStep:0, steps:[add learner peer 1636013 on store 1498967, promote learner peer 1636013 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447765(41447,41), createAt:2021-12-23 17:44:56.7751737 +0800 CST m=+11086392.632683121, startAt:2021-12-23 17:44:56.775202612 +0800 CST m=+11086392.632712033, currentStep:0, steps:[add learner peer 1636047 on store 1498967, promote learner peer 1636047 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447985(41454,47), createAt:2021-12-23 17:44:56.771713318 +0800 CST m=+11086392.629222738, startAt:2021-12-23 17:44:56.771742584 +0800 CST m=+11086392.629252007, currentStep:0, steps:[add learner peer 1636010 on store 1498967, promote learner peer 1636010 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:120220(1016,302), createAt:2021-12-23 17:41:15.11267679 +0800 CST m=+11086170.970186211, startAt:2021-12-23 17:41:15.112736452 +0800 CST m=+11086170.970245873, currentStep:0, steps:[add learner peer 1635942 on store 1, promote learner peer 1635942 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:936336(25544,66), createAt:2021-12-23 17:41:59.161574462 +0800 CST m=+11086215.019083883, startAt:2021-12-23 17:41:59.161676807 +0800 CST m=+11086215.019186233, currentStep:0, steps:[add learner peer 1635960 on store 1, promote learner peer 1635960 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:1374041(39726,24), createAt:2021-12-23 17:44:21.21192113 +0800 CST m=+11086357.069430547, startAt:2021-12-23 17:44:21.211958488 +0800 CST m=+11086357.069467909, currentStep:0, steps:[add learner peer 1635968 on store 1, promote learner peer 1635968 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447629(41446,33), createAt:2021-12-23 17:44:56.773998879 +0800 CST m=+11086392.631508296, startAt:2021-12-23 17:44:56.774036021 +0800 CST m=+11086392.631545442, currentStep:0, steps:[add learner peer 1636034 on store 1498967, promote learner peer 1636034 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447969(41442,37), createAt:2021-12-23 17:44:56.772563401 +0800 CST m=+11086392.630072819, startAt:2021-12-23 17:44:56.772599678 +0800 CST m=+11086392.630109100, currentStep:0, steps:[add learner peer 1636018 on store 1498967, promote learner peer 1636018 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448841(41449,35), createAt:2021-12-23 17:44:56.774823165 +0800 CST m=+11086392.632332582, startAt:2021-12-23 17:44:56.774853429 +0800 CST m=+11086392.632362851, currentStep:0, steps:[add learner peer 1636043 on store 1498967, promote learner peer 1636043 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [3]} (kind:region,replica, region:46644(389,94), createAt:2021-12-23 17:41:44.848051808 +0800 CST m=+11086200.705561228, startAt:2021-12-23 17:41:44.848103134 +0800 CST m=+11086200.705612557, currentStep:0, steps:[add learner peer 1635956 on store 3, promote learner peer 1635956 on store 3 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:31324(374,42), createAt:2021-12-23 17:41:43.144022489 +0800 CST m=+11086199.001531909, startAt:2021-12-23 17:41:43.14411031 +0800 CST m=+11086199.001619736, currentStep:0, steps:[add learner peer 1635955 on store 1, promote learner peer 1635955 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447993(41456,45), createAt:2021-12-23 17:44:56.771806157 +0800 CST m=+11086392.629315580, startAt:2021-12-23 17:44:56.771834626 +0800 CST m=+11086392.629344049, currentStep:0, steps:[add learner peer 1636011 on store 1498967, promote learner peer 1636011 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447521(41470,39), createAt:2021-12-23 17:44:56.772159709 +0800 CST m=+11086392.629669129, startAt:2021-12-23 17:44:56.772188375 +0800 CST m=+11086392.629697799, currentStep:0, steps:[add learner peer 1636015 on store 1498967, promote learner peer 1636015 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447793(41446,45), createAt:2021-12-23 17:44:56.77115426 +0800 CST m=+11086392.628663677, startAt:2021-12-23 17:44:56.77118317 +0800 CST m=+11086392.628692594, currentStep:0, steps:[add learner peer 1636004 on store 1498967, promote learner peer 1636004 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:103532(1028,16), createAt:2021-12-23 17:41:51.154153575 +0800 CST m=+11086207.011662993, startAt:2021-12-23 17:41:51.154235478 +0800 CST m=+11086207.011744901, currentStep:0, steps:[add learner peer 1635958 on store 1, promote learner peer 1635958 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:1444949(357,86), createAt:2021-12-23 17:46:47.487895593 +0800 CST m=+11086503.345405016, startAt:2021-12-23 17:46:47.487973235 +0800 CST m=+11086503.345482657, currentStep:0, steps:[add learner peer 1636072 on store 1, promote learner peer 1636072 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448093(41459,45), createAt:2021-12-23 17:44:56.771890398 +0800 CST m=+11086392.629399817, startAt:2021-12-23 17:44:56.771918489 +0800 CST m=+11086392.629427911, currentStep:0, steps:[add learner peer 1636012 on store 1498967, promote learner peer 1636012 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447849(41449,49), createAt:2021-12-23 17:44:56.771240655 +0800 CST m=+11086392.628750074, startAt:2021-12-23 17:44:56.77127065 +0800 CST m=+11086392.628780073, currentStep:0, steps:[add learner peer 1636005 on store 1498967, promote learner peer 1636005 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448789(41447,41), createAt:2021-12-23 17:44:56.775087318 +0800 CST m=+11086392.632596737, startAt:2021-12-23 17:44:56.775114956 +0800 CST m=+11086392.632624378, currentStep:0, steps:[add learner peer 1636046 on store 1498967, promote learner peer 1636046 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447825(41447,47), createAt:2021-12-23 17:44:56.776083321 +0800 CST m=+11086392.633592740, startAt:2021-12-23 17:44:56.776110907 +0800 CST m=+11086392.633620331, currentStep:0, steps:[add learner peer 1636057 on store 1498967, promote learner peer 1636057 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448053(41448,47), createAt:2021-12-23 17:44:56.776252904 +0800 CST m=+11086392.633762322, startAt:2021-12-23 17:44:56.776280945 +0800 CST m=+11086392.633790367, currentStep:0, steps:[add learner peer 1636059 on store 1498967, promote learner peer 1636059 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447917(41446,31), createAt:2021-12-23 17:44:56.772740536 +0800 CST m=+11086392.630249955, startAt:2021-12-23 17:44:56.772772364 +0800 CST m=+11086392.630281786, currentStep:0, steps:[add learner peer 1636020 on store 1498967, promote learner peer 1636020 on store 1498967 to voter, remove peer on store 2])”,
“merge-region {merge: region 10368 to 10476} (kind:leader,region,merge, region:10368(401,572), createAt:2021-12-23 17:47:54.864464239 +0800 CST m=+11086570.721973657, startAt:2021-12-23 17:47:54.864581059 +0800 CST m=+11086570.722090536, currentStep:0, steps:[add learner peer 1636074 on store 1, promote learner peer 1636074 on store 1 to voter, transfer leader from store 1498967 to store 3, remove peer on store 1498967, merge region 10368 into region 10476])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447889(41451,49), createAt:2021-12-23 17:44:56.771627715 +0800 CST m=+11086392.629137134, startAt:2021-12-23 17:44:56.771656229 +0800 CST m=+11086392.629165653, currentStep:0, steps:[add learner peer 1636009 on store 1498967, promote learner peer 1636009 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447885(41451,51), createAt:2021-12-23 17:44:56.771532875 +0800 CST m=+11086392.629042296, startAt:2021-12-23 17:44:56.771572143 +0800 CST m=+11086392.629081564, currentStep:0, steps:[add learner peer 1636008 on store 1498967, promote learner peer 1636008 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:126992(1108,12), createAt:2021-12-23 17:45:15.879943264 +0800 CST m=+11086411.737452686, startAt:2021-12-23 17:45:15.880020442 +0800 CST m=+11086411.737529865, currentStep:0, steps:[add learner peer 1636069 on store 1, promote learner peer 1636069 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [3]} (kind:region,replica, region:32208(330,302), createAt:2021-12-23 17:41:07.103579706 +0800 CST m=+11086162.961089127, startAt:2021-12-23 17:41:07.103650275 +0800 CST m=+11086162.961159701, currentStep:0, steps:[add learner peer 1635935 on store 3, promote learner peer 1635935 on store 3 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:63224(567,94), createAt:2021-12-23 17:41:47.149569313 +0800 CST m=+11086203.007078740, startAt:2021-12-23 17:41:47.14965659 +0800 CST m=+11086203.007166016, currentStep:0, steps:[add learner peer 1635957 on store 1, promote learner peer 1635957 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447905(41443,23), createAt:2021-12-23 17:44:56.752829971 +0800 CST m=+11086392.610339391, startAt:2021-12-23 17:44:56.752867291 +0800 CST m=+11086392.610376713, currentStep:0, steps:[add learner peer 1635977 on store 1498967, promote learner peer 1635977 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:948784(516,214), createAt:2021-12-23 17:45:05.76964191 +0800 CST m=+11086401.627151328, startAt:2021-12-23 17:45:05.769713243 +0800 CST m=+11086401.627222667, currentStep:0, steps:[add learner peer 1636065 on store 1, promote learner peer 1636065 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447593(41446,31), createAt:2021-12-23 17:44:56.773097029 +0800 CST m=+11086392.630606447, startAt:2021-12-23 17:44:56.773125314 +0800 CST m=+11086392.630634735, currentStep:0, steps:[add learner peer 1636024 on store 1498967, promote learner peer 1636024 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448001(41448,23), createAt:2021-12-23 17:44:56.753105942 +0800 CST m=+11086392.610615366, startAt:2021-12-23 17:44:56.753133248 +0800 CST m=+11086392.610642671, currentStep:0, steps:[add learner peer 1635980 on store 1498967, promote learner peer 1635980 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447757(41449,33), createAt:2021-12-23 17:44:56.774740222 +0800 CST m=+11086392.632249639, startAt:2021-12-23 17:44:56.774768351 +0800 CST m=+11086392.632277770, currentStep:0, steps:[add learner peer 1636042 on store 1498967, promote learner peer 1636042 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448289(41446,39), createAt:2021-12-23 17:44:56.772384434 +0800 CST m=+11086392.629893857, startAt:2021-12-23 17:44:56.77241943 +0800 CST m=+11086392.629928852, currentStep:0, steps:[add learner peer 1636016 on store 1498967, promote learner peer 1636016 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:818008(466,116), createAt:2021-12-23 17:46:17.653941876 +0800 CST m=+11086473.511451295, startAt:2021-12-23 17:46:17.654005721 +0800 CST m=+11086473.511515143, currentStep:0, steps:[add learner peer 1636071 on store 1, promote learner peer 1636071 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448161(41453,45), createAt:2021-12-23 17:44:56.753808307 +0800 CST m=+11086392.611317724, startAt:2021-12-23 17:44:56.75384591 +0800 CST m=+11086392.611355332, currentStep:0, steps:[add learner peer 1635988 on store 1498967, promote learner peer 1635988 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:16892(436,66), createAt:2021-12-23 17:46:51.493379354 +0800 CST m=+11086507.350888776, startAt:2021-12-23 17:46:51.493447678 +0800 CST m=+11086507.350957099, currentStep:0, steps:[add learner peer 1636073 on store 1, promote learner peer 1636073 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447561(41435,47), createAt:2021-12-23 17:44:56.754841497 +0800 CST m=+11086392.612350917, startAt:2021-12-23 17:44:56.754898993 +0800 CST m=+11086392.612408415, currentStep:0, steps:[add learner peer 1635995 on store 1498967, promote learner peer 1635995 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:1244113(14058,108), createAt:2021-12-23 17:41:23.120503041 +0800 CST m=+11086178.978012459, startAt:2021-12-23 17:41:23.120549279 +0800 CST m=+11086178.978058700, currentStep:0, steps:[add learner peer 1635946 on store 1, promote learner peer 1635946 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447857(41449,49), createAt:2021-12-23 17:44:56.771438418 +0800 CST m=+11086392.628947837, startAt:2021-12-23 17:44:56.77147728 +0800 CST m=+11086392.628986802, currentStep:0, steps:[add learner peer 1636007 on store 1498967, promote learner peer 1636007 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:47244(397,278), createAt:2021-12-23 17:41:11.108299725 +0800 CST m=+11086166.965809144, startAt:2021-12-23 17:41:11.108377227 +0800 CST m=+11086166.965886750, currentStep:0, steps:[add learner peer 1635940 on store 1, promote learner peer 1635940 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447597(41438,47), createAt:2021-12-23 17:44:56.755534129 +0800 CST m=+11086392.613043554, startAt:2021-12-23 17:44:56.755585672 +0800 CST m=+11086392.613095098, currentStep:0, steps:[add learner peer 1635998 on store 1498967, promote learner peer 1635998 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447773(41447,49), createAt:2021-12-23 17:44:56.775724511 +0800 CST m=+11086392.633233929, startAt:2021-12-23 17:44:56.775754781 +0800 CST m=+11086392.633264203, currentStep:0, steps:[add learner peer 1636053 on store 1498967, promote learner peer 1636053 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448233(41447,31), createAt:2021-12-23 17:44:56.774467211 +0800 CST m=+11086392.631976631, startAt:2021-12-23 17:44:56.77450306 +0800 CST m=+11086392.632012483, currentStep:0, steps:[add learner peer 1636039 on store 1498967, promote learner peer 1636039 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448237(41444,33), createAt:2021-12-23 17:44:56.772656103 +0800 CST m=+11086392.630165523, startAt:2021-12-23 17:44:56.772684919 +0800 CST m=+11086392.630194341, currentStep:0, steps:[add learner peer 1636019 on store 1498967, promote learner peer 1636019 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:1588319(34026,182), createAt:2021-12-23 17:41:27.123496592 +0800 CST m=+11086182.981006015, startAt:2021-12-23 17:41:27.123577483 +0800 CST m=+11086182.981086910, currentStep:0, steps:[add learner peer 1635948 on store 1, promote learner peer 1635948 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448205(41454,47), createAt:2021-12-23 17:44:56.7539011 +0800 CST m=+11086392.611410522, startAt:2021-12-23 17:44:56.753933493 +0800 CST m=+11086392.611442917, currentStep:0, steps:[add learner peer 1635989 on store 1498967, promote learner peer 1635989 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447853(41449,47), createAt:2021-12-23 17:44:56.771338304 +0800 CST m=+11086392.628847739, startAt:2021-12-23 17:44:56.771366855 +0800 CST m=+11086392.628876279, currentStep:0, steps:[add learner peer 1636006 on store 1498967, promote learner peer 1636006 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:109440(1095,200), createAt:2021-12-23 17:41:19.117013302 +0800 CST m=+11086174.974522721, startAt:2021-12-23 17:41:19.117066827 +0800 CST m=+11086174.974576251, currentStep:0, steps:[add learner peer 1635945 on store 1, promote learner peer 1635945 on store 1 to voter, remove peer on store 2])”,
“merge-region {merge: region 14096 to 12592} (kind:leader,region,merge, region:14096(409,566), createAt:2021-12-23 17:45:08.873188431 +0800 CST m=+11086404.730697851, startAt:2021-12-23 17:45:08.87326678 +0800 CST m=+11086404.730776231, currentStep:0, steps:[add learner peer 1636066 on store 1, promote learner peer 1636066 on store 1 to voter, transfer leader from store 1498967 to store 3, remove peer on store 1498967, merge region 14096 into region 12592])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:1145709(33703,256), createAt:2021-12-23 17:44:17.308403561 +0800 CST m=+11086353.165912981, startAt:2021-12-23 17:44:17.308471279 +0800 CST m=+11086353.165980702, currentStep:0, steps:[add learner peer 1635966 on store 1, promote learner peer 1635966 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448501(41447,47), createAt:2021-12-23 17:44:56.775343403 +0800 CST m=+11086392.632852820, startAt:2021-12-23 17:44:56.775386581 +0800 CST m=+11086392.632896063, currentStep:0, steps:[add learner peer 1636049 on store 1498967, promote learner peer 1636049 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448157(41453,47), createAt:2021-12-23 17:44:56.753716734 +0800 CST m=+11086392.611226153, startAt:2021-12-23 17:44:56.75374661 +0800 CST m=+11086392.611256086, currentStep:0, steps:[add learner peer 1635987 on store 1498967, promote learner peer 1635987 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [3]} (kind:region,replica, region:1269285(33698,212), createAt:2021-12-23 17:41:27.224251914 +0800 CST m=+11086183.081761336, startAt:2021-12-23 17:41:27.224333333 +0800 CST m=+11086183.081842759, currentStep:0, steps:[add learner peer 1635949 on store 3, promote learner peer 1635949 on store 3 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447753(41446,33), createAt:2021-12-23 17:44:56.774567146 +0800 CST m=+11086392.632076564, startAt:2021-12-23 17:44:56.774595571 +0800 CST m=+11086392.632104994, currentStep:0, steps:[add learner peer 1636040 on store 1498967, promote learner peer 1636040 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447909(41443,23), createAt:2021-12-23 17:44:56.752923657 +0800 CST m=+11086392.610433078, startAt:2021-12-23 17:44:56.752967715 +0800 CST m=+11086392.610477136, currentStep:0, steps:[add learner peer 1635978 on store 1498967, promote learner peer 1635978 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447553(41443,31), createAt:2021-12-23 17:44:56.773637696 +0800 CST m=+11086392.631147115, startAt:2021-12-23 17:44:56.773672748 +0800 CST m=+11086392.631182171, currentStep:0, steps:[add learner peer 1636030 on store 1498967, promote learner peer 1636030 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448017(41447,31), createAt:2021-12-23 17:44:56.773733129 +0800 CST m=+11086392.631242548, startAt:2021-12-23 17:44:56.773768573 +0800 CST m=+11086392.631277997, currentStep:0, steps:[add learner peer 1636031 on store 1498967, promote learner peer 1636031 on store 1498967 to voter, remove peer on store 2])”,
“merge-region {merge: region 10656 to 142536} (kind:leader,region,merge, region:10656(342,656), createAt:2021-12-23 17:45:05.670687165 +0800 CST m=+11086401.528196584, startAt:2021-12-23 17:45:05.670784419 +0800 CST m=+11086401.528293843, currentStep:0, steps:[add learner peer 1636064 on store 3, promote learner peer 1636064 on store 3 to voter, transfer leader from store 1498967 to store 1, remove peer on store 1498967, merge region 10656 into region 142536])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:1124041(422,190), createAt:2021-12-23 17:45:09.072342965 +0800 CST m=+11086404.929852385, startAt:2021-12-23 17:45:09.072415483 +0800 CST m=+11086404.929925006, currentStep:0, steps:[add learner peer 1636068 on store 1, promote learner peer 1636068 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448013(41448,29), createAt:2021-12-23 17:44:56.753356854 +0800 CST m=+11086392.610866274, startAt:2021-12-23 17:44:56.753402322 +0800 CST m=+11086392.610911765, currentStep:0, steps:[add learner peer 1635983 on store 1498967, promote learner peer 1635983 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447925(41443,35), createAt:2021-12-23 17:44:56.774181255 +0800 CST m=+11086392.631690675, startAt:2021-12-23 17:44:56.774209449 +0800 CST m=+11086392.631718871, currentStep:0, steps:[add learner peer 1636036 on store 1498967, promote learner peer 1636036 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447781(41440,47), createAt:2021-12-23 17:44:56.776622705 +0800 CST m=+11086392.634132125, startAt:2021-12-23 17:44:56.776658444 +0800 CST m=+11086392.634167866, currentStep:0, steps:[add learner peer 1636063 on store 1498967, promote learner peer 1636063 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448773(41448,31), createAt:2021-12-23 17:44:56.773460789 +0800 CST m=+11086392.630970209, startAt:2021-12-23 17:44:56.773494811 +0800 CST m=+11086392.631004233, currentStep:0, steps:[add learner peer 1636028 on store 1498967, promote learner peer 1636028 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448777(41447,37), createAt:2021-12-23 17:44:56.774655309 +0800 CST m=+11086392.632164728, startAt:2021-12-23 17:44:56.77468298 +0800 CST m=+11086392.632192403, currentStep:0, steps:[add learner peer 1636041 on store 1498967, promote learner peer 1636041 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [3]} (kind:region,replica, region:109100(1082,92), createAt:2021-12-23 17:41:19.116396354 +0800 CST m=+11086174.973905776, startAt:2021-12-23 17:41:19.116473199 +0800 CST m=+11086174.973982719, currentStep:0, steps:[add learner peer 1635944 on store 3, promote learner peer 1635944 on store 3 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448133(41447,47), createAt:2021-12-23 17:44:56.775995418 +0800 CST m=+11086392.633504836, startAt:2021-12-23 17:44:56.776023703 +0800 CST m=+11086392.633533126, currentStep:0, steps:[add learner peer 1636056 on store 1498967, promote learner peer 1636056 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447569(41437,47), createAt:2021-12-23 17:44:56.755262725 +0800 CST m=+11086392.612772145, startAt:2021-12-23 17:44:56.755339908 +0800 CST m=+11086392.612849337, currentStep:0, steps:[add learner peer 1635997 on store 1498967, promote learner peer 1635997 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447785(41446,47), createAt:2021-12-23 17:44:56.770959792 +0800 CST m=+11086392.628469209, startAt:2021-12-23 17:44:56.770999808 +0800 CST m=+11086392.628509230, currentStep:0, steps:[add learner peer 1636002 on store 1498967, promote learner peer 1636002 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447897(41444,17), createAt:2021-12-23 17:44:56.752554364 +0800 CST m=+11086392.610063783, startAt:2021-12-23 17:44:56.752589042 +0800 CST m=+11086392.610098463, currentStep:0, steps:[add learner peer 1635974 on store 1498967, promote learner peer 1635974 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447649(41440,47), createAt:2021-12-23 17:44:56.755737477 +0800 CST m=+11086392.613246894, startAt:2021-12-23 17:44:56.755782461 +0800 CST m=+11086392.613291883, currentStep:0, steps:[add learner peer 1635999 on store 1498967, promote learner peer 1635999 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [3]} (kind:region,replica, region:1342453(38982,182), createAt:2021-12-23 17:41:31.228585004 +0800 CST m=+11086187.086094422, startAt:2021-12-23 17 e-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448929(41447,49), createAt:2021-12-23 17:44:56.775635146 +0800 CST m=+11086392.633144566, startAt:2021-12-23 17:44:56.775664491 +0800 CST m=+11086392.633173914, currentStep:0, steps:[add learner peer 1636052 on store 1498967, promote learner peer 1636052 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448245(41456,49), createAt:2021-12-23 17:44:56.754074505 +0800 CST m=+11086392.611583925, startAt:2021-12-23 17:44:56.754102624 +0800 CST m=+11086392.611612048, currentStep:0, steps:[add learner peer 1635991 on store 1498967, promote learner peer 1635991 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448497(41447,43), createAt:2021-12-23 17:44:56.775259079 +0800 CST m=+11086392.632768498, startAt:2021-12-23 17:44:56.775286997 +0800 CST m=+11086392.632796420, currentStep:0, steps:[add learner peer 1636048 on store 1498967, promote learner peer 1636048 on store 1498967 to voter, remove peer on store 2])”,
“merge-region {merge: region 12384 to 2472} (kind:leader,region,merge, region:2472(432,5), createAt:2021-12-23 17:45:08.974023383 +0800 CST m=+11086404.831532801, startAt:2021-12-23 17:45:08.974130797 +0800 CST m=+11086404.831640220, currentStep:0, steps:[merge region 12384 into region 2472])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448457(41459,49), createAt:2021-12-23 17:44:56.754403462 +0800 CST m=+11086392.611912882, startAt:2021-12-23 17:44:56.754435772 +0800 CST m=+11086392.611945194, currentStep:0, steps:[add learner peer 1635993 on store 1498967, promote learner peer 1635993 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448369(41444,15), createAt:2021-12-23 17:44:56.752404666 +0800 CST m=+11086392.609914085, startAt:2021-12-23 17:44:56.752478722 +0800 CST m=+11086392.609988148, currentStep:0, steps:[add learner peer 1635973 on store 1498967, promote learner peer 1635973 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448009(41448,29), createAt:2021-12-23 17:44:56.753272974 +0800 CST m=+11086392.610782394, startAt:2021-12-23 17:44:56.753301167 +0800 CST m=+11086392.610810588, currentStep:0, steps:[add learner peer 1635982 on store 1498967, promote learner peer 1635982 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447921(41441,47), createAt:2021-12-23 17:44:56.77653802 +0800 CST m=+11086392.634047439, startAt:2021-12-23 17:44:56.776566573 +0800 CST m=+11086392.634075995, currentStep:0, steps:[add learner peer 1636062 on store 1498967, promote learner peer 1636062 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:1375093(39751,320), createAt:2021-12-23 17:44:21.212416448 +0800 CST m=+11086357.069925867, startAt:2021-12-23 17:44:21.212458725 +0800 CST m=+11086357.069968200, currentStep:0, steps:[add learner peer 1635969 on store 1, promote learner peer 1635969 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:1375981(39803,314), createAt:2021-12-23 17:44:21.212641506 +0800 CST m=+11086357.070150925, startAt:2021-12-23 17:44:21.212674048 +0800 CST m=+11086357.070183471, currentStep:0, steps:[add learner peer 1635970 on store 1, promote learner peer 1635970 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447777(41449,47), createAt:2021-12-23 17:44:56.775897058 +0800 CST m=+11086392.633406476, startAt:2021-12-23 17:44:56.775935872 +0800 CST m=+11086392.633445296, currentStep:0, steps:[add learner peer 1636055 on store 1498967, promote learner peer 1636055 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447961(41441,31), createAt:2021-12-23 17:44:56.772915273 +0800 CST m=+11086392.630424693, startAt:2021-12-23 17:44:56.772943439 +0800 CST m=+11086392.630452862, currentStep:0, steps:[add learner peer 1636022 on store 1498967, promote learner peer 1636022 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [3]} (kind:region,replica, region:46896(399,102), createAt:2021-12-23 17:42:18.789174488 +0800 CST m=+11086234.646683910, startAt:2021-12-23 17:42:18.789241626 +0800 CST m=+11086234.646751047, currentStep:0, steps:[add learner peer 1635963 on store 3, promote learner peer 1635963 on store 3 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447965(41443,31), createAt:2021-12-23 17:44:56.772999478 +0800 CST m=+11086392.630508895, startAt:2021-12-23 17:44:56.773039442 +0800 CST m=+11086392.630548865, currentStep:0, steps:[add learner peer 1636023 on store 1498967, promote learner peer 1636023 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447637(41441,31), createAt:2021-12-23 17:44:56.774094533 +0800 CST m=+11086392.631603953, startAt:2021-12-23 17:44:56.774122984 +0800 CST m=+11086392.631632406, currentStep:0, steps:[add learner peer 1636035 on store 1498967, promote learner peer 1636035 on store 1498967 to voter, remove peer on store 2])”,
"replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:1236(315,498),
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:64540(636,106), createAt:2021-12-23 17:44:02.691664235 +0800 CST m=+11086338.549173655, startAt:2021-12-23 17:44:02.691741073 +0800 CST m=+11086338.549250499, currentStep:0, steps:[add learner peer 1635964 on store 1, promote learner peer 1635964 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448569(41442,31), createAt:2021-12-23 17:44:56.773550201 +0800 CST m=+11086392.631059620, startAt:2021-12-23 17:44:56.773578267 +0800 CST m=+11086392.631087690, currentStep:0, steps:[add learner peer 1636029 on store 1498967, promote learner peer 1636029 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448257(41468,41), createAt:2021-12-23 17:44:56.772068725 +0800 CST m=+11086392.629578143, startAt:2021-12-23 17:44:56.772101036 +0800 CST m=+11086392.629610460, currentStep:0, steps:[add learner peer 1636014 on store 1498967, promote learner peer 1636014 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1447741(41442,31), createAt:2021-12-23 17:44:56.773831723 +0800 CST m=+11086392.631341141, startAt:2021-12-23 17:44:56.773860096 +0800 CST m=+11086392.631369519, currentStep:0, steps:[add learner peer 1636032 on store 1498967, promote learner peer 1636032 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [3]} (kind:region,replica, region:1214501(14010,92), createAt:2021-12-23 17:41:23.122391585 +0800 CST m=+11086178.979901004, startAt:2021-12-23 17:41:23.122450902 +0800 CST m=+11086178.979960450, currentStep:0, steps:[add learner peer 1635947 on store 3, promote learner peer 1635947 on store 3 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1]} (kind:region,replica, region:1374645(39736,294), createAt:2021-12-23 17:44:21.211547446 +0800 CST m=+11086357.069056865, startAt:2021-12-23 17:44:21.211633234 +0800 CST m=+11086357.069142662, currentStep:0, steps:[add learner peer 1635967 on store 1, promote learner peer 1635967 on store 1 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [1498967]} (kind:region,replica, region:1448905(41447,47), createAt:2021-12-23 17:44:56.77581275 +0800 CST m=+11086392.633322168, startAt:2021-12-23 17:44:56.775840689 +0800 CST m=+11086392.633350110, currentStep:0, steps:[add learner peer 1636054 on store 1498967, promote learner peer 1636054 on store 1498967 to voter, remove peer on store 2])”,
“replace-rule-offline-peer {mv peer: store [2] to [3]} (kind:region,replica, region:120768(1037,122), createAt:2021-12-23 17:41:15.113171269 +0800 CST m=+11086170.970680688, startAt:2021-12-23 17:41:15.11321252 +0800 CST m=+11086170.970721944, currentStep:0, steps:[add learner peer 1635943 on store 3, promote learner peer 1635943 on store 3 to voter, remove peer on store 2])”
]

执行tiup ctl pd -u http://ip:2379 operator show出来很多operator。这种也不正常吧？？？

[林先森cC]

我看3副本是均匀分布 不存在一个region 有两个副本在一个tikv的情况 我在测试环境测试了下 不管扩容还是缩容 region都是会自动调度转移的 分配也很均匀 生产这套就是之前tikv cpu爆了堵住了 我们重启了两个节点（因为写热点把cpu撑爆了 之前也重启过几次 都没问题） 后面就开始发现down 和pending 的region一直在增加 尝试新增一个tikv节点region迁移非常慢 一天过去 几乎只有leader飘过去一些 下线了怀疑有问题的 store2 发现store2上的region迁移不走 报我同事发的那个超时的报错 感觉是哪里堵住 后面我们设置驱逐了下线中的store2的leader发现过了快1小时 发现down的region突然下降了 然后pending在慢慢的下降了直到稳定在几十个 现在就是下线中的region飘不走 region分布不均匀 之前两个老节点leader和flower不往新节点移动

[林先森cC]

此问题已经解决
我们强制缩容了下线不掉的那个节点
tiup cluster scale-in hzxs-tidb-bi-1 --node tikvhost:20160 --force
执行tiup cluster display hzxs-tidb-bi-1发现已经没有store2 的信息了并扩容了一个节点
但是执行tiup ctl pd -u http://xxxx:2379 store 发现store2节点信息还在
于是执行了 stores remove-tombstone 和tiup cluster prune 集群名
似乎还是没效果
参考社区其他帖子
把 offline 状态的节点修改成 Tombstone 状态
执行了如下命令（4.x版本才有用）：

• 1.把 offline 状态的节点修改成 Tombstone 状态，curl -X POST http://{pd_ip}:2379/pd/api/v1/store/${store_id}/state?state=Tombstone

执行tiup ctl pd -u http://xxxx:2379 store 发现信息没了 但是表里面却还能查到store2的peer信息,region也没有迁移 其他节点一直报连接不上store2的region错误
最终选择在半夜业务低峰期使用了这条命令 这条命令很危险 执行完可能导致region不可能 我们执行了一个节点拉起来后tikv集群就报错了 因为后半夜有导入数据的任务 最后选择停掉所有节点 全部tikv执行了一遍这条命令清除store2信息 然后重启集群 集群恢复正常 region也开始往新节点迁移 之前region 一直是卡住的 不往新节点迁移

总结：
怀疑触发到了啥bug 下线的store2 里面的region都是pending状态 感觉是堵住了 down和pending数量一直是同步增加的 后面看到报错说flower信息落后leader太多 看了下其他帖子 也有很多遇到下线不成功的情况 即使是5.版本 在集群region调度异常的情况下 谨慎重启和下线tikv节点 我们之前滚动重启最后发现leader 都跑到一台去了，不会均匀调度。 有兴趣的同学可以看看我发在帖子里的报错日志和监控 再此感谢所有热情回复帖子的大佬们 给我们提供了很多思路。

[spc_monkey (carry@pingcap.com)]

先点个赞，我们也再看一下

[tidb_liu (Hacker S G Rwhv Xs)]

@林先森cC

Tombstone

我这边貌似也遇到了和你同样的问题，但是在执行
http://{pd_ip}:2379/pd/api/v1/store/${store_id}/state?state=Tombstone时返回了：
“invalid state tombstone”
我用的版本是5.2.2

[林先森cC]

你是几个节点呢

[林先森cC]

执行了 stores remove-tombstone 和tiup cluster prune 集群名 这两种方式试过吗。如果你只有3个tikv节点的话是下不掉