集群中一个kv节点下线时,出现其他有kv节点无法启动

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:

【TiDB 版本】
v4.0.6
【问题描述】
集群当前状态如图

其中12kv节点正处于下线状态,通过监控发现region确实在减少,可以认为处于正常下线状态,
147与236kv节点处于down状态,147节点的日志如下:

[2021/04/22 01:18:12.279 +00:00] [INFO] [raft.rs:833] ["became pre-candidate at term 1980"] [term=1980] [raft_id=246235410] [region_id=203974]

[2021/04/22 01:18:12.279 +00:00] [INFO] [raft.rs:902] ["246235410 received message from 246235410"] [term=1980] [msg=MsgRequestPreVote] [from=246235410] [id=246235410] [raft_id=246235410] [region_id=203974]

[2021/04/22 01:18:12.279 +00:00] [INFO] [raft.rs:923] ["[logterm: 1980, index: 15993] sent request to 247271224"] [msg=MsgRequestPreVote] [term=1980] [id=247271224] [log_index=15993] [log_term=1980] [raft_id=246235410] [region_id=203974]

[2021/04/22 01:18:12.279 +00:00] [INFO] [raft.rs:923] ["[logterm: 1980, index: 15993] sent request to 262012289"] [msg=MsgRequestPreVote] [term=1980] [id=262012289] [log_index=15993] [log_term=1980] [raft_id=246235410] [region_id=203974]

[2021/04/22 01:18:12.279 +00:00] [WARN] [raft_client.rs:199] ["send to 10.12.5.236:20160 fail, the gRPC connection could be broken"]

[2021/04/22 01:18:12.279 +00:00] [ERROR] [transport.rs:163] ["send raft msg err"] [err="Other(\"[src/server/raft_client.rs:208]: RaftClient send fail\")"]

[2021/04/22 01:18:12.291 +00:00] [INFO] [raft.rs:1208] ["[logterm: 2050, index: 2114, vote: 194021256] rejected vote from 268649611 [logterm: 1994, index: 2064] at term 2051"] ["msg type"=MsgRequestPreVote] [term=2051] [msg_index=2064] [msg_term=1994] [from=268649611] [vote=194021256] [log_index=2114] [log_term=2050] [raft_id=194021256] [region_id=55432]

[2021/04/22 01:18:12.295 +00:00] [INFO] [raft.rs:1208] ["[logterm: 532, index: 569, vote: 38845165] rejected vote from 268662127 [logterm: 463, index: 514] at term 532"] ["msg type"=MsgRequestPreVote] [term=532] [msg_index=514] [msg_term=463] [from=268662127] [vote=38845165] [log_index=569] [log_term=532] [raft_id=38845165] [region_id=18693881]

[2021/04/22 01:18:12.295 +00:00] [INFO] [raft.rs:1208] ["[logterm: 184, index: 220, vote: 247606726] rejected vote from 268649073 [logterm: 117, index: 166] at term 184"] ["msg type"=MsgRequestPreVote] [term=184] [msg_index=166] [msg_term=117] [from=268649073] [vote=247606726] [log_index=220] [log_term=184] [raft_id=247606726] [region_id=39147607]

[2021/04/22 01:18:12.298 +00:00] [INFO] [raft.rs:1208] ["[logterm: 1873, index: 15650, vote: 194243136] rejected vote from 268571645 [logterm: 1803, index: 15589] at term 1873"] ["msg type"=MsgRequestPreVote] [term=1873] [msg_index=15589] [msg_term=1803] [from=268571645] [vote=194243136] [log_index=15650] [log_term=1873] [raft_id=194243136] [region_id=195989]

[2021/04/22 01:18:12.304 +00:00] [INFO] [raft.rs:1208] ["[logterm: 200, index: 187, vote: 238154115] rejected vote from 268642327 [logterm: 140, index: 136] at term 200"] ["msg type"=MsgRequestPreVote] [term=200] [msg_index=136] [msg_term=140] [from=268642327] [vote=238154115] [log_index=187] [log_term=200] [raft_id=238154115] [region_id=80610864]

[2021/04/22 01:18:12.308 +00:00] [INFO] [raft.rs:1208] ["[logterm: 160, index: 157, vote: 246648589] rejected vote from 268592857 [logterm: 109, index: 117] at term 160"] ["msg type"=MsgRequestPreVote] [term=160] [msg_index=117] [msg_term=109] [from=268592857] [vote=246648589] [log_index=157] [log_term=160] [raft_id=246648589] [region_id=80616381]

[2021/04/22 01:18:12.345 +00:00] [INFO] [raft.rs:1208] ["[logterm: 11683, index: 13464, vote: 248646876] rejected vote from 268662264 [logterm: 11628, index: 13417] at term 11683"] ["msg type"=MsgRequestPreVote] [term=11683] [msg_index=13417] [msg_term=11628] [from=268662264] [vote=248646876] [log_index=13464] [log_term=11683] [raft_id=248646876] [region_id=10360541]

[2021/04/22 01:18:12.346 +00:00] [INFO] [raft.rs:1208] ["[logterm: 129, index: 138, vote: 249042865] rejected vote from 268642267 [logterm: 85, index: 99] at term 130"] ["msg type"=MsgRequestPreVote] [term=130] [msg_index=99] [msg_term=85] [from=268642267] [vote=249042865] [log_index=138] [log_term=129] [raft_id=249042865] [region_id=245377826]

[2021/04/22 01:18:12.346 +00:00] [INFO] [raft.rs:1208] ["[logterm: 149, index: 139, vote: 249219359] rejected vote from 271575602 [logterm: 77, index: 75] at term 149"] ["msg type"=MsgRequestPreVote] [term=149] [msg_index=75] [msg_term=77] [from=271575602] [vote=249219359] [log_index=139] [log_term=149] [raft_id=249219359] [region_id=249219358]

[2021/04/22 01:18:12.357 +00:00] [INFO] [raft.rs:1208] ["[logterm: 6878, index: 47514, vote: 249002190] rejected vote from 268438021 [logterm: 6804, index: 47456] at term 6878"] ["msg type"=MsgRequestPreVote] [term=6878] [msg_index=47456] [msg_term=6804] [from=268438021] [vote=249002190] [log_index=47514] [log_term=6878] [raft_id=249002190] [region_id=192181]

[2021/04/22 01:18:12.357 +00:00] [INFO] [raft.rs:1208] ["[logterm: 1264, index: 73332, vote: 245752076] rejected vote from 268656630 [logterm: 1211, index: 73284] at term 1265"] ["msg type"=MsgRequestPreVote] [term=1265] [msg_index=73284] [msg_term=1211] [from=268656630] [vote=245752076] [log_index=73332] [log_term=1264] [raft_id=245752076] [region_id=9373956]

[2021/04/22 01:18:12.408 +00:00] [INFO] [raft.rs:1208] ["[logterm: 4277, index: 17937, vote: 245840072] rejected vote from 268425072 [logterm: 4227, index: 17892] at term 4277"] ["msg type"=MsgRequestPreVote] [term=4277] [msg_index=17892] [msg_term=4227] [from=268425072] [vote=245840072] [log_index=17937] [log_term=4277] [raft_id=245840072] [region_id=253613]

[2021/04/22 01:18:12.456 +00:00] [INFO] [raft.rs:1208] ["[logterm: 184, index: 686626, vote: 248641677] rejected vote from 269739990 [logterm: 129, index: 686580] at term 184"] ["msg type"=MsgRequestPreVote] [term=184] [msg_index=686580] [msg_term=129] [from=269739990] [vote=248641677] [log_index=686626] [log_term=184] [raft_id=248641677] [region_id=80384650]

[2021/04/22 01:18:12.472 +00:00] [INFO] [raft.rs:1208] ["[logterm: 420, index: 465, vote: 38874788] rejected vote from 268455871 [logterm: 375, index: 425] at term 420"] ["msg type"=MsgRequestPreVote] [term=420] [msg_index=425] [msg_term=375] [from=268455871] [vote=38874788] [log_index=465] [log_term=420] [raft_id=38874788] [region_id=18585924]

[2021/04/22 01:18:12.472 +00:00] [INFO] [raft.rs:1208] ["[logterm: 1275, index: 1307, vote: 246919934] rejected vote from 268548211 [logterm: 1214, index: 1256] at term 1275"] ["msg type"=MsgRequestPreVote] [term=1275] [msg_index=1256] [msg_term=1214] [from=268548211] [vote=246919934] [log_index=1307] [log_term=1275] [raft_id=246919934] [region_id=1315353]

[2021/04/22 01:18:12.472 +00:00] [INFO] [raft.rs:1208] ["[logterm: 868, index: 1021, vote: 235736985] rejected vote from 268453941 [logterm: 809, index: 971] at term 869"] ["msg type"=MsgRequestPreVote] [term=869] [msg_index=971] [msg_term=809] [from=268453941] [vote=235736985] [log_index=1021] [log_term=868] [raft_id=235736985] [region_id=12770333]

[2021/04/22 01:18:12.472 +00:00] [INFO] [raft.rs:1208] ["[logterm: 306, index: 84677, vote: 177475651] rejected vote from 268434332 [logterm: 244, index: 84619] at term 306"] ["msg type"=MsgRequestPreVote] [term=306] [msg_index=84619] [msg_term=244] [from=268434332] [vote=177475651] [log_index=84677] [log_term=306] [raft_id=177475651] [region_id=18659826]

[2021/04/22 01:18:12.473 +00:00] [INFO] [raft.rs:1208] ["[logterm: 18227, index: 30397, vote: 248044077] rejected vote from 276633621 [logterm: 18149, index: 30326] at term 18227"] ["msg type"=MsgRequestPreVote] [term=18227] [msg_index=30326] [msg_term=18149] [from=276633621] [vote=248044077] [log_index=30397] [log_term=18227] [raft_id=248044077] [region_id=233766]

可以看到是因为236节点处于down导致无法通信

236节点的日志如下:

[2021/04/22 01:31:50.446 +00:00] [INFO] [raft.rs:783] ["became follower at term 147"] [term=147] [raft_id=249236196] [region_id=249236193]
[2021/04/22 01:31:50.446 +00:00] [INFO] [raft.rs:1192] ["[logterm: 146, index: 1498, vote: 0] cast vote for 249236195 [logterm: 146, index: 1498] at term 147"] ["msg type"=MsgRequestVote] [term=147] [msg_index=1498] [msg_term=146] [from=249236195] [vote=0] [log_index=1498] [log_term=146] [raft_id=249236196] [region_id=249236193]
[2021/04/22 01:31:50.446 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 247382971"] ["msg type"=MsgRequestVote] [message_term=194] [term=191] [from=247382971] [raft_id=80375255] [region_id=80375254]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 194"] [term=194] [raft_id=80375255] [region_id=80375254]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 191, index: 187, vote: 0] cast vote for 247382971 [logterm: 191, index: 187] at term 194"] ["msg type"=MsgRequestVote] [term=194] [msg_index=187] [msg_term=191] [from=247382971] [vote=0] [log_index=187] [log_term=191] [raft_id=80375255] [region_id=80375254]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 248597078"] ["msg type"=MsgRequestVote] [message_term=192] [term=191] [from=248597078] [raft_id=80278155] [region_id=80278154]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 192"] [term=192] [raft_id=80278155] [region_id=80278154]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 191, index: 196, vote: 0] cast vote for 248597078 [logterm: 191, index: 196] at term 192"] ["msg type"=MsgRequestVote] [term=192] [msg_index=196] [msg_term=191] [from=248597078] [vote=0] [log_index=196] [log_term=191] [raft_id=80278155] [region_id=80278154]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 248529899"] ["msg type"=MsgRequestVote] [message_term=734] [term=733] [from=248529899] [raft_id=248582566] [region_id=9359094]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 734"] [term=734] [raft_id=248582566] [region_id=9359094]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 733, index: 3075, vote: 0] cast vote for 248529899 [logterm: 733, index: 3075] at term 734"] ["msg type"=MsgRequestVote] [term=734] [msg_index=3075] [msg_term=733] [from=248529899] [vote=0] [log_index=3075] [log_term=733] [raft_id=248582566] [region_id=9359094]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 246235410"] ["msg type"=MsgRequestVote] [message_term=1982] [term=1981] [from=246235410] [raft_id=247271224] [region_id=203974]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 1982"] [term=1982] [raft_id=247271224] [region_id=203974]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 1981, index: 15994, vote: 0] cast vote for 246235410 [logterm: 1981, index: 15994] at term 1982"] ["msg type"=MsgRequestVote] [term=1982] [msg_index=15994] [msg_term=1981] [from=246235410] [vote=0] [log_index=15994] [log_term=1981] [raft_id=247271224] [region_id=203974]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 244839270"] ["msg type"=MsgRequestVote] [message_term=245] [term=243] [from=244839270] [raft_id=194127479] [region_id=20159492]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 245"] [term=245] [raft_id=194127479] [region_id=20159492]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 241, index: 777, vote: 0] cast vote for 244839270 [logterm: 241, index: 777] at term 245"] ["msg type"=MsgRequestVote] [term=245] [msg_index=777] [msg_term=241] [from=244839270] [vote=0] [log_index=777] [log_term=241] [raft_id=194127479] [region_id=20159492]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 246704011"] ["msg type"=MsgRequestVote] [message_term=174] [term=172] [from=246704011] [raft_id=139423876] [region_id=139423873]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 174"] [term=174] [raft_id=139423876] [region_id=139423873]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 172, index: 10871, vote: 0] cast vote for 246704011 [logterm: 172, index: 10871] at term 174"] ["msg type"=MsgRequestVote] [term=174] [msg_index=10871] [msg_term=172] [from=246704011] [vote=0] [log_index=10871] [log_term=172] [raft_id=139423876] [region_id=139423873]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 249359279"] ["msg type"=MsgRequestVote] [message_term=1227] [term=1226] [from=249359279] [raft_id=247165494] [region_id=2489002]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 1227"] [term=1227] [raft_id=247165494] [region_id=2489002]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 1226, index: 1266, vote: 0] cast vote for 249359279 [logterm: 1226, index: 1266] at term 1227"] ["msg type"=MsgRequestVote] [term=1227] [msg_index=1266] [msg_term=1226] [from=249359279] [vote=0] [log_index=1266] [log_term=1226] [raft_id=247165494] [region_id=2489002]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 80949886"] ["msg type"=MsgRequestVote] [message_term=6738] [term=6736] [from=80949886] [raft_id=24538947] [region_id=6567835]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 6738"] [term=6738] [raft_id=24538947] [region_id=6567835]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 6736, index: 14967, vote: 0] cast vote for 80949886 [logterm: 6736, index: 14967] at term 6738"] ["msg type"=MsgRequestVote] [term=6738] [msg_index=14967] [msg_term=6736] [from=80949886] [vote=0] [log_index=14967] [log_term=6736] [raft_id=24538947] [region_id=6567835]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 246187713"] ["msg type"=MsgRequestVote] [message_term=750] [term=749] [from=246187713] [raft_id=38943850] [region_id=12804477]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 750"] [term=750] [raft_id=38943850] [region_id=12804477]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 749, index: 768, vote: 0] cast vote for 246187713 [logterm: 749, index: 768] at term 750"] ["msg type"=MsgRequestVote] [term=750] [msg_index=768] [msg_term=749] [from=246187713] [vote=0] [log_index=768] [log_term=749] [raft_id=38943850] [region_id=12804477]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 248969324"] ["msg type"=MsgRequestVote] [message_term=146] [term=145] [from=248969324] [raft_id=248460060] [region_id=248460059]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 146"] [term=146] [raft_id=248460060] [region_id=248460059]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 145, index: 176, vote: 0] cast vote for 248969324 [logterm: 145, index: 176] at term 146"] ["msg type"=MsgRequestVote] [term=146] [msg_index=176] [msg_term=145] [from=248969324] [vote=0] [log_index=176] [log_term=145] [raft_id=248460060] [region_id=248460059]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 247714528"] ["msg type"=MsgRequestVote] [message_term=123] [term=122] [from=247714528] [raft_id=247995112] [region_id=247714527]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 123"] [term=123] [raft_id=247995112] [region_id=247714527]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 122, index: 362, vote: 0] cast vote for 247714528 [logterm: 122, index: 362] at term 123"] ["msg type"=MsgRequestVote] [term=123] [msg_index=362] [msg_term=122] [from=247714528] [vote=0] [log_index=362] [log_term=122] [raft_id=247995112] [region_id=247714527]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 249356550"] ["msg type"=MsgRequestVote] [message_term=178] [term=177] [from=249356550] [raft_id=248695936] [region_id=113248925]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 178"] [term=178] [raft_id=248695936] [region_id=113248925]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 176, index: 235, vote: 0] cast vote for 249356550 [logterm: 177, index: 236] at term 178"] ["msg type"=MsgRequestVote] [term=178] [msg_index=236] [msg_term=177] [from=249356550] [vote=0] [log_index=235] [log_term=176] [raft_id=248695936] [region_id=113248925]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 246811142"] ["msg type"=MsgRequestVote] [message_term=148] [term=147] [from=246811142] [raft_id=248903419] [region_id=246811141]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 148"] [term=148] [raft_id=248903419] [region_id=246811141]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 147, index: 6810, vote: 0] cast vote for 246811142 [logterm: 147, index: 6810] at term 148"] ["msg type"=MsgRequestVote] [term=148] [msg_index=6810] [msg_term=147] [from=246811142] [vote=0] [log_index=6810] [log_term=147] [raft_id=248903419] [region_id=246811141]
[2021/04/22 01:31:50.453 +00:00] [FATAL] [lib.rs:483] ["[region 249205597] 249205599 source_region id: 249205587 start_key: 74800000000002B1FF585F728000000000FF7BC0A10000000000FA end_key: 74800000000002B1FF585F728000000000FF7D50470000000000FA region_epoch { conf_ver: 91443 version: 66961 } peers { id: 249205588 store_id: 24590972 } peers { id: 249205589 store_id: 24478148 } peers { id: 276651401 store_id: 24480822 } not match exist region id: 249205587 start_key: 74800000000002B1FF585F728000000000FF7BC0A10000000000FA end_key: 74800000000002B1FF585F728000000000FF7D50470000000000FA region_epoch { conf_ver: 91443 version: 66961 } peers { id: 249205588 store_id: 24590972 } peers { id: 249205589 store_id: 24478148 } peers { id: 262355238 store_id: 38546296 } peers { id: 276651401 store_id: 24480822 }"] [backtrace="stack backtrace:\
   0: tikv_util::set_panic_hook::{{closure}}\
             at components/tikv_util/src/lib.rs:482\
   1: std::panicking::rust_panic_with_hook\
             at src/libstd/panicking.rs:475\
   2: rust_begin_unwind\
             at src/libstd/panicking.rs:375\
   3: std::panicking::begin_panic_fmt\
             at src/libstd/panicking.rs:326\
   4: raftstore::store::fsm::apply::ApplyDelegate::exec_commit_merge\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/<::std::macros::panic macros>:9\
      raftstore::store::fsm::apply::ApplyDelegate::exec_admin_cmd\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:1182\
   5: raftstore::store::fsm::apply::ApplyDelegate::exec_raft_cmd\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:1148\
      raftstore::store::fsm::apply::ApplyDelegate::apply_raft_cmd\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:1040\
      raftstore::store::fsm::apply::ApplyDelegate::process_raft_cmd\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:991\
   6: raftstore::store::fsm::apply::ApplyDelegate::handle_raft_entry_normal\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:871\
      raftstore::store::fsm::apply::ApplyDelegate::handle_raft_committed_entries\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:789\
   7: raftstore::store::fsm::apply::ApplyFsm::resume_pending\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:2709\
      <raftstore::store::fsm::apply::ApplyPoller<W> as batch_system::batch::PollHandler<raftstore::store::fsm::apply::ApplyFsm,raftstore::store::fsm::apply::ControlFsm>>::handle_normal\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:3027\
   8: batch_system::batch::Poller<N,C,Handler>::poll\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/batch-system/src/batch.rs:294\
      batch_system::batch::BatchSystem<N,C>::spawn::{{closure}}\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/batch-system/src/batch.rs:398\
      std::sys_common::backtrace::__rust_begin_short_backtrace\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/sys_common/backtrace.rs:136\
   9: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:469\
      <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:318\
      std::panicking::try::do_call\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panicking.rs:292\
      std::panicking::try\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8//src/libpanic_unwind/lib.rs:78\
      std::panic::catch_unwind\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:394\
      std::thread::Builder::spawn_unchecked::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:468\
      core::ops::function::FnOnce::call_once{{vtable.shim}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/ops/function.rs:232\
  10: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
  11: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
      std::sys_common::thread::start_thread\
             at src/libstd/sys_common/thread.rs:13\
      std::sys::unix::thread::Thread::new::thread_start\
             at src/libstd/sys/unix/thread.rs:80\
  12: start_thread\
  13: __clone\
"] [location=components/raftstore/src/store/fsm/apply.rs:1954] [thread_name=apply-0]
1 个赞
  1. 手动启动 147 和 236 节点可以正常启动么
  2. 尝试手动启动一下两个节点,然后拿一下 tikv.log 中,最近一次出现 Welcome 关键字那一行开始到最后的 tikv 节点日志信息看下

手动重启236结果如下

但display还是down状态

welcome开始日志如下
[2021/04/22 01:30:32.008 +00:00] [INFO] [lib.rs:92] [“Welcome to TiKV”]

[2021/04/22 01:30:32.009 +00:00] [INFO] [lib.rs:94] []

[2021/04/22 01:30:32.009 +00:00] [INFO] [lib.rs:94] [“Release Version: 4.0.6”]

[2021/04/22 01:30:32.009 +00:00] [INFO] [lib.rs:94] [“Edition: Community”]

[2021/04/22 01:30:32.009 +00:00] [INFO] [lib.rs:94] [“Git Commit Hash: ca2475bfbcb49a7c34cf783596acb3edd05fc88f”]

[2021/04/22 01:30:32.009 +00:00] [INFO] [lib.rs:94] [“Git Commit Branch: release-4.0”]

[2021/04/22 01:30:32.009 +00:00] [INFO] [lib.rs:94] [“UTC Build Time: 2020-09-15 10:51:45”]

[2021/04/22 01:30:32.009 +00:00] [INFO] [lib.rs:94] [“Rust Version: rustc 1.42.0-nightly (0de96d37f 2019-12-19)”]

[2021/04/22 01:30:32.009 +00:00] [INFO] [lib.rs:94] [“Enable Features: jemalloc portable sse protobuf-codec”]

[2021/04/22 01:30:32.009 +00:00] [INFO] [lib.rs:94] [“Profile: dist_release”]

[2021/04/22 01:30:32.039 +00:00] [INFO] [mod.rs:46] [“memory limit in bytes: 67543805952, cpu cores quota: 56”]

[2021/04/22 01:30:32.039 +00:00] [WARN] [lib.rs:530] [“environment variable TZ is missing, using /etc/localtime”]

[2021/04/22 01:30:32.039 +00:00] [WARN] [server.rs:852] [“check: kernel”] [err=“kernel parameters net.core.somaxconn got 128, expect 32768”]

[2021/04/22 01:30:32.039 +00:00] [WARN] [server.rs:852] [“check: kernel”] [err=“kernel parameters net.ipv4.tcp_syncookies got 1, expect 0”]

[2021/04/22 01:30:32.039 +00:00] [WARN] [server.rs:852] [“check: kernel”] [err=“kernel parameters vm.swappiness got 60, expect 0”]

[2021/04/22 01:31:08.265 +00:00] [INFO] [peer.rs:159] [“create peer”] [peer_id=246918346] [region_id=1586]

中间还有很多但最终还是一样的fatal报错

手动重启147失败,且原本up的节点状态变为disconnected

集群状态如下

此时240节点出现了之前147节点的报错,即与236节点通信不通

最终240节点状态为down,和之前的147节点完全一样

\\\

pd 上 region 249205587 和 249205597 是什么样的?

tikv ip 机器对应的 store id 是怎么样的

store_id: 38546296 这个是哪个store啊?

store状态如下:
store
{
“count”: 7,
“stores”: [
{
“store”: {
“id”: 24478148,
“address”: “10.12.5.236:20160”,
“version”: “4.0.6”,
“status_address”: “10.12.5.236:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1619063119,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1618914881248098385,
“state_name”: “Down”
},
“status”: {
“capacity”: “5.952TiB”,
“available”: “2.858TiB”,
“used_size”: “1.97TiB”,
“leader_count”: 7035,
“leader_weight”: 2,
“leader_score”: 3517.5,
“leader_size”: 403999,
“region_count”: 54200,
“region_weight”: 2,
“region_score”: 1813491,
“region_size”: 3626982,
“start_ts”: “2021-04-22T03:45:19Z”,
“last_heartbeat_ts”: “2021-04-20T10:34:41.248098385Z”
}
},
{
“store”: {
“id”: 24480822,
“address”: “10.12.5.239:20160”,
“version”: “4.0.6”,
“status_address”: “10.12.5.239:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1618917905,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619063083013828395,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.952TiB”,
“available”: “3.698TiB”,
“used_size”: “2.152TiB”,
“leader_count”: 27386,
“leader_weight”: 2,
“leader_score”: 13693,
“leader_size”: 2038079,
“region_count”: 92477,
“region_weight”: 2,
“region_score”: 2874272,
“region_size”: 5748544,
“start_ts”: “2021-04-20T11:25:05Z”,
“last_heartbeat_ts”: “2021-04-22T03:44:43.013828395Z”,
“uptime”: “40h19m38.013828395s”
}
},
{
“store”: {
“id”: 24590972,
“address”: “10.12.5.240:20160”,
“version”: “4.0.6”,
“status_address”: “10.12.5.240:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1618995718,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619058897687976254,
“state_name”: “Down”
},
“status”: {
“capacity”: “5.952TiB”,
“available”: “3.896TiB”,
“used_size”: “1.941TiB”,
“leader_count”: 38617,
“leader_weight”: 2,
“leader_score”: 19308.5,
“leader_size”: 2480564,
“region_count”: 88672,
“region_weight”: 2,
“region_score”: 2568008,
“region_size”: 5136016,
“start_ts”: “2021-04-21T09:01:58Z”,
“last_heartbeat_ts”: “2021-04-22T02:34:57.687976254Z”,
“uptime”: “17h32m59.687976254s”
}
},
{
“store”: {
“id”: 38833310,
“address”: “10.12.5.147:20160”,
“version”: “4.0.6”,
“status_address”: “10.12.5.147:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1619059119,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619063084123956461,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.952TiB”,
“available”: “3.569TiB”,
“used_size”: “2.179TiB”,
“leader_count”: 11811,
“leader_weight”: 2,
“leader_score”: 5905.5,
“leader_size”: 616817,
“region_count”: 99076,
“region_weight”: 2,
“region_score”: 2657826.5,
“region_size”: 5315653,
“start_ts”: “2021-04-22T02:38:39Z”,
“last_heartbeat_ts”: “2021-04-22T03:44:44.123956461Z”,
“uptime”: “1h6m5.123956461s”
}
},
{
“store”: {
“id”: 256634687,
“address”: “10.12.5.12:20160”,
“state”: 1,
“version”: “4.0.6”,
“status_address”: “10.12.5.12:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1618814078,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619063083131508880,
“state_name”: “Offline”
},
“status”: {
“capacity”: “2.952TiB”,
“available”: “2.846TiB”,
“used_size”: “77.92GiB”,
“leader_count”: 1130,
“leader_weight”: 1,
“leader_score”: 1130,
“leader_size”: 105746,
“region_count”: 2279,
“region_weight”: 1,
“region_score”: 225570,
“region_size”: 225570,
“start_ts”: “2021-04-19T06:34:38Z”,
“last_heartbeat_ts”: “2021-04-22T03:44:43.13150888Z”,
“uptime”: “69h10m5.13150888s”
}
},
{
“store”: {
“id”: 262397455,
“address”: “10.12.5.13:20160”,
“version”: “4.0.6”,
“status_address”: “10.12.5.13:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1618813933,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619063083206350142,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.952TiB”,
“available”: “4.564TiB”,
“used_size”: “1.367TiB”,
“leader_count”: 28234,
“leader_weight”: 1,
“leader_score”: 28234,
“leader_size”: 1786517,
“region_count”: 58450,
“region_weight”: 1,
“region_score”: 3449118,
“region_size”: 3449118,
“start_ts”: “2021-04-19T06:32:13Z”,
“last_heartbeat_ts”: “2021-04-22T03:44:43.206350142Z”,
“uptime”: “69h12m30.206350142s”
}
},
{
“store”: {
“id”: 268391998,
“address”: “10.12.5.119:20160”,
“version”: “4.0.6”,
“status_address”: “10.12.5.119:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1618813990,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619063087140236297,
“state_name”: “Up”
},
“status”: {
“capacity”: “320TiB”,
“available”: “312.1TiB”,
“used_size”: “1.444TiB”,
“leader_count”: 28239,
“leader_weight”: 1,
“leader_score”: 28239,
“leader_size”: 1551974,
“region_count”: 67467,
“region_weight”: 1,
“region_score”: 3449927,
“region_size”: 3449927,
“start_ts”: “2021-04-19T06:33:10Z”,
“last_heartbeat_ts”: “2021-04-22T03:44:47.140236297Z”,
“uptime”: “69h11m37.140236297s”
}
}
]
}

两个region的状态如下
» region 249205587
null

» region 249205597
{
“id”: 249205597,
“start_key”: “74800000000002B1FF585F728000000000FF7BC0A10000000000FA”,
“end_key”: “74800000000002B1FF585F728000000000FF81FD6A0000000000FA”,
“epoch”: {
“conf_ver”: 91437,
“version”: 66964
},
“peers”: [
{
“id”: 249205598,
“store_id”: 24590972
},
{
“id”: 262391762,
“store_id”: 24480822
},
{
“id”: 285798841,
“store_id”: 262397455
}
],
“leader”: {
“id”: 249205598,
“store_id”: 24590972
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 55,
“approximate_keys”: 398594
}

38546296 是我们集群之前强制下线的一个节点,可能问题就出在这里?

[2021/04/22 01:31:50.453 +00:00] [FATAL] [lib.rs:483]

["[region 249205597] 249205599 source_region id: 249205587
start_key: 74800000000002B1FF585F728000000000FF7BC0A10000000000FA
end_key: 74800000000002B1FF585F728000000000FF7D50470000000000FA
region_epoch { conf_ver: 91443 version: 66961 }
peers { id: 249205588 store_id: 24590972 }
peers { id: 249205589 store_id: 24478148 }
peers { id: 276651401 store_id: 24480822 }
not match exist
region id: 249205587
start_key: 74800000000002B1FF585F728000000000FF7BC0A10000000000FA
end_key: 74800000000002B1FF585F728000000000FF7D50470000000000FA
region_epoch { conf_ver: 91443 version: 66961 }
peers { id: 249205588 store_id: 24590972 }
peers { id: 249205589 store_id: 24478148 }
peers { id: 262355238 store_id: 38546296 }
peers { id: 276651401 store_id: 24480822 }"]

236panic 日志 peers { id: 262355238 store_id: 38546296 } 导致region信息不一致。

之前是不是执行过 unsafe-recover 清理下线的 store 38546296 关联的region? 然后 236漏掉了?

已在236上重新执行unsafe-recover清理store 38546296 region了,现在236处于disconnect状态

目前集群状态如下,236节点已经up

240节点日志如下,已无通信错误,且并未看到其他明显报错,是否需要手动重启

[quote=“abcd, post:14, topic:69770”]
240节点的日志如下,之前的通信错误已经没有,且无其他明显报错,是否需要手动重启

[2021/04/22 05:22:39.677 +00:00] [INFO] [raft.rs:923] [“[logterm: 112, index: 216541] sent request to 80903949”] [msg=MsgRequestPreVote] [term=112] [id=80903949] [log_index=216541] [log_term=112] [raft_id=80903950] [region_id=80903948]

[2021/04/22 05:22:39.677 +00:00] [INFO] [raft.rs:923] [“[logterm: 112, index: 216541] sent request to 261820199”] [msg=MsgRequestPreVote] [term=112] [id=261820199] [log_index=216541] [log_term=112] [raft_id=80903950] [region_id=80903948]

[2021/04/22 05:22:39.677 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=194] [raft_id=268559580] [region_id=80581511]

[2021/04/22 05:22:39.677 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 194”] [term=194] [raft_id=268559580] [region_id=80581511]

[2021/04/22 05:22:39.677 +00:00] [INFO] [raft.rs:902] [“268559580 received message from 268559580”] [term=194] [msg=MsgRequestPreVote] [from=268559580] [id=268559580] [raft_id=268559580] [region_id=80581511]

[2021/04/22 05:22:39.677 +00:00] [INFO] [raft.rs:923] [“[logterm: 132, index: 140] sent request to 246065129”] [msg=MsgRequestPreVote] [term=194] [id=246065129] [log_index=140] [log_term=132] [raft_id=268559580] [region_id=80581511]

[2021/04/22 05:22:39.677 +00:00] [INFO] [raft.rs:923] [“[logterm: 132, index: 140] sent request to 238147907”] [msg=MsgRequestPreVote] [term=194] [id=238147907] [log_index=140] [log_term=132] [raft_id=268559580] [region_id=80581511]

[2021/04/22 05:22:39.687 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=11584] [raft_id=249174513] [region_id=12538216]

[2021/04/22 05:22:39.687 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 11584”] [term=11584] [raft_id=249174513] [region_id=12538216]

[2021/04/22 05:22:39.687 +00:00] [INFO] [raft.rs:902] [“249174513 received message from 249174513”] [term=11584] [msg=MsgRequestPreVote] [from=249174513] [id=249174513] [raft_id=249174513] [region_id=12538216]

[2021/04/22 05:22:39.688 +00:00] [INFO] [raft.rs:923] [“[logterm: 11584, index: 11922] sent request to 261794592”] [msg=MsgRequestPreVote] [term=11584] [id=261794592] [log_index=11922] [log_term=11584] [raft_id=249174513] [region_id=12538216]

[2021/04/22 05:22:39.688 +00:00] [INFO] [raft.rs:923] [“[logterm: 11584, index: 11922] sent request to 246285753”] [msg=MsgRequestPreVote] [term=11584] [id=246285753] [log_index=11922] [log_term=11584] [raft_id=249174513] [region_id=12538216]

[2021/04/22 05:22:39.688 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=222] [raft_id=268429211] [region_id=38280296]

[2021/04/22 05:22:39.688 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 222”] [term=222] [raft_id=268429211] [region_id=38280296]

[2021/04/22 05:22:39.688 +00:00] [INFO] [raft.rs:902] [“268429211 received message from 268429211”] [term=222] [msg=MsgRequestPreVote] [from=268429211] [id=268429211] [raft_id=268429211] [region_id=38280296]

[2021/04/22 05:22:39.688 +00:00] [INFO] [raft.rs:923] [“[logterm: 134, index: 12330] sent request to 38879282”] [msg=MsgRequestPreVote] [term=222] [id=38879282] [log_index=12330] [log_term=134] [raft_id=268429211] [region_id=38280296]

[2021/04/22 05:22:39.688 +00:00] [INFO] [raft.rs:923] [“[logterm: 134, index: 12330] sent request to 246185055”] [msg=MsgRequestPreVote] [term=222] [id=246185055] [log_index=12330] [log_term=134] [raft_id=268429211] [region_id=38280296]

[2021/04/22 05:22:39.706 +00:00] [INFO] [raft.rs:971] [“[logterm: 985, index: 1068, vote: 238254985] ignored vote from 273269596 [logterm: 904, index: 999]: lease is not expired”] [“msg type”=MsgRequestPreVote] [“remaining ticks”=10] [term=985] [msg_index=999] [msg_term=904] [from=273269596] [vote=238254985] [log_index=1068] [log_term=985] [raft_id=238254985] [region_id=407443]

[2021/04/22 05:22:39.709 +00:00] [INFO] [raft.rs:1003] [“received a message with higher term from 248692306”] [“msg type”=MsgRequestVote] [message_term=7250] [term=7249] [from=248692306] [raft_id=248693779] [region_id=232207]

[2021/04/22 05:22:39.709 +00:00] [INFO] [raft.rs:783] [“became follower at term 7250”] [term=7250] [raft_id=248693779] [region_id=232207]

[2021/04/22 05:22:39.709 +00:00] [INFO] [raft.rs:1192] [“[logterm: 7249, index: 21276, vote: 0] cast vote for 248692306 [logterm: 7249, index: 21276] at term 7250”] [“msg type”=MsgRequestVote] [term=7250] [msg_index=21276] [msg_term=7249] [from=248692306] [vote=0] [log_index=21276] [log_term=7249] [raft_id=248693779] [region_id=232207]

[2021/04/22 05:22:39.743 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=191] [raft_id=268655983] [region_id=80289513]

[2021/04/22 05:22:39.743 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 191”] [term=191] [raft_id=268655983] [region_id=80289513]

[2021/04/22 05:22:39.743 +00:00] [INFO] [raft.rs:902] [“268655983 received message from 268655983”] [term=191] [msg=MsgRequestPreVote] [from=268655983] [id=268655983] [raft_id=268655983] [region_id=80289513]

[2021/04/22 05:22:39.743 +00:00] [INFO] [raft.rs:923] [“[logterm: 119, index: 130] sent request to 247590564”] [msg=MsgRequestPreVote] [term=191] [id=247590564] [log_index=130] [log_term=119] [raft_id=268655983] [region_id=80289513]

[2021/04/22 05:22:39.743 +00:00] [INFO] [raft.rs:923] [“[logterm: 119, index: 130] sent request to 175222675”] [msg=MsgRequestPreVote] [term=191] [id=175222675] [log_index=130] [log_term=119] [raft_id=268655983] [region_id=80289513]

[2021/04/22 05:22:39.743 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=73] [raft_id=246876075] [region_id=246876074]

[2021/04/22 05:22:39.743 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 73”] [term=73] [raft_id=246876075] [region_id=246876074]

[2021/04/22 05:22:39.743 +00:00] [INFO] [raft.rs:902] [“246876075 received message from 246876075”] [term=73] [msg=MsgRequestPreVote] [from=246876075] [id=246876075] [raft_id=246876075] [region_id=246876074]

[2021/04/22 05:22:39.743 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=808] [raft_id=246929722] [region_id=297091]

[2021/04/22 05:22:39.743 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 808”] [term=808] [raft_id=246929722] [region_id=297091]

[2021/04/22 05:22:39.743 +00:00] [INFO] [raft.rs:923] [“[logterm: 73, index: 7022] sent request to 246888259”] [msg=MsgRequestPreVote] [term=73] [id=246888259] [log_index=7022] [log_term=73] [raft_id=246876075] [region_id=246876074]

[2021/04/22 05:22:39.743 +00:00] [INFO] [raft.rs:902] [“246929722 received message from 246929722”] [term=808] [msg=MsgRequestPreVote] [from=246929722] [id=246929722] [raft_id=246929722] [region_id=297091]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 73, index: 7022] sent request to 261974804”] [msg=MsgRequestPreVote] [term=73] [id=261974804] [log_index=7022] [log_term=73] [raft_id=246876075] [region_id=246876074]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 808, index: 79260] sent request to 249311040”] [msg=MsgRequestPreVote] [term=808] [id=249311040] [log_index=79260] [log_term=808] [raft_id=246929722] [region_id=297091]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 808, index: 79260] sent request to 262057468”] [msg=MsgRequestPreVote] [term=808] [id=262057468] [log_index=79260] [log_term=808] [raft_id=246929722] [region_id=297091]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=169] [raft_id=268441267] [region_id=139451173]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 169”] [term=169] [raft_id=268441267] [region_id=139451173]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:902] [“268441267 received message from 268441267”] [term=169] [msg=MsgRequestPreVote] [from=268441267] [id=268441267] [raft_id=268441267] [region_id=139451173]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 99, index: 126] sent request to 246769436”] [msg=MsgRequestPreVote] [term=169] [id=246769436] [log_index=126] [log_term=99] [raft_id=268441267] [region_id=139451173]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 99, index: 126] sent request to 249236925”] [msg=MsgRequestPreVote] [term=169] [id=249236925] [log_index=126] [log_term=99] [raft_id=268441267] [region_id=139451173]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=163] [raft_id=268443417] [region_id=80496219]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 163”] [term=163] [raft_id=268443417] [region_id=80496219]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:902] [“268443417 received message from 268443417”] [term=163] [msg=MsgRequestPreVote] [from=268443417] [id=268443417] [raft_id=268443417] [region_id=80496219]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 108, index: 114] sent request to 246706953”] [msg=MsgRequestPreVote] [term=163] [id=246706953] [log_index=114] [log_term=108] [raft_id=268443417] [region_id=80496219]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 108, index: 114] sent request to 169484927”] [msg=MsgRequestPreVote] [term=163] [id=169484927] [log_index=114] [log_term=108] [raft_id=268443417] [region_id=80496219]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=1235] [raft_id=268646457] [region_id=7654074]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 1235”] [term=1235] [raft_id=268646457] [region_id=7654074]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:902] [“268646457 received message from 268646457”] [term=1235] [msg=MsgRequestPreVote] [from=268646457] [id=268646457] [raft_id=268646457] [region_id=7654074]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 1178, index: 5356] sent request to 248014436”] [msg=MsgRequestPreVote] [term=1235] [id=248014436] [log_index=5356] [log_term=1178] [raft_id=268646457] [region_id=7654074]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 1178, index: 5356] sent request to 246309043”] [msg=MsgRequestPreVote] [term=1235] [id=246309043] [log_index=5356] [log_term=1178] [raft_id=268646457] [region_id=7654074]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=17179] [raft_id=248854577] [region_id=6860407]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 17179”] [term=17179] [raft_id=248854577] [region_id=6860407]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:902] [“248854577 received message from 248854577”] [term=17179] [msg=MsgRequestPreVote] [from=248854577] [id=248854577] [raft_id=248854577] [region_id=6860407]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 17179, index: 23863] sent request to 246913752”] [msg=MsgRequestPreVote] [term=17179] [id=246913752] [log_index=23863] [log_term=17179] [raft_id=248854577] [region_id=6860407]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 17179, index: 23863] sent request to 261968137”] [msg=MsgRequestPreVote] [term=17179] [id=261968137] [log_index=23863] [log_term=17179] [raft_id=248854577] [region_id=6860407]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=5446] [raft_id=268410264] [region_id=2236870]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 5446”] [term=5446] [raft_id=268410264] [region_id=2236870]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:902] [“268410264 received message from 268410264”] [term=5446] [msg=MsgRequestPreVote] [from=268410264] [id=268410264] [raft_id=268410264] [region_id=2236870]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 5373, index: 83085] sent request to 247246776”] [msg=MsgRequestPreVote] [term=5446] [id=247246776] [log_index=83085] [log_term=5373] [raft_id=268410264] [region_id=2236870]

[2021/04/22 05:22:39.744 +00:00] [INFO] [raft.rs:923] [“[logterm: 5373, index: 83085] sent request to 248860269”] [msg=MsgRequestPreVote] [term=5446] [id=248860269] [log_index=83085] [log_term=5373] [raft_id=268410264] [region_id=2236870]

[2021/04/22 05:22:39.745 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=5778] [raft_id=249168096] [region_id=10375880]

[2021/04/22 05:22:39.745 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 5778”] [term=5778] [raft_id=249168096] [region_id=10375880]

[2021/04/22 05:22:39.745 +00:00] [INFO] [raft.rs:902] [“249168096 received message from 249168096”] [term=5778] [msg=MsgRequestPreVote] [from=249168096] [id=249168096] [raft_id=249168096] [region_id=10375880]

[2021/04/22 05:22:39.745 +00:00] [INFO] [raft.rs:923] [“[logterm: 5778, index: 52301] sent request to 262088098”] [msg=MsgRequestPreVote] [term=5778] [id=262088098] [log_index=52301] [log_term=5778] [raft_id=249168096] [region_id=10375880]

[2021/04/22 05:22:39.745 +00:00] [INFO] [raft.rs:923] [“[logterm: 5778, index: 52301] sent request to 245749735”] [msg=MsgRequestPreVote] [term=5778] [id=245749735] [log_index=52301] [log_term=5778] [raft_id=249168096] [region_id=10375880]

[2021/04/22 05:22:39.746 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=203] [raft_id=268661488] [region_id=80331351]

[2021/04/22 05:22:39.746 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 203”] [term=203] [raft_id=268661488] [region_id=80331351]

[2021/04/22 05:22:39.746 +00:00] [INFO] [raft.rs:902] [“268661488 received message from 268661488”] [term=203] [msg=MsgRequestPreVote] [from=268661488] [id=268661488] [raft_id=268661488] [region_id=80331351]

[2021/04/22 05:22:39.746 +00:00] [INFO] [raft.rs:923] [“[logterm: 126, index: 122] sent request to 238172329”] [msg=MsgRequestPreVote] [term=203] [id=238172329] [log_index=122] [log_term=126] [raft_id=268661488] [region_id=80331351]

[2021/04/22 05:22:39.746 +00:00] [INFO] [raft.rs:923] [“[logterm: 126, index: 122] sent request to 249001565”] [msg=MsgRequestPreVote] [term=203] [id=249001565] [log_index=122] [log_term=126] [raft_id=268661488] [region_id=80331351]

[2021/04/22 05:22:39.746 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=7726] [raft_id=268436507] [region_id=6664698]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=127] [raft_id=246898036] [region_id=80600448]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 7726”] [term=7726] [raft_id=268436507] [region_id=6664698]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 127”] [term=127] [raft_id=246898036] [region_id=80600448]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:902] [“268436507 received message from 268436507”] [term=7726] [msg=MsgRequestPreVote] [from=268436507] [id=268436507] [raft_id=268436507] [region_id=6664698]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:902] [“246898036 received message from 246898036”] [term=127] [msg=MsgRequestPreVote] [from=246898036] [id=246898036] [raft_id=246898036] [region_id=80600448]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:923] [“[logterm: 127, index: 141] sent request to 261949633”] [msg=MsgRequestPreVote] [term=127] [id=261949633] [log_index=141] [log_term=127] [raft_id=246898036] [region_id=80600448]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:923] [“[logterm: 7657, index: 29602] sent request to 38845769”] [msg=MsgRequestPreVote] [term=7726] [id=38845769] [log_index=29602] [log_term=7657] [raft_id=268436507] [region_id=6664698]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:923] [“[logterm: 127, index: 141] sent request to 238144958”] [msg=MsgRequestPreVote] [term=127] [id=238144958] [log_index=141] [log_term=127] [raft_id=246898036] [region_id=80600448]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:923] [“[logterm: 7657, index: 29602] sent request to 38382902”] [msg=MsgRequestPreVote] [term=7726] [id=38382902] [log_index=29602] [log_term=7657] [raft_id=268436507] [region_id=6664698]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=391] [raft_id=246796660] [region_id=246736591]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 391”] [term=391] [raft_id=246796660] [region_id=246736591]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:902] [“246796660 received message from 246796660”] [term=391] [msg=MsgRequestPreVote] [from=246796660] [id=246796660] [raft_id=246796660] [region_id=246736591]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=78] [raft_id=248692919] [region_id=248692918]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 78”] [term=78] [raft_id=248692919] [region_id=248692918]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:923] [“[logterm: 391, index: 308200] sent request to 261801469”] [msg=MsgRequestPreVote] [term=391] [id=261801469] [log_index=308200] [log_term=391] [raft_id=246796660] [region_id=246736591]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:902] [“248692919 received message from 248692919”] [term=78] [msg=MsgRequestPreVote] [from=248692919] [id=248692919] [raft_id=248692919] [region_id=248692918]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:923] [“[logterm: 391, index: 308200] sent request to 246796686”] [msg=MsgRequestPreVote] [term=391] [id=246796686] [log_index=308200] [log_term=391] [raft_id=246796660] [region_id=246736591]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:923] [“[logterm: 78, index: 76] sent request to 248692921”] [msg=MsgRequestPreVote] [term=78] [id=248692921] [log_index=76] [log_term=78] [raft_id=248692919] [region_id=248692918]

[2021/04/22 05:22:39.747 +00:00] [INFO] [raft.rs:923] [“[logterm: 78, index: 76] sent request to 261797366”] [msg=MsgRequestPreVote] [term=78] [id=261797366] [log_index=76] [log_term=78] [raft_id=248692919] [region_id=248692918]

[2021/04/22 05:22:39.760 +00:00] [INFO] [raft.rs:971] [“[logterm: 1455, index: 11908, vote: 247380702] ignored vote from 268559970 [logterm: 1389, index: 11855]: lease is not expired”] [“msg type”=MsgRequestPreVote] [“remaining ticks”=10] [term=1455] [msg_index=11855] [msg_term=1389] [from=268559970] [vote=247380702] [log_index=11908] [log_term=1455] [raft_id=247380702] [region_id=182656]

[2021/04/22 05:22:39.768 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=109] [raft_id=247257569] [region_id=245267968]

[2021/04/22 05:22:39.768 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 109”] [term=109] [raft_id=247257569] [region_id=245267968]

[2021/04/22 05:22:39.768 +00:00] [INFO] [raft.rs:902] [“247257569 received message from 247257569”] [term=109] [msg=MsgRequestPreVote] [from=247257569] [id=247257569] [raft_id=247257569] [region_id=245267968]

[2021/04/22 05:22:39.768 +00:00] [INFO] [raft.rs:923] [“[logterm: 108, index: 97] sent request to 248809204”] [msg=MsgRequestPreVote] [term=109] [id=248809204] [log_index=97] [log_term=108] [raft_id=247257569] [region_id=245267968]

[2021/04/22 05:22:39.768 +00:00] [INFO] [raft.rs:923] [“[logterm: 108, index: 97] sent request to 261803058”] [msg=MsgRequestPreVote] [term=109] [id=261803058] [log_index=97] [log_term=108] [raft_id=247257569] [region_id=245267968]

[2021/04/22 05:22:39.788 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=129] [raft_id=245901754] [region_id=80539332]

[2021/04/22 05:22:39.788 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 129”] [term=129] [raft_id=245901754] [region_id=80539332]

[2021/04/22 05:22:39.788 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=124] [raft_id=238276600] [region_id=38932731]

[2021/04/22 05:22:39.788 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 124”] [term=124] [raft_id=238276600] [region_id=38932731]

[2021/04/22 05:22:39.788 +00:00] [INFO] [raft.rs:902] [“238276600 received message from 238276600”] [term=124] [msg=MsgRequestPreVote] [from=238276600] [id=238276600] [raft_id=238276600] [region_id=38932731]

[2021/04/22 05:22:39.788 +00:00] [INFO] [raft.rs:902] [“245901754 received message from 245901754”] [term=129] [msg=MsgRequestPreVote] [from=245901754] [id=245901754] [raft_id=245901754] [region_id=80539332]

[2021/04/22 05:22:39.788 +00:00] [INFO] [raft.rs:923] [“[logterm: 124, index: 148] sent request to 245852897”] [msg=MsgRequestPreVote] [term=124] [id=245852897] [log_index=148] [log_term=124] [raft_id=238276600] [region_id=38932731]

[2021/04/22 05:22:39.788 +00:00] [INFO] [raft.rs:923] [“[logterm: 129, index: 234274] sent request to 247327248”] [msg=MsgRequestPreVote] [term=129] [id=247327248] [log_index=234274] [log_term=129] [raft_id=245901754] [region_id=80539332]

[2021/04/22 05:22:39.788 +00:00] [INFO] [raft.rs:923] [“[logterm: 129, index: 234274] sent request to 261900200”] [msg=MsgRequestPreVote] [term=129] [id=261900200] [log_index=234274] [log_term=129] [raft_id=245901754] [region_id=80539332]

[2021/04/22 05:22:39.789 +00:00] [INFO] [raft.rs:923] [“[logterm: 124, index: 148] sent request to 261795669”] [msg=MsgRequestPreVote] [term=124] [id=261795669] [log_index=148] [log_term=124] [raft_id=238276600] [region_id=38932731]

[2021/04/22 05:22:39.789 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=2011] [raft_id=268662927] [region_id=11082]

[2021/04/22 05:22:39.789 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 2011”] [term=2011] [raft_id=268662927] [region_id=11082]

[2021/04/22 05:22:39.789 +00:00] [INFO] [raft.rs:902] [“268662927 received message from 268662927”] [term=2011] [msg=MsgRequestPreVote] [from=268662927] [id=268662927] [raft_id=268662927] [region_id=11082]

[2021/04/22 05:22:39.789 +00:00] [INFO] [raft.rs:923] [“[logterm: 1925, index: 224556] sent request to 247138764”] [msg=MsgRequestPreVote] [term=2011] [id=247138764] [log_index=224556] [log_term=1925] [raft_id=268662927] [region_id=11082]

[2021/04/22 05:22:39.789 +00:00] [INFO] [raft.rs:923] [“[logterm: 1925, index: 224556] sent request to 247116655”] [msg=MsgRequestPreVote] [term=2011] [id=247116655] [log_index=224556] [log_term=1925] [raft_id=268662927] [region_id=11082]

[2021/04/22 05:22:39.792 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=1779] [raft_id=268546553] [region_id=93950]

[2021/04/22 05:22:39.792 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 1779”] [term=1779] [raft_id=268546553] [region_id=93950]

[2021/04/22 05:22:39.792 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=99] [raft_id=80388980] [region_id=80388978]

[2021/04/22 05:22:39.792 +00:00] [INFO] [raft.rs:902] [“268546553 received message from 268546553”] [term=1779] [msg=MsgRequestPreVote] [from=268546553] [id=268546553] [raft_id=268546553] [region_id=93950]

[2021/04/22 05:22:39.792 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 99”] [term=99] [raft_id=80388980] [region_id=80388978]

[2021/04/22 05:22:39.792 +00:00] [INFO] [raft.rs:902] [“80388980 received message from 80388980”] [term=99] [msg=MsgRequestPreVote] [from=80388980] [id=80388980] [raft_id=80388980] [region_id=80388978]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:923] [“[logterm: 1709, index: 4969] sent request to 248806721”] [msg=MsgRequestPreVote] [term=1779] [id=248806721] [log_index=4969] [log_term=1709] [raft_id=268546553] [region_id=93950]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:923] [“[logterm: 99, index: 670009] sent request to 261985396”] [msg=MsgRequestPreVote] [term=99] [id=261985396] [log_index=670009] [log_term=99] [raft_id=80388980] [region_id=80388978]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:923] [“[logterm: 1709, index: 4969] sent request to 166228842”] [msg=MsgRequestPreVote] [term=1779] [id=166228842] [log_index=4969] [log_term=1709] [raft_id=268546553] [region_id=93950]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:923] [“[logterm: 99, index: 670009] sent request to 80388979”] [msg=MsgRequestPreVote] [term=99] [id=80388979] [log_index=670009] [log_term=99] [raft_id=80388980] [region_id=80388978]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=155] [raft_id=268654425] [region_id=249208477]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 155”] [term=155] [raft_id=268654425] [region_id=249208477]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:902] [“268654425 received message from 268654425”] [term=155] [msg=MsgRequestPreVote] [from=268654425] [id=268654425] [raft_id=268654425] [region_id=249208477]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:923] [“[logterm: 76, index: 2529] sent request to 249208478”] [msg=MsgRequestPreVote] [term=155] [id=249208478] [log_index=2529] [log_term=76] [raft_id=268654425] [region_id=249208477]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:923] [“[logterm: 76, index: 2529] sent request to 249208479”] [msg=MsgRequestPreVote] [term=155] [id=249208479] [log_index=2529] [log_term=76] [raft_id=268654425] [region_id=249208477]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:1177] [“starting a new election”] [term=159] [raft_id=268554844] [region_id=117492170]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:833] [“became pre-candidate at term 159”] [term=159] [raft_id=268554844] [region_id=117492170]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:902] [“268554844 received message from 268554844”] [term=159] [msg=MsgRequestPreVote] [from=268554844] [id=268554844] [raft_id=268554844] [region_id=117492170]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:923] [“[logterm: 98, index: 113] sent request to 117492173”] [msg=MsgRequestPreVote] [term=159] [id=117492173] [log_index=113] [log_term=98] [raft_id=268554844] [region_id=117492170]

[2021/04/22 05:22:39.793 +00:00] [INFO] [raft.rs:923] [“[logterm: 98, index: 113] sent request to 247314357”] [msg=MsgRequestPreVote] [term=159] [id=247314357] [log_index=113] [log_term=98] [raft_id=268554844] [region_id=117492170]

那你把240启动起来观察下情况

是启动后还处于down状态吗? 通过 pd-ctl 查询状态呢?

单独启动240节点后,240启动成功,但其余几个节点又down了

以下是147,236,239的log
链接: 百度网盘-链接不存在 密码: fcv2

没有做任何操作,经过一天之后kv节点全部up了


但是现在反馈有大量查询报region is unavaible,请问是否与有节点在下线有关

正常情况下,因为集群还有好多个tikv,是不会出现大量 region is unavaibles的,你在 granfan pd 面板看下集群中 region 状态怎么样呢。 当出现 region is unvarible 时候查询下相关 region 是不是没有选主

请问一下在最开始 147 和 236 节点 down 的时候,在此之前这个集群做过些什么操作?
只是强制下线了store 38546296 一个节点么?还有做过其他操作么?比如 unsafe-recover 之类的?
我们想确认一下 236 出现 FATAL 信息的原因。

region is unavaible 的问题可以先参考这个排查一下:https://docs.pingcap.com/zh/tidb/v4.0/tidb-troubleshooting-map#11-客户端报-region-is-unavailable-错误

region状况如图