为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:
【TiDB 版本】
v4.0.6
【问题描述】
集群当前状态如图
其中12kv节点正处于下线状态,通过监控发现region确实在减少,可以认为处于正常下线状态,
147与236kv节点处于down状态,147节点的日志如下:
[2021/04/22 01:18:12.279 +00:00] [INFO] [raft.rs:833] ["became pre-candidate at term 1980"] [term=1980] [raft_id=246235410] [region_id=203974]
[2021/04/22 01:18:12.279 +00:00] [INFO] [raft.rs:902] ["246235410 received message from 246235410"] [term=1980] [msg=MsgRequestPreVote] [from=246235410] [id=246235410] [raft_id=246235410] [region_id=203974]
[2021/04/22 01:18:12.279 +00:00] [INFO] [raft.rs:923] ["[logterm: 1980, index: 15993] sent request to 247271224"] [msg=MsgRequestPreVote] [term=1980] [id=247271224] [log_index=15993] [log_term=1980] [raft_id=246235410] [region_id=203974]
[2021/04/22 01:18:12.279 +00:00] [INFO] [raft.rs:923] ["[logterm: 1980, index: 15993] sent request to 262012289"] [msg=MsgRequestPreVote] [term=1980] [id=262012289] [log_index=15993] [log_term=1980] [raft_id=246235410] [region_id=203974]
[2021/04/22 01:18:12.279 +00:00] [WARN] [raft_client.rs:199] ["send to 10.12.5.236:20160 fail, the gRPC connection could be broken"]
[2021/04/22 01:18:12.279 +00:00] [ERROR] [transport.rs:163] ["send raft msg err"] [err="Other(\"[src/server/raft_client.rs:208]: RaftClient send fail\")"]
[2021/04/22 01:18:12.291 +00:00] [INFO] [raft.rs:1208] ["[logterm: 2050, index: 2114, vote: 194021256] rejected vote from 268649611 [logterm: 1994, index: 2064] at term 2051"] ["msg type"=MsgRequestPreVote] [term=2051] [msg_index=2064] [msg_term=1994] [from=268649611] [vote=194021256] [log_index=2114] [log_term=2050] [raft_id=194021256] [region_id=55432]
[2021/04/22 01:18:12.295 +00:00] [INFO] [raft.rs:1208] ["[logterm: 532, index: 569, vote: 38845165] rejected vote from 268662127 [logterm: 463, index: 514] at term 532"] ["msg type"=MsgRequestPreVote] [term=532] [msg_index=514] [msg_term=463] [from=268662127] [vote=38845165] [log_index=569] [log_term=532] [raft_id=38845165] [region_id=18693881]
[2021/04/22 01:18:12.295 +00:00] [INFO] [raft.rs:1208] ["[logterm: 184, index: 220, vote: 247606726] rejected vote from 268649073 [logterm: 117, index: 166] at term 184"] ["msg type"=MsgRequestPreVote] [term=184] [msg_index=166] [msg_term=117] [from=268649073] [vote=247606726] [log_index=220] [log_term=184] [raft_id=247606726] [region_id=39147607]
[2021/04/22 01:18:12.298 +00:00] [INFO] [raft.rs:1208] ["[logterm: 1873, index: 15650, vote: 194243136] rejected vote from 268571645 [logterm: 1803, index: 15589] at term 1873"] ["msg type"=MsgRequestPreVote] [term=1873] [msg_index=15589] [msg_term=1803] [from=268571645] [vote=194243136] [log_index=15650] [log_term=1873] [raft_id=194243136] [region_id=195989]
[2021/04/22 01:18:12.304 +00:00] [INFO] [raft.rs:1208] ["[logterm: 200, index: 187, vote: 238154115] rejected vote from 268642327 [logterm: 140, index: 136] at term 200"] ["msg type"=MsgRequestPreVote] [term=200] [msg_index=136] [msg_term=140] [from=268642327] [vote=238154115] [log_index=187] [log_term=200] [raft_id=238154115] [region_id=80610864]
[2021/04/22 01:18:12.308 +00:00] [INFO] [raft.rs:1208] ["[logterm: 160, index: 157, vote: 246648589] rejected vote from 268592857 [logterm: 109, index: 117] at term 160"] ["msg type"=MsgRequestPreVote] [term=160] [msg_index=117] [msg_term=109] [from=268592857] [vote=246648589] [log_index=157] [log_term=160] [raft_id=246648589] [region_id=80616381]
[2021/04/22 01:18:12.345 +00:00] [INFO] [raft.rs:1208] ["[logterm: 11683, index: 13464, vote: 248646876] rejected vote from 268662264 [logterm: 11628, index: 13417] at term 11683"] ["msg type"=MsgRequestPreVote] [term=11683] [msg_index=13417] [msg_term=11628] [from=268662264] [vote=248646876] [log_index=13464] [log_term=11683] [raft_id=248646876] [region_id=10360541]
[2021/04/22 01:18:12.346 +00:00] [INFO] [raft.rs:1208] ["[logterm: 129, index: 138, vote: 249042865] rejected vote from 268642267 [logterm: 85, index: 99] at term 130"] ["msg type"=MsgRequestPreVote] [term=130] [msg_index=99] [msg_term=85] [from=268642267] [vote=249042865] [log_index=138] [log_term=129] [raft_id=249042865] [region_id=245377826]
[2021/04/22 01:18:12.346 +00:00] [INFO] [raft.rs:1208] ["[logterm: 149, index: 139, vote: 249219359] rejected vote from 271575602 [logterm: 77, index: 75] at term 149"] ["msg type"=MsgRequestPreVote] [term=149] [msg_index=75] [msg_term=77] [from=271575602] [vote=249219359] [log_index=139] [log_term=149] [raft_id=249219359] [region_id=249219358]
[2021/04/22 01:18:12.357 +00:00] [INFO] [raft.rs:1208] ["[logterm: 6878, index: 47514, vote: 249002190] rejected vote from 268438021 [logterm: 6804, index: 47456] at term 6878"] ["msg type"=MsgRequestPreVote] [term=6878] [msg_index=47456] [msg_term=6804] [from=268438021] [vote=249002190] [log_index=47514] [log_term=6878] [raft_id=249002190] [region_id=192181]
[2021/04/22 01:18:12.357 +00:00] [INFO] [raft.rs:1208] ["[logterm: 1264, index: 73332, vote: 245752076] rejected vote from 268656630 [logterm: 1211, index: 73284] at term 1265"] ["msg type"=MsgRequestPreVote] [term=1265] [msg_index=73284] [msg_term=1211] [from=268656630] [vote=245752076] [log_index=73332] [log_term=1264] [raft_id=245752076] [region_id=9373956]
[2021/04/22 01:18:12.408 +00:00] [INFO] [raft.rs:1208] ["[logterm: 4277, index: 17937, vote: 245840072] rejected vote from 268425072 [logterm: 4227, index: 17892] at term 4277"] ["msg type"=MsgRequestPreVote] [term=4277] [msg_index=17892] [msg_term=4227] [from=268425072] [vote=245840072] [log_index=17937] [log_term=4277] [raft_id=245840072] [region_id=253613]
[2021/04/22 01:18:12.456 +00:00] [INFO] [raft.rs:1208] ["[logterm: 184, index: 686626, vote: 248641677] rejected vote from 269739990 [logterm: 129, index: 686580] at term 184"] ["msg type"=MsgRequestPreVote] [term=184] [msg_index=686580] [msg_term=129] [from=269739990] [vote=248641677] [log_index=686626] [log_term=184] [raft_id=248641677] [region_id=80384650]
[2021/04/22 01:18:12.472 +00:00] [INFO] [raft.rs:1208] ["[logterm: 420, index: 465, vote: 38874788] rejected vote from 268455871 [logterm: 375, index: 425] at term 420"] ["msg type"=MsgRequestPreVote] [term=420] [msg_index=425] [msg_term=375] [from=268455871] [vote=38874788] [log_index=465] [log_term=420] [raft_id=38874788] [region_id=18585924]
[2021/04/22 01:18:12.472 +00:00] [INFO] [raft.rs:1208] ["[logterm: 1275, index: 1307, vote: 246919934] rejected vote from 268548211 [logterm: 1214, index: 1256] at term 1275"] ["msg type"=MsgRequestPreVote] [term=1275] [msg_index=1256] [msg_term=1214] [from=268548211] [vote=246919934] [log_index=1307] [log_term=1275] [raft_id=246919934] [region_id=1315353]
[2021/04/22 01:18:12.472 +00:00] [INFO] [raft.rs:1208] ["[logterm: 868, index: 1021, vote: 235736985] rejected vote from 268453941 [logterm: 809, index: 971] at term 869"] ["msg type"=MsgRequestPreVote] [term=869] [msg_index=971] [msg_term=809] [from=268453941] [vote=235736985] [log_index=1021] [log_term=868] [raft_id=235736985] [region_id=12770333]
[2021/04/22 01:18:12.472 +00:00] [INFO] [raft.rs:1208] ["[logterm: 306, index: 84677, vote: 177475651] rejected vote from 268434332 [logterm: 244, index: 84619] at term 306"] ["msg type"=MsgRequestPreVote] [term=306] [msg_index=84619] [msg_term=244] [from=268434332] [vote=177475651] [log_index=84677] [log_term=306] [raft_id=177475651] [region_id=18659826]
[2021/04/22 01:18:12.473 +00:00] [INFO] [raft.rs:1208] ["[logterm: 18227, index: 30397, vote: 248044077] rejected vote from 276633621 [logterm: 18149, index: 30326] at term 18227"] ["msg type"=MsgRequestPreVote] [term=18227] [msg_index=30326] [msg_term=18149] [from=276633621] [vote=248044077] [log_index=30397] [log_term=18227] [raft_id=248044077] [region_id=233766]
可以看到是因为236节点处于down导致无法通信
236节点的日志如下:
[2021/04/22 01:31:50.446 +00:00] [INFO] [raft.rs:783] ["became follower at term 147"] [term=147] [raft_id=249236196] [region_id=249236193]
[2021/04/22 01:31:50.446 +00:00] [INFO] [raft.rs:1192] ["[logterm: 146, index: 1498, vote: 0] cast vote for 249236195 [logterm: 146, index: 1498] at term 147"] ["msg type"=MsgRequestVote] [term=147] [msg_index=1498] [msg_term=146] [from=249236195] [vote=0] [log_index=1498] [log_term=146] [raft_id=249236196] [region_id=249236193]
[2021/04/22 01:31:50.446 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 247382971"] ["msg type"=MsgRequestVote] [message_term=194] [term=191] [from=247382971] [raft_id=80375255] [region_id=80375254]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 194"] [term=194] [raft_id=80375255] [region_id=80375254]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 191, index: 187, vote: 0] cast vote for 247382971 [logterm: 191, index: 187] at term 194"] ["msg type"=MsgRequestVote] [term=194] [msg_index=187] [msg_term=191] [from=247382971] [vote=0] [log_index=187] [log_term=191] [raft_id=80375255] [region_id=80375254]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 248597078"] ["msg type"=MsgRequestVote] [message_term=192] [term=191] [from=248597078] [raft_id=80278155] [region_id=80278154]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 192"] [term=192] [raft_id=80278155] [region_id=80278154]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 191, index: 196, vote: 0] cast vote for 248597078 [logterm: 191, index: 196] at term 192"] ["msg type"=MsgRequestVote] [term=192] [msg_index=196] [msg_term=191] [from=248597078] [vote=0] [log_index=196] [log_term=191] [raft_id=80278155] [region_id=80278154]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 248529899"] ["msg type"=MsgRequestVote] [message_term=734] [term=733] [from=248529899] [raft_id=248582566] [region_id=9359094]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 734"] [term=734] [raft_id=248582566] [region_id=9359094]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 733, index: 3075, vote: 0] cast vote for 248529899 [logterm: 733, index: 3075] at term 734"] ["msg type"=MsgRequestVote] [term=734] [msg_index=3075] [msg_term=733] [from=248529899] [vote=0] [log_index=3075] [log_term=733] [raft_id=248582566] [region_id=9359094]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 246235410"] ["msg type"=MsgRequestVote] [message_term=1982] [term=1981] [from=246235410] [raft_id=247271224] [region_id=203974]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 1982"] [term=1982] [raft_id=247271224] [region_id=203974]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 1981, index: 15994, vote: 0] cast vote for 246235410 [logterm: 1981, index: 15994] at term 1982"] ["msg type"=MsgRequestVote] [term=1982] [msg_index=15994] [msg_term=1981] [from=246235410] [vote=0] [log_index=15994] [log_term=1981] [raft_id=247271224] [region_id=203974]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 244839270"] ["msg type"=MsgRequestVote] [message_term=245] [term=243] [from=244839270] [raft_id=194127479] [region_id=20159492]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 245"] [term=245] [raft_id=194127479] [region_id=20159492]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 241, index: 777, vote: 0] cast vote for 244839270 [logterm: 241, index: 777] at term 245"] ["msg type"=MsgRequestVote] [term=245] [msg_index=777] [msg_term=241] [from=244839270] [vote=0] [log_index=777] [log_term=241] [raft_id=194127479] [region_id=20159492]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 246704011"] ["msg type"=MsgRequestVote] [message_term=174] [term=172] [from=246704011] [raft_id=139423876] [region_id=139423873]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 174"] [term=174] [raft_id=139423876] [region_id=139423873]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 172, index: 10871, vote: 0] cast vote for 246704011 [logterm: 172, index: 10871] at term 174"] ["msg type"=MsgRequestVote] [term=174] [msg_index=10871] [msg_term=172] [from=246704011] [vote=0] [log_index=10871] [log_term=172] [raft_id=139423876] [region_id=139423873]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 249359279"] ["msg type"=MsgRequestVote] [message_term=1227] [term=1226] [from=249359279] [raft_id=247165494] [region_id=2489002]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 1227"] [term=1227] [raft_id=247165494] [region_id=2489002]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 1226, index: 1266, vote: 0] cast vote for 249359279 [logterm: 1226, index: 1266] at term 1227"] ["msg type"=MsgRequestVote] [term=1227] [msg_index=1266] [msg_term=1226] [from=249359279] [vote=0] [log_index=1266] [log_term=1226] [raft_id=247165494] [region_id=2489002]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 80949886"] ["msg type"=MsgRequestVote] [message_term=6738] [term=6736] [from=80949886] [raft_id=24538947] [region_id=6567835]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 6738"] [term=6738] [raft_id=24538947] [region_id=6567835]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 6736, index: 14967, vote: 0] cast vote for 80949886 [logterm: 6736, index: 14967] at term 6738"] ["msg type"=MsgRequestVote] [term=6738] [msg_index=14967] [msg_term=6736] [from=80949886] [vote=0] [log_index=14967] [log_term=6736] [raft_id=24538947] [region_id=6567835]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 246187713"] ["msg type"=MsgRequestVote] [message_term=750] [term=749] [from=246187713] [raft_id=38943850] [region_id=12804477]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 750"] [term=750] [raft_id=38943850] [region_id=12804477]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 749, index: 768, vote: 0] cast vote for 246187713 [logterm: 749, index: 768] at term 750"] ["msg type"=MsgRequestVote] [term=750] [msg_index=768] [msg_term=749] [from=246187713] [vote=0] [log_index=768] [log_term=749] [raft_id=38943850] [region_id=12804477]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 248969324"] ["msg type"=MsgRequestVote] [message_term=146] [term=145] [from=248969324] [raft_id=248460060] [region_id=248460059]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 146"] [term=146] [raft_id=248460060] [region_id=248460059]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 145, index: 176, vote: 0] cast vote for 248969324 [logterm: 145, index: 176] at term 146"] ["msg type"=MsgRequestVote] [term=146] [msg_index=176] [msg_term=145] [from=248969324] [vote=0] [log_index=176] [log_term=145] [raft_id=248460060] [region_id=248460059]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 247714528"] ["msg type"=MsgRequestVote] [message_term=123] [term=122] [from=247714528] [raft_id=247995112] [region_id=247714527]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 123"] [term=123] [raft_id=247995112] [region_id=247714527]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 122, index: 362, vote: 0] cast vote for 247714528 [logterm: 122, index: 362] at term 123"] ["msg type"=MsgRequestVote] [term=123] [msg_index=362] [msg_term=122] [from=247714528] [vote=0] [log_index=362] [log_term=122] [raft_id=247995112] [region_id=247714527]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 249356550"] ["msg type"=MsgRequestVote] [message_term=178] [term=177] [from=249356550] [raft_id=248695936] [region_id=113248925]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 178"] [term=178] [raft_id=248695936] [region_id=113248925]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 176, index: 235, vote: 0] cast vote for 249356550 [logterm: 177, index: 236] at term 178"] ["msg type"=MsgRequestVote] [term=178] [msg_index=236] [msg_term=177] [from=249356550] [vote=0] [log_index=235] [log_term=176] [raft_id=248695936] [region_id=113248925]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1003] ["received a message with higher term from 246811142"] ["msg type"=MsgRequestVote] [message_term=148] [term=147] [from=246811142] [raft_id=248903419] [region_id=246811141]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:783] ["became follower at term 148"] [term=148] [raft_id=248903419] [region_id=246811141]
[2021/04/22 01:31:50.447 +00:00] [INFO] [raft.rs:1192] ["[logterm: 147, index: 6810, vote: 0] cast vote for 246811142 [logterm: 147, index: 6810] at term 148"] ["msg type"=MsgRequestVote] [term=148] [msg_index=6810] [msg_term=147] [from=246811142] [vote=0] [log_index=6810] [log_term=147] [raft_id=248903419] [region_id=246811141]
[2021/04/22 01:31:50.453 +00:00] [FATAL] [lib.rs:483] ["[region 249205597] 249205599 source_region id: 249205587 start_key: 74800000000002B1FF585F728000000000FF7BC0A10000000000FA end_key: 74800000000002B1FF585F728000000000FF7D50470000000000FA region_epoch { conf_ver: 91443 version: 66961 } peers { id: 249205588 store_id: 24590972 } peers { id: 249205589 store_id: 24478148 } peers { id: 276651401 store_id: 24480822 } not match exist region id: 249205587 start_key: 74800000000002B1FF585F728000000000FF7BC0A10000000000FA end_key: 74800000000002B1FF585F728000000000FF7D50470000000000FA region_epoch { conf_ver: 91443 version: 66961 } peers { id: 249205588 store_id: 24590972 } peers { id: 249205589 store_id: 24478148 } peers { id: 262355238 store_id: 38546296 } peers { id: 276651401 store_id: 24480822 }"] [backtrace="stack backtrace:\
0: tikv_util::set_panic_hook::{{closure}}\
at components/tikv_util/src/lib.rs:482\
1: std::panicking::rust_panic_with_hook\
at src/libstd/panicking.rs:475\
2: rust_begin_unwind\
at src/libstd/panicking.rs:375\
3: std::panicking::begin_panic_fmt\
at src/libstd/panicking.rs:326\
4: raftstore::store::fsm::apply::ApplyDelegate::exec_commit_merge\
at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/<::std::macros::panic macros>:9\
raftstore::store::fsm::apply::ApplyDelegate::exec_admin_cmd\
at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:1182\
5: raftstore::store::fsm::apply::ApplyDelegate::exec_raft_cmd\
at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:1148\
raftstore::store::fsm::apply::ApplyDelegate::apply_raft_cmd\
at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:1040\
raftstore::store::fsm::apply::ApplyDelegate::process_raft_cmd\
at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:991\
6: raftstore::store::fsm::apply::ApplyDelegate::handle_raft_entry_normal\
at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:871\
raftstore::store::fsm::apply::ApplyDelegate::handle_raft_committed_entries\
at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:789\
7: raftstore::store::fsm::apply::ApplyFsm::resume_pending\
at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:2709\
<raftstore::store::fsm::apply::ApplyPoller<W> as batch_system::batch::PollHandler<raftstore::store::fsm::apply::ApplyFsm,raftstore::store::fsm::apply::ControlFsm>>::handle_normal\
at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/raftstore/src/store/fsm/apply.rs:3027\
8: batch_system::batch::Poller<N,C,Handler>::poll\
at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/batch-system/src/batch.rs:294\
batch_system::batch::BatchSystem<N,C>::spawn::{{closure}}\
at /home/jenkins/agent/workspace/ld_tikv_multi_branch_release-4.0/tikv/components/batch-system/src/batch.rs:398\
std::sys_common::backtrace::__rust_begin_short_backtrace\
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/sys_common/backtrace.rs:136\
9: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:469\
<std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:318\
std::panicking::try::do_call\
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panicking.rs:292\
std::panicking::try\
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8//src/libpanic_unwind/lib.rs:78\
std::panic::catch_unwind\
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:394\
std::thread::Builder::spawn_unchecked::{{closure}}\
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:468\
core::ops::function::FnOnce::call_once{{vtable.shim}}\
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/ops/function.rs:232\
10: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
11: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
std::sys_common::thread::start_thread\
at src/libstd/sys_common/thread.rs:13\
std::sys::unix::thread::Thread::new::thread_start\
at src/libstd/sys/unix/thread.rs:80\
12: start_thread\
13: __clone\
"] [location=components/raftstore/src/store/fsm/apply.rs:1954] [thread_name=apply-0]