tikv cpu异常，raft选举失败

Inkjade · 2022 年9 月 20 日 11:02

【 TiDB 使用环境】生产环境
【 TiDB 版本】5.4.0
【遇到的问题】
cpu使用过高,无法自动恢复，日志错误
[peer.rs:553] [“handle raft message err”] [err_code=KV:Raft:StepLocalMsg] [err=“Raft raft: cannot step raft local message”] [peer_id=1456675] [region_id=1456674]

【问题现象及影响】

【附件】

xfworld · 2022 年9 月 21 日 00:23

贴图中的日志貌似毫无关联

全是锁冲突
raftStore 不正常

问题1：哪些节点不正常？
问题2：出现这些问题之前进行了什么操作？
问题3：tidb 节点，PD 节点是否还正常？
问题4：这些问题出现前后有没有关键日志可以参考？

Inkjade · 2022 年9 月 21 日 02:08

问题1：哪些节点不正常？
所有的TIkv节点
问题2：出现这些问题之前进行了什么操作？
唯有特殊异常操作
问题3：tidb 节点，PD 节点是否还正常？
PD节点正常，因为无法解决该问题，重启所有tikv节点故障小时
问题4：这些问题出现前后有没有关键日志可以参考
分析了所有tikv节点日志，发现该问题，产出在19号 ,日志如下：
[2022/09/19 21:29:00.746 +08:00] [INFO] [apply.rs:1895] [“add learner successfully”] [region=“id: 1386748 start_key: 7480000000000E94FF085F72038ACE4E5DFF380000010150524FFF4649545F48FF414EFF444645455F59FF48FF4B000000000000F9FF038000000001348BFFF700000000000000F8 region_epoch { conf_ver: 17207 version: 3917 } peers { id: 1404274 store_id: 5 } peers { id: 1412642 store_id: 1 } peers { id: 1417136 store_id: 6 }”] [peer=“id: 1418341 store_id: 4 role: Learner”] [peer_id=1404274] [region_id=1386748]
[2022/09/19 21:29:00.748 +08:00] [INFO] [raft.rs:2609] [“switched to configuration”] [config=“Configuration { voters: Configuration { incoming: Configuration { voters: {1417136, 1412642, 1404274} }, outgoing: Configuration { voters: {} } }, learners: {1418341}, learners_next: {}, auto_leave: false }”] [raft_id=1404274] [region_id=1386748]
[2022/09/19 21:29:00.844 +08:00] [INFO] [size.rs:168] [“approximate size over threshold, need to do split check”] [threshold=150994944] [size=155228686] [region_id=1412818]
[2022/09/19 21:29:01.026 +08:00] [INFO] [split_check.rs:289] [“update approximate size and keys with accurate value”] [keys=583983] [size=125929726] [region_id=1412818]
[2022/09/19 21:29:01.101 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000011155F698000000000000001013230323230393132FF323132345F593246FF7A61474A68593273FF7459574E3061585AFF6C4C584E6C636E5AFF70593255744D5441FF352E315F4E574D30FF5A6A6B354F574A68FF4E5441794E445135FF59324A6859546B31FF4E7A6C6C4D574579FF4D44493059324A41FF4D5463794C6A4977FF4C6A45774F433478FF4D446B3D00000000FB lock_version: 436101222375882805 key: 7480000000000011155F698000000000000003038000002F149A574C038000000000209EF2 lock_ttl: 3337 txn_size: 348 lock_type: Del min_commit_ts: 436101222375882806”]
[2022/09/19 21:29:01.435 +08:00] [INFO] [size.rs:168] [“approximate size over threshold, need to do split check”] [threshold=150994944] [size=186687892] [region_id=1375671]
[2022/09/19 21:29:01.437 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000011135F698000000000000001013230323230393132FF323132345F593246FF7A61474A68593273FF7459574E3061585AFF6C4C584E6C636E5AFF70593255744D5441FF352E315F4E574D30FF5A6A6B354F574A68FF4E5441794E445135FF59324A6859546B31FF4E7A6C6C4D574579FF4D44493059324A41FF4D5463794C6A4977FF4C6A45774F433478FF4D446B3D00000000FB lock_version: 436101222559383593 key: 7480000000000011135F698000000000000003038000002F149A574C038000000000209EF2 lock_ttl: 3093 txn_size: 348 lock_type: Del min_commit_ts: 436101222559383594”]
[2022/09/19 21:29:01.481 +08:00] [INFO] [raft.rs:1336] [“received a message with higher term from 1418340”] [“msg type”=MsgRequestVote] [message_term=1074] [term=1073] [from=1418340] [raft_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.481 +08:00] [INFO] [raft.rs:1092] [“became follower at term 1074”] [term=1074] [raft_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.482 +08:00] [INFO] [raft.rs:1532] ["[logterm: 1073, index: 990874, vote: 0] cast vote for 1418340 [logterm: 1073, index: 990874] at term 1074"] [“msg type”=MsgRequestVote] [term=1074] [msg_index=990874] [msg_term=1073] [from=1418340] [vote=0] [log_index=990874] [log_term=1073] [raft_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.482 +08:00] [INFO] [apply.rs:1383] [“execute admin command”] [command=“cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 1418340 store_id: 4 } } changes { change_type: AddLearnerNode peer { id: 1418271 store_id: 1 role: Learner } } }”] [index=990874] [term=1073] [peer_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.482 +08:00] [INFO] [apply.rs:1936] [“exec ConfChangeV2”] [epoch=“conf_ver: 17526 version: 3466”] [kind=EnterJoint] [peer_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.482 +08:00] [INFO] [apply.rs:2116] [“conf change successfully”] [“current region”=“id: 822789 start_key: 748000000000020DFF0F5F728AC93DDCA1FF4000010000000000FA end_key: 74800000000005A0FFE75F720132303232FF30383331FF313536FF3439303237FF3635FF303939343137FF36FF30310000000000FAFF038000000001348BFF9F00000000000000F8 region_epoch { conf_ver: 17528 version: 3466 } peers { id: 1411028 store_id: 7 } peers { id: 1414389 store_id: 5 } peers { id: 1418271 store_id: 1 role: DemotingVoter } peers { id: 1418340 store_id: 4 role: IncomingVoter }”] [“original region”=“id: 822789 start_key: 748000000000020DFF0F5F728AC93DDCA1FF4000010000000000FA end_key: 74800000000005A0FFE75F720132303232FF30383331FF313536FF3439303237FF3635FF303939343137FF36FF30310000000000FAFF038000000001348BFF9F00000000000000F8 region_epoch { conf_ver: 17526 version: 3466 } peers { id: 1411028 store_id: 7 } peers { id: 1414389 store_id: 5 } peers { id: 1418271 store_id: 1 } peers { id: 1418340 store_id: 4 role: Learner }”] [changes="[peer { id: 1418340 store_id: 4 }, change_type: AddLearnerNode peer { id: 1418271 store_id: 1 role: Learner }]"] [peer_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.488 +08:00] [INFO] [raft.rs:2609] [“switched to configuration”] [config=“Configuration { voters: Configuration { incoming: Configuration { voters: {1414389, 1411028, 1418340} }, outgoing: Configuration { voters: {1411028, 1414389, 1418271} } }, learners: {}, learners_next: {1418271}, auto_leave: false }”] [raft_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.507 +08:00] [INFO] [apply.rs:1383] [“execute admin command”] [command=“cmd_type: ChangePeerV2 change_peer_v2 {}”] [index=990876] [term=1074] [peer_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.507 +08:00] [INFO] [apply.rs:1936] [“exec ConfChangeV2”] [epoch=“conf_ver: 17528 version: 3466”] [kind=LeaveJoint] [peer_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.507 +08:00] [INFO] [apply.rs:2146] [“leave joint state successfully”] [region=“id: 822789 start_key: 748000000000020DFF0F5F728AC93DDCA1FF4000010000000000FA end_key: 74800000000005A0FFE75F720132303232FF30383331FF313536FF3439303237FF3635FF303939343137FF36FF30310000000000FAFF038000000001348BFF9F00000000000000F8 region_epoch { conf_ver: 17530 version: 3466 } peers { id: 1411028 store_id: 7 } peers { id: 1414389 store_id: 5 } peers { id: 1418271 store_id: 1 role: Learner } peers { id: 1418340 store_id: 4 }”] [peer_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.507 +08:00] [INFO] [apply.rs:1383] [“execute admin command”] [command=“cmd_type: ChangePeer change_peer { change_type: RemoveNode peer { id: 1418271 store_id: 1 role: Learner } }”] [index=990877] [term=1074] [peer_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.507 +08:00] [INFO] [apply.rs:1755] [“exec ConfChange”] [epoch=“conf_ver: 17530 version: 3466”] [type=RemoveNode] [peer_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.507 +08:00] [INFO] [apply.rs:1863] [“remove peer successfully”] [region=“id: 822789 start_key: 748000000000020DFF0F5F728AC93DDCA1FF4000010000000000FA end_key: 74800000000005A0FFE75F720132303232FF30383331FF313536FF3439303237FF3635FF303939343137FF36FF30310000000000FAFF038000000001348BFF9F00000000000000F8 region_epoch { conf_ver: 17530 version: 3466 } peers { id: 1411028 store_id: 7 } peers { id: 1414389 store_id: 5 } peers { id: 1418271 store_id: 1 role: Learner } peers { id: 1418340 store_id: 4 }”] [peer=“id: 1418271 store_id: 1 role: Learner”] [peer_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.508 +08:00] [INFO] [raft.rs:2609] [“switched to configuration”] [config=“Configuration { voters: Configuration { incoming: Configuration { voters: {1414389, 1411028, 1418340} }, outgoing: Configuration { voters: {} } }, learners: {1418271}, learners_next: {}, auto_leave: false }”] [raft_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.509 +08:00] [INFO] [raft.rs:2609] [“switched to configuration”] [config=“Configuration { voters: Configuration { incoming: Configuration { voters: {1414389, 1411028, 1418340} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }”] [raft_id=1414389] [region_id=822789]
[2022/09/19 21:29:01.629 +08:00] [ERROR] [peer.rs:553] [“handle raft message err”] [err_code=KV:Raft:StepLocalMsg] [err=“Raft raft: cannot step raft local message”] [peer_id=1417858] [region_id=450799]
[2022/09/19 21:29:01.629 +08:00] [ERROR] [peer.rs:553] [“handle raft message err”] [err_code=KV:Raft:StepLocalMsg] [err=“Raft raft: cannot step raft local message”] [peer_id=1378100] [region_id=1378099]
[2022/09/19 21:29:01.629 +08:00] [ERROR] [peer.rs:553] [“handle raft message err”] [err_code=KV:Raft:StepLocalMsg] [err=“Raft raft: cannot step raft local message”] [peer_id=443696] [region_id=3632]
[2022/09/19 21:29:01.629 +08:00] [ERROR] [peer.rs:553] [“handle raft message err”] [err_code=KV:Raft:StepLocalMsg] [err=“Raft raft: cannot step raft local message”] [peer_id=443696] [region_id=3632]
[2022/09/19 21:29:01.630 +08:00] [ERROR] [peer.rs:553] [“handle raft message err”] [err_code=KV:Raft:StepLocalMsg] [err=“Raft raft: cannot step raft local message”] [peer_id=1223182] [region_id=1193821]
[2022/09/19 21:29:01.630 +08:00] [ERROR] [peer.rs:553] [“handle raft message err”] [err_code=KV:Raft:StepLocalMsg] [err=“Raft raft: cannot step raft local message”] [peer_id=1417858] [region_id=450799]
[2022/09/19 21:29:01.632 +08:00] [INFO] [split_check.rs:289] [“update approximate size and keys with accurate value”] [keys=481443] [size=114426511] [region_id=1375671]
[2022/09/19 21:29:01.797 +08:00] [INFO] [apply.rs:1383] [“execute admin command”] [command=“cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 1418341 store_id: 4 } } changes { change_type: AddLearnerNode peer { id: 1412642 store_id: 1 role: Learner } } }”] [index=43347] [term=53] [peer_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.797 +08:00] [INFO] [apply.rs:1936] [“exec ConfChangeV2”] [epoch=“conf_ver: 17208 version: 3917”] [kind=EnterJoint] [peer_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.797 +08:00] [INFO] [apply.rs:2116] [“conf change successfully”] [“current region”=“id: 1386748 start_key: 7480000000000E94FF085F72038ACE4E5DFF380000010150524FFF4649545F48FF414EFF444645455F59FF48FF4B000000000000F9FF038000000001348BFFF700000000000000F8 region_epoch { conf_ver: 17210 version: 3917 } peers { id: 1404274 store_id: 5 } peers { id: 1412642 store_id: 1 role: DemotingVoter } peers { id: 1417136 store_id: 6 } peers { id: 1418341 store_id: 4 role: IncomingVoter }”] [“original region”=“id: 1386748 start_key: 7480000000000E94FF085F72038ACE4E5DFF380000010150524FFF4649545F48FF414EFF444645455F59FF48FF4B000000000000F9FF038000000001348BFFF700000000000000F8 region_epoch { conf_ver: 17208 version: 3917 } peers { id: 1404274 store_id: 5 } peers { id: 1412642 store_id: 1 } peers { id: 1417136 store_id: 6 } peers { id: 1418341 store_id: 4 role: Learner }”] [changes="[peer { id: 1418341 store_id: 4 }, change_type: AddLearnerNode peer { id: 1412642 store_id: 1 role: Learner }]"] [peer_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.798 +08:00] [INFO] [raft.rs:2609] [“switched to configuration”] [config=“Configuration { voters: Configuration { incoming: Configuration { voters: {1417136, 1418341, 1404274} }, outgoing: Configuration { voters: {1417136, 1412642, 1404274} } }, learners: {}, learners_next: {1412642}, auto_leave: false }”] [raft_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.827 +08:00] [INFO] [raft.rs:1336] [“received a message with higher term from 1418341”] [“msg type”=MsgRequestVote] [message_term=54] [term=53] [from=1418341] [raft_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.827 +08:00] [INFO] [raft.rs:1092] [“became follower at term 54”] [term=54] [raft_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.827 +08:00] [INFO] [raft.rs:1532] ["[logterm: 53, index: 43347, vote: 0] cast vote for 1418341 [logterm: 53, index: 43347] at term 54"] [“msg type”=MsgRequestVote] [term=54] [msg_index=43347] [msg_term=53] [from=1418341] [vote=0] [log_index=43347] [log_term=53] [raft_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.855 +08:00] [INFO] [apply.rs:1383] [“execute admin command”] [command=“cmd_type: ChangePeerV2 change_peer_v2 {}”] [index=43349] [term=54] [peer_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.855 +08:00] [INFO] [apply.rs:1936] [“exec ConfChangeV2”] [epoch=“conf_ver: 17210 version: 3917”] [kind=LeaveJoint] [peer_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.855 +08:00] [INFO] [apply.rs:2146] [“leave joint state successfully”] [region=“id: 1386748 start_key: 7480000000000E94FF085F72038ACE4E5DFF380000010150524FFF4649545F48FF414EFF444645455F59FF48FF4B000000000000F9FF038000000001348BFFF700000000000000F8 region_epoch { conf_ver: 17212 version: 3917 } peers { id: 1404274 store_id: 5 } peers { id: 1412642 store_id: 1 role: Learner } peers { id: 1417136 store_id: 6 } peers { id: 1418341 store_id: 4 }”] [peer_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.856 +08:00] [INFO] [raft.rs:2609] [“switched to configuration”] [config=“Configuration { voters: Configuration { incoming: Configuration { voters: {1417136, 1418341, 1404274} }, outgoing: Configuration { voters: {} } }, learners: {1412642}, learners_next: {}, auto_leave: false }”] [raft_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.890 +08:00] [INFO] [apply.rs:1383] [“execute admin command”] [command=“cmd_type: ChangePeer change_peer { change_type: RemoveNode peer { id: 1412642 store_id: 1 role: Learner } }”] [index=43350] [term=54] [peer_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.890 +08:00] [INFO] [apply.rs:1755] [“exec ConfChange”] [epoch=“conf_ver: 17212 version: 3917”] [type=RemoveNode] [peer_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.890 +08:00] [INFO] [apply.rs:1863] [“remove peer successfully”] [region=“id: 1386748 start_key: 7480000000000E94FF085F72038ACE4E5DFF380000010150524FFF4649545F48FF414EFF444645455F59FF48FF4B000000000000F9FF038000000001348BFFF700000000000000F8 region_epoch { conf_ver: 17212 version: 3917 } peers { id: 1404274 store_id: 5 } peers { id: 1412642 store_id: 1 role: Learner } peers { id: 1417136 store_id: 6 } peers { id: 1418341 store_id: 4 }”] [peer=“id: 1412642 store_id: 1 role: Learner”] [peer_id=1404274] [region_id=1386748]
[2022/09/19 21:29:01.891 +08:00] [INFO] [raft.rs:2609] [“switched to configuration”] [config=“Configuration { voters: Configuration { incoming: Configuration { voters: {1417136, 1418341, 1404274} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }”] [raft_id=1404274] [region_id=1386748]
[2022/09/19 21:29:02.087 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Key is locked (will clean up) primary_lock: 74800000000000112D5F698000000000000001013230323230393132FF323132345F593356FF7A6443317A5A584AFF3261574E6C4C5463FF342E315F4E7A6B79FF597A6B774F445134FF4F445A6A4E475534FF5A574935596A5930FF596D45794E545A6DFF4E54637A4D6A6441FF4D5463794C6A4977FF4C6A45774F433433FF4F413D3D00000000FB lock_version: 436101222742884398 key: 74800000000000112D5F698000000000000003038000002F149A574E038000000000072F71 lock_ttl: 3061 txn_size: 249 lock_type: Del min_commit_ts: 436101222742884399”]
[2022/09/19 21:29:02.311 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Key is locked (will clean up) primary_lock: 74800000000000112B5F698000000000000001013230323230393132FF323132345F593356FF7A6443317A5A584AFF3261574E6C4C5463FF342E315F4E7A6B79FF597A6B774F445134FF4F445A6A4E475534FF5A574935596A5930FF596D45794E545A6DFF4E54637A4D6A6441FF4D5463794C6A4977FF4C6A45774F433433FF4F413D3D00000000FB lock_version: 436101222795313207 key: 74800000000000112B5F698000000000000003038000002F149A574C038000000000072F4D lock_ttl: 3059 txn_size: 166 lock_type: Del min_commit_ts: 436101222795313208”]
[2022/09/19 21:29:06.931 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Key is locked (will clean up) primary_lock: 74800000000000110B5F698000000000000001013230323230393132FF323132345F593246FF7A61474A68593273FF7459574E3061585AFF6C4C584E6C636E5AFF70593255744D5441FF352E315F4E574D30FF5A6A6B354F574A68FF4E5441794E445135FF59324A6859546B31FF4E7A6C6C4D574579FF4D44493059324A41FF4D5463794C6A4977FF4C6A45774F433478FF4D446B3D00000000FB lock_version: 436101223896318016 key: 74800000000000110B5F698000000000000003038000002F149A574C038000000000209EF2 lock_ttl: 3296 txn_size: 348 lock_type: Del min_commit_ts: 436101223896318017”]
[2022/09/19 21:29:15.681 +08:00] [INFO] [compact.rs:113] [“compact range finished”] [time_takes=16.646063842s] [cf=default] [range_end=7A7480000000000011FF385F698000000000FF00001E0380001264FF0C77C35D03800000FF00130C9E24000000FC] [range_start=7A7480000000000011FF385F698000000000FF00001E0380001264FF0C68471703800000FF00122234AC000000FC]
[2022/09/19 21:29:19.444 +08:00] [WARN] [kv.rs:1092] [“call CheckLeader failed”] [err=Grpc(RemoteStopped)]
[2022/09/19 21:29:20.624 +08:00] [WARN] [kv.rs:1092] [“call CheckLeader failed”] [err=Grpc(RemoteStopped)]
[2022/09/19 21:29:24.784 +08:00] [INFO] [scheduler.rs:517] [“get snapshot failed”] [err=“Error(Request(message: “EpochNotMatch current epoch of region 469238 is conf_ver: 13583 version: 2193, but you sent conf_ver: 13583 version: 2192” epoch_not_match { current_regions { id: 469238 start_key: 7480000000000010FFB45F698000000000FF0000030380000000FF0000271003800000FF0000CD90C4000000FC end_key: 7480000000000010FFB45F698000000000FF0000040380000000FF0000000103800000FF0000242385000000FC region_epoch { conf_ver: 13583 version: 2193 } peers { id: 469239 store_id: 5 } peers { id: 469313 store_id: 1 } peers { id: 1385398 store_id: 6 } } current_regions { id: 1418277 start_key: 7480000000000010FFB45F698000000000FF0000030380000000FF0000271003800000FF0000BEC092000000FC end_key: 7480000000000010FFB45F698000000000FF0000030380000000FF0000271003800000FF0000CD90C4000000FC region_epoch { conf_ver: 13583 version: 2193 } peers { id: 1418278 store_id: 5 } peers { id: 1418279 store_id: 1 } peers { id: 1418280 store_id: 6 } } }))”] [cid=4158108172]
[2022/09/19 21:29:24.784 +08:00] [INFO] [scheduler.rs:517] [“get snapshot failed”] [err=“Error(Request(message: “EpochNotMatch current epoch of region 469238 is conf_ver: 13583 version: 2193, but you sent conf_ver: 13583 version: 2192” epoch_not_match { current_regions { id: 469238 start_key: 7480000000000010FFB45F698000000000FF0000030380000000FF0000271003800000FF0000CD90C4000000FC end_key: 7480000000000010FFB45F698000000000FF0000040380000000FF0000000103800000FF0000242385000000FC region_epoch { conf_ver: 13583 version: 2193 } peers { id: 469239 store_id: 5 } peers { id: 469313 store_id: 1 } peers { id: 1385398 store_id: 6 } } current_regions { id: 1418277 start_key: 7480000000000010FFB45F698000000000FF0000030380000000FF0000271003800000FF0000BEC092000000FC end_key: 7480000000000010FFB45F698000000000FF0000030380000000FF0000271003800000FF0000CD90C4000000FC region_epoch { conf_ver: 13583 version: 2193 } peers { id: 1418278 store_id: 5 } peers { id: 1418279 store_id: 1 } peers { id: 1418280 store_id: 6 } } }))”] [cid=4158108174]

未发现有特殊异常的日志。对于提出的任何思路，非常感谢

xfworld · 2022 年9 月 21 日 03:51

集群整体的拓扑是怎么样的？
region 822789 是哪个 table的所属？老投票，也不太正常了

image720×798 47.7 KB
按图中描述，你的tikv 集群有 4个节点？偶数？（太会玩了）

Inkjade · 2022 年9 月 21 日 06:12

整体拓扑结构：

3pd,3tidb server, 5 tikv

yilong · 2022 年9 月 26 日 04:31

请问在问题发生阶段，tikv-detail 监控中的 raft store 的 cpu 也打满了吗？
从 dashboard 中查看是否有消耗资源很多的 sql？比如 coprocessor