TiKV磁盘跑满，集群无法访问

[普罗米修斯]

排查tikv节点宕掉后无法重新拉起的原因，tikv日志中报last index < Applied index ，另一个因为与这个kv节点接口相互依赖无法启动；

[普罗米修斯]

[2022/09/13 10:36:02.900 +08:00] [FATAL] [lib.rs:499] [“[region 911971] 62089724 unexpected raft log index: last_index 341843299 < applied_index 341843497”] [backtrace="stack backtrace:
0: 0x5628f061551d - backtrace::backtrace::libunwind::trace::h0500f4f2825a5d17
at /rust/registry/src/github.com-1ecc6299db9ec823/backtrace-0.2.3/src/backtrace/libunwind.rs:54
- backtrace::backtrace::trace::h4187244de1605a06
at /rust/registry/src/github.com-1ecc6299db9ec823/backtrace-0.2.3/src/backtrace/mod.rs:70
1: 0x5628f0609cd0 - tikv_util::set_panic_hook::{{closure}}::h195100b0bbd49cfb
at /home/jenkins/.target/release/build/backtrace-e20a32a05fd0b8fe/out/capture.rs:79
2: 0x5628f07ae64f - std::panicking::rust_panic_with_hook::h8d2408723e9a2bd4
at src/libstd/panicking.rs:479
3: 0x5628f07ae42d - std::panicking::continue_panic_fmt::hb2aaa9386c4e5e80
at src/libstd/panicking.rs:382
4: 0x5628f07ae3db - std::panicking::begin_panic_fmt::h1c91fada5a982dcd
at src/libstd/panicking.rs:337
5: 0x5628efd9449a - tikv::raftstore::store::peer::Peer::h114b9c5233192fb4
at src/raftstore/store/peer.rs:0
6: 0x5628efc310c5 - tikv::raftstore::store::fsm::peer::PeerFsm::create::h796ac694c7f2d0e5
at src/raftstore/store/fsm/peer.rs:151
7: 0x5628efb1a973 - tikv::raftstore::store::fsm::store::RaftPollerBuilder<T,C>::init::{{closure}}::h4c08e0783f93ab81
at src/raftstore/store/fsm/store.rs:750
- engine::iterable::scan_impl::h0450662012195a8e
at /home/jenkins/workspace/release_tidb_3.0/tikv/components/engine/src/iterable.rs:198
- engine::iterable::Iterable::scan_cf::h6d7aa7d8bbcfb1ed
at /home/jenkins/workspace/release_tidb_3.0/tikv/components/engine/src/iterable.rs:174
- tikv::raftstore::store::fsm::store::RaftPollerBuilder<T,C>::init::h75225d858b84b1b1
at src/raftstore/store/fsm/store.rs:721
- tikv::raftstore::store::fsm::store::RaftBatchSystem::spawn::h4291f89945cb9126
at src/raftstore/store/fsm/store.rs:1003
8: 0x5628efaf6982 - tikv::server::node::Node::start_store::hb6e41ce5d17f8092
at src/server/node.rs:341
- tikv::server::node::Node::start::h502723d45c7f0962
at src/server/node.rs:148
- tikv::binutil::server::run_raft_server::he798388cc8852c3a
at src/binutil/server.rs:276
9: 0x5628efabf1b1 - tikv::binutil::server::run_tikv::h17cbdf211cde42b2
at src/binutil/server.rs:79
10: 0x5628efab4805 - tikv_server::main::h28eadcb59f5aa918
at src/bin/tikv-server.rs:159
11: 0x5628efa2b362 - std::rt::lang_start::{{closure}}::hd8df218522d5a046
at /rustc/0e4a56b4b04ea98bb16caada30cb2418dd06e250/src/libstd/rt.rs:64
12: 0x5628efab6128 - main
13: 0x7fb89070382f - __libc_start_main
14: 0x5628efa011a8 -
15:

[普罗米修斯]

我看提示的时raft不一致的问题

[TiDBer_jYQINSnf]

https://docs.pingcap.com/zh/tidb/stable/tidb-troubleshooting-map#41-tikv-panic-启动不了

[普罗米修斯]

unsafe-recover remove-fail-stores
指令在该tikv上执行，我看没有移除错误的region，提示如下：

[TiDBer_jYQINSnf]

这个我用的少，等其他人回复吧，或者你圈一下版主大人们。
如果你这个实在跑满的节点上执行，那也没用。磁盘都满了，什么也改变不了。