tikv故障启动不了 无法注册pd 有没有办法恢复数据

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:

【TiDB 版本】
v4.0.0
【问题描述】
刚开始是宕机pd启动不了
使用 pd-recover后 pd能启动但是 tikv依然启动不了。
有什么办法能够恢复数据 或者保留数据重建集群?

tiup部署
拓扑

.
global:
 user: "root"
 ssh_port: 22
 deploy_dir: "/opt/readm/tidb/dev"
 data_dir: "/opt/readm/tidb/dev/data"

# # Monitored variables are applied to all the machines.
monitored:
 node_exporter_port: 9100
 blackbox_exporter_port: 9115

server_configs:
 tidb:
   log.slow-threshold: 300
   performance.txn-total-size-limit: 10737418230
 tikv:
   readpool.storage.use-unified-pool: false
   readpool.coprocessor.use-unified-pool: true
 pd:
   replication.enable-placement-rules: true
   replication.location-labels: ["host"]
 tiflash:
   logger.level: "info"

pd_servers:
 - host: 192.168.2.88
tidb_servers:
 - host: 192.168.2.88


tikv_servers:
 - host: 192.168.2.88
   port: 20160
   status_port: 20180
   config:
     server.labels: { host: "logic-host-1" }

 - host: 192.168.2.88
   port: 20161
   status_port: 20181
   config:
     server.labels: { host: "logic-host-2" }

 - host: 192.168.2.88
   port: 20162
   status_port: 20182
   config:
     server.labels: { host: "logic-host-3" }

tiflash_servers:
 - host: 192.168.2.88
monitoring_servers:
 - host: 192.168.2.88

grafana_servers:
 - host: 192.168.2.88

tikv log

pd log

tidb log

若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

能描述一下 pd-recovery 恢复数据的具体步骤是怎么恢复的么?
另外提供一下目前 tiup cluster display 的输出结果看下

1 从 pd 日志获取 [info] cluster id
2从 tikv 日志获取 [info] alloc store id
3.停止了 pd组件 删除了 /opt/readm/tidb/dev/data/pd目录
运行 ./pd-recover -endpoints http://192.168.2.88:2379 -cluster-id 6904830971520179843 -alloc-id 100000
报错 看到文档下方的提示 需要pd
又启动了pd
然后再次运行
./pd-recover -endpoints http://192.168.2.88:2379 -cluster-id 6904830971520179843 -alloc-id 100000

tiup cluster display 结果

tikv的data目录和文件都在的没有去删除过 硬盘也是良好的状态 能恢复数据吗
看日志似乎是pd的问题
pd上报地址已经注册 tikv报pd注册不上 tidb报pd超时 似乎是一个死循环

pd.log
pd.log (121.2 KB)
tidb.log
tidb.log (30.1 KB)
tikv.log

[2021/03/16 10:14:48.003 +08:00] [ERROR] [status_server.rs:577] ["failed to register addr to pd"] [response="Response { url: \"http://192.168.2.88:2379/pd/api/v1/component\", status: 400, headers: {\"access-control-allow-headers\": \"accept, content-type, authorization\", \"access-control-allow-methods\": \"POST, GET, OPTIONS, PUT, DELETE\", \"access-control-allow-origin\": \"*\", \"content-type\": \"application/json; charset=UTF-8\", \"date\": \"Tue, 16 Mar 2021 02:14:48 GMT\", \"content-length\": \"67\"} }"]
[2021/03/16 10:14:48.003 +08:00] [INFO] [util.rs:398] ["connecting to PD endpoint"] [endpoints=http://192.168.2.88:2379]
[2021/03/16 10:14:48.004 +08:00] [INFO] [util.rs:398] ["connecting to PD endpoint"] [endpoints=http://192.168.2.88:2379]
[2021/03/16 10:14:48.005 +08:00] [INFO] [util.rs:457] ["connected to PD leader"] [endpoints=http://192.168.2.88:2379]
[2021/03/16 10:14:48.005 +08:00] [INFO] [util.rs:175] ["heartbeat sender and receiver are stale, refreshing ..."]
[2021/03/16 10:14:48.021 +08:00] [WARN] [util.rs:194] ["updating PD client done"] [spend=18.596013ms]
[2021/03/16 10:14:48.022 +08:00] [ERROR] [status_server.rs:577] ["failed to register addr to pd"] [response="Response { url: \"http://192.168.2.88:2379/pd/api/v1/component\", status: 400, headers: {\"access-control-allow-headers\": \"accept, content-type, authorization\", \"access-control-allow-methods\": \"POST, GET, OPTIONS, PUT, DELETE\", \"access-control-allow-origin\": \"*\", \"content-type\": \"application/json; charset=UTF-8\", \"date\": \"Tue, 16 Mar 2021 02:14:48 GMT\", \"content-length\": \"67\"} }"]
[2021/03/16 10:14:48.023 +08:00] [ERROR] [status_server.rs:577] ["failed to register addr to pd"] [response="Response { url: \"http://192.168.2.88:2379/pd/api/v1/component\", status: 400, headers: {\"access-control-allow-headers\": \"accept, content-type, authorization\", \"access-control-allow-methods\": \"POST, GET, OPTIONS, PUT, DELETE\", \"access-control-allow-origin\": \"*\", \"content-type\": \"application/json; charset=UTF-8\", \"date\": \"Tue, 16 Mar 2021 02:14:48 GMT\", \"content-length\": \"67\"} }"]
[2021/03/16 10:14:48.023 +08:00] [ERROR] [status_server.rs:577] ["failed to register addr to pd"] [response="Response { url: \"http://192.168.2.88:2379/pd/api/v1/component\", status: 400, headers: {\"access-control-allow-headers\": \"accept, content-type, authorization\", \"access-control-allow-methods\": \"POST, GET, OPTIONS, PUT, DELETE\", \"access-control-allow-origin\": \"*\", \"content-type\": \"application/json; charset=UTF-8\", \"date\": \"Tue, 16 Mar 2021 02:14:48 GMT\", \"content-length\": \"67\"} }"]
[2021/03/16 10:14:48.023 +08:00] [ERROR] [status_server.rs:577] ["failed to register addr to pd"] [response="Response { url: \"http://192.168.2.88:2379/pd/api/v1/component\", status: 400, headers: {\"access-control-allow-headers\": \"accept, content-type, authorization\", \"access-control-allow-methods\": \"POST, GET, OPTIONS, PUT, DELETE\", \"access-control-allow-origin\": \"*\", \"content-type\": \"application/json; charset=UTF-8\", \"date\": \"Tue, 16 Mar 2021 02:14:48 GMT\", \"content-length\": \"67\"} }"]
[2021/03/16 10:14:48.023 +08:00] [ERROR] [status_server.rs:586] ["failed to register addr to pd after 5 tries"]
[2021/03/16 10:14:48.123 +08:00] [FATAL] [lib.rs:481] ["entries[6:3766] is unavailable from storage, raft_id: 99011, region_id: 99009"] [backtrace="stack backtrace:\
   0: tikv_util::set_panic_hook::{{closure}}\
             at components/tikv_util/src/lib.rs:480\
   1: std::panicking::rust_panic_with_hook\
             at src/libstd/panicking.rs:475\
   2: rust_begin_unwind\
             at src/libstd/panicking.rs:375\
   3: std::panicking::begin_panic_fmt\
             at src/libstd/panicking.rs:326\
   4: raft::raft_log::RaftLog<T>::slice\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/<::std::macros::panic macros>:9\
   5: raft::raft_log::RaftLog<T>::next_entries_since\
             at /rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raft_log.rs:362\
      raft::raw_node::Ready::new\
             at /rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raw_node.rs:129\
      raft::raw_node::RawNode<T>::ready_since\
             at /rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raw_node.rs:346\
      raftstore::store::peer::Peer::handle_raft_ready_append\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/raftstore/src/store/peer.rs:1215\
   6: raftstore::store::fsm::peer::PeerFsmDelegate<T,C>::collect_ready\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/raftstore/src/store/fsm/peer.rs:628\
      <raftstore::store::fsm::store::RaftPoller<T,C> as batch_system::batch::PollHandler<raftstore::store::fsm::peer::PeerFsm<engine_rocks::engine::RocksEngine>,raftstore::store::fsm::store::StoreFsm>>::handle_normal\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/raftstore/src/store/fsm/store.rs:704\
   7: batch_system::batch::Poller<N,C,Handler>::poll\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/batch-system/src/batch.rs:324\
   8: batch_system::batch::BatchSystem<N,C>::spawn::{{closure}}\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/batch-system/src/batch.rs:398\
      std::sys_common::backtrace::__rust_begin_short_backtrace\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/sys_common/backtrace.rs:136\
   9: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:469\
      <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:318\
      std::panicking::try::do_call\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panicking.rs:292\
      std::panicking::try\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8//src/libpanic_unwind/lib.rs:78\
      std::panic::catch_unwind\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:394\
      std::thread::Builder::spawn_unchecked::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:468\
      core::ops::function::FnOnce::call_once{{vtable.shim}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/ops/function.rs:232\
  10: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
  11: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
      std::sys_common::thread::start_thread\
             at src/libstd/sys_common/thread.rs:13\
      std::sys::unix::thread::Thread::new::thread_start\
             at src/libstd/sys/unix/thread.rs:80\
  12: start_thread\
  13: __clone\
"] [location=/rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raft_log.rs:459] [thread_name=raftstore-5-0]
[2021/03/16 10:15:03.711 +08:00] [INFO] [lib.rs:93] ["Welcome to TiKV"]
[2021/03/16 10:15:03.711 +08:00] [INFO] [lib.rs:95] []
[2021/03/16 10:15:03.711 +08:00] [INFO] [lib.rs:95] ["Release Version:   4.0.0"]
[2021/03/16 10:15:03.711 +08:00] [INFO] [lib.rs:95] ["Edition:           Community"]
[2021/03/16 10:15:03.711 +08:00] [INFO] [lib.rs:95] ["Git Commit Hash:   198a2cea01734ce8f46d55a29708f123f9133944"]
[2021/03/16 10:15:03.711 +08:00] [INFO] [lib.rs:95] ["Git Commit Branch: heads/refs/tags/v4.0.0"]
[2021/03/16 10:15:03.711 +08:00] [INFO] [lib.rs:95] ["UTC Build Time:    2020-05-27 12:09:22"]
[2021/03/16 10:15:03.711 +08:00] [INFO] [lib.rs:95] ["Rust Version:      rustc 1.42.0-nightly (0de96d37f 2019-12-19)"]
[2021/03/16 10:15:03.711 +08:00] [INFO] [lib.rs:95] ["Enable Features:   jemalloc portable sse protobuf-codec"]
[2021/03/16 10:15:03.711 +08:00] [INFO] [lib.rs:95] ["Profile:           dist_release"]
[2021/03/16 10:15:03.805 +08:00] [INFO] [mod.rs:48] ["memory limit in bytes: 541015629824, cpu cores quota: 40"]
[2021/03/16 10:15:03.805 +08:00] [WARN] [lib.rs:528] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2021/03/16 10:15:03.805 +08:00] [WARN] [server.rs:826] ["check: kernel"] [err="kernel parameters net.core.somaxconn got 4096, expect 32768"]
[2021/03/16 10:15:03.805 +08:00] [WARN] [server.rs:826] ["check: kernel"] [err="kernel parameters net.ipv4.tcp_syncookies got 1, expect 0"]
[2021/03/16 10:15:03.805 +08:00] [WARN] [server.rs:826] ["check: kernel"] [err="kernel parameters vm.swappiness got 20, expect 0"]
[2021/03/16 10:15:03.807 +08:00] [INFO] [util.rs:398] ["connecting to PD endpoint"] [endpoints=192.168.2.88:2379]
[2021/03/16 10:15:03.807 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ff98f43a180 for subchannel 0x7ff9a4419a00"]
[2021/03/16 10:15:03.808 +08:00] [INFO] [util.rs:398] ["connecting to PD endpoint"] [endpoints=http://192.168.2.88:2379]
[2021/03/16 10:15:03.808 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ff98f43a240 for subchannel 0x7ff9a4419a00"]
[2021/03/16 10:15:03.809 +08:00] [INFO] [util.rs:398] ["connecting to PD endpoint"] [endpoints=http://192.168.2.88:2379]
[2021/03/16 10:15:03.812 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ff98f43a300 for subchannel 0x7ff9a4419a00"]
[2021/03/16 10:15:03.815 +08:00] [INFO] [util.rs:457] ["connected to PD leader"] [endpoints=http://192.168.2.88:2379]
[2021/03/16 10:15:03.815 +08:00] [INFO] [util.rs:386] ["all PD endpoints are consistent"] [endpoints="[\"192.168.2.88:2379\"]"]
[2021/03/16 10:15:03.816 +08:00] [INFO] [server.rs:242] ["connect to PD cluster"] [cluster_id=6904830971520179843]
[2021/03/16 10:15:04.620 +08:00] [INFO] [server.rs:834] ["beginning system configuration check"]
[2021/03/16 10:15:04.620 +08:00] [INFO] [config.rs:709] ["data dir"] [mount_fs="FsInfo { tp: \"xfs\", opts: \"rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota\", mnt_dir: \"/\", fsname: \"/dev/mapper/centos-root\" }"] [data_path=/opt/readm/tidb/dev/data/tikv-20160]
[2021/03/16 10:15:04.621 +08:00] [WARN] [config.rs:712] ["not on SSD device"] [data_path=/opt/readm/tidb/dev/data/tikv-20160]
[2021/03/16 10:15:04.621 +08:00] [INFO] [config.rs:709] ["data dir"] [mount_fs="FsInfo { tp: \"xfs\", opts: \"rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota\", mnt_dir: \"/\", fsname: \"/dev/mapper/centos-root\" }"] [data_path=/opt/readm/tidb/dev/data/tikv-20160/raft]
[2021/03/16 10:15:04.621 +08:00] [WARN] [config.rs:712] ["not on SSD device"] [data_path=/opt/readm/tidb/dev/data/tikv-20160/raft]
[2021/03/16 10:15:04.621 +08:00] [INFO] [server.rs:212] ["using config"] [config="{\"cfg-path\":\"conf/tikv.toml\",\"log-level\":\"info\",\"log-file\":\"/opt/readm/tidb/dev/tikv-20160/log/tikv.log\",\"slow-log-file\":\"\",\"slow-log-threshold\":\"1s\",\"log-rotation-timespan\":\"1d\",\"log-rotation-size\":\"300MiB\",\"panic-when-unexpected-key-or-data\":false,\"readpool\":{\"unified\":{\"min-thread-count\":1,\"max-thread-count\":32,\"stack-size\":\"10MiB\",\"max-tasks-per-worker\":2000},\"storage\":{\"use-unified-pool\":false,\"high-concurrency\":8,\"normal-concurrency\":8,\"low-concurrency\":8,\"max-tasks-per-worker-high\":2000,\"max-tasks-per-worker-normal\":2000,\"max-tasks-per-worker-low\":2000,\"stack-size\":\"10MiB\"},\"coprocessor\":{\"use-unified-pool\":true,\"high-concurrency\":32,\"normal-concurrency\":32,\"low-concurrency\":32,\"max-tasks-per-worker-high\":2000,\"max-tasks-per-worker-normal\":2000,\"max-tasks-per-worker-low\":2000,\"stack-size\":\"10MiB\"}},\"server\":{\"addr\":\"0.0.0.0:20160\",\"advertise-addr\":\"192.168.2.88:20160\",\"status-addr\":\"0.0.0.0:20180\",\"status-thread-pool-size\":1,\"max-grpc-send-msg-len\":10485760,\"grpc-compression-type\":\"none\",\"grpc-concurrency\":4,\"grpc-concurrent-stream\":1024,\"grpc-raft-conn-num\":1,\"grpc-memory-pool-quota\":9223372036854775807,\"grpc-stream-initial-window-size\":\"2MiB\",\"grpc-keepalive-time\":\"10s\",\"grpc-keepalive-timeout\":\"3s\",\"concurrent-send-snap-limit\":32,\"concurrent-recv-snap-limit\":32,\"end-point-recursion-limit\":1000,\"end-point-stream-channel-size\":8,\"end-point-batch-row-limit\":64,\"end-point-stream-batch-row-limit\":128,\"end-point-enable-batch-if-possible\":true,\"end-point-request-max-handle-duration\":\"1m\",\"end-point-max-concurrency\":40,\"snap-max-write-bytes-per-sec\":\"100MiB\",\"snap-max-total-size\":\"0KiB\",\"stats-concurrency\":1,\"heavy-load-threshold\":300,\"heavy-load-wait-duration\":\"1ms\",\"enable-request-batch\":true,\"request-batch-enable-cross-command\":false,\"request-batch-wait-duration\":\"1ms\",\"labels\":{\"host\":\"logic-host-1\"}},\"storage\":{\"data-dir\":\"/opt/readm/tidb/dev/data/tikv-20160\",\"gc-ratio-threshold\":1.1,\"max-key-size\":4096,\"scheduler-concurrency\":524288,\"scheduler-worker-pool-size\":8,\"scheduler-pending-write-threshold\":\"100MiB\",\"reserve-space\":\"2GiB\",\"block-cache\":{\"shared\":true,\"capacity\":\"209452MiB\",\"num-shard-bits\":6,\"strict-capacity-limit\":false,\"high-pri-pool-ratio\":0.8,\"memory-allocator\":\"nodump\"}},\"pd\":{\"endpoints\":[\"192.168.2.88:2379\"],\"retry-interval\":\"300ms\",\"retry-max-count\":9223372036854775807,\"retry-log-every\":10},\"metric\":{\"interval\":\"15s\",\"address\":\"\",\"job\":\"tikv\"},\"raftstore\":{\"sync-log\":true,\"prevote\":true,\"raftdb-path\":\"/opt/readm/tidb/dev/data/tikv-20160/raft\",\"capacity\":\"0KiB\",\"raft-base-tick-interval\":\"1s\",\"raft-heartbeat-ticks\":2,\"raft-election-timeout-ticks\":10,\"raft-min-election-timeout-ticks\":10,\"raft-max-election-timeout-ticks\":20,\"raft-max-size-per-msg\":\"1MiB\",\"raft-max-inflight-msgs\":256,\"raft-entry-max-size\":\"8MiB\",\"raft-log-gc-tick-interval\":\"10s\",\"raft-log-gc-threshold\":50,\"raft-log-gc-count-limit\":73728,\"raft-log-gc-size-limit\":\"72MiB\",\"raft-entry-cache-life-time\":\"30s\",\"raft-reject-transfer-leader-duration\":\"3s\",\"split-region-check-tick-interval\":\"10s\",\"region-split-check-diff\":\"6MiB\",\"region-compact-check-interval\":\"5m\",\"clean-stale-peer-delay\":\"11m\",\"region-compact-check-step\":100,\"region-compact-min-tombstones\":10000,\"region-compact-tombstones-percent\":30,\"pd-heartbeat-tick-interval\":\"1m\",\"pd-store-heartbeat-tick-interval\":\"10s\",\"snap-mgr-gc-tick-interval\":\"1m\",\"snap-gc-timeout\":\"4h\",\"lock-cf-compact-interval\":\"10m\",\"lock-cf-compact-bytes-threshold\":\"256MiB\",\"notify-capacity\":40960,\"messages-per-tick\":4096,\"max-peer-down-duration\":\"5m\",\"max-leader-missing-duration\":\"2h\",\"abnormal-leader-missing-duration\":\"10m\",\"peer-stale-state-check-interval\":\"5m\",\"leader-transfer-max-log-lag\":10,\"snap-apply-batch-size\":\"10MiB\",\"consistency-check-interval\":\"0s\",\"report-region-flow-interval\":\"1m\",\"raft-store-max-leader-lease\":\"9s\",\"right-derive-when-split\":true,\"allow-remove-leader\":false,\"merge-max-log-gap\":10,\"merge-check-tick-interval\":\"10s\",\"use-delete-range\":false,\"cleanup-import-sst-interval\":\"10m\",\"local-read-batch-size\":1024,\"apply-max-batch-size\":256,\"apply-pool-size\":2,\"apply-reschedule-duration\":\"5s\",\"store-max-batch-size\":256,\"store-pool-size\":2,\"store-reschedule-duration\":\"5s\",\"future-poll-size\":1,\"hibernate-regions\":false,\"early-apply\":true,\"apply-yield-duration\":\"500ms\"},\"coprocessor\":{\"split-region-on-table\":false,\"batch-split-limit\":10,\"region-max-size\":\"144MiB\",\"region-split-size\":\"96MiB\",\"region-max-keys\":1440000,\"region-split-keys\":960000},\"rocksdb\":{\"wal-recovery-mode\":2,\"wal-dir\":\"\",\"wal-ttl-seconds\":0,\"wal-size-limit\":\"0KiB\",\"max-total-wal-size\":\"4GiB\",\"max-background-jobs\":8,\"max-manifest-file-size\":\"128MiB\",\"create-if-missing\":true,\"max-open-files\":40960,\"enable-statistics\":true,\"stats-dump-period\":\"10m\",\"compaction-readahead-size\":\"0KiB\",\"info-log-max-size\":\"1GiB\",\"info-log-roll-time\":\"0s\",\"info-log-keep-log-file-num\":10,\"info-log-dir\":\"\",\"rate-bytes-per-sec\":\"0KiB\",\"rate-limiter-mode\":2,\"auto-tuned\":false,\"bytes-per-sync\":\"1MiB\",\"wal-bytes-per-sync\":\"512KiB\",\"max-sub-compactions\":3,\"writable-file-max-buffer-size\":\"1MiB\",\"use-direct-io-for-flush-and-compaction\":false,\"enable-pipelined-write\":true,\"enable-multi-batch-write\":true,\"enable-unordered-write\":false,\"defaultcf\":{\"block-size\":\"64KiB\",\"block-cache-size\":\"128988MiB\",\"disable-block-cache\":false,\"cache-index-and-filter-blocks\":true,\"pin-l0-filter-and-index-blocks\":true,\"use-bloom-filter\":true,\"optimize-filters-for-hits\":true,\"whole-key-filtering\":true,\"bloom-filter-bits-per-key\":10,\"block-based-bloom-filter\":false,\"read-amp-bytes-per-bit\":0,\"compression-per-level\":[\"no\",\"no\",\"lz4\",\"lz4\",\"lz4\",\"zstd\",\"zstd\"],\"write-buffer-size\":\"128MiB\",\"max-write-buffer-number\":5,\"min-write-buffer-number-to-merge\":1,\"max-bytes-for-level-base\":\"512MiB\",\"target-file-size-base\":\"8MiB\",\"level0-file-num-compaction-trigger\":4,\"level0-slowdown-writes-trigger\":20,\"level0-stop-writes-trigger\":36,\"max-compaction-bytes\":\"2GiB\",\"compaction-pri\":3,\"dynamic-level-bytes\":true,\"num-levels\":7,\"max-bytes-for-level-multiplier\":10,\"compaction-style\":0,\"disable-auto-compactions\":false,\"soft-pending-compaction-bytes-limit\":\"64GiB\",\"hard-pending-compaction-bytes-limit\":\"256GiB\",\"force-consistency-checks\":true,\"prop-size-index-distance\":4194304,\"prop-keys-index-distance\":40960,\"enable-doubly-skiplist\":true,\"titan\":{\"min-blob-size\":\"1KiB\",\"blob-file-compression\":\"lz4\",\"blob-cache-size\":\"0KiB\",\"min-gc-batch-size\":\"16MiB\",\"max-gc-batch-size\":\"64MiB\",\"discardable-ratio\":0.5,\"sample-ratio\":0.1,\"merge-small-file-threshold\":\"8MiB\",\"blob-run-mode\":\"normal\",\"level-merge\":false,\"range-merge\":true,\"max-sorted-runs\":20,\"gc-merge-rewrite\":false}},\"writecf\":{\"block-size\":\"64KiB\",\"block-cache-size\":\"77392MiB\",\"disable-block-cache\":false,\"cache-index-and-filter-blocks\":true,\"pin-l0-filter-and-index-blocks\":true,\"use-bloom-filter\":true,\"optimize-filters-for-hits\":false,\"whole-key-filtering\":false,\"bloom-filter-bits-per-key\":10,\"block-based-bloom-filter\":false,\"read-amp-bytes-per-bit\":0,\"compression-per-level\":[\"no\",\"no\",\"lz4\",\"lz4\",\"lz4\",\"zstd\",\"zstd\"],\"write-buffer-size\":\"128MiB\",\"max-write-buffer-number\":5,\"min-write-buffer-number-to-merge\":1,\"max-bytes-for-level-base\":\"512MiB\",\"target-file-size-base\":\"8MiB\",\"level0-file-num-compaction-trigger\":4,\"level0-slowdown-writes-trigger\":20,\"level0-stop-writes-trigger\":36,\"max-compaction-bytes\":\"2GiB\",\"compaction-pri\":3,\"dynamic-level-bytes\":true,\"num-levels\":7,\"max-bytes-for-level-multiplier\":10,\"compaction-style\":0,\"disable-auto-compactions\":false,\"soft-pending-compaction-bytes-limit\":\"64GiB\",\"hard-pending-compaction-bytes-limit\":\"256GiB\",\"force-consistency-checks\":true,\"prop-size-index-distance\":4194304,\"prop-keys-index-distance\":40960,\"enable-doubly-skiplist\":true,\"titan\":{\"min-blob-size\":\"1KiB\",\"blob-file-compression\":\"lz4\",\"blob-cache-size\":\"0KiB\",\"min-gc-batch-size\":\"16MiB\",\"max-gc-batch-size\":\"64MiB\",\"discardable-ratio\":0.5,\"sample-ratio\":0.1,\"merge-small-file-threshold\":\"8MiB\",\"blob-run-mode\":\"read-only\",\"level-merge\":false,\"range-merge\":true,\"max-sorted-runs\":20,\"gc-merge-rewrite\":false}},\"lockcf\":{\"block-size\":\"16KiB\",\"block-cache-size\":\"1GiB\",\"disable-block-cache\":false,\"cache-index-and-filter-blocks\":true,\"pin-l0-filter-and-index-blocks\":true,\"use-bloom-filter\":true,\"optimize-filters-for-hits\":false,\"whole-key-filtering\":true,\"bloom-filter-bits-per-key\":10,\"block-based-bloom-filter\":false,\"read-amp-bytes-per-bit\":0,\"compression-per-level\":[\"no\",\"no\",\"no\",\"no\",\"no\",\"no\",\"no\"],\"write-buffer-size\":\"32MiB\",\"max-write-buffer-number\":5,\"min-write-buffer-number-to-merge\":1,\"max-bytes-for-level-base\":\"128MiB\",\"target-file-size-base\":\"8MiB\",\"level0-file-num-compaction-trigger\":1,\"level0-slowdown-writes-trigger\":20,\"level0-stop-writes-trigger\":36,\"max-compaction-bytes\":\"2GiB\",\"compaction-pri\":0,\"dynamic-level-bytes\":true,\"num-levels\":7,\"max-bytes-for-level-multiplier\":10,\"compaction-style\":0,\"disable-auto-compactions\":false,\"soft-pending-compaction-bytes-limit\":\"64GiB\",\"hard-pending-compaction-bytes-limit\":\"256GiB\",\"force-consistency-checks\":true,\"prop-size-index-distance\":4194304,\"prop-keys-index-distance\":40960,\"enable-doubly-skiplist\":true,\"titan\":{\"min-blob-size\":\"1KiB\",\"blob-file-compression\":\"lz4\",\"blob-cache-size\":\"0KiB\",\"min-gc-batch-size\":\"16MiB\",\"max-gc-batch-size\":\"64MiB\",\"discardable-ratio\":0.5,\"sample-ratio\":0.1,\"merge-small-file-threshold\":\"8MiB\",\"blob-run-mode\":\"read-only\",\"level-merge\":false,\"range-merge\":true,\"max-sorted-runs\":20,\"gc-merge-rewrite\":false}},\"raftcf\":{\"block-size\":\"16KiB\",\"block-cache-size\":\"128MiB\",\"disable-block-cache\":false,\"cache-index-and-filter-blocks\":true,\"pin-l0-filter-and-index-blocks\":true,\"use-bloom-filter\":true,\"optimize-filters-for-hits\":true,\"whole-key-filtering\":true,\"bloom-filter-bits-per-key\":10,\"block-based-bloom-filter\":false,\"read-amp-bytes-per-bit\":0,\"compression-per-level\":[\"no\",\"no\",\"no\",\"no\",\"no\",\"no\",\"no\"],\"write-buffer-size\":\"128MiB\",\"max-write-buffer-number\":5,\"min-write-buffer-number-to-merge\":1,\"max-bytes-for-level-base\":\"128MiB\",\"target-file-size-base\":\"8MiB\",\"level0-file-num-compaction-trigger\":1,\"level0-slowdown-writes-trigger\":20,\"level0-stop-writes-trigger\":36,\"max-compaction-bytes\":\"2GiB\",\"compaction-pri\":0,\"dynamic-level-bytes\":true,\"num-levels\":7,\"max-bytes-for-level-multiplier\":10,\"compaction-style\":0,\"disable-auto-compactions\":false,\"soft-pending-compaction-bytes-limit\":\"64GiB\",\"hard-pending-compaction-bytes-limit\":\"256GiB\",\"force-consistency-checks\":true,\"prop-size-index-distance\":4194304,\"prop-keys-index-distance\":40960,\"enable-doubly-skiplist\":true,\"titan\":{\"min-blob-size\":\"1KiB\",\"blob-file-compression\":\"lz4\",\"blob-cache-size\":\"0KiB\",\"min-gc-batch-size\":\"16MiB\",\"max-gc-batch-size\":\"64MiB\",\"discardable-ratio\":0.5,\"sample-ratio\":0.1,\"merge-small-file-threshold\":\"8MiB\",\"blob-run-mode\":\"read-only\",\"level-merge\":false,\"range-merge\":true,\"max-sorted-runs\":20,\"gc-merge-rewrite\":false}},\"titan\":{\"enabled\":false,\"dirname\":\"\",\"disable-gc\":false,\"max-background-gc\":4,\"purge-obsolete-files-period\":\"10s\"}},\"raftdb\":{\"wal-recovery-mode\":2,\"wal-dir\":\"\",\"wal-ttl-seconds\":0,\"wal-size-limit\":\"0KiB\",\"max-total-wal-size\":\"4GiB\",\"max-background-jobs\":4,\"max-manifest-file-size\":\"20MiB\",\"create-if-missing\":true,\"max-open-files\":40960,\"enable-statistics\":true,\"stats-dump-period\":\"10m\",\"compaction-readahead-size\":\"0KiB\",\"info-log-max-size\":\"1GiB\",\"info-log-roll-time\":\"0s\",\"info-log-keep-log-file-num\":10,\"info-log-dir\":\"\",\"max-sub-compactions\":2,\"writable-file-max-buffer-size\":\"1MiB\",\"use-direct-io-for-flush-and-compaction\":false,\"enable-pipelined-write\":true,\"enable-unordered-write\":false,\"allow-concurrent-memtable-write\":true,\"bytes-per-sync\":\"1MiB\",\"wal-bytes-per-sync\":\"512KiB\",\"defaultcf\":{\"block-size\":\"64KiB\",\"block-cache-size\":\"2GiB\",\"disable-block-cache\":false,\"cache-index-and-filter-blocks\":true,\"pin-l0-filter-and-index-blocks\":true,\"use-bloom-filter\":false,\"optimize-filters-for-hits\":true,\"whole-key-filtering\":true,\"bloom-filter-bits-per-key\":10,\"block-based-bloom-filter\":false,\"read-amp-bytes-per-bit\":0,\"compression-per-level\":[\"no\",\"no\",\"lz4\",\"lz4\",\"lz4\",\"zstd\",\"zstd\"],\"write-buffer-size\":\"128MiB\",\"max-write-buffer-number\":5,\"min-write-buffer-number-to-merge\":1,\"max-bytes-for-level-base\":\"512MiB\",\"target-file-size-base\":\"8MiB\",\"level0-file-num-compaction-trigger\":4,\"level0-slowdown-writes-trigger\":20,\"level0-stop-writes-trigger\":36,\"max-compaction-bytes\":\"2GiB\",\"compaction-pri\":0,\"dynamic-level-bytes\":true,\"num-levels\":7,\"max-bytes-for-level-multiplier\":10,\"compaction-style\":0,\"disable-auto-compactions\":false,\"soft-pending-compaction-bytes-limit\":\"64GiB\",\"hard-pending-compaction-bytes-limit\":\"256GiB\",\"force-consistency-checks\":true,\"prop-size-index-distance\":4194304,\"prop-keys-index-distance\":40960,\"enable-doubly-skiplist\":true,\"titan\":{\"min-blob-size\":\"1KiB\",\"blob-file-compression\":\"lz4\",\"blob-cache-size\":\"0KiB\",\"min-gc-batch-size\":\"16MiB\",\"max-gc-batch-size\":\"64MiB\",\"discardable-ratio\":0.5,\"sample-ratio\":0.1,\"merge-small-file-threshold\":\"8MiB\",\"blob-run-mode\":\"normal\",\"level-merge\":false,\"range-merge\":true,\"max-sorted-runs\":20,\"gc-merge-rewrite\":false}},\"titan\":{\"enabled\":false,\"dirname\":\"\",\"disable-gc\":false,\"max-background-gc\":4,\"purge-obsolete-files-period\":\"10s\"}},\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":[],\"encryption\":{\"data-encryption-method\":\"plaintext\",\"data-key-rotation-period\":\"7d\",\"master-key\":{\"type\":\"plaintext\"},\"previous-master-key\":{\"type\":\"plaintext\"}}},\"import\":{\"num-threads\":8,\"stream-channel-window\":128},\"pessimistic-txn\":{\"enabled\":true,\"wait-for-lock-timeout\":\"1s\",\"wake-up-delay-duration\":\"20ms\",\"pipelined\":false},\"gc\":{\"ratio-threshold\":1.1,\"batch-keys\":512,\"max-write-bytes-per-sec\":\"0KiB\"},\"split\":{\"qps-threshold\":3000,\"split-balance-score\":0.25,\"split-contained-score\":0.5,\"detect-times\":10,\"sample-num\":20,\"sample-threshold\":100}}"]

可以参考下这个文档重新恢复一下 PD :【SOP 系列 17】TiUP 环境 PD-Recover 使用恢复案例

注意点:

  1. 找一下所有日志中的所有 cluster-id 信息,需要确认 pd-recovery 恢复时的 cluster-id 是故障前的 cluster-id
  2. pd-recovery 设置的 alloc-id 需要比找到的最大的 alloc-id 更大
  3. pd-recovery 运行完应该有 recover success! please restart the PD cluster 的提示,重启一下 PD 节点即可。

有 recover success 重启依然是报错的。
alloc-id也加大了。
请问还有别的办法吗

PD 一开始是什么时间点宕机无法使用的?
pd-recovery 操作是什么时间点操作的?

目前看到最新的 TiKV 无法启动的报错是

[2021/03/16 10:14:48.123 +08:00] [FATAL] [lib.rs:481] ["entries[6:3766] is unavailable from storage, raft_id: 99011, region_id: 99009"] [backtrace="stack backtrace:\
   0: tikv_util::set_panic_hook::{{closure}}\
             at components/tikv_util/src/lib.rs:480\
   1: std::panicking::rust_panic_with_hook\
             at src/libstd/panicking.rs:475\
   2: rust_begin_unwind\
             at src/libstd/panicking.rs:375\
   3: std::panicking::begin_panic_fmt\
             at src/libstd/panicking.rs:326\
   4: raft::raft_log::RaftLog<T>::slice\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/<::std::macros::panic macros>:9\
   5: raft::raft_log::RaftLog<T>::next_entries_since\
             at /rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raft_log.rs:362\
      raft::raw_node::Ready::new\
             at /rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raw_node.rs:129\
      raft::raw_node::RawNode<T>::ready_since\
             at /rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raw_node.rs:346\
      raftstore::store::peer::Peer::handle_raft_ready_append\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/raftstore/src/store/peer.rs:1215\
   6: raftstore::store::fsm::peer::PeerFsmDelegate<T,C>::collect_ready\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/raftstore/src/store/fsm/peer.rs:628\
      <raftstore::store::fsm::store::RaftPoller<T,C> as batch_system::batch::PollHandler<raftstore::store::fsm::peer::PeerFsm<engine_rocks::engine::RocksEngine>,raftstore::store::fsm::store::StoreFsm>>::handle_normal\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/raftstore/src/store/fsm/store.rs:704\
   7: batch_system::batch::Poller<N,C,Handler>::poll\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/batch-system/src/batch.rs:324\
   8: batch_system::batch::BatchSystem<N,C>::spawn::{{closure}}\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/batch-system/src/batch.rs:398\
      std::sys_common::backtrace::__rust_begin_short_backtrace\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/sys_common/backtrace.rs:136\
   9: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:469\
      <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:318\
      std::panicking::try::do_call\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panicking.rs:292\
      std::panicking::try\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8//src/libpanic_unwind/lib.rs:78\
      std::panic::catch_unwind\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:394\
      std::thread::Builder::spawn_unchecked::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:468\
      core::ops::function::FnOnce::call_once{{vtable.shim}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/ops/function.rs:232\
  10: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
  11: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
      std::sys_common::thread::start_thread\
             at src/libstd/sys_common/thread.rs:13\
      std::sys::unix::thread::Thread::new::thread_start\
             at src/libstd/sys/unix/thread.rs:80\
  12: start_thread\
  13: __clone\
"] [location=/rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raft_log.rs:459] [thread_name=raftstore-5-0]

这个错误可以参考这个 FAQ 恢复一下

宕机发生在一个月之前
因为项目原因 宕机之后我们就没有处理 因为之前有类似的情况 pd-recover后就恢复了。 原以为问题不大的,就当时没有处理 也没进行pd-recover。

现在开始要用到那批数据所以想要恢复 但是 昨日进行pd-recover的时候 无法成功恢复。
您发的帖子 我再看看试一下

嗯,目前看起来 PD 进程是正常的,TiKV 中的日志也不是因为无法注册 PD 而无法启动
可以先恢复目前的 TiKV FATAL 错误再看下

你好
我尝试 帖子中的方案 删除 这个region 但是报出信息是没有这个region 在三个tikv的目录下都是如此
是什么原因呢

出错的
region_id
99037
99009

这是因为 PD 被重装了, tikv 还没有启动起来,无法将 region 信息通过心跳形式上报到 PD 中,所以 PD 中目前还没有 region 信息。
可以先执行下 unsafe-recover 操作看下。

有unsafe-recover 的帖子推荐吗
因为数据只有一份 我也不太敢操作类似带有 删除 恢复的命令 总是担心数据文件真的删没了。
比如 有帖子说tikv-ctl --db /path/to/tikv/db unsafe-recover remove-fail-stores 1 执行这种
我就不太敢直接执行 不清楚是不是删除数据或者整体数据文件。
命令相关介绍也没有很明确的说明 小白看的很懵

你好 我这边执行了 但是重启tikv 依然是报 99009 region 有问题 好像没有删除掉这个region

  1. 集群的 max-replica 副本数设置的是 1 么?
  2. unsafe-recover 之后,重新启动 tikv 的日志是怎么样的?
  1. 在部署的yml里没有指定 副本数。 tiup cluster edit-config 查看也没有看到副本数 应该是默认的
    2.log文件
    tidb.log (425.7 KB)

这个是 tidb.log ,tikv 启动不了,应该提供 tikv.log

不好意思刚刚没看清楚
tikv的log很大 而且重复 我筛选了一下fatal的是这些 error的没有 其他的都是info级别 如果需要info的我找个共享的地方传一下 辛苦了
FATAL.LOG (77.4 KB)

[2021/03/16 12:02:01.527 +08:00] [FATAL] [lib.rs:481] ["entries[6:4932] is unavailable from storage, raft_id: 99039, region_id: 99037"] [backtrace="stack backtrace:\
   0: tikv_util::set_panic_hook::{{closure}}\
             at components/tikv_util/src/lib.rs:480\
   1: std::panicking::rust_panic_with_hook\
             at src/libstd/panicking.rs:475\
   2: rust_begin_unwind\
             at src/libstd/panicking.rs:375\
   3: std::panicking::begin_panic_fmt\
             at src/libstd/panicking.rs:326\
   4: raft::raft_log::RaftLog<T>::slice\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/<::std::macros::panic macros>:9\
   5: raft::raft_log::RaftLog<T>::next_entries_since\
             at /rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raft_log.rs:362\
      raft::raw_node::Ready::new\
             at /rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raw_node.rs:129\
      raft::raw_node::RawNode<T>::ready_since\
             at /rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raw_node.rs:346\
      raftstore::store::peer::Peer::handle_raft_ready_append\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/raftstore/src/store/peer.rs:1215\
   6: raftstore::store::fsm::peer::PeerFsmDelegate<T,C>::collect_ready\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/raftstore/src/store/fsm/peer.rs:628\
      <raftstore::store::fsm::store::RaftPoller<T,C> as batch_system::batch::PollHandler<raftstore::store::fsm::peer::PeerFsm<engine_rocks::engine::RocksEngine>,raftstore::store::fsm::store::StoreFsm>>::handle_normal\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/raftstore/src/store/fsm/store.rs:704\
   7: batch_system::batch::Poller<N,C,Handler>::poll\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/batch-system/src/batch.rs:324\
   8: batch_system::batch::BatchSystem<N,C>::spawn::{{closure}}\
             at /home/jenkins/agent/workspace/build_tikv_multi_branch_v4.0.0/tikv/components/batch-system/src/batch.rs:398\
      std::sys_common::backtrace::__rust_begin_short_backtrace\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/sys_common/backtrace.rs:136\
   9: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:469\
      <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:318\
      std::panicking::try::do_call\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panicking.rs:292\
      std::panicking::try\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8//src/libpanic_unwind/lib.rs:78\
      std::panic::catch_unwind\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:394\
      std::thread::Builder::spawn_unchecked::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:468\
      core::ops::function::FnOnce::call_once{{vtable.shim}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/ops/function.rs:232\
  10: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
  11: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
      std::sys_common::thread::start_thread\
             at src/libstd/sys_common/thread.rs:13\
      std::sys::unix::thread::Thread::new::thread_start\
             at src/libstd/sys/unix/thread.rs:80\
  12: start_thread\
  13: __clone\
"] [location=/rust/git/checkouts/raft-rs-841f8a6db665c5c0/b5f5830/src/raft_log.rs:459] [thread_name=raftstore-5-0]

看到最后一条 FATAL 日志是 12:02 的时候的,这个时间应该是在你执行 unsafe-recover 操作之前吧?
可以提供一下 12 点之后,完整的 tikv.log 日志么,包含 INFO 等级别信息的。

http://static.liqlab.cn/temp/log/tikv.log
你好这是完整 log文件 辛苦查看下

副本配置

» config show replication
{
“max-replicas”: 3,
“location-labels”: “host”,
“strictly-match-label”: “false”,
“enable-placement-rules”: “true”
}

完整
{
“replication”: {
“enable-placement-rules”: “true”,
“location-labels”: “host”,
“max-replicas”: 3,
“strictly-match-label”: “false”
},
“schedule”: {
“enable-cross-table-merge”: “false”,
“enable-debug-metrics”: “false”,
“enable-location-replacement”: “true”,
“enable-make-up-replica”: “true”,
“enable-one-way-merge”: “false”,
“enable-remove-down-replica”: “true”,
“enable-remove-extra-replica”: “true”,
“enable-replace-offline-replica”: “true”,
“high-space-ratio”: 0.7,
“hot-region-cache-hits-threshold”: 3,
“hot-region-schedule-limit”: 4,
“leader-schedule-limit”: 4,
“leader-schedule-policy”: “count”,
“low-space-ratio”: 0.8,
“max-merge-region-keys”: 200000,
“max-merge-region-size”: 20,
“max-pending-peer-count”: 16,
“max-snapshot-count”: 3,
“max-store-down-time”: “30m0s”,
“merge-schedule-limit”: 8,
“patrol-region-interval”: “100ms”,
“region-schedule-limit”: 2048,
“replica-schedule-limit”: 64,
“scheduler-max-waiting-operator”: 5,
“split-merge-interval”: “1h0m0s”,
“store-balance-rate”: 15,
“store-limit-mode”: “manual”,
“tolerant-size-ratio”: 0
}
}