tidb-lingtning导数据过程中导致tikv宕机，tikv无法启动

🪐 TiDB 技术问题 备份&数据迁移
1

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：

[TiDB 版本]
tidb v4.0.10
[问题描述]
tidb-lingtning 导入数据过程中，tikv宕机两台（共三台），然后启动不了。

报错信息如下：

Starting component ctl: /home/tidb/.tiup/components/ctl/v4.0.10/ctl tikv --db /tidb-data/tikv-20160/db
thread ‘main’ panicked at ‘called Result::unwrap() on an Err value: Os { code: 2, kind: NotFound, message: “No such file or directory” }’, src/libcore/result.rs:1188:5
stack backtrace:
0: backtrace::backtrace::libunwind::trace
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
1: backtrace::backtrace::trace_unsynchronized
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
2: std::sys_common::backtrace::_print_fmt
at src/libstd/sys_common/backtrace.rs:84
3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
at src/libstd/sys_common/backtrace.rs:61
4: core::fmt::write
at src/libcore/fmt/mod.rs:1025
5: std::io::Write::write_fmt
at src/libstd/io/mod.rs:1426
6: std::sys_common::backtrace::_print
at src/libstd/sys_common/backtrace.rs:65
7: std::sys_common::backtrace::print
at src/libstd/sys_common/backtrace.rs:50
8: std::panicking::default_hook::{{closure}}
at src/libstd/panicking.rs:193
9: std::panicking::default_hook
at src/libstd/panicking.rs:210
10: std::panicking::rust_panic_with_hook
at src/libstd/panicking.rs:471
11: rust_begin_unwind
at src/libstd/panicking.rs:375
12: core::panicking::panic_fmt
at src/libcore/panicking.rs:84
13: core::result::unwrap_failed
at src/libcore/result.rs:1188
14: core::result::Result<T,E>::unwrap
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/result.rs:956
15: tikv_ctl::new_debug_executor
at cmd/src/bin/tikv-ctl.rs:80
16: tikv_ctl::main
at cmd/src/bin/tikv-ctl.rs:1941
17: std::rt::lang_start::{{closure}}
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/rt.rs:67
18: main
19: __libc_start_main
20:
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.
Error: exit status 101
Error: run /home/tidb/.tiup/components/ctl/v4.0.10/ctl (wd:/home/tidb/.tiup/data/SMyrRav) failed: exit status 1
[tidb@localhost ~]$ tiup ctl tikv --db /tidb-data/tikv-20160/db
Found ctl newer version:

若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出的打印结果，请务必全选并复制粘贴上传。

2

1.确认下是不是以下问题

2.确保 tikv-ctl 版本与 tidb 集群版本一致
使用 tiup 对 ctl 组件进行升级

3

我之前是在线把ctl升级到了5.0，后面我离线安装为4.0.10.这个有影响没？

4

[tidb@localhost ~]$ tiup ctl tikv --db /tidb-data/tikv-20160/db tombstone -r 107638 --force
Starting component ctl: /home/tidb/.tiup/components/ctl/v4.0.10/ctl tikv --db /tidb-data/tikv-20160/db tombstone -r 107638 --force
thread ‘main’ panicked at ‘called Result::unwrap() on an Err value: Os { code: 2, kind: NotFound, message: “No such file or directory” }’, src/libcore/result.rs:1188:5
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.
Error: exit status 101
Error: run /home/tidb/.tiup/components/ctl/v4.0.10/ctl (wd:/home/tidb/.tiup/data/SMzJL84) failed: exit status 1

5

[2021/01/24 10:55:43.682 +08:00] [ERROR] [sst_importer.rs:92] [“ingest failed”] [err_code=KV:SSTImporter:RocksDB] [err=“RocksDB read metadata from /tidb-data/tikv-20160/import/2aa266b6-398f-4b6d-926d-799c21c30a30_142905_5_4862_write.sst: Os { code: 2, kind: NotFound, message: “No such file or directory” }”] [meta=“uuid: 2AA266B6398F4B6D926D799C21C30A30 range { start: 7480000000000004FF705F7280000000B4FF5346710000000000FA end: 7480000000000004FF705F7280000000B4FF6DB0640000000000FA } cf_name: “write” region_id: 142905 region_epoch { conf_ver: 5 version: 4862 }”]
[2021/01/24 10:55:43.682 +08:00] [INFO] [mod.rs:335] [“starting working thread”] [worker=consistency-check]
[2021/01/24 10:55:43.683 +08:00] [WARN] [store.rs:1272] [“set thread priority for raftstore failed”] [error=“Os { code: 13, kind: PermissionDenied, message: “Permission denied” }”]
[2021/01/24 10:55:43.683 +08:00] [INFO] [node.rs:173] [“put store to PD”] [store=“id: 5 address: “172.16.3.83:20160” version: “4.0.10” status_address: “172.16.3.83:20180” git_hash: “2ea4e608509150f8110b16d6e8af39284ca6c93a” start_timestamp: 1611456941 deploy_path: “/tidb-deploy/tikv-20160/bin””]

6

目前看就这个region_id: 142905 有问题，但是tikv ctl工具用不了。

7

这个命令要 在 tikv 实例本地执行。并满足 对应的 服务器权限

8

已解决，感谢支持。

9

:+1:

10

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。