【 TiDB 使用环境】测试
【 TiDB 版本】v6.3.0
【复现路径】做过哪些操作出现的问题
1、使用官方提供的LUA脚本在sbtest库里初始化4张表并插入数据，四张表大概一千万的样子
2、使用br备份sbtest，命令如下：
tiup br backup db --pd "192.168.2.124:2379" \
--db sbtest \
--backupts '2022-11-21 02:00:00' \
--storage "local:///usr/local/tidb/backup/tidb-test" \
--ratelimit 128 \
--log-file /usr/local/tidb/backup/backup.log
3、使用drop命令删除sbtest1表
4、使用如下命令恢复sbtest1
tiup br restore table --pd "192.168.2.124:2379" --db sbtest --table sbtest1 --storage "local:///usr/local/tidb/backup/tidb-test" --log-file restoredb.log
【遇到的问题：问题现象及影响】
恢复失败，报找不到某文件，报错日志如下：
[2022/11/21 01:46:13.948 +08:00] [INFO] [collector.go:69] ["Table Restore failed summary"] [total-ranges=1] [ranges-succeed=0] [ranges-failed=1] [split-region=684.545µs] [restore-ranges=21] [unit-name=file] [error="Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20160/import/.temp/c68f42e9-409f-4cd7-b672-4403c1241890_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/af8dbe53-14cc-485b-a40d-883918274417_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/982a35e7-ff00-4765-a003-d322145c4228_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/675c70db-8712-4c0c-9770-efdf66d69a1a_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/3287a30f-bee7-4356-b8e8-49b7c2538e35_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f3819034-8ff5-40be-a84d-df3e4adaa9b3_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f8251c07-f628-4f0e-bdc5-2a2ddca3b063_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/c50cb2d5-5df1-4998-b0c5-da95fdf4009a_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed"] [errorVerbose="the following errors occurred:\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20160/import/.temp/c68f42e9-409f-4cd7-b672-4403c1241890_11706_17_83_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/af8dbe53-14cc-485b-a40d-883918274417_11706_17_83_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/982a35e7-ff00-4765-a003-d322145c4228_11706_17_83_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/675c70db-8712-4c0c-9770-efdf66d69a1a_11706_17_83_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/3287a30f-bee7-4356-b8e8-49b7c2538e35_11706_17_83_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f3819034-8ff5-40be-a84d-df3e4adaa9b3_11706_17_83_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f8251c07-f628-4f0e-bdc5-2a2ddca3b063_11706_17_83_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/c50cb2d5-5df1-4998-b0c5-da95fdf4009a_11706_17_83_write.sst: No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594"]
Error: Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20160/import/.temp/c68f42e9-409f-4cd7-b672-4403c1241890_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/af8dbe53-14cc-485b-a40d-883918274417_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/982a35e7-ff00-4765-a003-d322145c4228_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/675c70db-8712-4c0c-9770-efdf66d69a1a_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/3287a30f-bee7-4356-b8e8-49b7c2538e35_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f3819034-8ff5-40be-a84d-df3e4adaa9b3_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f8251c07-f628-4f0e-bdc5-2a2ddca3b063_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/c50cb2d5-5df1-4998-b0c5-da95fdf4009a_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed
root@k8s-master:/usr/local/tidb/backup/tidb-test#
【资源配置】
【附件：截图/日志/监控】
集群拓扑信息：
