使用BR备份整库快照到本地后,进行恢复操作失败

【 TiDB 使用环境】测试
【 TiDB 版本】v6.3.0
【复现路径】做过哪些操作出现的问题
1、使用官方提供的LUA脚本在sbtest库里初始化4张表并插入数据,四张表大概一千万的样子
2、使用br备份sbtest,命令如下:

tiup br backup db --pd "192.168.2.124:2379" \
	--db sbtest \
	--backupts '2022-11-21 02:00:00' \ 
	--storage "local:///usr/local/tidb/backup/tidb-test" \
	--ratelimit 128 \
	--log-file /usr/local/tidb/backup/backup.log

3、使用drop命令删除sbtest1表
4、使用如下命令恢复sbtest1

tiup br restore table --pd "192.168.2.124:2379" --db sbtest --table sbtest1 --storage "local:///usr/local/tidb/backup/tidb-test" --log-file restoredb.log

【遇到的问题:问题现象及影响】
恢复失败,报找不到某文件,报错日志如下:

[2022/11/21 01:46:13.948 +08:00] [INFO] [collector.go:69] ["Table Restore failed summary"] [total-ranges=1] [ranges-succeed=0] [ranges-failed=1] [split-region=684.545µs] [restore-ranges=21] [unit-name=file] [error="Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20160/import/.temp/c68f42e9-409f-4cd7-b672-4403c1241890_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/af8dbe53-14cc-485b-a40d-883918274417_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/982a35e7-ff00-4765-a003-d322145c4228_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/675c70db-8712-4c0c-9770-efdf66d69a1a_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/3287a30f-bee7-4356-b8e8-49b7c2538e35_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f3819034-8ff5-40be-a84d-df3e4adaa9b3_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f8251c07-f628-4f0e-bdc5-2a2ddca3b063_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/c50cb2d5-5df1-4998-b0c5-da95fdf4009a_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed"] [errorVerbose="the following errors occurred:\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20160/import/.temp/c68f42e9-409f-4cd7-b672-4403c1241890_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/af8dbe53-14cc-485b-a40d-883918274417_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/982a35e7-ff00-4765-a003-d322145c4228_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/675c70db-8712-4c0c-9770-efdf66d69a1a_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/3287a30f-bee7-4356-b8e8-49b7c2538e35_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f3819034-8ff5-40be-a84d-df3e4adaa9b3_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f8251c07-f628-4f0e-bdc5-2a2ddca3b063_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/c50cb2d5-5df1-4998-b0c5-da95fdf4009a_11706_17_83_write.sst: No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594"]
Error: Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20160/import/.temp/c68f42e9-409f-4cd7-b672-4403c1241890_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/af8dbe53-14cc-485b-a40d-883918274417_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/982a35e7-ff00-4765-a003-d322145c4228_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/675c70db-8712-4c0c-9770-efdf66d69a1a_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/3287a30f-bee7-4356-b8e8-49b7c2538e35_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f3819034-8ff5-40be-a84d-df3e4adaa9b3_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/f8251c07-f628-4f0e-bdc5-2a2ddca3b063_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read local:///usr/local/tidb/backup/tidb-test/11001/11256_65_cebebc7e293a1f630f152f34070ec9240aff412a2a384700f8b2b9d8c4c7560a_1668965802925_write.sst into /usr/local/tidb/tidb-data/tikv-20161/import/.temp/c50cb2d5-5df1-4998-b0c5-da95fdf4009a_11706_17_83_write.sst: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed
root@k8s-master:/usr/local/tidb/backup/tidb-test#

【资源配置】
【附件:截图/日志/监控】
集群拓扑信息:

看了下备份的目录,只有如下截图文件,确实没有日志报错提示的文件,比较奇怪为什么会找这个文件

Backup & Restore 常见问题 | PingCAP Docs

找的那个文件是个sst 文件,存储的应该是数据,是不是备份的时候没有备份下来数据,看tikv有两个节点,两台服务器都有创建备份目录吗?

备份时可以备份到每个tikv的相同目录,恢复时必须在该目录下有其他所有tikv的备份数据,任何一个tikv上的备份目录都得有全部备份数据

嗯嗯,都创建了备份目录了,之前我备份的时候发现其余两个节点的备份目录并没有文件,就没管,刚才看了下有了,可能是之前的备份的有问题,问题已经解决了,谢谢

非常感谢,昨晚在做备份的时候忘了这茬了,之前做过一次备份,其余两个节点上的备份目录并没有数据,应该是备份的有问题,这次可以了,非常感谢

嗯嗯,可以了,非常感谢

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。