v6.5.0主库集群，使用br备份2张大表，再到v6.1.5版本集群从库，进行br恢复时，报错：

🪐 TiDB 技术问题
1

【 TiDB 使用环境】生产环境
【 TiDB 版本】主库集群:v6.5.0,集群从库:v6.1.5
【复现路径】
1.在主库集群上操作，使用tiup br backup full --filter 备份2张大表
2.然后，将备份到每个kv节点的备份文件，全部拷贝到共享盘上
3.再到从库集群上操作，把共享盘上的备份文件，拷贝给从库的每个kv节点
4.拷贝共享盘上的backupmeta放到从库的中控机上
5.执行tiup br restore full --filter ，进行恢复

【遇到的问题：问题现象及影响】报错如下：

Full Restore <----------------------------------------------------------------------------------------------> 100.00%
[2023/04/14 12:21:46.093 +08:00] [INFO] [collector.go:69] ["Full Restore failed summary"] [total-ranges=1] [ranges-succeed=0] [ranges-failed=1] [split-region=10.743447468s] [restore-ranges=2090] [unit-name=file] [error="No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed"] [errorVerbose="the following errors occurred:\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594\n -  [BR:KV:ErrKVDownloadFailed]download sst failed\n    No such file or directory (os error 2)\n    github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n    \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n    golang.org/x/sync/errgroup.(*Group).Go.func1\n    \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n    runtime.goexit\n    \t/usr/local/go/src/runtime/asm_amd64.s:1594"]
Error: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed

【资源配置】
【附件：截图/日志/监控】

2

Full Restore <----------------------------------------------------------------------------------------------> 100.00%
[2023/04/14 12:21:46.093 +08:00] [INFO] [collector.go:69] [“Full Restore failed summary”] [total-ranges=1] [ranges-succeed=0] [ranges-failed=1] [split-region=10.743447468s] [restore-ranges=2090] [unit-name=file] [error=“No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed”] [errorVerbose=“the following errors occurred:\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:607\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594”]
Error: No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed

3

看恢复步骤没撒问题，但是报错又是提示找不到文件，不知道你备份还原的路径是否指向是对的。

4

备份的路径和恢复的路径，都没有问题。这个问题比较奇葩，我先使用Dumpling备份和tidb-lingining恢复。br不稳定。

5

恢复时备份所在的目录有访问权限吗

6

您提供的错误消息表示 BR（备份和还原）恢复过程在下载 SST（SSTable）文件期间遇到故障。 具体错误是“没有这样的文件或目录（操作系统错误 2）”。

要解决此问题并成功完成还原，您可以执行以下步骤：

验证备份文件的可用性：确保复制到从库的每个KV节点的备份文件是可访问的，并且位于预期的目录中。 检查是否有任何不一致或丢失的文件。

检查文件权限和所有权：验证运行 BR 还原过程的用户是否具有足够的权限来访问和读取备份文件。 确保正确设置备份文件的所有权和权限。

确认backupmeta文件：仔细检查放置在从库中控机上的backupmeta文件。 确保文件有效并包含恢复过程所需的元数据。

验证网络连通性：确保中控机与从库KV节点之间的网络连通稳定可靠。 确认没有导致下载失败的网络相关问题。

检查存储容量：确认从库的KV节点有足够的存储容量来容纳恢复的数据。 磁盘空间不足会导致下载失败和错误。

查看 BR 还原配置：查看 BR 还原过程中提供的配置参数。 注意配置中指定的存储路径和文件位置。 确保它们与备份文件的实际设置和路径相匹配。

查阅 BR 文档和社区：请参阅 TiDB 的官方备份和还原文档以获取更详细的说明和故障排除步骤。 您还可以从 TiDB 社区或论坛寻求帮助，有经验的用户和开发人员可以在其中提供指导并帮助解决特定问题。

通过仔细检查和解决以上几点，您应该能够识别并解决导致 BR 恢复过程中下载失败的问题。

7

备份的文件系统是什么 ？ 目前 您可能 仅仅用 nfs 吧

8

您搭建nfs 没。 是否挂在到各个 节点上了呢？