【 TiDB 使用环境】新集群
【 TiDB 版本】V6.5.2
【复现路径】
现有两套tidb集群环境,均为内网环境,版本v6.5.2,操作如下:
老集群:备份
挂载NFS磁盘,老集群BR所在机器及所有TIKV节点新建文件目录/br,均挂载NFS,
mount -t nfs xxx.xxx.xxx.148:/data /br
在上述/data目录下新建br_back目录,已使用各节点做读写操作,均通过。
root用户执行备份语句如下(br所在目录):
./br backup full -pd “xx.xx.xx.39:2379” -storage “local:///br/br_backup” -ratelimit 128 -log-file brbackup.log
执行完成后,显示
Full Backup <----->100.00%,
Checksum <…>100.00%
检查日志,未发现错误。
恢复到新集群:
br所在节点(中控机),所有tikv节点,均新建目录 /data 挂载NFS目录
mount -t nfs xxx.xxx.xxx.xxx:/data /data
测试读写没有问题,查看/br_backup文件夹及其下所有目录,在所有新集群节点授权文件夹root权限,
chown -R root:root /data/br_backup
按
https://docs.pingcap.com/zh/tidb/stable/backup-and-restore-faq#遇到-permission-denied-或者-no-such-file-or-directory-错误即使用-root-运行-br-命令行工具也无法解决该如何处理
文档,检查集群用户
ps aux | grep tikv-server
root
查询启动信息
tiup cluster list
user显示为tidb,但每次启动我使用的都是root用户,这里很奇怪。
root用户执行恢复语句(新集群br所在节点,中控机):
./br restore full -s local:///data/br_backup/ --pd xx.xx.xx.207:2379 --ratelimit 128 --log-file br-restore.log
【遇到的问题:问题现象及影响】
BR恢复启动后,进度条显示,大概10.00%左右,显示报错,
[2025/02/26 11:53:05.067 +08:00] [ERROR] [main.go:59] [“br failed”] [error=“No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed; No such file or directory (os error 2): [BR:KV:ErrKVDownloadFailed]download sst failed”] [errorVerbose=“the following errors occurred:\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:663\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:663\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:663\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:663\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:663\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:663\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:663\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594\n - [BR:KV:ErrKVDownloadFailed]download sst failed\n No such file or directory (os error 2)\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).downloadSST.func1\n \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:663\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1594”] [stack=“main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:59\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250”]
详细日志见附件
br-restore.log (15.2 MB)