br 备份报错

【 TiDB 使用环境】生产环境
【 TiDB 版本】
【复现路径】做过哪些操作出现的问题
【遇到的问题:问题现象及影响】
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【附件:截图/日志/监控】
PD 节点报的问题

PD 节点是正常的

br备份
[“br failed”] [error=“cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found; cannot find region from pd client: [BR:PD:ErrPDLeaderNotFound]PD leader not found”] [errorVerbose=“the following errors occurred:\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDLeaderNotFound]PD leader not found\n cannot find region from pd client\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1023\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:234\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:215\n github.com/pingcap/tidb/br/pkg/backup.(*Client).FindTargetPeer\n \t/workspace/source/tidb/br/pkg/backup/client.go:973\n github.com/pingcap/tidb/br/pkg/backup.(*Client).handleFineGrained\n \t/workspace/source/tidb/br/pkg/backup/client.go:1228\n github.com/pingcap/tidb/br/pkg/backup.(*Client).fineGrainedBackup.func2\n \t/workspace/source/tidb/br/pkg/backup/client.go:1107\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650”] [stack=“main.main\n\t/workspace/source/tidb/br/cmd/br/main.go:38\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267”]

看看 2379 2380 端口都通吗,br 到 pd 的,和 pd 间的

都是通的

这报错看起来是连不上PD哎,地址没写错吧,网络都再测测是不是都通

找不到某个 region 的 leader 有很多可能,比如 TiKV 到 PD 的网络不通畅,gRPC 线程被阻塞,或者 TiKV 太忙导致心跳受阻。

楼主的日志里面还有说找不到 PD leader,因此也可能是 PD 集群出了问题,可以看看 PD 日志里面有没有 panic 以及 PD 有没有重启一类的。


对于 region 找不到 leader 的具体情况需要找到出问题的 region:

pd-ctl region key $key

(这里会需要一个 hex string,你可能得想办法把 PD 日志打出来的 key 转化一下。)
随后找到 region 的 leader 所在的 TiKV,搜索一下相关的日志,然后看看有没有什么异常:

grep -E "(region|id)( ?| ?= ?|: ?)$region_id"

但是考虑到备份已经过去一段时间了,region 的分布可能也产生了变化。找到具体原因应该比较困难了,如果后续备份不再失败可以不管它。