【 TiDB 使用环境】生产\测试环境\ POC
【 TiDB 版本】 5.1.4
【遇到的问题】
【复现路径】做过哪些操作出现的问题
【问题现象及影响】
使用tidb备份数据 到海外的s3
/home/tidb/bin/br backup full --pd xxxx:2379 --storage s3://x-x-backup-tokyo/90736/ --s3.region ‘tokyo’ --ratelimit 120 --concurrency 2 --check-requirements=false --s3.endpoint http://s3.ap-northeast-1.amazonaws.com
由于我的aws s3根iam绑定了 所以我在地执行aws s3 ls 等其他命令可以正常执行
我使用br备份后，s3路径下有backup.lock产生
但是数据在tikv节点上传到s3时返回s3的报错
[error="rpc error: code = Canceled desc = context canceled"] [errorVerbose="rpc error: code = Canceled desc = context canceled\
github.com/tikv/pd/client.(*client).GetAllStores\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20210323121136-78679e5e209d/client/client.go:1196\
github.com/pingcap/br/pkg/conn.GetAllTiKVStores\
\tgithub.com/pingcap/br/pkg/conn/conn.go:142\
github.com/pingcap/br/pkg/conn.GetAllTiKVStoresWithRetry.func1\
\tgithub.com/pingcap/br/pkg/conn/conn.go:180\
github.com/pingcap/br/pkg/utils.WithRetry\
\tgithub.com/pingcap/br/pkg/utils/retry.go:47\
github.com/pingcap/br/pkg/conn.GetAllTiKVStoresWithRetry\
\tgithub.com/pingcap/br/pkg/conn/conn.go:177\
github.com/pingcap/br/pkg/backup.(*Client).BackupRange\
\tgithub.com/pingcap/br/pkg/backup/client.go:459\
github.com/pingcap/br/pkg/backup.(*Client).BackupRanges.func1\
\tgithub.com/pingcap/br/pkg/backup/client.go:424\
github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\
\tgithub.com/pingcap/br/pkg/utils/worker.go:63\
golang.org/x/sync/errgroup.(*Group).Go.func1\
\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\
runtime.goexit\
\truntime/asm_amd64.s:1371"] [unit-name="range start:7480000000000000155f69800000000000000100 end:7480000000000000155f698000000000000001fb"] [error="rpc error: code = Canceled desc = context canceled"] [errorVerbose="rpc error: code = Canceled desc = context canceled\
github.com/tikv/pd/client.(*client).GetAllStores\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20210323121136-78679e5e209d/client/client.go:1196\
github.com/pingcap/br/pkg/conn.GetAllTiKVStores\
\tgithub.com/pingcap/br/pkg/conn/conn.go:142\
github.com/pingcap/br/pkg/conn.GetAllTiKVStoresWithRetry.func1\
\tgithub.com/pingcap/br/pkg/conn/conn.go:180\
github.com/pingcap/br/pkg/utils.WithRetry\
\tgithub.com/pingcap/br/pkg/utils/retry.go:47\
github.com/pingcap/br/pkg/conn.GetAllTiKVStoresWithRetry\
\tgithub.com/pingcap/br/pkg/conn/conn.go:177\
github.com/pingcap/br/pkg/backup.(*Client).BackupRange\
\tgithub.com/pingcap/br/pkg/backup/client.go:459\
github.com/pingcap/br/pkg/backup.(*Client).BackupRanges.func1\
\tgithub.com/pingcap/br/pkg/backup/client.go:424\
github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\
\tgithub.com/pingcap/br/pkg/utils/worker.go:63\
golang.org/x/sync/errgroup.(*Group).Go.func1\
\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\
runtime.goexit\
\truntime/asm_amd64.s:1371"] [unit-name="range start:7480000000000000195f720000000000000000 end:7480000000000000195f72ffffffffffffffff00"] [error="rpc error: code = Canceled desc = context canceled"] [errorVerbose="rpc error: code = Canceled desc = context canceled\
github.com/tikv/pd/client.(*client).GetAllStores\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20210323121136-78679e5e209d/client/client.go:1196\
github.com/pingcap/br/pkg/conn.GetAllTiKVStores\
\tgithub.com/pingcap/br/pkg/conn/conn.go:142\
github.com/pingcap/br/pkg/conn.GetAllTiKVStoresWithRetry.func1\
\tgithub.com/pingcap/br/pkg/conn/conn.go:180\
github.com/pingcap/br/pkg/utils.WithRetry\
\tgithub.com/pingcap/br/pkg/utils/retry.go:47\
github.com/pingcap/br/pkg/conn.GetAllTiKVStoresWithRetry\
\tgithub.com/pingcap/br/pkg/conn/conn.go:177\
github.com/pingcap/br/pkg/backup.(*Client).BackupRange\
\tgithub.com/pingcap/br/pkg/backup/client.go:459\
github.com/pingcap/br/pkg/backup.(*Client).BackupRanges.func1\
\tgithub.com/pingcap/br/pkg/backup/client.go:424\
github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\
\tgithub.com/pingcap/br/pkg/utils/worker.go:63\
golang.org/x/sync/errgroup.(*Group).Go.func1\
\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\
runtime.goexit\
\truntime/asm_amd64.s:1371"] [unit-name="range start:74800000000000001b5f720000000000000000 end:74800000000000001b5f72ffffffffffffffff00"] [error="rpc error: code = Canceled desc = context canceled"] [errorVerbose="rpc error: code = Canceled desc = context canceled\
github.com/tikv/pd/client.(*client).GetAllStores\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20210323121136-78679e5e209d/client/client.go:1196\
github.com/pingcap/br/pkg/conn.GetAllTiKVStores\
\tgithub.com/pingcap/br/pkg/conn/conn.go:142\
github.com/pingcap/br/pkg/conn.GetAllTiKVStoresWithRetry.func1\
\tgithub.com/pingcap/br/pkg/conn/conn.go:180\
github.com/pingcap/br/pkg/utils.WithRetry\
\tgithub.com/pingcap/br/pkg/utils/retry.go:47\
github.com/pingcap/br/pkg/conn.GetAllTiKVStoresWithRetry\
\tgithub.com/pingcap/br/pkg/conn/conn.go:177\
github.com/pingcap/br/pkg/backup.(*Client).BackupRange\
\tgithub.com/pingcap/br/pkg/backup/client.go:459\
github.com/pingcap/br/pkg/backup.(*Client).BackupRanges.func1\
\tgithub.com/pingcap/br/pkg/backup/client.go:424\
github.com/pingcap/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\
\tgithub.com/pingcap/br/pkg/utils/worker.go:63\
golang.org/x/sync/errgroup.(*Group).Go.func1\
\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\
runtime.goexit\
\truntime/asm_amd64.s:1371"]
Error: error happen in store 1 at 10.67.103.198:20160: Io(Custom { kind: Other, error: "failed to put object Request ID: None Body: <?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>ASIAQ5JMF4VBYY5FP5C7</AWSAccessKeyId><RequestId>GN9P0YA4BNQ1G65S</RequestId><HostId>T3Lj/lDuZbbFKek/AMAcNLwX0I6n4tBYedwMmMkamLStlmsMjCxVPdDTNy6DLUe4Gvom3lpSarM=</HostId></Error>" }): [BR:KV:ErrKVStorage]tikv storage occur I/O error
Error: error happen in store 1 at 10.67.103.198:20160: Io(Custom { kind: Other, error: "failed to put object Request ID: None Body: <?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>ASIAQ5JMF4VBYY5FP5C7</AWSAccessKeyId><RequestId>GN9P0YA4BNQ1G65S</RequestId><HostId>T3Lj/lDuZbbFKek/AMAcNLwX0I6n4tBYedwMmMkamLStlmsMjCxVPdDTNy6DLUe4Gvom3lpSarM=</HostId></Error>" }): [BR:KV:ErrKVStorage]tikv storage occur I/O error