【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】
【复现路径】做过哪些操作出现的问题
使用br 进行s3备份，备份到一半报错
br命令如下：
export AWS_ACCESS_KEY_ID=******************** && export AWS_SECRET_ACCESS_KEY=******************** && ./br backup full --log-file ./brlog/backupfull.log-instance-164672146297720230512T010635H --pd http://10.0.0.1:2379 --ratelimit 250 --s3.endpoint http://10.0.0.132:7480 --storage s3://tidb-backup/instance1646721462977-20230512T010635H
【遇到的问题：问题现象及影响】
每次都报错，并且每次报错的store和region不一样
br日志：
[2023/05/12 02:58:43.826 +08:00] [ERROR] [main.go:58] ["br failed"] [error="error happen in store 126 at 10.0.0.7:20160: Io(Custom { kind: Other, error: \"failed to put object Request ID: None Body: <?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>RequestTimeout</Code><BucketName>tidb-backup</BucketName><RequestId>tx00000000000000aff0e61-00645d3a09-fbf7c7</RequestId><HostId>fbf7c7-hubeijiangxiazone1-hubeijiangxiazone1-zonegroup</HostId></Error>\" }): [BR:KV:ErrKVStorage]tikv storage occur I/O error"] [errorVerbose="[BR:KV:ErrKVStorage]tikv storage occur I/O error\nerror happen in store 126 at 10.0.0.7:20160: Io(Custom { kind: Other, error: \"failed to put object Request ID: None Body: <?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>RequestTimeout</Code><BucketName>tidb-backup</BucketName><RequestId>tx00000000000000aff0e61-00645d3a09-fbf7c7</RequestId><HostId>fbf7c7-zonegroup</HostId></Error>\" })\ngithub.com/pingcap/tidb/br/pkg/backup.(*pushDown).pushBackup\n\t/data1/jenkins/workspace/tibr/br/pkg/backup/push.go:176\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRange\n\t/data1/jenkins/workspace/tibr/br/pkg/backup/client.go:499\ngithub.com/pingcap/tidb/br/pkg/backup.(*Client).BackupRanges.func1\n\t/data1/jenkins/workspace/tibr/br/pkg/backup/client.go:453\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/data1/jenkins/workspace/tibr/br/pkg/utils/worker.go:73\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/data1/debezium/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go1.16/src/runtime/asm_amd64.s:1371"] [stack="main.main\n\t/data1/jenkins/workspace/tibr/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go1.16/src/runtime/proc.go:225"]
tikv的日志：
[2023/05/12 02:58:43.818 +08:00] [ERROR] [endpoint.rs:828] ["backup region failed"] [err_code=KV:Unknown] [err="Io(Custom { kind: Other, error: \"failed to put object Request ID: None Body: <?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>RequestTimeout</Code><BucketName>tidb-backup</BucketName><RequestId>tx00000000000000aff0e61-00645d3a09-fbf7c7</RequestId><HostId>fbf7c7-zonegroup</HostId></Error>\" })"] [end_key=7480000000000000705F72A0000000666073F5] [start_key=7480000000000000705F72A0000000664D78DC] [region="id: 343115 start_key: 7480000000000000FF705F72A000000066FF4D78DC0000000000FA end_key: 7480000000000000FF705F72A000000066FF6073F50000000000FA region_epoch { conf_ver: 1631 version: 3540 } peers { id: 343117 store_id: 120 } peers { id: 343118 store_id: 126 } peers { id: 562292 store_id: 122 }"]
【资源配置】
【附件：截图/日志/监控】