br备份数据时一直卡

TiDBer_LusPLVRz · 2024 年11 月 22 日 03:18

在做数据备份时一直卡住不动

场景：
将tidb中指定表备份到MINIO上

异常信息：
[“backup occur storage error”] [range-sn=10] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]

命令：
tiup br backup table --pd “10.x.x.12:2379” --db test3 --table user_userinfo -s “s3://tidbbackup/snapshot-20241113-02?access-key=xxx&secret-access-key=xxx&endpoint=http://172.16.x.x:9032&force-path-style=true” --concurrency=20 --log-file user_userinfo_bak.log

日志：
[2024/11/22 10:27:40.613 +08:00] [INFO] [info.go:49] [“Welcome to Backup & Restore (BR)”] [release-version=v7.5.0] [git-hash=069631e2ecfedc000ffb92c67207bea81380f020] [git-branch=heads/refs/tags/v7.5.0] [go-version=go1.21.3] [utc-build-time=“2023-11-24 08:42:41”] [race-enabled=false]
[2024/11/22 10:27:40.613 +08:00] [INFO] [common.go:754] [arguments] [__command=“br backup table”] [concurrency=20] [db=test3] [log-file=user_userinfo_bak2.log] [pd=“[10.1.1.12:2379]”] [storage=s3://tidbbackup/snapshot-20241113-02] [table=user_userinfo]
[2024/11/22 10:27:40.616 +08:00] [INFO] [conn.go:149] [“new mgr”] [pdAddrs=10.1.1.12:2379]
[2024/11/22 10:27:40.624 +08:00] [INFO] [pd_service_discovery.go:606] [“[pd] update member urls”] [old-urls=“[http://10.1.1.12:2379]”] [new-urls=“[http://10.0.186.11:2379,http://10.0.186.75:2379,http://10.1.1.12:2379]”]
[2024/11/22 10:27:40.624 +08:00] [INFO] [pd_service_discovery.go:632] [“[pd] switch leader”] [new-leader=http://10.0.186.75:2379] [old-leader=]
[2024/11/22 10:27:40.624 +08:00] [INFO] [pd_service_discovery.go:197] [“[pd] init cluster id”] [cluster-id=7320182539641244320]
[2024/11/22 10:27:40.625 +08:00] [INFO] [client.go:600] [“[pd] changing service mode”] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2024/11/22 10:27:40.625 +08:00] [INFO] [tso_client.go:230] [“[tso] switch dc tso global allocator serving address”] [dc-location=global] [new-address=http://10.0.186.75:2379]
[2024/11/22 10:27:40.626 +08:00] [INFO] [tso_dispatcher.go:318] [“[tso] tso dispatcher created”] [dc-location=global]
[2024/11/22 10:27:40.626 +08:00] [INFO] [client.go:648] [“[pd] service mode changed”] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2024/11/22 10:27:40.629 +08:00] [INFO] [conn.go:123] [“checked alive KV stores”] [aliveStores=28] [totalStores=28]
[2024/11/22 10:27:40.633 +08:00] [INFO] [pd_service_discovery.go:606] [“[pd] update member urls”] [old-urls=“[http://10.1.1.12:2379]”] [new-urls=“[http://10.0.186.11:2379,http://10.0.186.75:2379,http://10.1.1.12:2379]”]
[2024/11/22 10:27:40.633 +08:00] [INFO] [pd_service_discovery.go:632] [“[pd] switch leader”] [new-leader=http://10.0.186.75:2379] [old-leader=]
[2024/11/22 10:27:40.633 +08:00] [INFO] [pd_service_discovery.go:197] [“[pd] init cluster id”] [cluster-id=7320182539641244320]
[2024/11/22 10:27:40.634 +08:00] [INFO] [client.go:600] [“[pd] changing service mode”] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2024/11/22 10:27:40.634 +08:00] [INFO] [tso_client.go:230] [“[tso] switch dc tso global allocator serving address”] [dc-location=global] [new-address=http://10.0.186.75:2379]
[2024/11/22 10:27:40.635 +08:00] [INFO] [tso_dispatcher.go:318] [“[tso] tso dispatcher created”] [dc-location=global]
[2024/11/22 10:27:40.635 +08:00] [INFO] [client.go:648] [“[pd] service mode changed”] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2024/11/22 10:27:40.636 +08:00] [INFO] [tikv_driver.go:200] [“using API V1.”]
[2024/11/22 10:27:40.637 +08:00] [INFO] [tidb.go:80] [“new domain”] [store=tikv-7320182539641244320] [“ddl lease”=1s] [“stats lease”=-1ns] [“index usage sync lease”=0s]
[2024/11/22 10:27:40.668 +08:00] [WARN] [info.go:302] [“init TiFlashReplicaManager”] [“pd addrs”=“[10.0.186.11:2379,10.0.186.75:2379,10.1.1.12:2379]”]
[2024/11/22 10:27:40.759 +08:00] [INFO] [domain.go:2754] [acquireServerID] [serverID=813] [“lease id”=495690fcaa7219b5]
[2024/11/22 10:27:40.770 +08:00] [INFO] [controller.go:163] [“load resource controller config”] [config=“{"degraded-mode-wait-duration":"0s","ltb-max-wait-duration":"30s","request-unit":{"read-base-cost":0.125,"read-per-batch-base-cost":0.5,"read-cost-per-byte":0.0000152587890625,"write-base-cost":1,"write-per-batch-base-cost":1,"write-cost-per-byte":0.0009765625,"read-cpu-ms-cost":0.3333333333333333}}”] [ru-config=“{"ReadBaseCost":0.125,"ReadPerBatchBaseCost":0.5,"ReadBytesCost":0.0000152587890625,"WriteBaseCost":1,"WritePerBatchBaseCost":1,"WriteBytesCost":0.0009765625,"CPUMsCost":0.3333333333333333,"LTBMaxWaitDuration":30000000000,"DegradedModeWaitDuration":0}”]
[2024/11/22 10:27:40.772 +08:00] [INFO] [region_cache.go:2682] [“change store resolve state”] [store=3688215] [addr=10.1.1.15:20161] [from=unresolved] [to=resolved] [liveness-state=reachable]
[2024/11/22 10:27:40.772 +08:00] [INFO] [region_cache.go:2682] [“change store resolve state”] [store=3688220] [addr=10.1.1.19:20160] [from=unresolved] [to=resolved] [liveness-state=reachable]
[2024/11/22 10:27:40.772 +08:00] [INFO] [region_cache.go:2682] [“change store resolve state”] [store=3688230] [addr=10.1.1.16:20163] [from=unresolved] [to=resolved] [liveness-state=reachable]
[2024/11/22 10:27:40.785 +08:00] [INFO] [region_cache.go:2682] [“change store resolve state”] [store=3640845] [addr=10.1.1.20:20160] [from=unresolved] [to=resolved] [liveness-state=reachable]
[2024/11/22 10:27:40.785 +08:00] [INFO] [region_cache.go:2682] [“change store resolve state”] [store=3688221] [addr=10.1.1.19:20162] [from=unresolved] [to=resolved] [liveness-state=reachable]
[2024/11/22 10:27:40.797 +08:00] [INFO] [region_cache.go:2682] [“change store resolve state”] [store=3688219] [addr=10.1.1.14:20161] [from=unresolved] [to=resolved] [liveness-state=reachable]
[2024/11/22 10:27:42.403 +08:00] [INFO] [domain.go:295] [“full load InfoSchema success”] [currentSchemaVersion=0] [neededSchemaVersion=62666] [“start time”=1.619229118s]
[2024/11/22 10:27:42.438 +08:00] [INFO] [domain.go:610] [“full load and reset schema validator”]
[2024/11/22 10:27:42.438 +08:00] [WARN] [domain.go:622] [“loading schema takes a long time”] [“take time”=1.667552099s]
[2024/11/22 10:27:42.438 +08:00] [INFO] [ddl.go:764] [“start DDL”] [category=ddl] [ID=3163d085-9ebb-48af-8c97-4b9a16915bf9] [runWorker=false]
[2024/11/22 10:27:42.438 +08:00] [INFO] [ddl.go:727] [“start delRangeManager OK”] [category=ddl] [“is a emulator”=false]
[2024/11/22 10:27:42.453 +08:00] [WARN] [env.go:53] [“initialize environment failed”] [category=ddl-ingest] [“storage limitation”=“only support TiKV storage”] [“current storage”=unistore] [“lightning is initialized”=false]
[2024/11/22 10:27:42.453 +08:00] [WARN] [domain.go:1247] [“loading schema takes a long time, we do a new reload”] [“take time”=1.682786248s]
[2024/11/22 10:27:42.455 +08:00] [INFO] [job_table.go:325] [“get global state and global state change”] [category=ddl] [oldState=false] [currState=false]
[2024/11/22 10:27:42.461 +08:00] [INFO] [wait_group_wrapper.go:130] [“background process started”] [source=domain] [process=loadSchemaInLoop]
[2024/11/22 10:27:42.461 +08:00] [INFO] [wait_group_wrapper.go:130] [“background process started”] [source=domain] [process=mdlCheckLoop]
[2024/11/22 10:27:42.461 +08:00] [INFO] [wait_group_wrapper.go:130] [“background process started”] [source=domain] [process=topNSlowQueryLoop]
[2024/11/22 10:27:42.461 +08:00] [INFO] [wait_group_wrapper.go:130] [“background process started”] [source=domain] [process=infoSyncerKeeper]
[2024/11/22 10:27:42.461 +08:00] [INFO] [wait_group_wrapper.go:130] [“background process started”] [source=domain] [process=globalConfigSyncerKeeper]
[2024/11/22 10:27:42.461 +08:00] [INFO] [wait_group_wrapper.go:130] [“background process started”] [source=domain] [process=runawayRecordFlushLoop]
[2024/11/22 10:27:42.461 +08:00] [INFO] [wait_group_wrapper.go:130] [“background process started”] [source=domain] [process=runawayWatchSyncLoop]
[2024/11/22 10:27:42.461 +08:00] [INFO] [wait_group_wrapper.go:130] [“background process started”] [source=domain] [process=closestReplicaReadCheckLoop]
[2024/11/22 10:27:42.461 +08:00] [INFO] [owner_daemon.go:70] [“begin advancer daemon”] [daemon-id=LogBackup::Advancer]
[2024/11/22 10:27:42.461 +08:00] [INFO] [manager.go:187] [“start campaign owner”] [ownerInfo=“[log-backup] /tidb/br-stream/owner”]
[2024/11/22 10:27:42.510 +08:00] [INFO] [wait_group_wrapper.go:130] [“background process started”] [source=domain] [process=logBackupAdvancer]
[2024/11/22 10:27:42.510 +08:00] [INFO] [owner_daemon.go:81] [“begin running daemon”] [id=2c5602ef-c8aa-49a6-8e43-7cf1afec5be3] [daemon-id=LogBackup::Advancer]
[2024/11/22 10:27:42.514 +08:00] [INFO] [region_cache.go:2682] [“change store resolve state”] [store=3688229] [addr=10.1.1.16:20160] [from=unresolved] [to=resolved] [liveness-state=reachable]
[2024/11/22 10:27:42.515 +08:00] [INFO] [region_cache.go:2682] [“change store resolve state”] [store=3640848] [addr=10.1.1.21:20163] [from=unresolved] [to=resolved] [liveness-state=reachable]
[2024/11/22 10:27:42.515 +08:00] [INFO] [region_cache.go:2682] [“change store resolve state”] [store=3688218] [addr=10.1.1.14:20163] [from=unresolved] [to=resolved] [liveness-state=reachable]
[2024/11/22 10:27:42.521 +08:00] [INFO] [backup.go:431] [“get new_collation_enabled config from mysql.tidb table”] [new_collation_enabled=True]
[2024/11/22 10:27:42.970 +08:00] [INFO] [region_cache.go:2682] [“change store resolve state”] [store=3640843] [addr=10.1.1.20:20161] [from=unresolved] [to=resolved] [liveness-state=reachable]
[2024/11/22 10:27:42.971 +08:00] [INFO] [region_cache.go:2682] [“change store resolve state”] [store=3688226] [addr=10.1.1.17:20161] [from=unresolved] [to=resolved] [liveness-state=reachable]
[2024/11/22 10:27:42.971 +08:00] [INFO] [region_cache.go:2682] [“change store resolve state”] [store=3640849] [addr=10.1.1.21:20162] [from=unresolved] [to=resolved] [liveness-state=reachable]
[2024/11/22 10:27:43.468 +08:00] [INFO] [ddl_workerpool.go:83] [“closing workerPool”] [category=ddl]
[2024/11/22 10:27:43.468 +08:00] [INFO] [ddl_workerpool.go:83] [“closing workerPool”] [category=ddl]
[2024/11/22 10:27:43.468 +08:00] [INFO] [delete_range.go:150] [“closing delRange”] [category=ddl]
[2024/11/22 10:27:43.468 +08:00] [INFO] [session_pool.go:99] [“closing session pool”] [category=ddl]
[2024/11/22 10:27:43.468 +08:00] [INFO] [ddl.go:887] [“DDL closed”] [category=ddl] [ID=3163d085-9ebb-48af-8c97-4b9a16915bf9] [“take time”=947.636049ms]
[2024/11/22 10:27:43.468 +08:00] [INFO] [ddl.go:719] [“stop DDL”] [category=ddl] [ID=3163d085-9ebb-48af-8c97-4b9a16915bf9]
[2024/11/22 10:27:43.493 +08:00] [INFO] [domain.go:2775] [“releaseServerID succeed”] [serverID=813]
[2024/11/22 10:27:43.493 +08:00] [INFO] [wait_group_wrapper.go:137] [“background process exited”] [source=domain] [process=runawayRecordFlushLoop]
[2024/11/22 10:27:43.493 +08:00] [INFO] [wait_group_wrapper.go:137] [“background process exited”] [source=domain] [process=mdlCheckLoop]
[2024/11/22 10:27:43.493 +08:00] [INFO] [domain.go:879] [“loadSchemaInLoop exited.”]
[2024/11/22 10:27:43.493 +08:00] [INFO] [domain.go:712] [“globalConfigSyncerKeeper exited.”]
[2024/11/22 10:27:43.493 +08:00] [INFO] [wait_group_wrapper.go:137] [“background process exited”] [source=domain] [process=runawayWatchSyncLoop]
[2024/11/22 10:27:43.493 +08:00] [INFO] [wait_group_wrapper.go:137] [“background process exited”] [source=domain] [process=globalConfigSyncerKeeper]
[2024/11/22 10:27:43.493 +08:00] [INFO] [manager.go:291] [“failed to campaign”] [“owner info”=“[log-backup] /tidb/br-stream/owner ownerManager 2c5602ef-c8aa-49a6-8e43-7cf1afec5be3”] [error=“lost watcher waiting for delete”]
[2024/11/22 10:27:43.493 +08:00] [INFO] [domain.go:686] [“infoSyncerKeeper exited.”]
[2024/11/22 10:27:43.493 +08:00] [INFO] [advancer.go:349] [“Meet task event”] [category=“log backup advancer”] [event=“Err(, err = EOF)”]
[2024/11/22 10:27:43.493 +08:00] [INFO] [wait_group_wrapper.go:137] [“background process exited”] [source=domain] [process=infoSyncerKeeper]
[2024/11/22 10:27:43.493 +08:00] [INFO] [domain.go:2839] [“serverIDKeeper exited.”]
[2024/11/22 10:27:43.493 +08:00] [INFO] [wait_group_wrapper.go:137] [“background process exited”] [source=domain] [process=loadSchemaInLoop]
[2024/11/22 10:27:43.493 +08:00] [ERROR] [advancer.go:352] [“listen task meet error, would reopen.”] [error=EOF] [stack=“github.com/pingcap/tidb/br/pkg/streamhelper.(*CheckpointAdvancer).StartTaskListener.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/streamhelper/advancer.go:352”]
[2024/11/22 10:27:43.493 +08:00] [INFO] [advancer.go:355] [“Task watcher exits due to some error.”] [category=“log backup advancer”] [error=EOF]
[2024/11/22 10:27:43.493 +08:00] [INFO] [domain.go:658] [“topNSlowQueryLoop exited.”]
[2024/11/22 10:27:43.494 +08:00] [INFO] [wait_group_wrapper.go:137] [“background process exited”] [source=domain] [process=topNSlowQueryLoop]
[2024/11/22 10:27:43.493 +08:00] [INFO] [owner_daemon.go:87] [“daemon loop exits”] [id=2c5602ef-c8aa-49a6-8e43-7cf1afec5be3] [daemon-id=LogBackup::Advancer]
[2024/11/22 10:27:43.494 +08:00] [INFO] [wait_group_wrapper.go:137] [“background process exited”] [source=domain] [process=logBackupAdvancer]
[2024/11/22 10:27:43.493 +08:00] [INFO] [manager.go:291] [“failed to campaign”] [“owner info”=“[log-backup] /tidb/br-stream/owner ownerManager 2c5602ef-c8aa-49a6-8e43-7cf1afec5be3”] [error=“context canceled”]
[2024/11/22 10:27:43.494 +08:00] [INFO] [domain.go:1330] [“closestReplicaReadCheckLoop exited.”]
[2024/11/22 10:27:43.494 +08:00] [INFO] [wait_group_wrapper.go:137] [“background process exited”] [source=domain] [process=closestReplicaReadCheckLoop]
[2024/11/22 10:27:43.494 +08:00] [INFO] [manager.go:256] [“etcd session is done, creates a new one”] [“owner info”=“[log-backup] /tidb/br-stream/owner ownerManager 2c5602ef-c8aa-49a6-8e43-7cf1afec5be3”]
[2024/11/22 10:27:43.494 +08:00] [INFO] [domain.go:1046] [“domain closed”] [“take time”=973.021188ms]
[2024/11/22 10:27:43.494 +08:00] [INFO] [glue.go:158] [“one shot domain closed”]
[2024/11/22 10:27:43.494 +08:00] [INFO] [glue.go:141] [“one shot session closed”]
[2024/11/22 10:27:43.494 +08:00] [INFO] [client.go:108] [“new backup client”]
[2024/11/22 10:27:43.494 +08:00] [INFO] [manager.go:260] [“break campaign loop, NewSession failed”] [“owner info”=“[log-backup] /tidb/br-stream/owner ownerManager 2c5602ef-c8aa-49a6-8e43-7cf1afec5be3”] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\ngithub.com/pingcap/tidb/pkg/util.contextDone\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/etcd.go:90\ngithub.com/pingcap/tidb/pkg/util.NewSession\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/etcd.go:50\ngithub.com/pingcap/tidb/pkg/owner.(*ownerManager).campaignLoop\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/owner/manager.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650”]
[2024/11/22 10:27:43.496 +08:00] [INFO] [s3.go:406] [“succeed to get bucket region from s3”] [“bucket region”=us-east-1]
[2024/11/22 10:27:43.498 +08:00] [WARN] [s3.go:509] [“failed to check object lock for bucket”] [bucket=tidbbackup] [error=“ObjectLockConfigurationNotFoundError: Object Lock configuration does not exist for this bucket\n\tstatus code: 404, request id: 180A2949C6FDF5A4, host id: 96d8bee26a55b6f5a060f077db2749007ab248b0e5b02bb2c06d38b5c2fdfd05”]
[2024/11/22 10:27:43.524 +08:00] [INFO] [backup.go:470] [“use checkpoint’s default GC TTL”] [“GC TTL”=4320]
[2024/11/22 10:27:43.526 +08:00] [INFO] [client.go:176] [“backup encode timestamp”] [BackupTS=454096968346238980]
[2024/11/22 10:27:43.526 +08:00] [INFO] [backup.go:492] [“current backup safePoint job”] [safePoint=“{ID=br-528f9255-3220-4b87-bcc7-d6e08637d985,TTL=1h12m0s,BackupTime="2024-11-22 10:27:43.479 +0800 CST",BackupTS=454096968346238980}”]
[2024/11/22 10:27:43.569 +08:00] [INFO] [backup.go:579] [“get placement policies”] [count=0]
[2024/11/22 10:27:43.593 +08:00] [INFO] [checkpoint.go:610] [“start to flush the checkpoint lock”] [lock-at=1732242463579] [expire-at=1732242763579]
[2024/11/22 10:27:44.743 +08:00] [INFO] [manager.go:316] [“revoke session”] [“owner info”=“[log-backup] /tidb/br-stream/owner ownerManager 2c5602ef-c8aa-49a6-8e43-7cf1afec5be3”] [error=“rpc error: code = Canceled desc = grpc: the client connection is closing”]
[2024/11/22 10:27:46.615 +08:00] [INFO] [client.go:850] [“Backup Ranges Started”] [ranges=“{total=15,ranges="[\"[74800000000000B0AA5F7201, 74800000000000B0AA5F72FB)\",\"(skip 13)\",\"[74800000000000B0AA5F69800000000000000E00, 74800000000000B0AA5F69800000000000000EFB)\"]",totalFiles=0,totalKVs=0,totalBytes=0,totalSize=0}”]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=14] [startKey=74800000000000B0AA5F69800000000000000E00] [endKey=74800000000000B0AA5F69800000000000000EFB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=6] [startKey=74800000000000B0AA5F69800000000000000600] [endKey=74800000000000B0AA5F698000000000000006FB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=10] [startKey=74800000000000B0AA5F69800000000000000A00] [endKey=74800000000000B0AA5F69800000000000000AFB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=8] [startKey=74800000000000B0AA5F69800000000000000800] [endKey=74800000000000B0AA5F698000000000000008FB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=12] [startKey=74800000000000B0AA5F69800000000000000C00] [endKey=74800000000000B0AA5F69800000000000000CFB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=3] [startKey=74800000000000B0AA5F69800000000000000300] [endKey=74800000000000B0AA5F698000000000000003FB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=4] [startKey=74800000000000B0AA5F69800000000000000400] [endKey=74800000000000B0AA5F698000000000000004FB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=13] [startKey=74800000000000B0AA5F69800000000000000D00] [endKey=74800000000000B0AA5F69800000000000000DFB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=0] [startKey=74800000000000B0AA5F7201] [endKey=74800000000000B0AA5F72FB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=7] [startKey=74800000000000B0AA5F69800000000000000700] [endKey=74800000000000B0AA5F698000000000000007FB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=11] [startKey=74800000000000B0AA5F69800000000000000B00] [endKey=74800000000000B0AA5F69800000000000000BFB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=2] [startKey=74800000000000B0AA5F69800000000000000200] [endKey=74800000000000B0AA5F698000000000000002FB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=1] [startKey=74800000000000B0AA5F69800000000000000100] [endKey=74800000000000B0AA5F698000000000000001FB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=9] [startKey=74800000000000B0AA5F69800000000000000900] [endKey=74800000000000B0AA5F698000000000000009FB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.616 +08:00] [INFO] [client.go:912] [“backup range started”] [range-sn=5] [startKey=74800000000000B0AA5F69800000000000000500] [endKey=74800000000000B0AA5F698000000000000005FB] [rateLimit=0] [concurrency=4]
[2024/11/22 10:27:46.617 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=4]
[2024/11/22 10:27:46.617 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=12]
[2024/11/22 10:27:46.618 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=8]
[2024/11/22 10:27:46.618 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=13]
[2024/11/22 10:27:46.618 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=10]
[2024/11/22 10:27:46.618 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=6]
[2024/11/22 10:27:46.618 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=7]
[2024/11/22 10:27:46.618 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=9]
[2024/11/22 10:27:46.618 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=3]
[2024/11/22 10:27:46.618 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=0]
[2024/11/22 10:27:46.618 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=11]
[2024/11/22 10:27:46.618 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=1]
[2024/11/22 10:27:46.619 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=5]
[2024/11/22 10:27:46.619 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.20:20160] [store-id=3640845]
[2024/11/22 10:27:46.619 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=14]
[2024/11/22 10:27:46.619 +08:00] [INFO] [client.go:939] [“backup push down started”] [range-sn=2]
[2024/11/22 10:27:46.620 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=4] [store-id=3640845] [“retry time”=0]
[2024/11/22 10:27:46.620 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.14:20160] [store-id=3688216]
[2024/11/22 10:27:46.622 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=4] [store-id=3688216] [“retry time”=0]
[2024/11/22 10:27:46.622 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.20:20161] [store-id=3640843]
[2024/11/22 10:27:46.623 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=12] [store-id=3640843] [“retry time”=0]
[2024/11/22 10:27:46.623 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=8] [store-id=3688216] [“retry time”=0]
[2024/11/22 10:27:46.623 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.20:20163] [store-id=3640844]
[2024/11/22 10:27:46.625 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=13] [store-id=3640844] [“retry time”=0]
[2024/11/22 10:27:46.625 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.17:20163] [store-id=3688225]
[2024/11/22 10:27:46.626 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=10] [store-id=3688225] [“retry time”=0]
[2024/11/22 10:27:46.626 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.16:20163] [store-id=3688230]
[2024/11/22 10:27:46.628 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=6] [store-id=3688230] [“retry time”=0]
[2024/11/22 10:27:46.628 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.15:20160] [store-id=3688212]
[2024/11/22 10:27:46.629 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=7] [store-id=3688212] [“retry time”=0]
[2024/11/22 10:27:46.629 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=9] [store-id=3688225] [“retry time”=0]
[2024/11/22 10:27:46.629 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.15:20162] [store-id=3688213]
[2024/11/22 10:27:46.630 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=3] [store-id=3688213] [“retry time”=0]
[2024/11/22 10:27:46.631 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.17:20162] [store-id=3688227]
[2024/11/22 10:27:46.632 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=0] [store-id=3688227] [“retry time”=0]
[2024/11/22 10:27:46.632 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.21:20161] [store-id=3640847]
[2024/11/22 10:27:46.633 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=11] [store-id=3640847] [“retry time”=0]
[2024/11/22 10:27:46.633 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=1] [store-id=3640845] [“retry time”=0]
[2024/11/22 10:27:46.634 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.19:20161] [store-id=3688222]
[2024/11/22 10:27:46.635 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=5] [store-id=3688222] [“retry time”=0]
[2024/11/22 10:27:46.635 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.21:20160] [store-id=3640850]
[2024/11/22 10:27:46.636 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=14] [store-id=3640850] [“retry time”=0]
[2024/11/22 10:27:46.636 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=2] [store-id=3688230] [“retry time”=0]
[2024/11/22 10:27:46.636 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.17:20160] [store-id=3688224]
[2024/11/22 10:27:46.638 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=4] [store-id=3688224] [“retry time”=0]
[2024/11/22 10:27:46.638 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.20:20162] [store-id=3640846]
[2024/11/22 10:27:46.639 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=12] [store-id=3640846] [“retry time”=0]
[2024/11/22 10:27:46.639 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=10] [store-id=3688227] [“retry time”=0]
[2024/11/22 10:27:46.639 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=13] [store-id=3640845] [“retry time”=0]
[2024/11/22 10:27:46.639 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=8] [store-id=3688224] [“retry time”=0]
[2024/11/22 10:27:46.639 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.15:20161] [store-id=3688215]
[2024/11/22 10:27:46.641 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=6] [store-id=3688215] [“retry time”=0]
[2024/11/22 10:27:46.641 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.14:20161] [store-id=3688219]
[2024/11/22 10:27:46.642 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=7] [store-id=3688219] [“retry time”=0]
[2024/11/22 10:27:46.642 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=9] [store-id=3688227] [“retry time”=0]
[2024/11/22 10:27:46.643 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.16:20162] [store-id=3688228]
[2024/11/22 10:27:46.644 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=0] [store-id=3688225] [“retry time”=0]
[2024/11/22 10:27:46.644 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=3] [store-id=3688228] [“retry time”=0]
[2024/11/22 10:27:46.644 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.19:20163] [store-id=3688223]
[2024/11/22 10:27:46.645 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=1] [store-id=3688216] [“retry time”=0]
[2024/11/22 10:27:46.645 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=11] [store-id=3688223] [“retry time”=0]
[2024/11/22 10:27:46.646 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.21:20163] [store-id=3640848]
[2024/11/22 10:27:46.647 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=5] [store-id=3640848] [“retry time”=0]
[2024/11/22 10:27:46.647 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.15:20163] [store-id=3688214]
[2024/11/22 10:27:46.648 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=14] [store-id=3688214] [“retry time”=0]
[2024/11/22 10:27:46.648 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=2] [store-id=3688215] [“retry time”=0]
[2024/11/22 10:27:46.648 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.17:20161] [store-id=3688226]
[2024/11/22 10:27:46.649 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=4] [store-id=3688226] [“retry time”=0]
[2024/11/22 10:27:46.650 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.14:20163] [store-id=3688218]
[2024/11/22 10:27:46.651 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=10] [store-id=3688216] [“retry time”=0]
[2024/11/22 10:27:46.651 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=8] [store-id=3688226] [“retry time”=0]
[2024/11/22 10:27:46.651 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=6] [store-id=3688213] [“retry time”=0]
[2024/11/22 10:27:46.651 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=12] [store-id=3688218] [“retry time”=0]
[2024/11/22 10:27:46.651 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=13] [store-id=3688216] [“retry time”=0]
[2024/11/22 10:27:46.651 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.16:20160] [store-id=3688229]
[2024/11/22 10:27:46.660 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=7] [store-id=3688229] [“retry time”=0]
[2024/11/22 10:27:46.660 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=0] [store-id=3640844] [“retry time”=0]
[2024/11/22 10:27:46.660 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=1] [store-id=3688224] [“retry time”=0]
[2024/11/22 10:27:46.660 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=3] [store-id=3640843] [“retry time”=0]
[2024/11/22 10:27:46.660 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=9] [store-id=3688226] [“retry time”=0]
[2024/11/22 10:27:46.660 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.16:20161] [store-id=3688231]
[2024/11/22 10:27:46.661 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=5] [store-id=3640847] [“retry time”=0]
[2024/11/22 10:27:46.661 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=11] [store-id=3688231] [“retry time”=0]
[2024/11/22 10:27:46.662 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.19:20160] [store-id=3688220]
[2024/11/22 10:27:46.663 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=14] [store-id=3688220] [“retry time”=0]
[2024/11/22 10:27:46.663 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=8] [store-id=3640844] [“retry time”=0]
[2024/11/22 10:27:46.663 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=10] [store-id=3688224] [“retry time”=0]
[2024/11/22 10:27:46.663 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=6] [store-id=3688228] [“retry time”=0]
[2024/11/22 10:27:46.663 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=4] [store-id=3640844] [“retry time”=0]
[2024/11/22 10:27:46.663 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=2] [store-id=3688213] [“retry time”=0]
[2024/11/22 10:27:46.663 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.19:20162] [store-id=3688221]
[2024/11/22 10:27:46.664 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=12] [store-id=3688221] [“retry time”=0]
[2024/11/22 10:27:46.664 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=0] [store-id=3640845] [“retry time”=0]
[2024/11/22 10:27:46.664 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=13] [store-id=3688224] [“retry time”=0]
[2024/11/22 10:27:46.664 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=7] [store-id=3688225] [“retry time”=0]
[2024/11/22 10:27:46.664 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=1] [store-id=3688226] [“retry time”=0]
[2024/11/22 10:27:46.664 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=3] [store-id=3688218] [“retry time”=0]
[2024/11/22 10:27:46.664 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=9] [store-id=3640844] [“retry time”=0]
[2024/11/22 10:27:46.665 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.14:20162] [store-id=3688217]
[2024/11/22 10:27:46.666 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=5] [store-id=3688223] [“retry time”=0]
[2024/11/22 10:27:46.666 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=11] [store-id=3688217] [“retry time”=0]
[2024/11/22 10:27:46.666 +08:00] [INFO] [store_manager.go:151] [“StoreManager: dialing to store.”] [address=10.1.1.21:20162] [store-id=3640849]
[2024/11/22 10:27:46.667 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=14] [store-id=3640849] [“retry time”=0]
[2024/11/22 10:27:46.667 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=14] [store-id=3688222] [“retry time”=0]
…
[2024/11/22 10:27:46.671 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=9] [store-id=3640846] [“retry time”=0]
[2024/11/22 10:27:46.670 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=12] [store-id=3640849] [“retry time”=0]
[2024/11/22 10:27:46.671 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=4] [store-id=3640843] [“retry time”=0]
[2024/11/22 10:27:46.671 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=6] [store-id=3688217] [“retry time”=0]
[2024/11/22 10:27:46.670 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=7] [store-id=3688227] [“retry time”=0]
[2024/11/22 10:27:46.670 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=1] [store-id=3640844] [“retry time”=0]
[2024/11/22 10:27:46.673 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=4] [store-id=3688212] [“retry time”=0]
[2024/11/22 10:27:46.674 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=9] [store-id=3640848] [“retry time”=0]
[2024/11/22 10:27:46.670 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=13] [store-id=3688219] [“retry time”=0]
[2024/11/22 10:27:46.671 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=13] [store-id=3688220] [“retry time”=0]
[2024/11/22 10:27:46.749 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=1] [small-range-count=1]
[2024/11/22 10:27:46.749 +08:00] [INFO] [client.go:984] [“transactional range backup completed”] [range-sn=1] [StartTS=0] [EndTS=454096968346238980]
[2024/11/22 10:27:46.749 +08:00] [INFO] [client.go:904] [“backup range completed”] [range-sn=1] [startKey=74800000000000B0AA5F69800000000000000100] [endKey=74800000000000B0AA5F698000000000000001FB] [take=132.868015ms]
[2024/11/22 10:31:46.620 +08:00] [INFO] [checkpoint.go:610] [“start to flush the checkpoint lock”] [lock-at=1732242706579] [expire-at=1732243006579]
[2024/11/22 10:35:46.620 +08:00] [INFO] [checkpoint.go:610] [“start to flush the checkpoint lock”] [lock-at=1732242946579] [expire-at=1732243246579]
[2024/11/22 10:39:46.620 +08:00] [INFO] [checkpoint.go:610] [“start to flush the checkpoint lock”] [lock-at=1732243186579] [expire-at=1732243486579]
[2024/11/22 10:43:46.619 +08:00] [INFO] [checkpoint.go:610] [“start to flush the checkpoint lock”] [lock-at=1732243426578] [expire-at=1732243726578]
[2024/11/22 10:47:46.620 +08:00] [INFO] [checkpoint.go:610] [“start to flush the checkpoint lock”] [lock-at=1732243666578] [expire-at=1732243966578]
[2024/11/22 10:51:46.619 +08:00] [INFO] [checkpoint.go:610] [“start to flush the checkpoint lock”] [lock-at=1732243906578] [expire-at=1732244206578]
[2024/11/22 10:55:46.620 +08:00] [INFO] [checkpoint.go:610] [“start to flush the checkpoint lock”] [lock-at=1732244146579] [expire-at=1732244446579]
[2024/11/22 10:59:46.620 +08:00] [INFO] [checkpoint.go:610] [“start to flush the checkpoint lock”] [lock-at=1732244386579] [expire-at=1732244686579]
[2024/11/22 11:03:46.620 +08:00] [INFO] [checkpoint.go:610] [“start to flush the checkpoint lock”] [lock-at=1732244626579] [expire-at=1732244926579]
[2024/11/22 11:05:00.050 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=3] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:00.050 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=3] [small-range-count=0]
[2024/11/22 11:05:00.051 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=3] [incomplete=1]
[2024/11/22 11:05:00.051 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=3] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF0000030000000000FB]
[2024/11/22 11:05:00.051 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=3] [“retry time”=0]
[2024/11/22 11:05:00.434 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=14] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:00.435 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=14] [small-range-count=0]
[2024/11/22 11:05:00.435 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=14] [incomplete=1]
[2024/11/22 11:05:00.435 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=14] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF00000E0000000000FB]
[2024/11/22 11:05:00.435 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=14] [“retry time”=0]
[2024/11/22 11:05:00.562 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=9] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:00.562 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=9] [small-range-count=0]
[2024/11/22 11:05:00.562 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=9] [incomplete=1]
[2024/11/22 11:05:00.563 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=9] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF0000090000000000FB]
[2024/11/22 11:05:00.563 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=9] [“retry time”=0]
[2024/11/22 11:05:00.690 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=13] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:00.690 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=13] [small-range-count=0]
[2024/11/22 11:05:00.691 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=13] [incomplete=1]
[2024/11/22 11:05:00.691 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=13] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF00000D0000000000FB]
[2024/11/22 11:05:00.691 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=13] [“retry time”=0]
[2024/11/22 11:05:01.074 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=5] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:01.074 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=5] [small-range-count=0]
[2024/11/22 11:05:01.075 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=5] [incomplete=1]
[2024/11/22 11:05:01.075 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=5] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF0000050000000000FB]
[2024/11/22 11:05:01.075 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=5] [“retry time”=0]
[2024/11/22 11:05:01.202 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=0] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:01.202 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=0] [small-range-count=0]
[2024/11/22 11:05:01.202 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=0] [incomplete=1]
[2024/11/22 11:05:01.203 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=0] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F720100000000FB]
[2024/11/22 11:05:01.203 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=0] [“retry time”=0]
[2024/11/22 11:05:01.458 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=12] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:01.458 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=12] [small-range-count=0]
[2024/11/22 11:05:01.458 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=12] [incomplete=1]
[2024/11/22 11:05:01.459 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=12] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF00000C0000000000FB]
[2024/11/22 11:05:01.459 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=12] [“retry time”=0]
[2024/11/22 11:05:01.714 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=8] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:01.714 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=8] [small-range-count=0]
[2024/11/22 11:05:01.714 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=8] [incomplete=1]
[2024/11/22 11:05:01.715 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=8] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF0000080000000000FB]
[2024/11/22 11:05:01.715 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=8] [“retry time”=0]
[2024/11/22 11:05:01.842 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=11] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:01.842 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=4] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:01.843 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=4] [small-range-count=0]
[2024/11/22 11:05:01.843 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=11] [small-range-count=0]
[2024/11/22 11:05:01.843 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=11] [incomplete=1]
[2024/11/22 11:05:01.843 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=4] [incomplete=1]
[2024/11/22 11:05:01.843 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=11] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF00000B0000000000FB]
[2024/11/22 11:05:01.843 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=11] [“retry time”=0]
[2024/11/22 11:05:01.843 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=4] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF0000040000000000FB]
[2024/11/22 11:05:01.843 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=4] [“retry time”=0]
[2024/11/22 11:05:02.610 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=6] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:02.610 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=6] [small-range-count=0]
[2024/11/22 11:05:02.610 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=6] [incomplete=1]
[2024/11/22 11:05:02.611 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=6] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF0000060000000000FB]
[2024/11/22 11:05:02.611 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=6] [“retry time”=0]
[2024/11/22 11:05:03.122 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=7] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:03.122 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=7] [small-range-count=0]
[2024/11/22 11:05:03.122 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=7] [incomplete=1]
[2024/11/22 11:05:03.123 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=7] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF0000070000000000FB]
[2024/11/22 11:05:03.123 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=7] [“retry time”=0]
[2024/11/22 11:05:03.762 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=2] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:03.763 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=2] [small-range-count=0]
[2024/11/22 11:05:03.763 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=2] [incomplete=1]
[2024/11/22 11:05:03.763 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=2] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF0000020000000000FB]
[2024/11/22 11:05:03.763 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=2] [“retry time”=0]
[2024/11/22 11:05:04.146 +08:00] [WARN] [push.go:198] [“backup occur storage error”] [range-sn=10] [error=“Io(Custom { kind: Other, error: "failed to put object rusoto error Error during dispatch: error trying to connect: tcp connect error: Connection timed out (os error 110)" })”]
[2024/11/22 11:05:04.147 +08:00] [INFO] [client.go:967] [“backup push down completed”] [range-sn=10] [small-range-count=0]
[2024/11/22 11:05:04.147 +08:00] [INFO] [client.go:1093] [“start fine grained backup”] [range-sn=10] [incomplete=1]
[2024/11/22 11:05:04.148 +08:00] [INFO] [client.go:1028] [“find leader”] [range-sn=10] [Leader=“{"id":5444982189,"store_id":3640843}”] [key=74800000000000B0FFAA5F698000000000FF00000A0000000000FB]
[2024/11/22 11:05:04.148 +08:00] [INFO] [client.go:1417] [“try backup”] [range-sn=10] [“retry time”=0]

WalterWj · 2024 年11 月 22 日 07:33

OS ERROR？是不是 br 备份目标数据存储不稳定啊。

TiDBer_xTvoCh2f · 2024 年11 月 22 日 09:18

先看下数据有没有传输过去
如果没有任何数据传输，大概率是访问的端口不通或者权限问题

kang · 2024 年11 月 23 日 10:48

备份到MINIO时卡住的问题可能与以下几个因素有关：

网络连接超时：
- 错误信息中提到“Connection timed out (os error 110)”，这表明在尝试连接到MINIO服务器时发生了TCP连接超时。这可能是由于网络不稳定、MINIO服务器不可达或者网络配置错误导致的。
MINIO服务器配置：
- 请确保MINIO服务器运行正常，并且可以正常访问。检查MINIO服务器的日志，看是否有任何错误或警告信息。
网络配置：
- 检查您的网络配置，确保TiDB集群可以访问MINIO服务器的IP地址和端口。如果MINIO服务器在不同的网络或VPC中，确保相应的网络策略和路由规则允许访问。
防火墙或安全组规则：
- 检查是否有任何防火墙或安全组规则阻止了TiDB集群与MINIO服务器之间的通信。
TiDB BR工具配置：
- 确保您使用的TiDB BR工具版本与TiDB版本兼容，并且正确配置了S3存储的访问密钥和端点。
并发连接数：
- 由于您设置了--concurrency=20，这可能会导致大量的并发连接到MINIO服务器。如果MINIO服务器配置有连接数限制，可能需要调整这个参数。
BR日志和监控：
- 查看BR工具的日志文件user_userinfo_bak.log，可能会有更多的错误信息或提示。同时，监控TiDB集群和MINIO服务器的性能指标，看是否有资源瓶颈。
重试策略：
- 根据日志，BR工具在遇到错误时会尝试重试备份。您可以检查重试策略是否有效，或者是否需要调整重试间隔和重试次数。
MINIO客户端日志：
- 如果可能，启用MINIO客户端的详细日志记录，这可能会提供更多关于连接失败的详细信息。

TiDBer_steven · 2024 年11 月 23 日 16:51

IO性能不行吧，看下PD或者KV，可能在不断的重启。

小青年er · 2024 年11 月 24 日 04:26

能看一下机器这个时候的性能指标吗

有猫万事足 · 2024 年11 月 24 日 07:23

没限速吧，16个并发就可以跑到900mb/s。你20个并发，如果是1g的网络轻松打满。

[2024/11/22 10:27:42.438 +08:00] [WARN] [domain.go:622] [“loading schema takes a long time”] [“take time”=1.667552099s]

日志里面看也像是这样。

加

--ratelimit

参数，限制到一个合理的数值，再试一下。

上面那个文章的3.3节也写了这部分内容