BR全量备份异常

Queena · 2021 年1 月 19 日 12:00

为提高效率，提问时请提供以下信息，问题描述清晰可优先响应。

【TiDB 版本】： v4.0.8
【问题描述】：BR全量备份执行完成后，并没有任何报错，但是没有备份文件

tidb-toolkit-v4.0.8-linux-amd64/bin/br backup table --pd “172.23.4.9:2379” --db mer --table details --log-file /20210119_back/backupfull.log --storage “local:///logs/” --log-level debug

Queena · 2021 年1 月 19 日 12:01

GC时间已修改

yilong · 2021 年1 月 19 日 12:11

1.完整的backup日志上传下。
2. 备份的 db mer 和表 details 检查下是否在库里存在，且有数据，多谢。

Queena · 2021 年1 月 20 日 01:28

第二点已确认，是有数据的

第一点的日志信息如下

[2021/01/19 14:22:37.786 +08:00] [INFO] [version.go:37] [“Welcome to Backup & Restore (BR)”]
[2021/01/19 14:22:37.786 +08:00] [INFO] [version.go:38] [BR] [release-version=v4.0.8]
[2021/01/19 14:22:37.786 +08:00] [INFO] [version.go:39] [BR] [git-hash=c2ed897feadaae1ae27a4111cd44b1840941e9be]
[2021/01/19 14:22:37.786 +08:00] [INFO] [version.go:40] [BR] [git-branch=heads/refs/tags/v4.0.8]
[2021/01/19 14:22:37.786 +08:00] [INFO] [version.go:41] [BR] [go-version=go1.13]
[2021/01/19 14:22:37.786 +08:00] [INFO] [version.go:42] [BR] [utc-build-time=“2020-10-30 08:14:21”]
[2021/01/19 14:22:37.786 +08:00] [INFO] [version.go:43] [BR] [race-enabled=false]
[2021/01/19 14:22:37.786 +08:00] [INFO] [common.go:429] [arguments] [db=mer] [log-file=/export/servers/logs/20210119_back/backupfull.log] [log-level=debug] [pd="[172.23.4.9:2379]"] [storage=local:///export/servers/logs/] [table=details]
[2021/01/19 14:22:37.786 +08:00] [INFO] [client.go:166] ["[pd] create pd client with endpoints"] [pd-address="[172.23.4.9:2379]"]
[2021/01/19 14:22:37.789 +08:00] [INFO] [base_client.go:236] ["[pd] update member urls"] [old-urls="[http://172.23.4.9:2379]"] [new-urls="[http://
172.23.4.9:2379,http://172.23.4.7:2379,http://172.23.4.1:2379]"]
[2021/01/19 14:22:37.789 +08:00] [INFO] [base_client.go:252] ["[pd] switch leader"] [new-leader=http://172.23.4.9:2379] [old-leader=]
[2021/01/19 14:22:37.789 +08:00] [INFO] [base_client.go:102] ["[pd] init cluster id"] [cluster-id=6871057888683601243]
[2021/01/19 14:22:37.791 +08:00] [INFO] [client.go:166] ["[pd] create pd client with endpoints"] [pd-address="[172.23.4.9:2379]"]
[2021/01/19 14:22:37.792 +08:00] [INFO] [base_client.go:236] ["[pd] update member urls"] [old-urls="[http://172.23.4.9:2379]"] [new-urls="[http://
172.23.4.9:2379,http://172.23.4.7:2379,http://172.23.4.1:2379]"]
[2021/01/19 14:22:37.792 +08:00] [INFO] [base_client.go:252] ["[pd] switch leader"] [new-leader=http://172.23.4.9:2379] [old-leader=]
[2021/01/19 14:22:37.792 +08:00] [INFO] [base_client.go:102] ["[pd] init cluster id"] [cluster-id=6871057888683601243]
[2021/01/19 14:22:37.792 +08:00] [DEBUG] [version.go:102] [“checking compatibility of store in cluster”] [ID=1] [TiFlash?=false] [address=172.23.43.8:20160] [version=4.0.8]
[2021/01/19 14:22:37.792 +08:00] [DEBUG] [version.go:102] [“checking compatibility of store in cluster”] [ID=2] [TiFlash?=false] [address=172.23.4.8:20160] [version=4.0.8]
[2021/01/19 14:22:37.792 +08:00] [DEBUG] [version.go:102] [“checking compatibility of store in cluster”] [ID=7] [TiFlash?=false] [address=172.23.4.79:20160] [version=4.0.8]
[2021/01/19 14:22:37.792 +08:00] [INFO] [conn.go:129] [“new mgr”] [pdAddrs=172.23.4.9:2379]
[2021/01/19 14:22:37.792 +08:00] [INFO] [tidb.go:71] [“new domain”] [store=tikv-6871057888683601243] [“ddl lease”=1s] [“stats lease”=-1ns] [“index usage sync lease”=1m0s]
[2021/01/19 14:22:37.796 +08:00] [INFO] [ddl.go:323] ["[ddl] start DDL"] [ID=ec58a374-1ab2-4d33-b586-d5843dae3ea0] [runWorker=false]
[2021/01/19 14:22:37.805 +08:00] [DEBUG] [scan.go:157] [“txn getData”] [nextStartKey=6d4442730000000000fa0000000000000068] [nextEndKey=6d4442730000000000fa0000000000000069] [reverse=false] [txnStartTS=422323777122598913]
[2021/01/19 14:22:37.806 +08:00] [DEBUG] [scan.go:157] [“txn getData”] [nextStartKey=6d44423a3100000000fb0000000000000068] [nextEndKey=6d44423a3100000000fb0000000000000069] [reverse=false] [txnStartTS=422323777122598913]
[2021/01/19 14:22:37.806 +08:00] [DEBUG] [scan.go:157] [“txn getData”] [nextStartKey=6d44423a3300000000fb0000000000000068] [nextEndKey=6d44423a3300000000fb0000000000000069] [reverse=false] [txnStartTS=422323777122598913]
[2021/01/19 14:22:37.810 +08:00] [DEBUG] [scan.go:157] [“txn getData”] [nextStartKey=6d44423a3539000000fc0000000000000068] [nextEndKey=6d44423a3539000000fc0000000000000069] [reverse=false] [txnStartTS=422323777122598913]
[2021/01/19 14:22:37.813 +08:00] [INFO] [domain.go:147] [“full load InfoSchema success”] [usedSchemaVersion=0] [neededSchemaVersion=148] [“start time”=8.445124ms]
[2021/01/19 14:22:37.814 +08:00] [INFO] [domain.go:381] [“full load and reset schema validator”]
[2021/01/19 14:22:37.814 +08:00] [DEBUG] [schema_validator.go:156] [“update schema validator”] [oldVer=0] [currVer=148] [changedTableIDs="[]"] [changedActionTypes="[]"]
[2021/01/19 14:22:37.814 +08:00] [INFO] [client.go:83] [“new backup client”]
[2021/01/19 14:22:37.815 +08:00] [INFO] [client.go:127] [“backup encode timestamp”] [BackupTS=422323777122598914]
[2021/01/19 14:22:37.815 +08:00] [INFO] [backup.go:214] [“current backup safePoint job”] [safePoint="{ID=br-85cd7e90-7c42-41ae-98f1-fd0fde811eab,TTL=5m0s,BackupTime=“2021-01-19 14:22:37.798 +0800 CST”,BackupTS=422323777122598914}"]
[2021/01/19 14:22:37.815 +08:00] [DEBUG] [safe_point.go:79] [“update PD safePoint limit with TTL”] [safePoint="{ID=br-85cd7e90-7c42-41ae-98f1-fd0fde811eab,TTL=5m0s,BackupTime=“2021-01-19 14:22:37.798 +0800 CST”,BackupTS=422323777122598914}"]
[2021/01/19 14:22:37.816 +08:00] [WARN] [safe_point.go:85] [“service GC safe point lost, we may fail to back up if GC lifetime isn’t long enough”] [lastSafePoint=18446744073709551615] [safePoint="{ID=br-85cd7e90-7c42-41ae-98f1-fd0fde811eab,TTL=5m0s,BackupTime=“2021-01-19 14:22:37.798 +0800 CST”,BackupTS=422323777122598914}"]
[2021/01/19 14:22:37.816 +08:00] [DEBUG] [scan.go:157] [“txn getData”] [nextStartKey=6d4442730000000000fa0000000000000068] [nextEndKey=6d44427300000
00000fa0000000000000069] [reverse=false] [txnStartTS=422323777122598914]
[2021/01/19 14:22:37.817 +08:00] [DEBUG] [scan.go:157] [“txn getData”] [nextStartKey=6d44423a3100000000fb0000000000000068] [nextEndKey=6d44423a31000
00000fb0000000000000069] [reverse=false] [txnStartTS=422323777122598914]
[2021/01/19 14:22:37.817 +08:00] [DEBUG] [scan.go:157] [“txn getData”] [nextStartKey=6d44423a3300000000fb0000000000000068] [nextEndKey=6d44423a33000
00000fb0000000000000069] [reverse=false] [txnStartTS=422323777122598914]
[2021/01/19 14:22:37.820 +08:00] [DEBUG] [scan.go:157] [“txn getData”] [nextStartKey=6d44423a3539000000fc0000000000000068] [nextEndKey=6d44423a35390
00000fc0000000000000069] [reverse=false] [txnStartTS=422323777122598914]
[2021/01/19 14:22:37.823 +08:00] [INFO] [domain.go:147] [“full load InfoSchema success”] [usedSchemaVersion=0] [neededSchemaVersion=148] [“start time”=6.753263ms]
[2021/01/19 14:22:37.823 +08:00] [INFO] [client.go:349] [“nothing to backup”]
[2021/01/19 14:22:37.824 +08:00] [DEBUG] [client.go:203] [“backup meta”] [meta="{“end_version”:422323777122598914,“ddls”:“bnVsbA==”}"]
[2021/01/19 14:22:37.824 +08:00] [INFO] [client.go:205] [“save backup meta”] [path=local:///export/servers/logs/] [size=16]
[2021/01/19 14:22:37.824 +08:00] [INFO] [ddl.go:391] ["[ddl] DDL closed"] [ID=ec58a374-1ab2-4d33-b586-d5843dae3ea0] [“take time”=608.199µs]
[2021/01/19 14:22:37.824 +08:00] [INFO] [ddl.go:304] ["[ddl] stop DDL"] [ID=ec58a374-1ab2-4d33-b586-d5843dae3ea0]
[2021/01/19 14:22:37.826 +08:00] [INFO] [domain.go:457] [“infoSyncerKeeper exited.”]
[2021/01/19 14:22:37.826 +08:00] [INFO] [domain.go:516] [“loadSchemaInLoop exited.”]
[2021/01/19 14:22:37.826 +08:00] [INFO] [domain.go:428] [“topNSlowQueryLoop exited.”]
[2021/01/19 14:22:37.826 +08:00] [INFO] [domain.go:637] [“domain closed”] [“take time”=1.822063ms]
[2021/01/19 14:22:37.826 +08:00] [INFO] [collector.go:187] [“Table backup Failed summary : total backup ranges: 0, total success: 0, total failed: 0”] [BackupTS=422323777122598914]

yilong · 2021 年1 月 20 日 02:21

看到有这样的告警，你的gc是备份前调整的吗？
目前 gc 是多久，试试再备份一次。

Queena · 2021 年1 月 20 日 02:25

GC是在备份前调整的，

全量full备份是可以的，kv节点有备份的数据

就是指定到库和表的时候，没有报错，也没有备份的数据
–db mer --table details

yilong · 2021 年1 月 20 日 02:35

select count(*) from mer.merchant_account_details; 展示下结果
pd-ctl 反馈下 store 查询结果
mer.merchant_account_details 这个表是否副本同步到 tiflash 了？
展示下 tiup cluster display 的信息

yuqi1129 · 2021 年7 月 14 日 02:27

这个问题有更新吗，这两天同样遇到这个问题，排查之后发现原因出现在这里

这里获取到的schema信息很老，与通过PD http 和TiDB show databases 信息不一致，不知道官方有没有修复这个问题

版本: v4.0.8

yuqi1129 · 2021 年7 月 14 日 02:29

备份表、db 不成功是因为db找不到，备份整个集群可以成功是因为不需要过滤db, 所以可以成功，但最终的结果肯定是少了:joy: