BR备份异常：GC safepoint exceeded

🪐 TiDB 技术问题 备份&数据迁移
1

【 TiDB 使用环境】
TIDB V4.0.15

【概述】 场景 + 问题概述
使用BR全量备份后，过了大于10分钟后，在进行增量备份时报错如下：
Error: GC safepoint 429977118583554048 exceed TS 429976261162958849: [BR:Backup:ErrBackupGCSafepointExceeded]backup GC safepoint exceeded

【备份和数据迁移策略逻辑】
BR标准备份

【背景】 做过哪些操作
使用以下命令修改了tikv_gc_life_time为24小时：
UPDATE mysql.tidb SET VARIABLE_VALUE = ‘24h’ WHERE VARIABLE_NAME = ‘tikv_gc_life_time’;

【问题】 当前遇到的问题
无法完成增量备份

2

指定一个和全备不一样的新目录

3

运行增量备份命令时，我是已经指定了新目录的

4

这个错误就是说那个备份元数据文件存在。 如果可以的话整个全备和增量备份日志都贴出来。

5

我日志贴错了 以下是正确的错误日志：
Error: GC safepoint 429977118583554048 exceed TS 429976261162958849: [BR:Backup:ErrBackupGCSafepointExceeded]backup GC safepoint exceeded

6

增量备份日志如下：
[2021/12/23 14:39:47.905 +08:00] [INFO] [info.go:40] [“Welcome to Backup & Restore (BR)”] [release-version=v4.0.15] [git-hash=2ca9fa25f15aec061e281ad84f77dacf622ac9c3] [git-branch=heads/refs/tags/v4.0.15] [go-version=go1.13] [utc-build-time=“2021-09-23 09:48:49”] [race-enabled=false]
[2021/12/23 14:39:47.905 +08:00] [INFO] [common.go:469] [arguments] [__command=“br backup db”] [db=ish-im2] [lastbackupts=429976261162958849] [log-file=/data12/test_tidb_add.log] [pd="[172.17.0.136:2379]"] [ratelimit=30] [storage=local:///data12/ossfs/ish_im2_add_20211223]
[2021/12/23 14:39:47.905 +08:00] [INFO] [client.go:167] ["[pd] create pd client with endpoints"] [pd-address="[172.17.0.136:2379]"]
[2021/12/23 14:39:47.907 +08:00] [INFO] [base_client.go:236] ["[pd] update member urls"] [old-urls="[http://172.17.0.136:2379]"] [new-urls="[http://172.17.0.136:2379,http://172.17.0.137:2379]"]
[2021/12/23 14:39:47.907 +08:00] [INFO] [base_client.go:252] ["[pd] switch leader"] [new-leader=http://172.17.0.137:2379] [old-leader=]
[2021/12/23 14:39:47.907 +08:00] [INFO] [base_client.go:102] ["[pd] init cluster id"] [cluster-id=7023199013344700358]
[2021/12/23 14:39:47.910 +08:00] [INFO] [client.go:167] ["[pd] create pd client with endpoints"] [pd-address="[172.17.0.136:2379]"]
[2021/12/23 14:39:47.911 +08:00] [INFO] [base_client.go:236] ["[pd] update member urls"] [old-urls="[http://172.17.0.136:2379]"] [new-urls="[http://172.17.0.136:2379,http://172.17.0.137:2379]"]
[2021/12/23 14:39:47.911 +08:00] [INFO] [base_client.go:252] ["[pd] switch leader"] [new-leader=http://172.17.0.137:2379] [old-leader=]
[2021/12/23 14:39:47.911 +08:00] [INFO] [base_client.go:102] ["[pd] init cluster id"] [cluster-id=7023199013344700358]
[2021/12/23 14:39:47.912 +08:00] [INFO] [conn.go:162] [“new mgr”] [pdAddrs=172.17.0.136:2379]
[2021/12/23 14:39:47.913 +08:00] [INFO] [client.go:91] [“new backup client”]
[2021/12/23 14:39:48.613 +08:00] [INFO] [client.go:135] [“backup encode timestamp”] [BackupTS=429979490999861249]
[2021/12/23 14:39:48.613 +08:00] [INFO] [backup.go:258] [“current backup safePoint job”] [safePoint="{ID=br-f296ba3b-c703-4c75-8f7c-6f0e699778bf,TTL=5m0s,BackupTime=“2021-12-23 11:14:27.742 +0800 CST”,BackupTS=429976261162958849}"]
[2021/12/23 14:39:48.614 +08:00] [INFO] [collector.go:195] [“Database backup failed summary”] [total-ranges=0] [ranges-succeed=0] [ranges-failed=0]
[2021/12/23 14:39:48.614 +08:00] [ERROR] [backup.go:32] [“failed to backup”] [error=“GC safepoint 429977118583554048 exceed TS 429976261162958849: [BR:Backup:ErrBackupGCSafepointExceeded]backup GC safepoint exceeded”] [errorVerbose="[BR:Backup:ErrBackupGCSafepointExceeded]backup GC safepoint exceeded\ GC safepoint 429977118583554048 exceed TS 429976261162958849\ngithub.com/pingcap/br/pkg/utils.CheckGCSafePoint\ \tgithub.com/pingcap/br@/pkg/utils/safe_point.go:72\ github.com/pingcap/br/pkg/utils.StartServiceSafePointKeeper\ \tgithub.com/pingcap/br@/pkg/utils/safe_point.go:101\ github.com/pingcap/br/pkg/task.RunBackup\ \tgithub.com/pingcap/br@/pkg/task/backup.go:259\ main.runBackupCommand\ \tgithub.com/pingcap/br@/cmd/br/backup.go:31\ main.newDBBackupCommand.func1\ \tgithub.com/pingcap/br@/cmd/br/backup.go:107\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:842\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.0.0/command.go:950\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:887\ main.main\ \tgithub.com/pingcap/br@/cmd/br/main.go:56\ runtime.main\ \truntime/proc.go:203\ runtime.goexit\ \truntime/asm_amd64.s:1357"] [stack=“main.runBackupCommand\ \tgithub.com/pingcap/br@/cmd/br/backup.go:32\ main.newDBBackupCommand.func1\ \tgithub.com/pingcap/br@/cmd/br/backup.go:107\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:842\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.0.0/command.go:950\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:887\ main.main\ \tgithub.com/pingcap/br@/cmd/br/main.go:56\ runtime.main\ \truntime/proc.go:203”]
[2021/12/23 14:39:48.615 +08:00] [ERROR] [main.go:58] [“br failed”] [error=“GC safepoint 429977118583554048 exceed TS 429976261162958849: [BR:Backup:ErrBackupGCSafepointExceeded]backup GC safepoint exceeded”] [errorVerbose="[BR:Backup:ErrBackupGCSafepointExceeded]backup GC safepoint exceeded\ GC safepoint 429977118583554048 exceed TS 429976261162958849\ngithub.com/pingcap/br/pkg/utils.CheckGCSafePoint\ \tgithub.com/pingcap/br@/pkg/utils/safe_point.go:72\ github.com/pingcap/br/pkg/utils.StartServiceSafePointKeeper\ \tgithub.com/pingcap/br@/pkg/utils/safe_point.go:101\ github.com/pingcap/br/pkg/task.RunBackup\ \tgithub.com/pingcap/br@/pkg/task/backup.go:259\ main.runBackupCommand\ \tgithub.com/pingcap/br@/cmd/br/backup.go:31\ main.newDBBackupCommand.func1\ \tgithub.com/pingcap/br@/cmd/br/backup.go:107\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:842\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.0.0/command.go:950\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:887\ main.main\ \tgithub.com/pingcap/br@/cmd/br/main.go:56\ runtime.main\ \truntime/proc.go:203\ runtime.goexit\ \truntime/asm_amd64.s:1357"] [stack=“main.main\ \tgithub.com/pingcap/br@/cmd/br/main.go:58\ runtime.main\ \truntime/proc.go:203”]

7

tidb的版本和BR的版本一致么

8

BR和TIDB版本是一致的

br -V

Release Version: v4.0.15
Git Commit Hash: 2ca9fa25f15aec061e281ad84f77dacf622ac9c3
Git Branch: heads/refs/tags/v4.0.15
Go Version: go1.13
UTC Build Time: 2021-09-23 09:48:49
Race Enabled: false

tiup cluster display tidb-stage

Starting component cluster: /root/.tiup/components/cluster/v1.8.1/tiup-cluster display tidb-stage
Cluster type: tidb
Cluster name: tidb-stage
Cluster version: v4.0.15
Deploy user: root
SSH type: builtin
Dashboard URL: http://172.17.0.137:2379/dashboard

9

可以调大gc life time试一下

10

GC safepoint 必须在 lastbackupts 之前

11

这里显示你的GC safepoint 大于 lastbackupts了

12

gc life time 我改成240都不行

image
image784×195 14.1 KB

13

看了下你的日志大概清楚了，你是11点14分那会全备份完成的，然后14：39开始增量备份。增量备份需要基于全量备份的时间点，也就是需要11点14分以后的数据版本一直存在，而你那会的GC的safetime时间是12:08，相当于清除了12：08之前的数据版本，所以报这个错误。至于你后来修改GC时间只对后面的操作生效，你需要再全备份一次、再做增量备份

2 个赞
14

明显gc safepoint时间不够

15

明白了！
请问 增量备份是否也依赖GC safetime时间？

16

是的，想像一下你开始备份的时间是21:00 ，一张表一张表备份，如果到最后一张表是22:00，GC是21:30，那它就读不到最后一张表的21:00的数据了。就会报这个错误失败。

17

好的 感谢大佬~
咨询下 可以针对某个session设定tikv_gc_life_time吗？

18

session可以设置saft time时间内的点用于查询历史版本数据、方便恢复，set @@tidb_snapshot=“2021-12-24 13:45:26”; 再去查询。 官方文档这块描述的很详细

19

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。