TiKV cdc组件 panic The CommitTs must be greater than the resolvedTs

【 TiKV 版本】
v6.1.0
【遇到的问题】
通过tidbkv组件拉取历史数据,并且会往tikv中入数据。会使用cdc组件puller进行数据同步,在cdc client侧出现panic The CommitTs must be greater than the resolvedTs。
我有搜索到一条类似的bug,该bug已经在5.2上进行修复,但未查到合入到6.1版本的信息,求解答该问题是否与下列问题一致: https://github.com/pingcap/tiflow/issues/3075 非常感谢您的关注和解答

【复现路径】
cdc client通过Iter接口恢复大量历史数据,同时在往tikv中入数据,cdc puller组件同时在拉取数据变更。

【问题现象及影响】
现象: cdc puller client侧panic导致服务重启。client侧首先报几个resolved ts fallen back的日志,随后panic。详细日志如下:

[2022/09/06 05:20:24.072 +08:00] [DEBUG] [region_worker.go:778] [“The resolvedTs is fallen back in kvclient”] [namesapce=xxx] [changefeed=] [EventType=RESOLVED] [resolvedTs=435760821527379971] [lastResolvedTs=435785568274874381] [regionID=76069]

[2022/09/06 05:20:24.110 +08:00] [WARN] [region_worker.go:360] [“region not receiving resolved event from tikv or resolved ts is not pushing for too long time, try to resolve lock”] [namesapce=xxx] [changefeed=] [addr=A16969629704832:17513] [regionID=76069] [span="[0000000000000001ff00000d1000000182ffffbbfc023765007cff846a000000000000f9, 0000000000000001ff00000d2000000182ffff6a8dfe3765ff86ff9f43000000000000f9)"] [duration=6h20m8.3s] [lastEvent=37.565138ms] [resolvedTs=435785568274874381]

[2022/09/06 05:20:29.078 +08:00] [DEBUG] [region_worker.go:778] [“The resolvedTs is fallen back in kvclient”] [namesapce=xxx] [changefeed=] [EventType=RESOLVED] [resolvedTs=435760821527379971] [lastResolvedTs=435785568274874381] [regionID=76069]

[2022/09/06 05:20:30.079 +08:00] [DEBUG] [region_worker.go:778] [“The resolvedTs is fallen back in kvclient”] [namesapce=xxx] [changefeed=] [EventType=RESOLVED] [resolvedTs=435760821527379971] [lastResolvedTs=435785568274874381] [regionID=76069]

[2022/09/06 05:20:31.080 +08:00] [DEBUG] [region_worker.go:778] [“The resolvedTs is fallen back in kvclient”] [namesapce=xxx] [changefeed=] [EventType=RESOLVED] [resolvedTs=435760821527379971] [lastResolvedTs=435785568274874381] [regionID=76069]

[2022/09/06 05:20:31.787 +08:00] [PANIC] [region_worker.go:716] [“The CommitTs must be greater than the resolvedTs”] [EventType=COMMIT] [CommitTs=435760862156032795] [resolvedTs=435785568274874381] [regionID=76069] [stack=“github.com/pingcap/tiflow/cdc/kv.(*regionWorker).handleEventEntry\ \t/go/pkg/mod/github.com/pingcap/tiflow@v0.0.0-20220714025705-c7e31d529131/cdc/kv/region_worker.go:716\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).processEvent\ \t/go/pkg/mod/github.com/pingcap/tiflow@v0.0.0-20220714025705-c7e31d529131/cdc/kv/region_worker.go:398\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).eventHandler\ \t/go/pkg/mod/github.com/pingcap/tiflow@v0.0.0-20220714025705-c7e31d529131/cdc/kv/region_worker.go:495\ngithub.com/pingcap/tiflow/cdc/kv.(*regionWorker).run.func4\ \t/go/pkg/mod/github.com/pingcap/tiflow@v0.0.0-20220714025705-c7e31d529131/cdc/kv/region_worker.go:633\ngolang.org/x/sync/errgroup.(*Group).Go.func1\ \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75”]

【附件】

tiflow版本 github.com/pingcap/tiflow v0.0.0-20220714025705-c7e31d529131

这个应该是已经被 GC 掉了。

1 个赞