TiCDC TSO不往前推进

【 TiDB 使用环境】生产\测试环境\ POC
【 TiDB 版本】
4.0.7

cdc server 以及cdc cli均为4.0.7

【遇到的问题】
基于时间点的同步,新建的复制任务会出现卡住的情况,tso不往前推进
【复现路径】做过哪些操作出现的问题
update mysql.tidb set VARIABLE_VALUE= '24h' where VARIABLE_NAME= "tikv_gc_life_time" ;

br备份,作完整的数据库恢复,根据br备份文件metabackup中的commitTS作为cdc的tart-ts

./cdc cli changefeed create --pd=http://192.168.0.5:2379 --sink-uri=“mysql://test:test@192.168.0.1:4000/” --changefeed-id “test-2018” --config=changefeed.toml --start-ts=433921993657810984 --sort-engine=unified

复制任务不往前推进,暂停之后在启动也不行

【问题现象及影响】
【附件】

  • 相关日志、配置文件、Grafana 监控(https://metricstool.pingcap.com/)
  • TiUP Cluster Display 信息
  • TiUP CLuster Edit config 信息
  • TiDB-Overview 监控
  • 对应模块的 Grafana 监控(如有 BR、TiDB-binlog、TiCDC 等)
  • 对应模块日志(包含问题前后 1 小时日志)
    [2022/06/14 15:18:18.321 +00:00] [DEBUG] [client.go:643] [“dispatching region”] [regionID=22649]
    [2022/06/14 15:18:18.321 +00:00] [INFO] [client.go:713] [“creating new stream to store to send request”] [regionID=22649] [requestID=12] [storeID=4] [addr=tidb-z3x18cr0q6-tikv-1.tidb-z3x18cr0q6-tikv-peer.tidb-z3
    [2022/06/14 15:18:18.323 +00:00] [DEBUG] [client.go:364] [“created stream to store”] [addr=tidb-z3x18cr0q6-tikv-1.tidb-z3x18cr0q6-tikv-peer.tidb-z3x18cr0q6.svc:20160]
    [2022/06/14 15:18:18.323 +00:00] [DEBUG] [client.go:364] [“created stream to store”] [addr=tidb-z3x18cr0q6-tikv-0.tidb-z3x18cr0q6-tikv-peer.tidb-z3x18cr0q6.svc:20160]
    [2022/06/14 15:18:18.323 +00:00] [DEBUG] [client.go:748] [“start new request”] [request="{“header”:{“cluster_id”:7102381364438096930,“ticdc_version”:“4.0.7”},“region_id”:22649,“region_epoch”:{“conf_
    [2022/06/14 15:18:18.323 +00:00] [DEBUG] [client.go:748] [“start new request”] [request=”{“header”:{“cluster_id”:7102381364438096930,“ticdc_version”:“4.0.7”},“region_id”:16697,“region_epoch”:{“conf_
    [2022/06/14 15:18:18.324 +00:00] [INFO] [puller.go:247] [“puller is initialized”] [duration=4.990905ms] [changefeedid=billing-2247] [tableID=594] [spans=”["[7480000000000002ff525f000000000000f9, 748000000000000
    [2022/06/14 15:18:18.324 +00:00] [INFO] [puller.go:247] [“puller is initialized”] [duration=5.099369ms] [changefeedid=billing-2247] [tableID=413] [spans="["[7480000000000001ff9d5f000000000000f9, 748000000000000
    [2022/06/14 15:18:18.766 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:19.265 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:19.765 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:20.265 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:20.765 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:21.266 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:21.765 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:22.266 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:22.765 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:23.267 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:23.765 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:24.265 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:24.765 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:25.265 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:25.767 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:26.265 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:26.765 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:27.265 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:27.765 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:28.265 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]
    [2022/06/14 15:18:28.765 +00:00] [DEBUG] [changefeed.go:716] [“skip update resolved ts”] [taskPositions=0] [taskStatus=1]

若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

1、下游是mysql还是tidb呢,看你用的4000端口可能是tidb吧,sink url换成tidb试试?
2、start-ts用0试试能不能正常同步

再参考下类似帖子:

4.0.7的CDC版本有点低了,建议升级。

1 个赞

经过验证,4.0.10+以上的版本不会出现卡住的问题,所以只能升级tidb了

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。