CDC任务checkpoint-ts不推进

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:
【 TiDB 使用环境】
v4.0.5

【概述】 场景 + 问题概述
新建cdc任务后,状态显示正常,但是checkpoint-ts一直不推进,新增数据无法同步到下游集群。
changefeed信息如下,cdc日志请见附件。
{
“info”: {
“sink-uri”: “mysql://XXX@XXX/”,
“opts”: {},
“create-time”: “2022-02-07T18:48:54.688253051+08:00”,
“start-ts”: 431024065001553923,
“target-ts”: 0,
“admin-job-type”: 2,
“sort-engine”: “unified”,
“sort-dir”: “.”,
“config”: {
“case-sensitive”: true,
“enable-old-value”: true,
“filter”: {
“rules”: [
“biz_mall.*”
],
“ignore-txn-start-ts”: null,
“ddl-allow-list”: null
},
“mounter”: {
“worker-num”: 16
},
“sink”: {
“dispatchers”: null,
“protocol”: “default”
},
“cyclic-replication”: {
“enable”: false,
“replica-id”: 0,
“filter-replica-ids”: null,
“id-buckets”: 0,
“sync-ddl”: false
},
“scheduler”: {
“type”: “table-number”,
“polling-time”: -1
}
},
“state”: “normal”,
“history”: [
1644235022251,
1644235029315,
1644235036380,
1644235043435,
1644235050507,
1644235149747,
1644235156803,
1644235163867,
1644235170931,
1644235177995,
1644235277251,
1644235284315,
1644235291371,
1644235298435,
1644235305491,
1644235404747,
1644235411811,
1644235418867,
1644235425923,
1644235432995
],
“error”: null
},
“status”: {
“resolved-ts”: 0,
“checkpoint-ts”: 431024065001553923,
“admin-job-type”: 0
},
“count”: 0,
“task-status”: [
{
“capture-id”: “cc76fd01-4dac-4d01-8ec8-72fa3e1d5460”,
“status”: {
“tables”: {
“65”: {
“start-ts”: 431024065001553923,
“mark-table-id”: 0
},
“86”: {
“start-ts”: 431024065001553923,
“mark-table-id”: 0
}
},
“operation”: {
“65”: {
“delete”: false,
“boundary_ts”: 431024065001553923,
“done”: false
},
“86”: {
“delete”: false,
“boundary_ts”: 431024065001553923,
“done”: false
}
},
“admin-job-type”: 1
}
}
]
}cdc.log.tar.gz (299.6 KB)

【备份和数据迁移策略逻辑】

【背景】 做过哪些操作

【现象】 业务和数据库现象

【问题】 当前遇到的问题

【业务影响】

【TiDB 版本】

【附件】

  • 相关日志、配置文件、Grafana 监控(https://metricstool.pingcap.com/)
  • TiUP Cluster Display 信息
  • TiUP CLuster Edit config 信息
  • TiDB-Overview 监控
  • 对应模块的 Grafana 监控(如有 BR、TiDB-binlog、TiCDC 等)
  • 对应模块日志(包含问题前后 1 小时日志)

若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

看了下日志,你这个应该是 2022/02/07 启动的 changefeed,changefeed 一直处于创建失败重试,10.61.67.105:4000: connect: connection timed out 建议检查下。

[2022/02/07 18:29:57.516 +08:00] [INFO] [client.go:714] ["start new request"] [request="{\"header\":{\"cluster_id\":6920257471096983859},\"region_id\":6,\"region_epoch\":{\"conf_ver\":8,\"version\":2},\"checkpoint_ts\":431024065001553923,\"start_key\":\"bURETEpvYkH/ZP9kSWR4TGn/c3T/AAAAAAD/AAAA9wAAAAD/AAAAbAAAAAD7\",\"end_key\":\"bURETEpvYkH/ZP9kSWR4TGn/c3T/AAAAAAD/AAAA9wAAAAD/AAAAbQAAAAD7\",\"request_id\":30}"] [addr=10.20.35.134:20160]
[2022/02/07 18:29:57.516 +08:00] [INFO] [client.go:714] ["start new request"] [request="{\"header\":{\"cluster_id\":6920257471096983859},\"region_id\":6,\"region_epoch\":{\"conf_ver\":8,\"version\":2},\"checkpoint_ts\":431024065001553923,\"start_key\":\"bURETEpvYkz/af9zdAAAAAD/AAD5AAAAAAD/AABsAAAAAAD6\",\"end_key\":\"bURETEpvYkz/af9zdAAAAAD/AAD5AAAAAAD/AABtAAAAAAD6\",\"request_id\":29}"] [addr=10.20.35.134:20160]
[2022/02/07 18:30:04.536 +08:00] [WARN] [owner.go:497] ["create changefeed failed, retry later"] [changefeed=tidb-common-cluster-wx-to-k9] [error="fail to query sink for support of auto-random: dial tcp 10.61.67.105:4000: connect: connection timed out"] [errorVerbose="dial tcp 10.61.67.105:4000: connect: connection timed out\
fail to query sink for support of auto-random\
github.com/pingcap/ticdc/cdc/sink.configureSinkURI\
\tgithub.com/pingcap/ticdc@/cdc/sink/mysql.go:303\
github.com/pingcap/ticdc/cdc/sink.newMySQLSink\
\tgithub.com/pingcap/ticdc@/cdc/sink/mysql.go:425\
github.com/pingcap/ticdc/cdc/sink.NewSink\
\tgithub.com/pingcap/ticdc@/cdc/sink/sink.go:69\
github.com/pingcap/ticdc/cdc.(*Owner).newChangeFeed\
\tgithub.com/pingcap/ticdc@/cdc/owner.go:314\
github.com/pingcap/ticdc/cdc.(*Owner).loadChangeFeeds\
\tgithub.com/pingcap/ticdc@/cdc/owner.go:471\
github.com/pingcap/ticdc/cdc.(*Owner).run\
\tgithub.com/pingcap/ticdc@/cdc/owner.go:990\
github.com/pingcap/ticdc/cdc.(*Owner).Run\
\tgithub.com/pingcap/ticdc@/cdc/owner.go:942\
github.com/pingcap/ticdc/cdc.(*Server).campaignOwnerLoop\
\tgithub.com/pingcap/ticdc@/cdc/server.go:266\
github.com/pingcap/ticdc/cdc.(*Server).run.func1\
\tgithub.com/pingcap/ticdc@/cdc/server.go:297\
golang.org/x/sync/errgroup.(*Group).Go.func1\
\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57\
runtime.goexit\
\truntime/asm_amd64.s:1357"]
[2022/02/07 18:31:00.199 +08:00] [WARN] [owner.go:448] ["changefeed fails reach rate limit, try to initialize it later"] [history="[1644229776251,1644229783323,1644229790403,1644229797459,1644229804531]"]
[2022/02/07 18:31:00.700 +08:00] [WARN] [owner.go:448] ["changefeed fails reach rate limit, try to initialize it later"] [history="[1644229776251,1644229783323,1644229790403,1644229797459,1644229804531]"]

友情提示:cdc 在 4.0 早期版本存在一些问题,建议升级到 4.0.x 最新版本使用~ 功能更加完善。

1 个赞

这个必须要升级,TiCDC是在4.0.6 GA的。否则夜不能寐!

2 个赞

是的,升级吧

2 个赞

能直接升级到最新版本吗?

1 个赞

推荐将 TiCDC 升级到最新版本

4.0.6 还是有问题,若在4.0.13之前用还是有各种问题的

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。