TiCDC创建任务后，没有任何报错信息，但数据始终不同步

🌞 建议反馈 产品缺陷
1

为提高效率，提问时请提供以下信息，问题描述清晰可优先响应。

  • 【TiDB 版本】：TiDB v4.0.8 TiCDC v4.0.8 MySQL8.0.14
  • 【问题描述】：TiCDC添加同步任务成功后，mysql始终没有数据，没有任何报错信息

tiup cdc cli changefeed create --pd=http://10.8.114.91:2379 --sink-uri=“mysql://miaoyamin:123123@10.8.4.117:3306/?time-zone=” --changefeed-id=“simple-replication-task” --tz=Asia/Shanghai
Create changefeed successfully!

以下是查询同步状态信息
[root@10-8-4-117 ~]# tiup cdc cli changefeed query --pd=http://10.8.114.91:2379 --changefeed-id=simple-replication-task
Starting component cdc: cli changefeed query --pd=http://10.8.114.91:2379 --changefeed-id=simple-replication-task
{
“info”: {
“sink-uri”: “mysql://miaoyamin:123123@10.8.4.117:3306/?time-zone=”,
“opts”: {
“_changefeed_id”: “cli-verify”
},
“create-time”: “2020-12-30T14:29:00.244660749+08:00”,
“start-ts”: 421870892538396674,
“target-ts”: 0,
“admin-job-type”: 0,
“sort-engine”: “memory”,
“sort-dir”: “.”,
“config”: {
“case-sensitive”: true,
“enable-old-value”: false,
“force-replicate”: false,
“check-gc-safe-point”: true,
“filter”: {
“rules”: [
“.”
],
“ignore-txn-start-ts”: null,
“ddl-allow-list”: null
},
“mounter”: {
“worker-num”: 16
},
“sink”: {
“dispatchers”: null,
“protocol”: “default”
},
“cyclic-replication”: {
“enable”: false,
“replica-id”: 0,
“filter-replica-ids”: null,
“id-buckets”: 0,
“sync-ddl”: false
},
“scheduler”: {
“type”: “table-number”,
“polling-time”: -1
}
},
“state”: “normal”,
“history”: null,
“error”: null,
“sync-point-enabled”: false,
“sync-point-interval”: 600000000000
},
“status”: {
“resolved-ts”: 421871518145052678,
“checkpoint-ts”: 421871411596623873,
“admin-job-type”: 0
},
“count”: 0,
“task-status”: [
{
“capture-id”: “be4ca1e8-507e-4fd7-8685-edd703b39265”,
“status”: {
“tables”: {
“1088”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1090”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1092”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1094”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1096”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1098”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1100”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1102”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1104”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1106”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1108”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1110”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1112”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1114”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1116”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1118”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1120”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1122”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1124”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1126”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1128”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1130”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1132”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1134”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1136”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1138”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1140”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1142”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1144”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1146”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1148”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1150”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1152”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1154”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1156”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1158”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1160”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1162”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1164”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1166”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1168”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1170”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1172”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1174”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1176”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1178”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1180”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1182”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1184”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1186”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1188”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1190”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1192”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1194”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1196”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1198”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1200”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1202”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1204”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1206”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1208”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1210”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1212”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1214”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1216”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1218”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1220”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1222”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1224”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1226”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1228”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1230”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1232”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1234”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1236”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1238”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1240”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1242”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1244”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1246”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1248”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1257”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1263”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1274”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1280”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1282”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1284”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
}
},
“operation”: null,
“admin-job-type”: 0
}
}
]
}

请问该如何处理？谢谢

1 个赞
2

请问有 ticdc log 可以提供看下吗。
再确认上游是否有正确写入数据呢

3

cdc服务器里面的日志文件 logs.zip (1.7 MB)

帮忙看下，感谢

4

上游TiDB数据库一直有数据写入，这是我们的生产环境，下游是mysql数据库

5

请问当前 cdc 环境还在吗，可以提供下 resolve ts 看是多少

6

这是我们的生产环境，cdc环境还在的，这个命令具体怎么使用的

7

这个命令即可

8

[root@10-8-4-117 ~]# tiup cdc cli changefeed query --pd=http://10.8.114.91:2379 --changefeed-id=simple-replication-task
Starting component cdc: cli changefeed query --pd=http://10.8.114.91:2379 --changefeed-id=simple-replication-task
{
“info”: {
“sink-uri”: “mysql://miaoyamin:123123@10.8.4.117:3306/?time-zone=”,
“opts”: {
“_changefeed_id”: “cli-verify”
},
“create-time”: “2020-12-30T14:29:00.244660749+08:00”,
“start-ts”: 421870892538396674,
“target-ts”: 0,
“admin-job-type”: 0,
“sort-engine”: “memory”,
“sort-dir”: “.”,
“config”: {
“case-sensitive”: true,
“enable-old-value”: false,
“force-replicate”: false,
“check-gc-safe-point”: true,
“filter”: {
“rules”: [
“.”
],
“ignore-txn-start-ts”: null,
“ddl-allow-list”: null
},
“mounter”: {
“worker-num”: 16
},
“sink”: {
“dispatchers”: null,
“protocol”: “default”
},
“cyclic-replication”: {
“enable”: false,
“replica-id”: 0,
“filter-replica-ids”: null,
“id-buckets”: 0,
“sync-ddl”: false
},
“scheduler”: {
“type”: “table-number”,
“polling-time”: -1
}
},
“state”: “normal”,
“history”: null,
“error”: null,
“sync-point-enabled”: false,
“sync-point-interval”: 600000000000
},
“status”: {
“resolved-ts”: 421871518145052678,
“checkpoint-ts”: 421871411596623873,
“admin-job-type”: 0
},
“count”: 0,
“task-status”: [
{
“capture-id”: “be4ca1e8-507e-4fd7-8685-edd703b39265”,
“status”: {
“tables”: {
“1088”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1090”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1092”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1094”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1096”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1098”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1100”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1102”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1104”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1106”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1108”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1110”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1112”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1114”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1116”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1118”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1120”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1122”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1124”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1126”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1128”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1130”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1132”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1134”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1136”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1138”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1140”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1142”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1144”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1146”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1148”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1150”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1152”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1154”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1156”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1158”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1160”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1162”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1164”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1166”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1168”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1170”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1172”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1174”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1176”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1178”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1180”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1182”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1184”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1186”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1188”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1190”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1192”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1194”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1196”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1198”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1200”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1202”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1204”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1206”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1208”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1210”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1212”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1214”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1216”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1218”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1220”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1222”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1224”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1226”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1228”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1230”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1232”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1234”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1236”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1238”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1240”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1242”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1244”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1246”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1248”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1257”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1263”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1274”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1280”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1282”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
},
“1284”: {
“start-ts”: 421870892538396674,
“mark-table-id”: 0
}
},
“operation”: null,
“admin-job-type”: 0
}
}
]
}

9

当前 ticdc 一共多少实例？

10

cdc目前就一台实例

image
image1201×330 12.2 KB

11

@rona 从日志里面看，比较像是 cdc 的 owner 卡住了，麻烦抓一下进程当前的 goroutine 看一下：

访问 10.8.114.92:8300/debug/pprof/goroutine?debug=2

贴一下对应页面的内容，多谢

12

内容挺多的，请看下附近文件debug.txt (7.2 MB)

13

麻烦截一下近两天的TiCDC监控的图

14

https://metricstool.pingcap.com/ 可以按照这个说明拉取监控信息

15

监控信息比较多，截图统一放在word文档里面了，请查看附件ticdc监控图表.doc (1.5 MB)

16

按方案导出了一份json格式文档，
tidb-mediamz-CDC_2021-01-05T08_59_40.994Z.json (5.9 MB)

17

好的，问题已经定位了，您现在可以通过重启 cdc server 来恢复同步。

tiup cluster restart -R cdc
1 个赞
18

刚通过命令重启cdc成功了，但mysql里面还没数据，是不是要等一段时间才会有？

image
image1261×474 11.5 KB

19

是的 要等一段时间，有一个初始化过程。可以 关注监控中 checkpoint 面板，当 checkpoint 开始推了，就意味着初始化过程结束了

20

好的，我再跟踪看下，如果还有问题我再请教下