CDC 延迟问题

🪐 TiDB 技术问题
1

【TiDB 使用环境】生产环境
【TiDB 版本】v 6.5.5
【操作系统】UnionTech 20
【部署方式】
【集群数据量】
【集群节点数】
【问题复现路径】做过哪些操作出现的问题
【遇到的问题：问题现象及影响】
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面

图片
图片1785×736 37 KB

【复制黏贴 ERROR 报错的日志】
[2025/09/10 09:31:57.446 +08:00] [INFO] [replication_manager.go:635] [“schedulerv3: slow table”] [namespace=default] [changefeed=xxxxx-9-2] [tableID=87194] [tableStatus=] [checkpointTs=460704012171476997] [resolvedTs=460704012171476997] [checkpointLag=5h59m41.90929966s]
[2025/09/10 09:31:57.446 +08:00] [INFO] [replication_manager.go:635] [“schedulerv3: slow table”] [namespace=default] [changefeed=xxxxx-9-2] [tableID=55303] [tableStatus=] [checkpointTs=460704012171476997] [resolvedTs=460704012171476997] [checkpointLag=5h59m41.90929966s]
[2025/09/10 09:31:57.446 +08:00] [INFO] [replication_manager.go:635] [“schedulerv3: slow table”] [namespace=default] [changefeed=xxxxx-9-2] [tableID=52723] [tableStatus=] [checkpointTs=460704012171476997] [resolvedTs=460704012171476997] [checkpointLag=5h59m41.90929966s]
[2025/09/10 09:31:57.446 +08:00] [INFO] [replication_manager.go:635] [“schedulerv3: slow table”] [namespace=default] [changefeed=xxxxx-9-2] [tableID=55363] [tableStatus=] [checkpointTs=460704012171476997] [resolvedTs=460704012171476997] [checkpointLag=5h59m41.90929966s]
[2025/09/10 09:32:57.498 +08:00] [INFO] [replication_manager.go:635] [“schedulerv3: slow table”] [namespace=default] [changefeed=xxxxx-9-2] [tableID=87194] [tableStatus=] [checkpointTs=460704012171476997] [resolvedTs=460704012171476997] [checkpointLag=6h0m42.00110468s]
[2025/09/10 09:32:57.498 +08:00] [INFO] [replication_manager.go:635] [“schedulerv3: slow table”] [namespace=default] [changefeed=xxxxx-9-2] [tableID=55303] [tableStatus=] [checkpointTs=460704012171476997] [resolvedTs=460704012171476997] [checkpointLag=6h0m42.00110468s]
[2025/09/10 09:32:57.498 +08:00] [INFO] [replication_manager.go:635] [“schedulerv3: slow table”] [namespace=default] [changefeed=xxxxx-9-2] [tableID=52723] [tableStatus=] [checkpointTs=460704012171476997] [resolvedTs=460704012171476997] [checkpointLag=6h0m42.00110468s]
[2025/09/10 09:32:57.498 +08:00] [INFO] [replication_manager.go:635] [“schedulerv3: slow table”] [namespace=default] [changefeed=xxxxx-9-2] [tableID=55363] [tableStatus=] [checkpointTs=460704012171476997] [resolvedTs=460704012171476997] [checkpointLag=6h0m42.00110468s]
[2025/09/10 09:33:57.547 +08:00] [INFO] [replication_manager.go:635] [“schedulerv3: slow table”] [namespace=default] [changefeed=xxxxx-9-2] [tableID=87194] [tableStatus=] [checkpointTs=460704012171476997] [resolvedTs=460704012171476997] [checkpointLag=6h1m42.050682396s]
[2025/09/10 09:33:57.547 +08:00] [INFO] [replication_manager.go:635] [“schedulerv3: slow table”] [namespace=default] [changefeed=xxxxx-9-2] [tableID=55303] [tableStatus=] [checkpointTs=460704012171476997] [resolvedTs=460704012171476997] [checkpointLag=6h1m42.050682396s]
[2025/09/10 09:33:57.547 +08:00] [INFO] [replication_manager.go:635] [“schedulerv3: slow table”] [namespace=default] [changefeed=xxxxx-9-2] [tableID=52723] [tableStatus=] [checkpointTs=460704012171476997] [resolvedTs=460704012171476997] [checkpointLag=6h1m42.050682396s]
[2025/09/10 09:33:57.547 +08:00] [INFO] [replication_manager.go:635] [“schedulerv3: slow table”] [namespace=default] [changefeed=xxxxx-9-2] [tableID=55363] [tableStatus=] [checkpointTs=460704012171476997] [resolvedTs=460704012171476997] [checkpointLag=6h1m42.050682396s]

【其他附件：截图/日志/监控】

2

对应得 changefeed 也重启过了， cdc得日志显示有slow table ， checkpoint_tso 变化很慢。 如何找到慢得原因呢

3

下游是什么数据库？如果是tidb admin show ddl jobs看看是不是有ddl卡住了

4

有正在备份吗

5

下游也是tidb，tidb的异地备份。在备库查看ddl嘛，

6

看日志好像是上游region读取阻塞? 或者下游sink写入失败，在可能是ticdc内部调度器卡死？

7

异地容灾从库？那延迟大是正常的

8

1、同步中断了？
2、从库上面再做什么？

9

看看上游正在干什么，资源都用在什么服务上了

10

网络问题吗

11

下游也是tidb的， 单独处理下同步lag 慢的表。

12

处理 OK 了？