cdc changefeed出现较大延迟

【 TiDB 使用环境】生产环境
【 TiDB 版本】v5.2.4
【遇到的问题:问题现象及影响】cdc 同步MySQL出现较大的延迟,另外两个changefeed也是同步到同样的MySQL实例上,没出现较大的延迟
【资源配置】
3台 tidb/pd 16c * 64g, 其中每台机器部署一个cdc节点
3台 tikv 16c * 64g

【附件:截图/日志/监控】


eeo-tidb002-TiCDC_2022-11-23T02_34_03.571Z.json (208.3 KB)

[2022/11/22 23:30:46.205 +08:00] [INFO] [region_worker.go:243] [“single region event feed disconnected”] [changefeed=eoosfile-dqs] [regionID=2562584] [requestID=855] [span=“[7480000000000001ff475f72800000000cff9e27da0000000000fa, 7480000000000001ff475f72800000000cffa821480000000000fa)”] [checkpoint=437552688802299918] [error=“[CDC:ErrEventFeedEventError]not_leader:<region_id:2562584 leader:<id:2562586 store_id:1 > > : not_leader:<region_id:2562584 leader:<id:2562586 store_id:1 > > “]
[2022/11/22 23:30:46.205 +08:00] [INFO] [region_range_lock.go:383] [“unlocked range”] [changefeed=eoosfile-dqs] [lockID=3] [regionID=2562584] [startKey=7480000000000001ff475f72800000000cff9e27da0000000000fa] [endKey=7480000000000001ff475f72800000000cffa821480000000000fa] [checkpointTs=437552688802299918]
[2022/11/22 23:30:46.205 +08:00] [INFO] [region_cache.go:1102] [“switch region leader to specific leader due to kv return NotLeader”] [regionID=2562584] [currIdx=2] [leaderStoreID=1]
[2022/11/22 23:30:46.205 +08:00] [INFO] [region_range_lock.go:222] [“range locked”] [changefeed=eoosfile-dqs] [lockID=3] [regionID=2562584] [startKey=7480000000000001ff475f72800000000cff9e27da0000000000fa] [endKey=7480000000000001ff475f72800000000cffa821480000000000fa] [checkpointTs=437552688802299918]
[2022/11/22 23:30:46.205 +08:00] [INFO] [client.go:926] [“cannot get rpcCtx, retry span”] [changefeed=eoosfile-dqs] [regionID=2562584] [span=”[7480000000000001ff475f72800000000cff9e27da0000000000fa, 7480000000000001ff475f72800000000cffa821480000000000fa)”]
[2022/11/22 23:30:46.205 +08:00] [INFO] [region_range_lock.go:383] [“unlocked range”] [changefeed=eoosfile-dqs] [lockID=3] [regionID=2562584] [startKey=7480000000000001ff475f72800000000cff9e27da0000000000fa] [endKey=7480000000000001ff475f72800000000cffa821480000000000fa] [checkpointTs=437552688802299918]
[2022/11/22 23:30:46.205 +08:00] [INFO] [region_range_lock.go:222] [“range locked”] [changefeed=eoosfile-dqs] [lockID=3] [regionID=2562584] [startKey=7480000000000001ff475f72800000000cff9e27da0000000000fa] [endKey=7480000000000001ff475f72800000000cffa821480000000000fa] [checkpointTs=437552688802299918]
[2022/11/22 23:30:46.206 +08:00] [INFO] [client.go:825] [“start new request”] [changefeed=eoosfile-dqs] [request=“{"header":{"cluster_id":7039262237001066123,"ticdc_version":"5.2.4"},"region_id":2562584,"region_epoch":{"conf_ver":2609,"version":4353},"checkpoint_ts":437552688802299918,"start_key":"dIAAAAAAAAH/R19ygAAAAAz/nifaAAAAAAD6","end_key":"dIAAAAAAAAH/R19ygAAAAAz/qCFIAAAAAAD6","request_id":2075,"extra_op":1,"Request":null}”] [addr=10.1.38.111:20161]
[2022/11/22 23:30:46.207 +08:00] [INFO] [region_worker.go:243] [“single region event feed disconnected”] [changefeed=eoosfile-dqs] [regionID=2562584] [requestID=2075] [span=“[7480000000000001ff475f72800000000cff9e27da0000000000fa, 7480000000000001ff475f72800000000cffa821480000000000fa)”] [checkpoint=437552688802299918] [error="[CDC:ErrEventFeedEventError]not_leader:<region_id:2562584 leader:<id:2562586 store_id:1 > > : not_leader:<region_id:2562584 leader:<id:2562586 store_id:1 > > "]

如果是同一个tidb分三个changefeed同步到同一个MySQL,其他两个没有延时,说明大概率不是tidb或者MySQL的问题,可以考虑看下出现延时的那个changefeed上的表是不是有大事务。看你贴的曲线,直线上升,断崖下降,非常像大事务。