ticdc 状态正常无法推送数据到kafka

🪐 TiDB 技术问题 备份&数据迁移
1

cdc状态正常 但是无法推送数据到kafka

ticdc状态如下：
cdc cli changefeed list --pd=http://172.16.1.30:2379
[
{
“id”: “6dca52ea-c092-48b6-9588-355af5472613”,
“summary”: {
“state”: “normal”,
“tso”: 427193854633705472,
“checkpoint”: “2021-08-22 14:53:49.163”,
“error”: null
}
}
]

cdc.log 一直报此错误，日志刷得非常快：
[2021/08/23 10:30:51.859 +08:00] [INFO] [client.go:868] [“start new request”] [request="{“header”:{“cluster_id”:6994643796087371628,“ticdc_version”:“5.1.0”},“region_id”:5135,“region_epoch”:{“conf_ver”:5,“version”:287},“checkpoint_ts”:427193854633705472,“start_key”:“dIAAAAAAAAH/RV9ygAAAAAD/HNB+AAAAAAD6”,“end_key”:“dIAAAAAAAAH/RV9ygAAAAAD/HO4YAAAAAAD6”,“request_id”:7702927,“extra_op”:1,“Request”:null}"] [addr=10.49.1.216:20160]
[2021/08/23 10:30:51.860 +08:00] [INFO] [region_range_lock.go:218] [“range locked”] [lockID=54] [regionID=1309] [startKey=7480000000000000ffb35f720000000000fa] [endKey=7480000000000000ffb35f730000000000fa] [checkpointTs=427193854633705472]
[2021/08/23 10:30:51.860 +08:00] [INFO] [client.go:868] [“start new request”] [request="{“header”:{“cluster_id”:6994643796087371628,“ticdc_version”:“5.1.0”},“region_id”:1309,“region_epoch”:{“conf_ver”:5,“version”:175},“checkpoint_ts”:427193854633705472,“start_key”:“dIAAAAAAAAD/s19yAAAAAAD6”,“end_key”:“dIAAAAAAAAD/s19zAAAAAAD6”,“request_id”:7702928,“extra_op”:1,“Request”:null}"] [addr=10.49.1.216:20160]
[2021/08/23 10:30:51.860 +08:00] [INFO] [region_worker.go:206] [“single region event feed disconnected”] [regionID=977] [requestID=7702268] [span="[7480000000000001ff245f728000000003ff3a3d8c0000000000fa, 7480000000000001ff245f730000000000fa)"] [checkpoint=427193854633705472] [error="[CDC:ErrEventFeedEventError]region_not_found:<region_id:977 > “]
[2021/08/23 10:30:51.860 +08:00] [INFO] [region_range_lock.go:370] [“unlocked range”] [lockID=111] [regionID=977] [startKey=7480000000000001ff245f728000000003ff3a3d8c0000000000fa] [endKey=7480000000000001ff245f730000000000fa] [checkpointTs=427193854633705472]
[2021/08/23 10:30:51.861 +08:00] [INFO] [region_range_lock.go:218] [“range locked”] [lockID=111] [regionID=977] [startKey=7480000000000001ff245f728000000003ff3a3d8c0000000000fa] [endKey=7480000000000001ff245f730000000000fa] [checkpointTs=427193854633705472]
[2021/08/23 10:30:51.861 +08:00] [INFO] [client.go:868] [“start new request”] [request=”{“header”:{“cluster_id”:6994643796087371628,“ticdc_version”:“5.1.0”},“region_id”:977,“region_epoch”:{“conf_ver”:5,“version”:245},“checkpoint_ts”:427193854633705472,“start_key”:“dIAAAAAAAAH/JF9ygAAAAAP/Oj2MAAAAAAD6”,“end_key”:“dIAAAAAAAAH/JF9zAAAAAAD6”,“request_id”:7702929,“extra_op”:1,“Request”:null}"] [addr=10.49.0.66:20160]
[2021/08/23 10:30:51.863 +08:00] [INFO] [region_worker.go:206] [“single region event feed disconnected”] [regionID=4949] [requestID=7702301] [span="[7480000000000000ffcb5f728000000000ff02fd810000000000fa, 7480000000000000ffcb5f728000000000ff031acd0000000000fa)"] [checkpoint=427193854633705472] [error="[CDC:ErrEventFeedEventError]region_not_found:<region_id:4949 > “]
[2021/08/23 10:30:51.863 +08:00] [INFO] [region_range_lock.go:370] [“unlocked range”] [lockID=114] [regionID=4949] [startKey=7480000000000000ffcb5f728000000000ff02fd810000000000fa] [endKey=7480000000000000ffcb5f728000000000ff031acd0000000000fa] [checkpointTs=427193854633705472]
[2021/08/23 10:30:51.864 +08:00] [INFO] [region_range_lock.go:218] [“range locked”] [lockID=114] [regionID=4949] [startKey=7480000000000000ffcb5f728000000000ff02fd810000000000fa] [endKey=7480000000000000ffcb5f728000000000ff031acd0000000000fa] [checkpointTs=427193854633705472]
[2021/08/23 10:30:51.864 +08:00] [INFO] [client.go:868] [“start new request”] [request=”{“header”:{“cluster_id”:6994643796087371628,“ticdc_version”:“5.1.0”},“region_id”:4949,“region_epoch”:{“conf_ver”:5,“version”:167},“checkpoint_ts”:427193854633705472,“start_key”:“dIAAAAAAAAD/y19ygAAAAAD/Av2BAAAAAAD6”,“end_key”:“dIAAAAAAAAD/y19ygAAAAAD/AxrNAAAAAAD6”,“request_id”:7702930,“extra_op”:1,“Request”:null}"] [addr=10.49.0.66:20160]
[2021/08/23 10:30:51.879 +08:00] [INFO] [region_worker.go:206] [“single region event feed disconnected”] [regionID=1561] [requestID=7702155] [span="[7480000000000001ff325f728000000000ff1de46b0000000000fa, 7480000000000001ff325f728000000000ff1e0b7a0000000000fa)"] [checkpoint=427193854633705472] [error="[CDC:ErrEventFeedEventError]region_not_found:<region_id:1561 > “]
[2021/08/23 10:30:51.879 +08:00] [INFO] [region_range_lock.go:370] [“unlocked range”] [lockID=76] [regionID=1561] [startKey=7480000000000001ff325f728000000000ff1de46b0000000000fa] [endKey=7480000000000001ff325f728000000000ff1e0b7a0000000000fa] [checkpointTs=427193854633705472]
[2021/08/23 10:30:51.879 +08:00] [INFO] [region_range_lock.go:218] [“range locked”] [lockID=76] [regionID=1561] [startKey=7480000000000001ff325f728000000000ff1de46b0000000000fa] [endKey=7480000000000001ff325f728000000000ff1e0b7a0000000000fa] [checkpointTs=427193854633705472]
[2021/08/23 10:30:51.880 +08:00] [INFO] [client.go:868] [“start new request”] [request=”{“header”:{“cluster_id”:6994643796087371628,“ticdc_version”:“5.1.0”},“region_id”:1561,“region_epoch”:{“conf_ver”:5,“version”:253},“checkpoint_ts”:427193854633705472,“start_key”:“dIAAAAAAAAH/Ml9ygAAAAAD/HeRrAAAAAAD6”,“end_key”:“dIAAAAAAAAH/Ml9ygAAAAAD/Hgt6AAAAAAD6”,“request_id”:7702931,“extra_op”:1,“Request”:null}"] [addr=10.49.1.216:20160]
[2021/08/23 10:30:51.884 +08:00] [INFO] [region_worker.go:206] [“single region event feed disconnected”] [regionID=2021] [requestID=7702157] [span="[7480000000000001ff405f728000000000ff1407fa0000000000fa, 7480000000000001ff405f728000000000ff142c9c0000000000fa)"] [checkpoint=427193854633705472] [error="[CDC:ErrEventFeedEventError]region_not_found:<region_id:2021 > "]

2

TiDB 集群的版本是多少？先看下具体同步任务的状态，命令：
cdc cli changefeed query --pd=http://10.0.10.25:2379 --changefeed-id={task-name}

3

v5.1.1 现在是正常的 重启集群后 region找不到 应该是调度有问题

4

日志中是一直提示 region is unvaliable 吗？可以使用 pd-ctl 看下具体 region 的状态情况。

5

嗯嗯 目前已经是正常状态不好复现 但是可以分享下你分析的问题

6

这个不好复现的话排查起来就比较困难了，重启后 region 找不到，猜测是 tidb 中缓存的 region 信息过旧，重试后刷新一遍 region cache 就恢复正常了；下次再遇到的话可以保留下具体日志、任务详细的状态信息以及集群的日志等信息。

7

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。