cdc 不同步数据

【 TiDB 使用环境】生产环境 or 测试环境 or POC
【 TiDB 版本】
【遇到的问题】
集群版本:4.0.6
搭建完cdc以后,上游表插入数据,下游查询不到任何数据,不知道是哪里有问题。
如下是我的配置过程:

1、上游创建数据库和表
create database cdcdb;
use cdcdb;

create table test(id int not null primary key, name varchar(100));

2、下游创建数据库和表
create database cdcdb;
use cdcdb;

create table test(id int not null primary key, name varchar(100));

3、下游数据库添加复制用户
mysql> create user ‘root’@‘xxx.xxx.xxx.136’ identified by ‘admin’;
Query OK, 0 rows affected (0.03 sec)

mysql> grant all on . to ‘root’@‘xxx.xxx.xxx.136’;
Query OK, 0 rows affected (0.02 sec)

测试,从上游连接到下游的数据库,可以连通!
[tidb@TIDBjsjdsjkxt01 ~]$ /home/tidb/tidb_init/mysql-5.7.26-el7-x86_64/bin/mysql -h xxx.xxx.xxx.128 -uroot -P 4000 -p

4、使用非配置文件,创建同步任务
[tidb@TIDBjsjdsjkxt01 ~]$ cdc cli changefeed create --pd=http://xxx.xxx.xxx.136:2379 --sink-uri=“tidb://root:admin@xxx.xxx.xxx.128:4000/” --changefeed-id=“cdc2” --sort-engine=“unified”
[WARN] some tables are not eligible to replicate, model.TableName{model.TableName{Schema:“dbs”, Table:“t4”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“g4”, TableID:0, IsPartition:false}, model.TableName{Schema:“zjp”, Table:“zjps”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“yf10.29”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“t22”, TableID:0, IsPartition:false}, model.TableName{Schema:“gpxtest”, Table:“zhang100”, TableID:0, IsPartition:false}, model.TableName{Schema:“zjp”, Table:“test”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“g6”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“g2”, TableID:0, IsPartition:false}, model.TableName{Schema:“gpxtest”, Table:“t1”, TableID:0, IsPartition:false}, model.TableName{Schema:“gpxtest”, Table:“guo001”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“bbb”, TableID:0, IsPartition:false}, model.TableName{Schema:“zjp”, Table:“seq_test”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“t33”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“t01”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“yftest1”, TableID:0, IsPartition:false}, model.TableName{Schema:“gpxtest”, Table:“guo004”, TableID:0, IsPartition:false}, model.TableName{Schema:“dbs”, Table:“t2”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“t2”, TableID:0, IsPartition:false}, model.TableName{Schema:“gpxtest”, Table:“guo002”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“g5”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“zhang100”, TableID:0, IsPartition:false}, model.TableName{Schema:“zjp”, Table:“time”, TableID:0, IsPartition:false}, model.TableName{Schema:“gpxtest”, Table:“zhang1001”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“yftest”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“ee”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“g3”, TableID:0, IsPartition:false}, model.TableName{Schema:“dbs”, Table:“t3”, TableID:0, IsPartition:false}, model.TableName{Schema:“zjp”, Table:“zhang2”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“t333”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“g7”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“a”, TableID:0, IsPartition:false}, model.TableName{Schema:“gpxtest”, Table:“guo003”, TableID:0, IsPartition:false}, model.TableName{Schema:“zjp”, Table:“t2”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“g1”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“test”, TableID:0, IsPartition:false}, model.TableName{Schema:“dbs”, Table:“t5”, TableID:0, IsPartition:false}, model.TableName{Schema:“zjp”, Table:“seq_zjp”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“backup_status”, TableID:0, IsPartition:false}, model.TableName{Schema:“test”, Table:“zhang”, TableID:0, IsPartition:false}}
Could you agree to ignore those tables, and continue to replicate [Y/N]
Y
Create changefeed successfully!
ID: cdc2
Info: {“sink-uri”:“tidb://root:admin@xxx.xxx.xxx.128:4000/”,“opts”:{},“create-time”:“2022-06-22T11:24:42.813427811+08:00”,“start-ts”:434075935120556033,“target-ts”:0,“admin-job-type”:0,“sort-engine”:“unified”,“sort-dir”:“.”,“config”:{“case-sensitive”:true,“enable-old-value”:false,“filter”:{“rules”:[“.”],“ignore-txn-start-ts”:null,“ddl-allow-list”:null},“mounter”:{“worker-num”:16},“sink”:{“dispatchers”:null,“protocol”:“default”},“cyclic-replication”:{“enable”:false,“replica-id”:0,“filter-replica-ids”:null,“id-buckets”:0,“sync-ddl”:false},“scheduler”:{“type”:“table-number”,“polling-time”:-1}},“state”:“normal”,“history”:null,“error”:null}

5、查询同步状态
[tidb@TIDBjsjdsjkxt01 ~]$ cdc cli changefeed list --pd=http://xxx.xxx.xxx.136:2379
[
{
“id”: “cdc2”,
“summary”: {
“state”: “normal”,
“tso”: 434075935120556033,
“checkpoint”: “2022-06-22 11:24:42.778”,
“error”: null
}
}
]

6、插入测试数据,下游没有查询到。

相关cdc日志里面没有发现有报错信息,请见附件。
cdc.log (9.9 KB)

【复现路径】做过哪些操作出现的问题
【问题现象及影响】

【附件】

请提供各个组件的 version 信息,如 cdc/tikv,可通过执行 cdc version/tikv-server --version 获取。

检查一下cdc owner是否有问题

你指的是哪一方面?

cdc cli capture list --pd=http://10.0.10.25:2379 找到owner节点 看下日志

我也遇到过这个现象,我当时是4.0.15版本。后来发现是cdc请求pd设计上的问题,后来4.0.16修复了。

4.0.16 版本之前的版本都会有这个问题吗?

cdc.log 里有这段日志,这张表被忽略了,不会同步:
[2022/06/22 13:15:45.841 +08:00] [INFO] [owner.go:306] ["ignore known table"] [tid=455] [table=cdcdb.test] [ts=434075935120556033]

检查下changefeed或者配置,是不是之前同步过?

这个表之前没有同步过,新搭建的同步任务。

对比cdc日志内容
[2022/06/22 13:15:45.841 +08:00] [WARN] [owner.go:298] [“skip ineligible table”] [tid=287] [table=test.g1]
[2022/06/22 13:15:45.841 +08:00] [WARN] [owner.go:298] [“skip ineligible table”] [tid=363] [table=dbs.t3]
[2022/06/22 13:15:45.841 +08:00] [INFO] [owner.go:306] [“ignore known table”] [tid=455] [table=cdcdb.test] [ts=434075935120556033]

skip ineligible table:这个是忽略表,不同步表的意思。
ignore known table:这个信息看到过,对比skip警告的信息,我以为这个是没有问题呢。

我这个表是有主键的呀,不应该有问题吧?

并且我创建同步任务的时候,[WARN] some tables are not eligible to replicate,这个提示信息里面没有提到我这个表。

提示了

你配置文件是怎么样的?不指定配置文件默认是什么行为?

难道是 test.test 和 cdcdb.test 冲突了?

我没有指定配置文件,直接命令行创建同步任务,如下:
cdc cli changefeed create --pd=http://xxx.xxx.xxx.136:2379 --sink-uri=“tidb://root:admin@xxx.xxx.xxx.128:4000/” --changefeed-id=“cdc2” --sort-engine=“unified”

创建同步任务以后,我在上游创建了一个所有库中不存在的一个表,这个表没有同步到下游。

tikv日志里面,偶尔会报如下警告信息:
[2022/06/23 09:02:23.765 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.765 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.765 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.765 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.765 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.765 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.765 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.773 +08:00] [INFO] [owner.go:1159] [“cleanup stale task”] [captureid=891cda67-f042-4027-a1b5-da80cdd13204] [changefeedid=cdc2]
[2022/06/23 09:02:23.773 +08:00] [WARN] [owner.go:1135] [“task position not found, fallback to use original start ts”] [capture=f292a158-03e5-4b79-a6cc-461076b567a6] [changefeed=cdc2] [“task status”="{“tables”:{“258”:{“start-ts”:434075935120556033,“mark-table-id”:0},“260”:{“start-ts”:434075935120556033,“mark-table-id”:0},“288”:{“start-ts”:434075935120556033,“mark-table-id”:0},“291”:{“start-ts”:434075935120556033,“mark-table-id”:0},“369”:{“start-ts”:434075935120556033,“mark-table-id”:0},“389”:{“start-ts”:434075935120556033,“mark-table-id”:0},“443”:{“start-ts”:434075935120556033,“mark-table-id”:0}},“operation”:{“258”:{“delete”:false,“boundary_ts”:434075935120556033,“done”:false},“260”:{“delete”:false,“boundary_ts”:434075935120556033,“done”:false},“288”:{“delete”:false,“boundary_ts”:434075935120556033,“done”:false},“291”:{“delete”:false,“boundary_ts”:434075935120556033,“done”:false},“369”:{“delete”:false,“boundary_ts”:434075935120556033,“done”:false},“389”:{“delete”:false,“boundary_ts”:434075935120556033,“done”:false},“443”:{“delete”:false,“boundary_ts”:434075935120556033,“done”:false}},“admin-job-type”:1}"]
[2022/06/23 09:02:23.773 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.773 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.773 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.773 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.773 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.773 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.773 +08:00] [WARN] [owner.go:186] [“changefeed not found”] [changefeed=cdc2]
[2022/06/23 09:02:23.781 +08:00] [INFO] [owner.go:1159] [“cleanup stale task”] [captureid=f292a158-03e5-4b79-a6cc-461076b567a6] [changefeedid=cdc2]
[2022/06/23 09:02:23.781 +08:00] [INFO] [owner.go:1196] [“monitoring captures”] [key=/tidb/cdc/capture] [rev=23938946]

使用tiup ctl cdc创建同步任务,也不行。
谁能帮我看看这个问题。

经过测试,得出结论如下:
v4.0.6环境下配置CDC同步任务完成以后,数据同步不能同步到下游。
升级集群到v4.0.16以后,数据同步正常。

看来你说的没错。

1 个赞

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。