Drainer 集群同步数据到 MariaDB 产生重复数据

🪐 TiDB 技术问题 备份&数据迁移
1
前置条件

TiDB 3.0.2 binlog同步数据到 MariaDB 10.3

场景

三台 drainer 服务器

问题发生时间
  1. drainer 刚启动时；注：过了大概10分钟以后 这个问题就会不存在
  2. 创建的列不添加主键
我做了什么
  1. 在TiDB新建数据库 eric-tidb
  2. 创建表 表名 t1
  3. 添加列 列名 ids （不设置主键）
  4. 向表中添加一条数据 1

查看MariaDB同步的数据，eric-tidb → t1 → 出现内容都是 1的三条数据

尝试排查 查看drainer.log 异常信息如下：
[2019/08/21 11:23:08.057 +08:00] [INFO] [syncer.go:254] ["write save point"] [ts=410611295353307137]
[2019/08/21 11:23:11.777 +08:00] [INFO] [syncer.go:254] ["write save point"] [ts=410611296507002881]
[2019/08/21 11:23:16.114 +08:00] [INFO] [syncer.go:254] ["write save point"] [ts=410611297542209537]
[2019/08/21 11:23:17.079 +08:00] [INFO] [client.go:758] ["recycle idle connection"] [target=192.168.180.47:20160]
[2019/08/21 11:23:17.090 +08:00] [INFO] [collector.go:280] ["get ddl job"] [job="ID:1130, Type:create table, State:synced, SchemaState:public, SchemaID:1125, TableID:1129, RowCount:0, ArgLen:0, start time: 2019-08-21 11:23:14.769 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0"]
[2019/08/21 11:23:17.090 +08:00] [INFO] [syncer.go:403] ["add ddl item to syncer, you can add this commit ts to `ignore-txn-commit-ts` to skip this ddl if needed"] [sql="CREATE TABLE `eric-tidb`.`t2`  (\r
  `ids` varchar(255) NULL\r
)"] ["commit ts"=410611297751924747]
[2019/08/21 11:23:22.181 +08:00] [WARN] [load.go:592] ["ignore ddl"] [error="Error 1050: Table 't2' already exists"] [errorVerbose="Error 1050: Table 't2' already exists
github.com/pingcap/errors.AddStack
	/home/jenkins/workspace/release_tidb_3.0/go/pkg/mod/github.com/pingcap/errors@v0.11.4/errors.go:174
github.com/pingcap/errors.Trace
	/home/jenkins/workspace/release_tidb_3.0/go/pkg/mod/github.com/pingcap/errors@v0.11.4/juju_adaptor.go:15
github.com/pingcap/tidb-binlog/pkg/loader.(*loaderImpl).execDDL
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-binlog/pkg/loader/load.go:337
github.com/pingcap/tidb-binlog/pkg/loader.(*batchManager).execDDL
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-binlog/pkg/loader/load.go:587
github.com/pingcap/tidb-binlog/pkg/loader.(*batchManager).put
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-binlog/pkg/loader/load.go:610
github.com/pingcap/tidb-binlog/pkg/loader.(*loaderImpl).Run
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-binlog/pkg/loader/load.go:473
github.com/pingcap/tidb-binlog/drainer/sync.(*MysqlSyncer).run
	/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-binlog/drainer/sync/mysql.go:118
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1337"] [ddl="CREATE TABLE `eric-tidb`.`t2`  (\r
  `ids` varchar(255) NULL\r
)"]
[2019/08/21 11:23:22.181 +08:00] [INFO] [syncer.go:254] ["write save point"] [ts=410611297751924747]
[2019/08/21 11:23:22.181 +08:00] [INFO] [load.go:241] ["refresh table info"] [schema=eric-tidb] [table=t2]
[2019/08/21 11:23:22.183 +08:00] [WARN] [load.go:257] ["table has no any primary key and unique index, it may be slow when syncing data to downstream, we highly recommend add primary key or unique key for table"] [table=`eric-tidb`.`t2`]
[2019/08/21 11:23:26.097 +08:00] [INFO] [syncer.go:254] ["write save point"] [ts=410611300085006338]
[2019/08/21 11:23:29.814 +08:00] [INFO] [syncer.go:254] ["write save point"] [ts=410611301238439937]
[tidb@test1 deploy]$

tidb

2

我也遇到了同样的问题

3

问题解决了，原来 一个drainer 只能对应一个下游， 是使用上的问题

4

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。