dm-worker 启动报错no current UUID set

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。

  • 【TiDB 版本】:3.0.4
  • 【问题描述】:dm搭建启动work后报错

[2020/02/17 16:35:32.846 +08:00] [ERROR] [metrics.go:84] [“fail to start status server return”] [error=“mux: listener closed”] [2020/02/17 16:35:32.883 +08:00] [ERROR] [relay.go:646] [“flush checkpoint”] [component=“relay log”] [error="[code=30005:class=relay-unit:scope=internal:level=high] no current UUID set"] [errorVerbose="[code=30005:class=relay-unit:scope=internal:level=high] no current UUID set\ngithub.com/pingcap/dm/pkg/terror.(*Error).Generate\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:232\ngithub.com/pingcap/dm/relay.(*LocalMeta).doFlush\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/meta.go:233\ngithub.com/pingcap/dm/relay.(*LocalMeta).Flush\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/meta.go:227\ngithub.com/pingcap/dm/relay.(*Relay).stopSync\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:645\ngithub.com/pingcap/dm/relay.(*Relay).Close\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:666\ngithub.com/pingcap/dm/dm/worker.(*realRelayHolder).Close\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:158\ngithub.com/pingcap/dm/dm/worker.(*Worker).Close\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/worker.go:214\ngithub.com/pingcap/dm/dm/worker.(*Server).Close\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/server.go:141\ main.main.func2\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/cmd/dm-worker/main.go:73\ runtime.goexit\ \t/usr/local/go/src/runtime/asm_amd64.s:1357"] [2020/02/17 17:40:25.720 +08:00] [INFO] [syncer.go:1992] [“print status routine exits”] [task=test] [unit=“binlog replication”] [error=“context canceled”] [2020/02/17 17:40:25.738 +08:00] [WARN] [reader.go:137] [“parse relay finished”] [task=test] [unit=“binlog replication”] [component=“binlog reader”] [error=“parse relay log file mysql-bin.001533 from offset 2043 in dir /data/tidb/dm/relay_log/1bfe5e41-adfe-11e9-988b-b8ca3a621608.000001: parse relay log file mysql-bin.001533 from offset 0 in dir /data/tidb/dm/relay_log/1bfe5e41-adfe-11e9-988b-b8ca3a621608.000001: context canceled”] [2020/02/17 17:40:25.754 +08:00] [ERROR] [subtask.go:255] [“unit process error”] [subtask=test] [unit=Sync] [“error information”="{“msg”:"[code=36057:class=sync-unit:scope=internal:level=high] reopen *syncer.localBinlogReader not supported\ngithub.com/pingcap/dm/pkg/terror.(*Error).Generate\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:232\ngithub.com/pingcap/dm/syncer.(*Syncer).reopen\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:2157\ngithub.com/pingcap/dm/syncer.(*Syncer).reopenWithRetry\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:2134\ngithub.com/pingcap/dm/syncer.(*Syncer).Run\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1244\ngithub.com/pingcap/dm/syncer.(*Syncer).Process\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:598\ runtime.goexit\ \t/usr/local/go/src/runtime/asm_amd64.s:1357",“error”:{“ErrCode”:36057,“ErrClass”:11,“ErrScope”:3,“ErrLevel”:3,“Message”:“reopen *syncer.localBinlogReader not supported”}}"] [2020/02/17 17:51:28.371 +08:00] [INFO] [syncer.go:1992] [“print status routine exits”] [task=test] [unit=“binlog replication”] [error=“context canceled”] [2020/02/17 17:51:28.384 +08:00] [WARN] [reader.go:137] [“parse relay finished”] [task=test] [unit=“binlog replication”] [component=“binlog reader”] [error=“parse relay log file mysql-bin.001533 from offset 2043 in dir /data/tidb/dm/relay_log/1bfe5e41-adfe-11e9-988b-b8ca3a621608.000001: parse relay log file mysql-bin.001533 from offset 0 in dir /data/tidb/dm/relay_log/1bfe5e41-adfe-11e9-988b-b8ca3a621608.000001: context canceled”]

工具问题麻烦提供工具的对应的版本,这边辛苦提供下 dm 的版本,另外提供下 inverntory.ini 中的配置,看报错是没有设置 UUID。这边可以检查下上游的 UUID 信息,和目录中的 UUID 信息是否相对应。

[root@emarsys105016 dm-ansible-v1.0.3]# more inventory.ini

DM modules

[dm_master_servers] dm_master ansible_host=xxx.xxx.242.16

[dm_worker_servers] dm-worker1 ansible_host=xxx.xxx.242.17 source_id=“mysql-replica-01” server_id=101 mysql_host=xxx.xxx.127.190 mysql_user=root mysql_p assword= mysql_port=3308

#dm-worker2 ansible_host=172.16.10.73 source_id=“mysql-replica-02” server_id=102 mysql_host=172.16.10.73 mysql_user=root mysql_passw ord=123456 mysql_port=3306

[dm_portal_servers] dm_portal ansible_host=xxx.xxx.242.16

Monitoring modules

[prometheus_servers] prometheus ansible_host=xxx.xxx.242.16

[grafana_servers] grafana ansible_host=xxx.xxx.242.16

[alertmanager_servers] alertmanager ansible_host=xxx.xxx.242.16

Global variables

[all:vars] cluster_name = gs-cluster ansible_user = chenlei dm_version = v1.0.3

deploy_dir = /data/tidb/dm

grafana_admin_user = “admin” grafana_admin_password = “admin”

现在worker好像正常的,只是 totalEvents 一直是 0 。 [root@emarsys105016 bin]# dmctl --master-addr xxx.xxx.242.16:8261 query-status 127190_to_gs_heming { “result”: true, “msg”: “”, “workers”: [ { “result”: true, “worker”: “xxx.xxx.242.17:8262”, “msg”: “”, “subTaskStatus”: [ { “name”: “127190_to_gs_heming”, “stage”: “Running”, “unit”: “Sync”, “result”: null, “unresolvedDDLLockID”: “”, “sync”: { “totalEvents”: “0”, “totalTps”: “0”, “recentTps”: “0”, “masterBinlog”: “(mysql-bin.001533, 50172)”, “masterBinlogGtid”: “”, “syncerBinlog”: “(mysql-bin.001533, 50172)”, “syncerBinlogGtid”: “”, “blockingDDLs”: [ ], “unresolvedGroups”: [ ], “synced”: true } } ], “relayStatus”: { “masterBinlog”: “(mysql-bin.001533, 50172)”, “masterBinlogGtid”: “”, “relaySubDir”: “1bfe5e41-adfe-11e9-988b-b8ca3a621608.000001”, “relayBinlog”: “(mysql-bin.001533, 50172)”, “relayBinlogGtid”: “”, “relayCatchUpMaster”: true, “stage”: “Running”, “result”: null }, “sourceID”: “mysql-replica-01” } ] } dm-worker日志 总是忽略事件 没有同步相关的事件 [2020/02/18 10:33:11.966 +08:00] [INFO] [relay.go:425] [“ignore event by transformer”] [component=“relay log”] [header="{“Timestamp”:0,“EventType”:27,“ServerID”:13308,“EventSize”:39,“LogPos”:50172,“Flags”:0}"] [reason=“heartbeat event”]

[2020/02/18 10:27:04.267 +08:00] [WARN] [loader.go:800] [“invalid mydumper files for there are no -schema-create.sql files found, and will generate later”] [task=127190_to_gs_heming] [unit=load] [2020/02/18 10:27:04.267 +08:00] [WARN] [loader.go:803] [“no available -schema-create.sql files, check mydumper parameter matches black-white-list in task config, will generate later”] [task=127190_to_gs_heming] [unit=load]

发现了事件怎么都 skip event 都跳过去了 ,没有同步 ,另外之前也没有成功初始化dm_tidb_test库的同步 [2020/02/18 10:54:15.240 +08:00] [INFO] [relay.go:113] [“current earliest active relay log”] [task=127190_to_gs_heming] [unit=“binlo g replication”] [“active relay log”=1bfe5e41-adfe-11e9-988b-b8ca3a621608.000001/mysql-bin.001533] [2020/02/18 10:54:15.449 +08:00] [INFO] [syncer.go:1599] [task=127190_to_gs_heming] [unit=“binlog replication”] [event=query] [state ment=“CREATE TABLE user_hem (\ id int(11) NOT NULL DEFAULT ‘0’,\ card varchar(32) CHARACTER SET utf8 DEFAULT NULL COMMENT ‘编号’,\ name varchar(32) CHARACTER SET utf8 DEFAULT NULL COMMENT ‘姓名’,\ equipment_id int(11) DEFAULT NULL COMMENT '设备id ',\ area_id int(10) DEFAULT NULL COMMENT ‘区域ID’,\ start_time datetime DEFAULT NULL COMMENT ‘开始时间’,\ end_time datet ime DEFAULT NULL COMMENT ‘结束时间’,\ ifdel int(11) DEFAULT ‘0’ COMMENT ‘0:使用,1:已移除’\ ) ENGINE=InnoDB DEFAULT CHARSET=utf 8 COLLATE=utf8_bin”] [schema=dm_tidb_test] [“last position”="(mysql-bin|000001.001533, 50384)"] [position="(mysql-bin|000001.001533, 51093)"] [“gtid set”=NULL] [2020/02/18 10:54:15.455 +08:00] [INFO] [syncer.go:1615] [“resolve sql”] [task=127190_to_gs_heming] [unit=“binlog replication”] [eve nt=query] [“raw statement”=“CREATE TABLE user_hem (\ id int(11) NOT NULL DEFAULT ‘0’,\ card varchar(32) CHARACTER SET utf8 DEFAULT NULL COMMENT ‘编号’,\ name varchar(32) CHARACTER SET utf8 DEFAULT NULL COMMENT ‘姓名’,\ equipment_id int(11) DEFAUL T NULL COMMENT ‘设备id’,\ area_id int(10) DEFAULT NULL COMMENT ‘区域ID’,\ start_time datetime DEFAULT NULL COMMENT '开始时间 ',\ end_time datetime DEFAULT NULL COMMENT ‘结束时间’,\ ifdel int(11) DEFAULT ‘0’ COMMENT ‘0:使用,1:已移除’\ ) ENGINE=Inno DB DEFAULT CHARSET=utf8 COLLATE=utf8_bin”] [statements="[“CREATE TABLE IF NOT EXISTS dm_tidb_test.user_hem (id INT(11) NOT NU LL DEFAULT ‘0’,card VARCHAR(32) CHARACTER SET UTF8 DEFAULT NULL COMMENT ‘编号’,name VARCHAR(32) CHARACTER SET UTF8 DEFAULT NULL COMMENT ‘姓名’,equipment_id INT(11) DEFAULT NULL COMMENT ‘设备id’,area_id INT(10) DEFAULT NULL COMMENT ‘区域ID’,start_time DAT ETIME DEFAULT NULL COMMENT ‘开始时间’,end_time DATETIME DEFAULT NULL COMMENT ‘结束时间’,ifdel INT(11) DEFAULT ‘0’ COMMENT ‘0:使 ,1:已移除’) ENGINE = InnoDB DEFAULT CHARACTER SET = UTF8 DEFAULT COLLATE = UTF8_BIN”]"] [schema=dm_tidb_test] [“last position”="( mysql-bin|000001.001533, 51093)"] [position="(mysql-bin|000001.001533, 51093)"] [“gtid set”=NULL] [2020/02/18 10:54:15.457 +08:00] [WARN] [syncer.go:1647] [“skip event”] [task=127190_to_gs_heming] [unit=“binlog replication”] [even t=query] [statement=“CREATE TABLE IF NOT EXISTS dm_tidb_test.user_hem (id INT(11) NOT NULL DEFAULT ‘0’,card VARCHAR(32) CHAR ACTER SET UTF8 DEFAULT NULL COMMENT ‘编号’,name VARCHAR(32) CHARACTER SET UTF8 DEFAULT NULL COMMENT ‘姓名’,equipment_id INT(11) DEFAULT NULL COMMENT ‘设备id’,area_id INT(10) DEFAULT NULL COMMENT ‘区域ID’,start_time DATETIME DEFAULT NULL COMMENT ‘开始时间’, end_time DATETIME DEFAULT NULL COMMENT ‘结束时间’,ifdel INT(11) DEFAULT ‘0’ COMMENT ‘0:使用,1:已移除’) ENGINE = InnoDB DEFAULT CHARACTER SET = UTF8 DEFAULT COLLATE = UTF8_BIN”] [schema=dm_tidb_test] [2020/02/18 10:54:15.457 +08:00] [INFO] [syncer.go:1700] [“prepare to handle ddls”] [task=127190_to_gs_heming] [unit=“binlog replica tion”] [event=query] [ddls="[]"] [“raw statement”=“CREATE TABLE user_hem (\ id int(11) NOT NULL DEFAULT ‘0’,\ card varchar (32) CHARACTER SET utf8 DEFAULT NULL COMMENT ‘编号’,\ name varchar(32) CHARACTER SET utf8 DEFAULT NULL COMMENT ‘姓名’,\ equip ment_id int(11) DEFAULT NULL COMMENT ‘设备id’,\ area_id int(10) DEFAULT NULL COMMENT ‘区域ID’,\ start_time datetime DEFAULT NULL COMMENT ‘开始时间’,\ end_time datetime DEFAULT NULL COMMENT ‘结束时间’,\ ifdel int(11) DEFAULT ‘0’ COMMENT ‘0:使用,1 已移除’\ ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin”] [position="(mysql-bin|000001.001533, 51093)"] [2020/02/18 10:54:15.457 +08:00] [INFO] [syncer.go:1702] [“skip event, need handled ddls is empty”] [task=127190_to_gs_heming] [unit =“binlog replication”] [event=query] [“raw statement”=“CREATE TABLE user_hem (\ id int(11) NOT NULL DEFAULT ‘0’,\ card var char(32) CHARACTER SET utf8 DEFAULT NULL COMMENT ‘编号’,\ name varchar(32) CHARACTER SET utf8 DEFAULT NULL COMMENT ‘姓名’,\ e quipment_id int(11) DEFAULT NULL COMMENT ‘设备id’,\ area_id int(10) DEFAULT NULL COMMENT ‘区域ID’,\ start_time datetime DEF AULT NULL COMMENT ‘开始时间’,\ end_time datetime DEFAULT NULL COMMENT ‘结束时间’,\ ifdel int(11) DEFAULT ‘0’ COMMENT ‘0:使用 ,1:已移除’\ ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin”] [position="(mysql-bin|000001.001533, 51093)"] [2020/02/18 10:54:15.471 +08:00] [INFO] [syncer.go:1599] [task=127190_to_gs_heming] [unit=“binlog replication”] [event=query] [state ment="/*!40000 ALTER TABLE user_hem DISABLE KEYS /"] [schema=dm_tidb_test] [“last position”="(mysql-bin|000001.001533, 51093)"] [ position="(mysql-bin|000001.001533, 51295)"] [“gtid set”=NULL] [2020/02/18 10:54:15.472 +08:00] [INFO] [syncer.go:1615] [“resolve sql”] [task=127190_to_gs_heming] [unit=“binlog replication”] [eve nt=query] [“raw statement”="/!40000 ALTER TABLE user_hem DISABLE KEYS /"] [statements="[“ALTER TABLE dm_tidb_test.user_hem DISABLE KEYS”]"] [schema=dm_tidb_test] [“last position”="(mysql-bin|000001.001533, 51295)"] [position="(mysql-bin|000001.001533, 51 295)"] [“gtid set”=NULL] [2020/02/18 10:54:15.472 +08:00] [WARN] [syncer.go:1647] [“skip event"] [task=127190_to_gs_heming] [unit=“binlog replication”] [even t=query] [statement=“ALTER TABLE dm_tidb_test.user_hem DISABLE KEYS”] [schema=dm_tidb_test] [2020/02/18 10:54:15.472 +08:00] [INFO] [syncer.go:1700] [“prepare to handle ddls”] [task=127190_to_gs_heming] [unit=“binlog replica tion”] [event=query] [ddls=”[]"] [“raw statement”="/!40000 ALTER TABLE user_hem DISABLE KEYS /"] [position="(mysql-bin|000001.00 1533, 51295)"] [2020/02/18 10:54:15.472 +08:00] [INFO] [syncer.go:1702] [“skip event, need handled ddls is empty”] [task=127190_to_gs_heming] [unit =“binlog replication”] [event=query] [“raw statement”="/!40000 ALTER TABLE user_hem DISABLE KEYS /"] [position="(mysql-bin|00000 1.001533, 51295)"] [2020/02/18 10:54:15.649 +08:00] [INFO] [syncer.go:1599] [task=127190_to_gs_heming] [unit=“binlog replication”] [event=query] [state ment="/!40000 ALTER TABLE user_hem ENABLE KEYS /"] [schema=dm_tidb_test] [“last position”="(mysql-bin|000001.001533, 55162)"] [p osition="(mysql-bin|000001.001533, 55363)"] [“gtid set”=NULL] [2020/02/18 10:54:15.649 +08:00] [INFO] [syncer.go:1615] [“resolve sql”] [task=127190_to_gs_heming] [unit=“binlog replication”] [eve nt=query] [“raw statement”="/!40000 ALTER TABLE user_hem ENABLE KEYS /"] [statements="[“ALTER TABLE dm_tidb_test.user_hem E NABLE KEYS”]"] [schema=dm_tidb_test] [“last position”="(mysql-bin|000001.001533, 55363)"] [position="(mysql-bin|000001.001533, 5536 3)"] [“gtid set”=NULL] [2020/02/18 10:54:15.649 +08:00] [WARN] [syncer.go:1647] [“skip event”] [task=127190_to_gs_heming] [unit=“binlog replication”] [even t=query] [statement=“ALTER TABLE dm_tidb_test.user_hem ENABLE KEYS”] [schema=dm_tidb_test] [2020/02/18 10:54:15.649 +08:00] [INFO] [syncer.go:1700] [“prepare to handle ddls”] [task=127190_to_gs_heming] [unit=“binlog replica tion”] [event=query] [ddls="[]"] [“raw statement”="/!40000 ALTER TABLE user_hem ENABLE KEYS /"] [position="(mysql-bin|000001.001 533, 55363)"] [2020/02/18 10:54:15.649 +08:00] [INFO] [syncer.go:1702] [“skip event, need handled ddls is empty”] [task=127190_to_gs_heming] [unit =“binlog replication”] [event=query] [“raw statement”="/!40000 ALTER TABLE user_hem ENABLE KEYS */"] [position="(mysql-bin|000001 .001533, 55363)"] [2020/02/18 10:54:20.080 +08:00] [INFO] [syncer.go:2023] [“binlog replication progress”] [task=127190_to_gs_heming] [unit=“binlog re plication”] [“total binlog size”=5829] [“last binlog size”=162] [“cost time”=30] [bytes/Second=188] [“unsynced binlog size”=0] [“est imate time to catch up”=0] [2020/02/18 10:54:20.081 +08:00] [INFO] [syncer.go:2048] [“binlog replication status”] [task=127190_to_gs_heming] [unit=“binlog repl ication”] [total_events=0] [total_tps=0] [tps=0] [master_position="(mysql-bin.001533, 55363)"] [master_gtid=] [checkpoint="(mysql-bi n|000001.001533, 55162)(flushed (mysql-bin.001533, 50172))"] [2020/02/18 10:54:21.330 +08:00] [INFO] [relay.go:564] [“flush meta finished”] [component=“relay log”] [meta=“master-uuid = 1bfe5e41 -adfe-11e9-988b-b8ca3a621608.000001, relay-binlog = (mysql-bin.001533, 55363), relay-binlog-gtid = “] [2020/02/18 10:54:45.575 +08:00] [INFO] [relay.go:425] [“ignore event by transformer”] [component=“relay log”] [header=”{“Timestamp “:0,“EventType”:27,“ServerID”:13308,“EventSize”:39,“LogPos”:55363,“Flags”:0}”] [reason=“heartbeat event”] [2020/02/18 10:54:50.081 +08:00] [INFO] [syncer.go:2048] [“binlog replication status”] [task=127190_to_gs_heming] [unit=“binlog repl ication”] [total_events=0] [total_tps=0] [tps=0] [master_position=”(mysql-bin.001533, 55363)”] [master_gtid=] [checkpoint="(mysql-bi n|000001.001533, 55162)(flushed (mysql-bin.001533, 50172))"] [2020/02/18 10:55:15.575 +08:00] [INFO] [relay.go:425] [“ignore event by transformer”] [component=“relay log”] [header="{“Timestamp “:0,“EventType”:27,“ServerID”:13308,“EventSize”:39,“LogPos”:55363,“Flags”:0}”] [reason=“heartbeat event”] [2020/02/18 10:55:15.650 +08:00] [INFO] [syncer.go:1231] [“deadline exceeded when fetching binlog event”] [task=127190_to_gs_heming] [unit=“binlog replication”] [2020/02/18 10:55:15.650 +08:00] [INFO] [syncer.go:2117] [“flush all jobs”] [task=127190_to_gs_heming] [unit=“binlog replication”] [ “global checkpoint”=”(mysql-bin|000001.001533, 55162)(flushed (mysql-bin.001533, 50172))"] [2020/02/18 10:55:15.959 +08:00] [INFO] [syncer.go:870] [“flushed checkpoint”] [task=127190_to_gs_heming] [unit=“binlog replication” ] [checkpoint="(mysql-bin|000001.001533, 55162)(flushed (mysql-bin|000001.001533, 55162))"] [2020/02/18 10:55:15.960 +08:00] [INFO] [relay.go:113] [“current earliest active relay log”] [task=127190_to_gs_heming] [unit=“binlo g replication”] [“active relay log”=1bfe5e41-adfe-11e9-988b-b8ca3a621608.000001/mysql-bin.001533] [2020/02/18 10:55:20.081 +08:00] [INFO] [syncer.go:2048] [“binlog replication status”] [task=127190_to_gs_heming] [unit=“binlog repl ication”] [total_events=0] [total_tps=0] [tps=0] [master_position="(mysql-bin.001533, 55363)"] [master_gtid=] [checkpoint="(mysql-bi n|000001.001533, 55162)(flushed (mysql-bin|000001.001533, 55162))"]

DM-worker 启动成功,则表示能够从上游拉取 binlog 保存在本地。如果需要同步数据,是需要配置并开启同步任务的。启动同步任务后,才可以进行数据同步。
https://pingcap.com/docs-cn/stable/reference/tools/data-migration/deploy/

如果是全量同步,或者 全量 + 增量同步,可以检查下 dump 下来的全量数据中,是否包含 -schema-create.sql 文件,当前 dm-worker 的设计是 无此类文件则判定为无数据需要 load,这边可以检查下。

另外 task 配置麻烦整理下格式,多谢。

task配置文件
127190_to_gs_heming.yaml (1.1 KB)

这个规则注释的 没启用 #route-rules: [“instance-1-user-rules-table”, “instance-1-user-rules-schema”]

name: 127190_to_gs_heming
task-mode: all  # full/incremental/all
meta-schema: "dm_meta"
remove-meta: false

target-database:
  host: "xxx.xxx.242.17"
  port: 4000
  user: "root"
  password: ""

mysql-instances:
  -
    source-id: "mysql-replica-01"
    #route-rules: ["instance-1-user-rules-table", "instance-1-user-rules-schema"]
    black-white-list:  "bw-rule-1"
    mydumper-config-name: "global"
    loader-config-name: "global"
    syncer-config-name: "global"

routes:
  instance-1-user-rules-table:
    schema-pattern: "dm_tidb_test"
    table-pattern: u*
    target-schema: "dm_tidb_test_gs"
    target-table: u_gs

  instance-1-user-rules-schema:
    schema-pattern: "dm_tidb_test"
    target-schema: "dm_tidb_test_gs"




black-white-list:
  bw-rule-1:
    do-dbs: ["dm_tidb_test" ]
    do-tables:
    - db-name: "dm_tidb_test"
      tbl-name: "*"


mydumpers:
  global:
    threads: 4
        chunk-filesize: 64
    skip-tz-utc: true

loaders:
  global:
    pool-size: 16
    dir: "./dumped_data"

syncers:
  global:
    worker-count: 16
    batch: 100
    max-retry: 100

dumped 目录基本是空的 只有元数据文件

    [root@emarsys105017 dumped_data.127190_to_gs_heming]# l
    total 4.0K
    drwx------  2 chenlei chenlei  22 Feb 18 10:37 .
    drwxr-xr-x 12 chenlei chenlei 203 Feb 18 10:37 ..
    -rw-r--r--  1 chenlei chenlei 138 Feb 18 10:37 metadata
    [root@emarsys105017 dumped_data.127190_to_gs_heming]# more metadata 
    Started dump at: 2020-02-18 10:37:19
    SHOW MASTER STATUS:
            Log: mysql-bin.001533
            Pos: 50172
            GTID:

    Finished dump at: 2020-02-18 10:37:20 

可以从几个方向排查下 1.如果只同步 dm_tidb_test 库的话,可以把黑白名单部分修改成以下方式:

black-white-list:
  bw-rule-1:
    do-dbs: ["dm_tidb_test"]

重新启动 task 配置后,如仍不能正常 dump ,可以看下 dm-worker.log 日志中,dump 阶段的一些参数以及 table list

2.上游数据库的账号是否有对应的权限 ?可以参考链接 https://pingcap.com/docs-cn/stable/reference/tools/data-migration/dm-worker-intro/#dm-worker-所需权限

改了之后 可以同步上次手动创建的最后一个表 user_h 表了 ,之前的user user_heming 表依然没有同步 ,

另外dump目录还是只有元数据文件

改task文件之后 start-task之前 ./dumped_data 是否需要删除

权限没问题 GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, RELOAD, REFERENCES, INDEX, ALTER, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER ON . TO ‘chenlei’@’%’

麻烦提供下 dm-worker 的日志,以及当前 task 任务配置。以及你这边需要同步的库表情况。

另外你这边 routes 中 是把上游 dm_tidb_test 库下面的所有 以 u开头的表,同步到下游的 dm_tidb_test_gs 库下的 u_gs 表中,上游以 u 开头的表是只有一个吗 ? 多个的话会有问题的,

这个路由规则没有启用 我关闭的 。

源站 
(user:root  time: 12:32)[db: dm_tidb_test]show tables;
+------------------------+
| Tables_in_dm_tidb_test |
+------------------------+
| user                   |
| user_h                 |
| user_hem               |
| user_heming            |
| user_heming_new        |
+------------------------+
5 rows in set (0.00 sec)
目标库:
mysql> show tables;    #正常同步这两个表是最后建的 ,
+------------------------+
| Tables_in_dm_tidb_test |
+------------------------+
| user_h                 |   #这个表是修改 black-white-list:  bw-rule-1:    do-dbs: ["dm_tidb_test"] 之前建的空表 
| user_heming_new        |   #这个是修改 black-white-list:  bw-rule-1:    do-dbs: ["dm_tidb_test"] 之后重启之后导入的表 
+------------------------+
2 rows in set (0.00 sec)

根据日志记录 [2020/02/18 11:38:56.996 +08:00] [INFO] [mydumper.go:276] [“create mydumper”] [task=127190_to_gs_heming] [unit=dump] [argument="[–host,xxx.xxx.127.190,–port,3308,–user,chenlei,–outputdir,./dumped_data.127190_to_gs_heming,–logfile,/dev/stderr,–verbose,3,–threads,4,–chunk-filesize,64,–skip-tz-utc,–tables-list,dm_tidb_test.user,dm_tidb_test.user_h,dm_tidb_test.user_hem,dm_tidb_test.user_heming]"] 我手动执行命令 确实能生成sql文件 ,但是dm-worker目录下只有元数据文件

./mydumper --host xxx.xxx.127.190 --port 3308 --user chenlei -p che --outputdir ./dumped_data.127190_to_gs_heming --logfile /dev/stderr --verbose 3 --threads 4 --chunk-filesize 64 --skip-tz-utc --tables-list dm_tidb_test.user,dm_tidb_test.user_h,dm_tidb_test.user_hem,dm_tidb_test.user_heming
[root@emarsys105016 dumped_data.127190_to_gs_heming]# l
total 52K
drwx------ 2 root    root     308 Feb 18 12:42 .
drwxr-xr-x 3 chenlei chenlei  204 Feb 18 12:40 ..
-rw-r--r-- 1 root    root     626 Feb 18 12:42 dm_tidb_test.user_heming-schema.sql
-rw-r--r-- 1 root    root    6.9K Feb 18 12:42 dm_tidb_test.user_heming.sql
-rw-r--r-- 1 root    root     623 Feb 18 12:42 dm_tidb_test.user_hem-schema.sql
-rw-r--r-- 1 root    root    6.9K Feb 18 12:42 dm_tidb_test.user_hem.sql
-rw-r--r-- 1 root    root     621 Feb 18 12:42 dm_tidb_test.user_h-schema.sql
-rw-r--r-- 1 root    root    6.9K Feb 18 12:42 dm_tidb_test.user_h.sql
-rw-r--r-- 1 root    root     631 Feb 18 12:42 dm_tidb_test.user-schema.sql
-rw-r--r-- 1 root    root    6.9K Feb 18 12:42 dm_tidb_test.user.sql
-rw-r--r-- 1 root    root     138 Feb 18 12:42 metadata

image

看你这边提供的 task 配置中缩进有些问题,可以按照这个模版修改下修改下配置规则 https://github.com/pingcap/dm/blob/v1.0.3/dm/dm-ansible/conf/task_basic.yaml.example

修改后仍不能 dump 的话,麻烦提供下 dm-worker.log 的日志,多谢。

dm-worker.log (27.2 KB)
改了 stop-task 再 start-task 还是没有dump出正确的文件
重启task之前把 这个dump目录改了 要不要清理掉relay_log 重启一下dm-worker
image

dm-worker.log 有这个 需要dump的表的列表
–tables-list,dm_tidb_test.user_heming_new,dm_tidb_test.user,dm_tidb_test.user_h,dm_tidb_test.user_hem,dm_tidb_test.user_heming
你们的程序能不能挨个检查一下 有没有生成相应的文件 ,能不能把mydump的日志也打印到dm-worker.log 或者 mydump.log,如果没有的话能不能打印一下错误日志 。 让你们的迁移程序更健壮

感谢提供建议,日志问题也在继续优化中。

辛苦检查下 relay_log 下是否有正常拉取 binlog,日志中看类似的相关信息。

方便操作的话,也可以的。

新的表同步没问题 ,就是历史表 没有dump成功 所以就没有初始化 之前的表
可以加微信沟通吗? 我私信你我的微信号了
image

  1. 麻烦提供下 query-status task-name 的结果吧 ? 目前暂停是已经到 sync 阶段了吗 ?
  2. 看你这边提供的 relay_log 的下文件的更新时间,binlog 拉取到 14:54 后续就没有更新了,检查下 当前是否正常拉取 binlog (上游仍有写入的情况下)
[root@emarsys105016 ~]# dmctl --master-addr xxx.xxx.242.16:8261 query-status    127190_to_gs_heming 
{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "xxx.xxx..242.17:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "127190_to_gs_heming",
                    "stage": "Running",
                    "unit": "Sync",
                    "result": null,
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "0",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.001533, 2135606)",
                        "masterBinlogGtid": "",
                        "syncerBinlog": "(mysql-bin|000001.001533, 2104901)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.001533, 2135606)",
                "masterBinlogGtid": "",
                "relaySubDir": "1bfe5e41-adfe-11e9-988b-b8ca3a621608.000001",
                "relayBinlog": "(mysql-bin.001533, 2135606)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "mysql-replica-01"
        }
    ]
}

主库是一个 测试数据库 几乎没有什么操作 ,只有我测试同步操作 。所以一般后续都没有更新 。

关键是初始化mydumper 不能正常同步历史数据 。

请问每次修改时,是否有删除下游 dm_meta 数据库 ? 目前看正在 sync 阶段,增量同步,stop-task start-task 之后,会从 dm_meta 中的 增量同步的 checkpoint 位点获取增量同步信息。可以按照以下方式操作:

  • 1.stop-task 停止同步任务
  • 2.删除下游已经同步的数据,以及 dm_meta 数据库
  • 3.start-task 重启同步任务,使其可以重新进行 mydumper 以及 loader 以及 sync