DM:dumpling全量导出时, 全量阶段准备开始的时间非常长,是怎么回事?

DM2.0:在全量阶段使用的mydumpers配置。
描述:
开始任务后,dump目录很长时间没有数据。
正常情况启动任务后,就会输出日志【All the dumping transactions have started】,然后就能看到dump下的文件了,但是出现的问题是 这个【All the dumping transactions have started】等了接近两个小时。

查看日志
[2023/01/18 01:44:31.490 +08:00] [INFO] [checkpoint.go:660] [“create checkpoint schema”] [task=ph_kefu-10.100.73.42] [unit=“binlog replication”] [component=“remote checkpoint”] [statement=“CREATE SCHEMA IF NOT EXISTS dm_meta”]
[2023/01/18 01:44:33.068 +08:00] [WARN] [db.go:264] [“execute transaction”] [task=kefu] [unit=“binlog replication”] [query=“[CREATE TABLE IF NOT EXISTS dm_meta.kefu_syncer_checkpoint (\n\t\t\tid VARCHAR(64) NOT NULL,\n\t\t\tcp_schema VARCHAR(128) NOT NULL,\n\t\t\tcp_table VARCHAR(128) NOT NULL,\n\t\t\tbinlog_name VARCHAR(128),\n\t\t\tbinlog_pos INT UNSIGNED,\n\t\t\tbinlog_gtid TEXT,\n\t\t\texit_safe_binlog_name VARCHAR(128) DEFAULT ‘’,\n\t\t\texit_safe_binlog_pos INT UNSIGNED DEFAULT 0,\n\t\t\texit_safe_binlog_gtid TEXT,\n\t\t\ttable_info JSON NOT NULL,\n\t\t\tis_global BOOLEAN,\n\t\t\tcreate_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,\n\t\t\tupdate_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,\n\t\t\tUNIQUE KEY uk_id_schema_table (id, cp_schema, cp_table)\n\t\t)]”] [argument=“[]”] [“cost time”=1.577981416s]
[2023/01/18 01:44:33.068 +08:00] [INFO] [checkpoint.go:685] [“create checkpoint table”] [task=kefu] [unit=“binlog replication”] [component=“remote checkpoint”] [statements=“["CREATE TABLE IF NOT EXISTS dm_meta.kefu_syncer_checkpoint (\n\t\t\tid VARCHAR(64) NOT NULL,\n\t\t\tcp_schema VARCHAR(128) NOT NULL,\n\t\t\tcp_table VARCHAR(128) NOT NULL,\n\t\t\tbinlog_name VARCHAR(128),\n\t\t\tbinlog_pos INT UNSIGNED,\n\t\t\tbinlog_gtid TEXT,\n\t\t\texit_safe_binlog_name VARCHAR(128) DEFAULT ‘’,\n\t\t\texit_safe_binlog_pos INT UNSIGNED DEFAULT 0,\n\t\t\texit_safe_binlog_gtid TEXT,\n\t\t\ttable_info JSON NOT NULL,\n\t\t\tis_global BOOLEAN,\n\t\t\tcreate_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,\n\t\t\tupdate_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,\n\t\t\tUNIQUE KEY uk_id_schema_table (id, cp_schema, cp_table)\n\t\t)"]”]
[2023/01/18 01:44:33.077 +08:00] [INFO] [subtask.go:223] [“start to run”] [subtask=kefu] [unit=Dump]
[2023/01/18 01:44:33.079 +08:00] [INFO] [log.go:77] [“detect server type”] [task=kefu] [unit=dump] [type=MySQL]
[2023/01/18 01:44:33.079 +08:00] [INFO] [log.go:77] [“detect server version”] [task=kefu] [unit=dump] [version=8.0.21]

[2023/01/18 03:07:14.680 +08:00] [INFO] [log.go:77] [“All the dumping transactions have started. Start to unlock tables”] [task=kefu] [unit=dump]
[2023/01/18 04:29:15.717 +08:00] [INFO] [log.go:77] [“backup Success summary: total backup ranges: 69, total success: 69, total failed: 0, total take(backup time): 1h22m1.007362539s, total take(real time): 1h22m1.007400392s, total size(MB): 13671.35, avg speed(MB/s): 2.78, total rows: 52397765”] [task=kefu] [unit=dump]
[2023/01/18 04:29:15.804 +08:00] [INFO] [dumpling.go:135] [“dump data finished”] [task=kefu] [unit=dump] [“cost time”=2h44m42.726804241s]

DM 的版本过于老旧了,近期准备升级一下吗?

近期没有升级计划。

排查可能是因为mysql有其他进程锁表,导致dumper 无法获取读锁。

是导mysql还是tidb啊?

从mysql 同步数据到TiDB

dumpling导出mysql时一致性保证方式用的是 flush ,这个过程会flush tables with read lock
具体如下:

如果这时候有其他进程,比如有canal同步数据,长链接会阻塞,导致这个read lock获取不到。就会出现这种问题吧?

长连接不一定会阻塞。
活跃事务、ddl 、刷缓存到磁盘,这些都是加锁前要完成的,完成后才能上锁成功。
具体可度娘下 mysql 的 flush tables with read lock

1 个赞

嗯,我先看看。

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。