DM-worker切换上游MYSQL后无法获取gtid

Hacker_MMJa6UpY · 2020 年5 月 9 日 03:50

【TiDB 版本】：v4.0.0-rc
【DM 版本】：v1.0.4-hotfix
【问题描述】：由于上游mysql日志缺失，打算手动切换上游mysql实例。但是重启worker时报错

操作： 1、修改inventory.ini，手动指定新实例binlog和gtid：mysql_host ，relay_binlog_name，relay_binlog_gtid 2、更新dm-worker：ansible-playbook rolling_update.yml --tags=dm-worker

dm-worker日志： [2020/05/09 11:30:20.876 +08:00] [ERROR] [relay.go:395] [“the requested binlog files have purged in the master server or the master server have switched, currently DM do no support to handle this error”] [component=“relay log”] [“db host”=r3312_mydb.lzdb.com] [“db port”=3312] [error="[code=30015:class=relay-unit:scope=upstream:level=high] TCPReader get relay event with error: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires."]

[2020/05/09 11:30:20.879 +08:00] [ERROR] [relay.go:264] [“fail to close binlog event reader”] [component=“relay log”] [error="[code=10001:class=database:scope=upstream:level=high] kill connection 3628260 for master r3312_mydb.lzdb.com:3312: database driver error: Error 1094: Unknown thread id: 3628260"] [errorVerbose="[code=10001:class=database:scope=upstream:level=high] kill connection 3628260 for master r3312_mydb.lzdb.com:3312: database driver error: Error 1094: Unknown thread id: 3628260\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/terror.DBErrorAdaptArgs\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/adapter.go:39\ github.com/pingcap/dm/pkg/terror.DBErrorAdapt\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/adapter.go:46\ github.com/pingcap/dm/pkg/utils.KillConn\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/utils/db.go:356\ github.com/pingcap/dm/pkg/binlog/reader.(*TCPReader).Close\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/binlog/reader/tcp.go:131\ngithub.com/pingcap/dm/relay/reader.(*reader).Close\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/reader/reader.go:126\ngithub.com/pingcap/dm/relay.(*Relay).process.func1\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:262\ngithub.com/pingcap/dm/relay.(*Relay).process\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:296\ngithub.com/pingcap/dm/relay.(*Relay).Process\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:191\ngithub.com/pingcap/dm/dm/worker.(*realRelayHolder).run\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:167\ngithub.com/pingcap/dm/dm/worker.(*realRelayHolder).Start.func1\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:143\ runtime.goexit\ \t/usr/local/go/src/runtime/asm_amd64.s:1357"] [2020/05/09 11:30:20.879 +08:00] [ERROR] [relay.go:194] [“process exit”] [component=“relay log”] [error="[code=30015:class=relay-unit:scope=upstream:level=high] TCPReader get relay event with error: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires."] [errorVerbose="[code=30015:class=relay-unit:scope=upstream:level=high] TCPReader get relay event with error: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/binlog/reader.(*TCPReader).GetEvent\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/binlog/reader/tcp.go:151\ngithub.com/pingcap/dm/relay/reader.(*reader).GetEvent\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/reader/reader.go:144\ngithub.com/pingcap/dm/relay.(*Relay).handleEvents\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:382\ngithub.com/pingcap/dm/relay.(*Relay).process\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:292\ngithub.com/pingcap/dm/relay.(*Relay).Process\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:191\ngithub.com/pingcap/dm/dm/worker.(*realRelayHolder).run\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:167\ngithub.com/pingcap/dm/dm/worker.(*realRelayHolder).Start.func1\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:143\ runtime.goexit\ \t/usr/local/go/src/runtime/asm_amd64.s:1357"] [2020/05/09 11:30:20.879 +08:00] [ERROR] [relay.go:173] [“process error”] [component=“relay holder”] [type=“msg:”[code=30015:class=relay-unit:scope=upstream:level=high] TCPReader get relay event with error: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/binlog/reader.(*TCPReader).GetEvent\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/binlog/reader/tcp.go:151\ngithub.com/pingcap/dm/relay/reader.(*reader).GetEvent\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/reader/reader.go:144\ngithub.com/pingcap/dm/relay.(*Relay).handleEvents\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:382\ngithub.com/pingcap/dm/relay.(*Relay).process\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:292\ngithub.com/pingcap/dm/relay.(*Relay).Process\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:191\ngithub.com/pingcap/dm/dm/worker.(*realRelayHolder).run\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:167\ngithub.com/pingcap/dm/dm/worker.(*realRelayHolder).Start.func1\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:143\ runtime.goexit\ \t/usr/local/go/src/runtime/asm_amd64.s:1357" error:<ErrCode:30015 ErrClass:8 ErrScope:1 ErrLevel:3 Message:“TCPReader get relay event with error: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.” RawCause:“ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.” > "]

来了老弟 · 2020 年5 月 9 日 04:19

你好，

你的问题已收到，正在分析，请稍等

来了老弟 · 2020 年5 月 9 日 06:25

你好，

从日志中看，该 gtid 已经被 purged

请确认上游 mysql 缺失 binlog 时间点
看下对应时间点的 mysql2 的日志是否存在
将 gtid 往前设置一点，开启 safe_mode 并 stop-task 、start-task，看是否可恢复数据。

原则就是

最好的结果就是将数据从断电就能接上。
再差一些就是将复制点位提前并开启 safe_mode 进行同步。如果判断数据已经没有冲突，记得将 safe_mode 修改为 false。

请检查下是否严格按照文档要求操作。

Hacker_MMJa6UpY · 2020 年5 月 9 日 06:43

上游mysql1是缺失日志了。mysql2是确定包含该gtid信息的，目前卡在worker这一步，指定mysql2的gtid重启worker后就报找不到gtid。是否需要清除tidb中 dm_meta.table_checkpoint表中的记录

来了老弟 · 2020 年5 月 9 日 07:06

你好，

从目前的情况，建议修改 inventory 文件中 relay_binlog_gtid 设置为 mysql2 最早的一个点（在现存 binlog 文件找到最早的 gtid 点），开启 safe_mode 进行将数据补全，或者是复制。

以防数据丢失。因为你已经无法从 mysql 中确定具体的 xid 以及 mysql2 所对应的的 gtid 具体是什么。

PS：还是想问下具体的操作，因为如果操作不当可能导致数据丢失，是否严格按照文档操作进行的，是否存在偏差

Hacker_MMJa6UpY · 2020 年5 月 9 日 07:18

具体是这样的：mysql1发生网络问题恢复后，dm并没有继续同步。当发现时mysql1的binlog已经被purge了。但是这个时候执行到的gtid是可以在query-status里面找到的。第一次尝试使用文档的直接切换方式，结果日志报错找不到gtid

Hacker_MMJa6UpY · 2020 年5 月 9 日 07:21

第二次尝试直接在dm-worker.toml中配置relay-binlog-name,relay-binlog-gtid。发现启动直接报错，也是找不到gtid。如果我不配置这两个参数，应该会从mysql2最新的binlog开始同步，这样可能会导致数据丢失

来了老弟 · 2020 年5 月 9 日 07:35

你好，

按照，上面提到的方式，将数据从 mysql2 最早的 binglog 开始同步如果还会丢数据，除非有备份，在 tidb 恢复下，在进行同步。

ps：在开启 enable_gtid ，relay_binlog_name 是不生效的。因为你已经开启了 gtid 复制模式。

【解决方案】

stop-task
修改 inventory 文件中 relay_binlog_gtid 并将其设置为 mysql2 的最早的点（从最早的 binlog 中找下）
修改 task 中 safe_mode: true
滚动 dm-woker ，cat conf/dm-worker.toml 看配置是否生效
start-task
将 safe_mode 修改回 false

Hacker_MMJa6UpY · 2020 年5 月 9 日 07:37

谢谢，我先试试

Hacker_MMJa6UpY · 2020 年5 月 9 日 08:30

第4步还是没有生效，我发现就算配置了relay_binlog_gtid。重启worker后还是会从上次中断的gtid开始读去，（relay.meta里面），结果还是会报一样的错

来了老弟 · 2020 年5 月 9 日 08:37

你好，

执行

ll {dm-worker_deploy_dir}/relay_log/
cat {dm-worker_deploy_dir}/relay_log/server-uuid.index

看下 .index 提示的目录下的 relay.meta 是否为需要的 gtid 点。

例子:

如果你已经将 gtid：mysql_host mysql_port 更换，就是已经切换了数据源，在 dm-worker/relay_log/ 下会增加一个文件夹，用来同步新的上游数据。

Hacker_MMJa6UpY · 2020 年5 月 9 日 08:45

Hacker_MMJa6UpY · 2020 年5 月 9 日 08:47

有生产新的文件夹，位置不是我设置的最早的binlog中的gtid，而是切换前执行到的gtid。

来了老弟 · 2020 年5 月 9 日 09:08

你好，

请帮忙确认

修改 inventory 文件之后，是否已经 rolling_update，将配置刷到 dm-worker 上
检查你的 dm-worker - conf - dm-worker.toml 是否已经更新了配置。
尝试 start-task 进行恢复。

Hacker_MMJa6UpY · 2020 年5 月 9 日 09:15

1,2 都已经确认了。只是worker启动报错，也没有拉到relay log。所以没哟执行3

来了老弟 · 2020 年5 月 9 日 09:18

反馈下 dm-worker 的 log/dm-worker.log 日志，这边看下。

Hacker_MMJa6UpY · 2020 年5 月 9 日 09:28

[2020/05/09 17:05:40.555 +08:00] [INFO] [server.go:284] [request=FetchDDLInfo]
[2020/05/09 17:23:11.510 +08:00] [INFO] [main.go:72] [“got signal to exit”] [signal=terminated]
[2020/05/09 17:23:11.510 +08:00] [ERROR] [metrics.go:87] [“fail to start status server return”] [error=“mux: listener closed”]
[2020/05/09 17:23:11.510 +08:00] [INFO] [worker.go:189] [“status print process exits!”] [component=“worker controller”]
[2020/05/09 17:23:11.510 +08:00] [INFO] [worker.go:781] [“handle task process exits!”] [component=“worker controller”]
[2020/05/09 17:23:11.510 +08:00] [INFO] [relay.go:668] [“relay unit is closing”] [component=“relay log”]
[2020/05/09 17:23:11.511 +08:00] [INFO] [relay.go:675] [“relay unit closed”] [component=“relay log”]
[2020/05/09 17:23:11.511 +08:00] [INFO] [purger.go:151] [“closing relay log purger”] [component=“relay purger”]
[2020/05/09 17:23:11.511 +08:00] [INFO] [task_checker.go:251] [“worker task checker exits”] [component=“task checker”]
[2020/05/09 17:23:11.511 +08:00] [INFO] [log.go:304] [“gc routine exits!”] [component=“operator log”]
[2020/05/09 17:23:11.511 +08:00] [INFO] [main.go:81] [“dm-worker exit”]
[2020/05/09 17:23:32.593 +08:00] [INFO] [printer.go:54] [“Welcome to dm-worker”] [“Release Version”=v1.0.4-hotfix] [“Git Commit Hash”=be5b54faec39c30e8161760db014224398f63156] [“Git Branch”=release-1.0] [“UTC Build Time”=“2020-03-19 09:30:23”] [“Go Version”=“go version go1.13 linux/amd64”]
[2020/05/09 17:23:32.593 +08:00] [INFO] [main.go:58] [“dm-worker config”=“{"log-level":"info","log-file":"/data_db1/dm/dm_worker1_1/log/dm-worker.log","log-rotate":"","worker-addr":":8262","enable-gtid":true,"auto-fix-gtid":false,"relay-dir":"/data_db1/dm/dm_worker1_1/relay_log","meta-dir":"./dm_worker_meta","server-id":429536752,"flavor":"mysql","charset":"","relay-binlog-name":"","relay-binlog-gtid":"db27cec9-0f05-11e9-84d2-6c92bf53d384:503232332","source-id":"mysql-replica-lizhipayment","from":{"host":"192.168.100.203","port":3312,"user":"tidbrepl","max-allowed-packet":67108864},"purge":{"interval":3600,"expires":0,"remain-space":15},"checker":{"check-enable":true,"backoff-rollback":{"duration":"5m0s"},"backoff-max":{"duration":"5m0s"}},"tracer":{"enable":false,"source":"mysql-replica-lizhipayment","tracer-addr":"","batch-size":20,"checksum":false},"config-file":"conf/dm-worker.toml"}”]
[2020/05/09 17:23:32.596 +08:00] [INFO] [db.go:54] [“kv config”=“{"block-cache-capacity":8388608,"block-restart-interval":16,"block-size":4096,"compaction-L0-trigger":8,"compaction-table-size":67108864,"compaction-total-size":536870912,"compaction-total-size-multiplier":8,"write-buffer":67108864,"write-L0-pause-trigger":24,"write-L0-slowdown-trigger":17}”]
[2020/05/09 17:23:32.605 +08:00] [INFO] [upgrade.go:169] [component=“bootstrap upgrade”] [“previous version”=“{"internal-no":1,"release-version":"v1.0.4-hotfix"}”]
[2020/05/09 17:23:32.605 +08:00] [INFO] [upgrade.go:174] [“previous and current versions are same, no need to upgrade”] [component=“bootstrap upgrade”] [version=“{"internal-no":1,"release-version":"v1.0.4-hotfix"}”]
[2020/05/09 17:23:32.606 +08:00] [INFO] [db.go:54] [“kv config”=“{"block-cache-capacity":8388608,"block-restart-interval":16,"block-size":4096,"compaction-L0-trigger":8,"compaction-table-size":67108864,"compaction-total-size":536870912,"compaction-total-size-multiplier":8,"write-buffer":67108864,"write-L0-pause-trigger":24,"write-L0-slowdown-trigger":17}”]
[2020/05/09 17:23:32.612 +08:00] [INFO] [log.go:199] [“finish initialization”] [component=“operator log”] [“handle pointer”=“{"Location":0}”] [“end pointer”=“{"Location":1}”]
[2020/05/09 17:23:32.612 +08:00] [INFO] [worker.go:146] [initialized] [component=“worker controller”]
[2020/05/09 17:23:32.612 +08:00] [INFO] [purger.go:115] [“starting relay log purger”] [component=“relay purger”] [config=“{"interval":3600,"expires":0,"remain-space":15}”]
[2020/05/09 17:23:32.612 +08:00] [INFO] [worker.go:182] [“start running”] [component=“worker controller”]
[2020/05/09 17:23:32.612 +08:00] [INFO] [server.go:112] [“start gRPC API”] [“listened address”=:8262]
[2020/05/09 17:23:32.614 +08:00] [WARN] [relay.go:320] [“no relay log file need to recover”] [component=“relay log”] [position=“(, 4)”] [“gtid set”=6a25d501-6991-11ea-9895-80615f0468d4:54697432]
[2020/05/09 17:23:32.614 +08:00] [INFO] [reader.go:159] [“start sync”] [component=“relay reader”] [master=192.168.100.203:3312] [“from GTID set”=6a25d501-6991-11ea-9895-80615f0468d4:54697432]
[2020/05/09 17:23:32.616 +08:00] [INFO] [reader.go:104] [“set up binlog reader”] [component=“relay reader”] [master=192.168.100.203:3312] [status=“{"stage":"prepared","connection":3642459}”]
[2020/05/09 17:23:32.616 +08:00] [INFO] [relay.go:617] [“started underlying reader”] [component=“relay log”] [UUID=5633ea06-14cc-11ea-afcf-50af73303989.000001]
[2020/05/09 17:23:32.616 +08:00] [INFO] [relay.go:634] [“started underlying writer”] [component=“relay log”] [UUID=5633ea06-14cc-11ea-afcf-50af73303989.000001] [config=“{"RelayDir":"/data_db1/dm/dm_worker1_1/relay_log/5633ea06-14cc-11ea-afcf-50af73303989.000001","Filename":""}”]
[2020/05/09 17:23:32.617 +08:00] [ERROR] [relay.go:395] [“the requested binlog files have purged in the master server or the master server have switched, currently DM do no support to handle this error”] [component=“relay log”] [“db host”=192.168.100.203] [“db port”=3312] [error=“[code=30015:class=relay-unit:scope=upstream:level=high] TCPReader get relay event with error: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.”]
[2020/05/09 17:23:32.617 +08:00] [INFO] [relay.go:400] [“current master status”] [component=“relay log”] [position=“(mysql-bin.000624, 522196882)”] [“GTID sets”=5633ea06-14cc-11ea-afcf-50af73303989:1-772,6a25d501-6991-11ea-9895-80615f0468d4:14-68374767,db27cec9-0f05-11e9-84d2-6c92bf53d384:369297016-478031801:479244719-504603615]
[2020/05/09 17:23:32.619 +08:00] [ERROR] [relay.go:264] [“fail to close binlog event reader”] [component=“relay log”] [error=“[code=10001:class=database:scope=upstream:level=high] kill connection 3642459 for master 192.168.100.203:3312: database driver error: Error 1094: Unknown thread id: 3642459”] [errorVerbose=“[code=10001:class=database:scope=upstream:level=high] kill connection 3642459 for master 192.168.100.203:3312: database driver error: Error 1094: Unknown thread id: 3642459
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/terror.DBErrorAdaptArgs
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/adapter.go:39
github.com/pingcap/dm/pkg/terror.DBErrorAdapt
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/adapter.go:46
github.com/pingcap/dm/pkg/utils.KillConn
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/utils/db.go:356
github.com/pingcap/dm/pkg/binlog/reader.(*TCPReader).Close
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/binlog/reader/tcp.go:131
github.com/pingcap/dm/relay/reader.(*reader).Close
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/reader/reader.go:126
github.com/pingcap/dm/relay.(*Relay).process.func1
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:262
github.com/pingcap/dm/relay.(*Relay).process
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:296
github.com/pingcap/dm/relay.(*Relay).Process
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:191
github.com/pingcap/dm/dm/worker.(*realRelayHolder).run
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:167
github.com/pingcap/dm/dm/worker.(*realRelayHolder).Start.func1
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:143
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”]
[2020/05/09 17:23:32.619 +08:00] [ERROR] [relay.go:194] [“process exit”] [component=“relay log”] [error=“[code=30015:class=relay-unit:scope=upstream:level=high] TCPReader get relay event with error: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.”] [errorVerbose=“[code=30015:class=relay-unit:scope=upstream:level=high] TCPReader get relay event with error: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/binlog/reader.(*TCPReader).GetEvent
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/binlog/reader/tcp.go:151
github.com/pingcap/dm/relay/reader.(*reader).GetEvent
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/reader/reader.go:144
github.com/pingcap/dm/relay.(*Relay).handleEvents
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:382
github.com/pingcap/dm/relay.(*Relay).process
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:292
github.com/pingcap/dm/relay.(*Relay).Process
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:191
github.com/pingcap/dm/dm/worker.(*realRelayHolder).run
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:167
github.com/pingcap/dm/dm/worker.(*realRelayHolder).Start.func1
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:143
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”]
[2020/05/09 17:23:32.619 +08:00] [ERROR] [relay.go:173] [“process error”] [component=“relay holder”] [type="msg:"[code=30015:class=relay-unit:scope=upstream:level=high] TCPReader get relay event with error: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/binlog/reader.(*TCPReader).GetEvent\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/binlog/reader/tcp.go:151\
github.com/pingcap/dm/relay/reader.(*reader).GetEvent\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/reader/reader.go:144\
github.com/pingcap/dm/relay.(*Relay).handleEvents\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:382\
github.com/pingcap/dm/relay.(*Relay).process\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:292\
github.com/pingcap/dm/relay.(*Relay).Process\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:191\
github.com/pingcap/dm/dm/worker.(*realRelayHolder).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:167\
github.com/pingcap/dm/dm/worker.(*realRelayHolder).Start.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:143\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357" error:<ErrCode:30015 ErrClass:8 ErrScope:1 ErrLevel:3 Message:"TCPReader get relay event with error: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires." RawCause:"ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires." > "]
[2020/05/09 17:23:36.513 +08:00] [INFO] [server.go:284] [request=FetchDDLInfo]

来了老弟 · 2020 年5 月 9 日 09:47

你好，

检查 mysql2 中的 dm 复制的账号权限是否正确，检查 mysql2 和 dm-worker 的防火墙是否关闭 2.mv relay_log relay_log_bak ，在执行 rolling_update ，看下 dm-woker 是否正常启动

Hacker_MMJa6UpY · 2020 年5 月 9 日 10:31

检查过了，还是不行。我先重新部署一次同步吧，后面再测试切换的

来了老弟 · 2020 年5 月 9 日 12:22