dm 数据同步报错

Hacker_CbSLlt2B · 2021 年9 月 13 日 03:18

tidb 5.2 版本，dm 使用快速安装方式，1台物理机上启了1个master，1个work

----------tidb 安装方式
wget http://download.pingcap.org/dm-nightly-linux-amd64.tar.gz
tar -xzvf dm-nightly-linux-amd64.tar.gz
cd dm-nightly-linux-amd64

nohup bin/dm-master --master-addr=‘127.0.0.1:8261’ --log-file=/tmp/dm-master.log --name=“master1” >> /tmp/dm-master.log 2>&1 &

nohup bin/dm-worker --worker-addr=‘127.0.0.1:8262’ --log-file=/tmp/dm-worker.log --join=‘127.0.0.1:8261’ --name=“worker1” >> /tmp/dm-worker.log 2>&1 &

----------配置文件

数据在同步过程中报错，如下
master —
[2021/09/08 17:55:48.964 +08:00] [WARN] [util.go:121] [“failed to apply request”] [component=“embed etcd”] [took=21.213µs] [request=“header:<ID:15696588155148742474 > compaction:<revision:20 > “] [response=] [error=“mvcc: required revision has been compacted”]
[2021/09/08 19:42:09.758 +08:00] [INFO] [server.go:2259] [payload=“name:“o2o_comment” “] [request=QueryStatus]
[2021/09/08 19:42:11.758 +08:00] [INFO] [server.go:2259] [payload=“name:“o2o_comment” “] [request=QueryStatus]
[2021/09/08 19:55:48.976 +08:00] [WARN] [util.go:121] [“failed to apply request”] [component=“embed etcd”] [took=14.637µs] [request=“header:<ID:15696588155148742476 > compaction:<revision:24 > “] [response=] [error=“mvcc: required revision has been compacted”]
[2021/09/08 20:21:20.798 +08:00] [INFO] [main.go:84] [“got signal to exit”] [signal=hangup]
[2021/09/08 20:21:20.798 +08:00] [INFO] [server.go:260] [“closing server”]
[2021/09/08 20:21:20.798 +08:00] [INFO] [scheduler.go:2076] [“observeLoadTask will quit now”]
[2021/09/08 20:21:20.798 +08:00] [INFO] [election.go:190] [“election is closing”] [component=election] [“current member”=”{“id”:“master1”,“addr”:“127.0.0.1:8261”}”]
[2021/09/08 20:21:20.798 +08:00] [INFO] [pessimist.go:123] [“pessimist will quit now”]
[2021/09/08 20:21:20.798 +08:00] [INFO] [election.go:336] [“retire from leader”] [component=election] [“current member”=”{“id”:“master1”,“addr”:“127.0.0.1:8261”}”]
[2021/09/08 20:21:20.798 +08:00] [INFO] [election.go:240] [“break campaign loop, context is done”] [component=election] [“current member”=”{“id”:“master1”,“addr”:“127.0.0.1:8261”}”] [error=“context canceled”]
[2021/09/08 20:21:20.798 +08:00] [INFO] [keepalive.go:203] [“watch keepalive worker quit due to context canceled”]
[2021/09/08 20:21:20.798 +08:00] [INFO] [scheduler.go:1642] [“observeWorkerEvent will quit now”]
[2021/09/08 20:21:20.800 +08:00] [INFO] [election.go:198] [“election is closed”] [component=election] [“current member”=”{“id”:“master1”,“addr”:“127.0.0.1:8261”}”]
[2021/09/08 20:21:20.800 +08:00] [WARN] [grpclog.go:60] [“grpc: addrConn.createTransport failed to connect to {127.0.0.1:8261 0 }. Err :connection error: desc = “transport: Error while dialing dial tcp 127.0.0.1:8261: connect: connection refused”. Reconnecting…”] [component=“embed etcd”]
[2021/09/08 20:21:20.800 +08:00] [WARN] [grpclog.go:60] [“transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:8261->127.0.0.1:56406: use of closed network connection”] [component=“embed etcd”]
[2021/09/08 20:21:20.800 +08:00] [WARN] [grpclog.go:60] [“transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:8261->127.0.0.1:56400: use of closed network connection”] [component=“embed etcd”]
[2021/09/08 20:21:20.802 +08:00] [INFO] [server.go:262] [“server closed”]
[2021/09/08 20:21:20.802 +08:00] [INFO] [main.go:91] [“dm-master exit”]
nohup: ignoring input

work-----

[2021/09/08 20:21:00.160 +08:00] [INFO] [syncer.go:3076] [“binlog replication status”] [task=o2o_comment] [unit=“binlog replication”] [total_events=109687] [total_tps=10] [tps=10] [master_position="(mysql-bin.000318, 259247110)"] [master_gtid=] [checkpoint=“position: (mysql-bin.000318, 259247110), gtid-set: (flushed position: (mysql-bin.000318, 259243371), gtid-set: )”]
[2021/09/08 20:21:20.798 +08:00] [INFO] [main.go:86] [“got signal to exit”] [signal=hangup]
[2021/09/08 20:21:20.798 +08:00] [INFO] [keepalive.go:144] [“ctx is canceled, keepalive will exit now”]
[2021/09/08 20:21:20.799 +08:00] [WARN] [join.go:105] [“keepalive with master goroutine paused”] []
[2021/09/08 20:21:20.800 +08:00] [INFO] [worker.go:683] [“worker is closed, handleSubTaskStage will quit now”]
[2021/09/08 20:21:20.800 +08:00] [INFO] [worker.go:143] [“status print process exits!”] [component=“worker controller”]
[2021/09/08 20:21:20.800 +08:00] [INFO] [worker.go:644] [“observeSubtaskStage will quit now”]
[2021/09/08 20:21:20.800 +08:00] [INFO] [subtask.go:464] [closing] [subtask=o2o_comment]
[2021/09/08 20:21:20.800 +08:00] [INFO] [syncer.go:1503] [“received subtask’s done”] [task=o2o_comment] [unit=“binlog replication”]
[2021/09/08 20:21:20.800 +08:00] [INFO] [syncer.go:1509] [“the last job is transaction end, done directly”] [task=o2o_comment] [unit=“binlog replication”]
[2021/09/08 20:21:20.800 +08:00] [INFO] [syncer.go:3016] [“print status routine exits”] [task=o2o_comment] [unit=“binlog replication”] [error=“context canceled”]
[2021/09/08 20:21:20.800 +08:00] [INFO] [syncer.go:1809] [“binlog replication main routine quit(context canceled)!”] [task=o2o_comment] [unit=“binlog replication”] [“last location”=“position: (mysql-bin.000318, 259395022), gtid-set: “]
[2021/09/08 20:21:20.831 +08:00] [INFO] [syncer.go:1154] [“flushed checkpoint”] [task=o2o_comment] [unit=“binlog replication”] [checkpoint=“position: (mysql-bin.000318, 259395022), gtid-set: (flushed position: (mysql-bin.000318, 259395022), gtid-set: )”]
[2021/09/08 20:21:20.832 +08:00] [INFO] [syncer.go:1169] [“after last flushing checkpoint, DM has ignored row changes by expression filter”] [task=o2o_comment] [unit=“binlog replication”] [“number of filtered insert”=0] [“number of filtered update”=0] [“number of filtered delete”=0]
[2021/09/08 20:21:20.832 +08:00] [INFO] [syncer.go:1715] [“flush checkpoints when exit task”] [task=o2o_comment] [unit=“binlog replication”]
[2021/09/08 20:21:20.832 +08:00] [INFO] [checkpoint.go:695] [“rollback checkpoint”] [task=o2o_comment] [unit=“binlog replication”] [component=“remote checkpoint”] [schema=o2o_comment] [table=tagitem_dictionary] [from=“position: (mysql-bin.000318, 170752453), gtid-set: “] [to=“position: (mysql-bin.000318, 170752453), gtid-set: “]
[2021/09/08 20:21:20.833 +08:00] [WARN] [checkpoint.go:698] [“failed to drop table from schema tracker”] [task=o2o_comment] [unit=“binlog replication”] [component=“remote checkpoint”] [schema=o2o_comment] [table=tagitem_dictionary] [error=”[schema:1146]Table ‘o2o_comment.tagitem_dictionary’ doesn’t exist”]
[2021/09/08 20:21:20.841 +08:00] [INFO] [subtask.go:293] [“unit process returned”] [subtask=o2o_comment] [unit=Sync] [stage=Stopped] [status=”{“totalEvents”:109893,“totalTps”:10,“recentTps”:10,“syncerBinlog”:”(mysql-bin.000318, 259395022)”,“binlogType”:“remote”}”]
[2021/09/08 20:21:20.841 +08:00] [INFO] [subtask.go:365] [“closing unit process”] [subtask=o2o_comment] [unit=Sync]
[2021/09/08 20:21:20] [error] binlogsyncer.go:671 io.ReadFull(header) failed. err EOF: connection was bad
[2021/09/08 20:21:20] [warn] binlogsyncer.go:681 retry sync is disabled
[2021/09/08 20:21:20] [error] binlogstreamer.go:78 close sync with err: io.ReadFull(header) failed. err EOF: connection was bad
[2021/09/08 20:21:20] [info] binlogsyncer.go:180 syncer is closing…
[2021/09/08 20:21:20] [error] binlogsyncer.go:858 kill connection 22480941 error ERROR 1094 (HY000): Unknown thread id: 22480941
[2021/09/08 20:21:20] [info] binlogsyncer.go:864 kill last connection id 22480941
[2021/09/08 20:21:20] [info] binlogsyncer.go:210 syncer is closed
[2021/09/08 20:21:20.853 +08:00] [INFO] [task_checker.go:213] [“worker task checker exits”] [component=“task checker”]
[2021/09/08 20:21:20.853 +08:00] [INFO] [worker.go:189] [“Stop worker”] [component=“worker controller”]
[2021/09/08 20:21:20.853 +08:00] [INFO] [join.go:118] [“keepalive with master goroutine exited!”]
[2021/09/08 20:21:20.853 +08:00] [INFO] [server.go:602] [“handleSourceBound will quit now”]
[2021/09/08 20:21:20.853 +08:00] [INFO] [server.go:641] [“worker server is closed, handleRelayConfig will quit now”]
[2021/09/08 20:21:20.853 +08:00] [ERROR] [metrics.go:138] [“status server returned”] [error=“mux: listener closed”]
[2021/09/08 20:21:20.853 +08:00] [INFO] [server.go:383] [“observeRelayConfig will quit now”]
[2021/09/08 20:21:20.853 +08:00] [INFO] [server.go:469] [“observeSourceBound will quit now”]
[2021/09/08 20:21:20.854 +08:00] [INFO] [main.go:94] [“dm-worker exit”]
nohup: ignoring input

songxuecheng · 2021 年9 月 13 日 04:06

table ‘o2o_comment.tagitem_dictionary’ doesn’t exist 表不存在看下这个

Hacker_CbSLlt2B · 2021 年9 月 13 日 05:38

看过了，表是存在的

songxuecheng · 2021 年9 月 13 日 05:44

下游是mysql还是tidb,表结构也发一下

Hacker_CbSLlt2B · 2021 年9 月 13 日 06:33

下游是tidb ,我现在重新开始了同步，看看效果吧

小王同学Plus · 2021 年9 月 13 日 08:05

为什么选择二进制方式部署呀？TiUP 不能用吗

Hacker_CbSLlt2B · 2021 年9 月 13 日 09:44

机器紧张，还有就是不影响现有集群

小王同学Plus · 2021 年9 月 13 日 09:48

还是建议使用 TiUP 部署，如果没有监控的话，不方便排查问题，比如说延迟之类的

Hacker_CbSLlt2B · 2021 年9 月 14 日 08:06

用tiup 已经开始同步数据了，有个疑问，网上说 tidb cluster监控和dm server监控非要装在同一台机器上，端口要不同 确认下这样说对吗，如果端口相同会有什么问题呢？

小王同学Plus · 2021 年9 月 14 日 08:29

会冲突的。不同服务用同一个 IP 端口，其他服务也也一样啊。其中一套起不来。

Hacker_CbSLlt2B · 2021 年9 月 14 日 08:30

dm 的监控融入不到tidb 集群？

小王同学Plus · 2021 年9 月 14 日 08:30

不能。两套集群，如果你想整合，可以搜索一下，有相关文章。

Hacker_CbSLlt2B · 2021 年9 月 14 日 08:31

好的，谢谢了

小王同学Plus · 2021 年9 月 14 日 08:32

system · 2022 年10 月 31 日 19:15

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。