DM同步报错,丢失一个事务内的数据

【 TiDB 使用环境`】生产

【 TiDB 版本】

./dm-worker -V

Release Version: v5.3.0

Git Commit Hash: 20626babf21fc381d4364646c40dd84598533d66

Git Branch: heads/refs/tags/v5.3.0

UTC Build Time: 2021-11-29 08:29:35

Go Version: go version go1.16.4 linux/amd64

【遇到的问题】

发现 TiDB 数据存在丢失情况,分析丢失数据,发现问题时间点有如下报错,暂发现 1e97de7a-d324-11eb-a45e-1c34da5039fc:244050799 事务数据丢失,部分报错日志如下,详见附件

并解析GTID 244050799 binlog

需要帮忙确认分析问题原因

[2022/04/01 22:43:45.736 +08:00] [ERROR] [streamer_controller.go:287] [“meet error when get binlog event”] [task=task-mysql-scm-tms] [unit=“binlog replication”] [error=“io.CopyN failed. err unexpected EOF, copied 0, expected 278: connection was bad”] [errorVerbose=“connection was bad\ngithub.com/go-mysql-org/go-mysql/mysql.init\n\tgithub.com/go-mysql-org/go-mysql@v1.1.3-0.20210705101833-83965e516929/mysql/error.go:10\nruntime.doInit\n\truntime/proc.go:6309\nruntime.doInit\n\truntime/proc.go:6286\nruntime.doInit\n\truntime/proc.go:6286\nruntime.doInit\n\truntime/proc.go:6286\nruntime.main\n\truntime/proc.go:208\nruntime.goexit\n\truntime/asm_amd64.s:1371\nio.CopyN failed. err unexpected EOF, copied 0, expected 278\ngithub.com/go-mysql-org/go-mysql/packet.(*Conn).ReadPacketTo\n\tgithub.com/go-mysql-org/go-mysql@v1.1.3-0.20210705101833-83965e516929/packet/conn.go:151\ngithub.com/go-mysql-org/go-mysql/packet.(*Conn).ReadPacketReuseMem\n\tgithub.com/go-mysql-org/go-mysql@v1.1.3-0.20210705101833-83965e516929/packet/conn.go:98\ngithub.com/go-mysql-org/go-mysql/packet.(*Conn).ReadPacket\n\tgithub.com/go-mysql-org/go-mysql@v1.1.3-0.20210705101833-83965e516929/packet/conn.go:90\ngithub.com/go-mysql-org/go-mysql/replication.(*BinlogSyncer).onStream\n\tgithub.com/go-mysql-org/go-mysql@v1.1.3-0.20210705101833-83965e516929/replication/binlogsyncer.go:662\nruntime.goexit\n\truntime/asm_amd64.s:1371”]

[2022/04/01 22:43:45.736 +08:00] [ERROR] [syncer.go:1616] [“fail to fetch binlog”] [task=task-mysql-scm-tms] [unit=“binlog replication”] [error=“io.CopyN failed. err unexpected EOF, copied 0, expected 278: connection was bad”]

[2022/04/01 22:43:45.736 +08:00] [ERROR] [streamer_controller.go:354] [“fail to kill last connection”] [task=task-mysql-scm-tms] [unit=“binlog replication”] [“connection ID”=9145067] [error="[code=10001:class=database:scope=not-set:level=high], Message: database driver error, RawCause: Error 1094: Unknown thread id: 9145067, Workaround: Please check the database connection and the database config in configuration file."]

【复现路径】做过哪些操作出现的问题

当时在源端MySQL 有做过加索引操作,但问题事务解析出来有较多的DML操作

【问题现象及影响】

DM日志有报错,但是运行状态正常,出现数据丢失。

dm_err.zip (71.9 KB)

https://github.com/pingcap/tiflow/issues/3711

非常抱歉,这个是 v5.3.0 版本已知问题。您可以尝试升级到 v5.3.1 或者更新版本