【 TiDB 使用环境`】生产
【 TiDB 版本】
./dm-worker -V
Release Version: v5.3.0
Git Commit Hash: 20626babf21fc381d4364646c40dd84598533d66
Git Branch: heads/refs/tags/v5.3.0
UTC Build Time: 2021-11-29 08:29:35
Go Version: go version go1.16.4 linux/amd64
【遇到的问题】
发现 TiDB 数据存在丢失情况,分析丢失数据,发现问题时间点有如下报错,暂发现 1e97de7a-d324-11eb-a45e-1c34da5039fc:244050799 事务数据丢失,部分报错日志如下,详见附件
并解析GTID 244050799 binlog
需要帮忙确认分析问题原因
[2022/04/01 22:43:45.736 +08:00] [ERROR] [streamer_controller.go:287] [“meet error when get binlog event”] [task=task-mysql-scm-tms] [unit=“binlog replication”] [error=“io.CopyN failed. err unexpected EOF, copied 0, expected 278: connection was bad”] [errorVerbose=“connection was bad\ngithub.com/go-mysql-org/go-mysql/mysql.init\ \tgithub.com/go-mysql-org/go-mysql@v1.1.3-0.20210705101833-83965e516929/mysql/error.go:10\ runtime.doInit\ \truntime/proc.go:6309\ runtime.doInit\ \truntime/proc.go:6286\ runtime.doInit\ \truntime/proc.go:6286\ runtime.doInit\ \truntime/proc.go:6286\ runtime.main\ \truntime/proc.go:208\ runtime.goexit\ \truntime/asm_amd64.s:1371\ io.CopyN failed. err unexpected EOF, copied 0, expected 278\ngithub.com/go-mysql-org/go-mysql/packet.(*Conn).ReadPacketTo\ \tgithub.com/go-mysql-org/go-mysql@v1.1.3-0.20210705101833-83965e516929/packet/conn.go:151\ github.com/go-mysql-org/go-mysql/packet.(*Conn).ReadPacketReuseMem\ \tgithub.com/go-mysql-org/go-mysql@v1.1.3-0.20210705101833-83965e516929/packet/conn.go:98\ github.com/go-mysql-org/go-mysql/packet.(*Conn).ReadPacket\ \tgithub.com/go-mysql-org/go-mysql@v1.1.3-0.20210705101833-83965e516929/packet/conn.go:90\ github.com/go-mysql-org/go-mysql/replication.(*BinlogSyncer).onStream\ \tgithub.com/go-mysql-org/go-mysql@v1.1.3-0.20210705101833-83965e516929/replication/binlogsyncer.go:662\ runtime.goexit\ \truntime/asm_amd64.s:1371”]
[2022/04/01 22:43:45.736 +08:00] [ERROR] [syncer.go:1616] [“fail to fetch binlog”] [task=task-mysql-scm-tms] [unit=“binlog replication”] [error=“io.CopyN failed. err unexpected EOF, copied 0, expected 278: connection was bad”]
[2022/04/01 22:43:45.736 +08:00] [ERROR] [streamer_controller.go:354] [“fail to kill last connection”] [task=task-mysql-scm-tms] [unit=“binlog replication”] [“connection ID”=9145067] [error="[code=10001:class=database:scope=not-set:level=high], Message: database driver error, RawCause: Error 1094: Unknown thread id: 9145067, Workaround: Please check the database connection and the database config in configuration file."]
【复现路径】做过哪些操作出现的问题
当时在源端MySQL 有做过加索引操作,但问题事务解析出来有较多的DML操作
【问题现象及影响】
DM日志有报错,但是运行状态正常,出现数据丢失。