- 【TiDB 版本】:V3.0.12
- 【问题描述】:
用loader导入过程中发现大量的错误信息:
[2020/04/16 00:29:42.553 +00:00] [WARN] [session.go:435] [“can not retry txn”] [conn=99723] [label=general] [error=“[kv:1062]Duplicate entry ‘dae442-infinity.if_add_item_source.00000
9858.sql’ for key ‘uk_id_f’”] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=false] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2020/04/16 00:29:42.553 +00:00] [WARN] [session.go:461] [“commit failed”] [conn=99723] [“finished txn”=“Txn{state=invalid}”] [error=“[kv:1062]Duplicate entry ‘dae442-infinity_boe.if_add
_item_source.000009858.sql’ for key ‘uk_id_f’”]
导入完成后配置 syncer 配置同步报错:
2020/04/16 02:05:51 main.go:79: [error] Near line 1 (last key parsed ‘Started’): expected key separator ‘=’, but got ‘d’ instead
/home/jenkins/agent/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/meta.go:97:
/home/jenkins/agent/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/syncer.go:163:
syncer-config.toml (2.8 KB)
loader 的时候出现大量的Duplicate 是因为的导入完成后,自动化导入任务没关,所以数据重新拉起导入了~,这个可以忽视。现在的问题就是syncer 配置同步报错
[root@db export-20200410-212243]# cat /data2/mydumper/export-20200410-212243/metadata
Started dump at: 2020-04-10 21:22:43
SHOW MASTER STATUS:
Log: mysql-bin.006834
Pos: 13846233
GTID:1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,
22863285-e03a-11e9-a452-ac1f6b403494:1-8454425,
704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890826277
SHOW SLAVE STATUS:
Host: 10.1.13.3
Log: mysql-bin.006311
Pos: 819569478
GTID:1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,
22863285-e03a-11e9-a452-ac1f6b403494:1-8454425,
704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890826277
Finished dump at: 2020-04-11 02:46:49
还有我怎么看同步的进度了呢?有没有类似show slave status呢
[2020/04/16 03:24:09] [error] binlogsyncer.go:843 kill connection 15878586 error ERROR 1094 (HY000): Unknown thread id: 15878586
[2020/04/16 03:24:09] [info] binlogsyncer.go:849 kill last connection id 15878586
[2020/04/16 03:24:09] [info] binlogsyncer.go:776 rotate to (mysql-bin.006834, 4)
[2020/04/16 03:25:43] [error] binlogsyncer.go:656 io.CopyN failed. err EOF, copied 7954, expected 8207: connection was bad
[2020/04/16 03:25:44] [info] binlogsyncer.go:581 begin to re-sync from 704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890859362,1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425 (last read GTID=704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890859363,1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425)
可能是网络抖动带来的链接中断,但是会进行重试。
另外,Syncer 官方已不再维护,如果有数据同步的需求,建议使用 DM 代替~
因为这个第一次使用,所以先用这个,下个项目就考虑用DM 了,确实syncer 导入数据过程中太慢了
还有问题syncer同步的数据不一致的问题:
这个是tidb 库里面的数据
这个是Mysql 库里面的数据:
我看了一下tidb 同步都是用REPLACE INTO 的模式同步数据,我们表的主键都是自增ID ,这个会受到影响吗?
您好,现在报错误,无法同步~ 帮忙看一下
[root@db-146-0-235-79 mydumper]# /bin/syncer -config config.toml --enable-gtid
[2020/04/16 04:10:44] [info] binlogsyncer.go:144 create BinlogSyncer with config {101 mysql 10.1.13.34 3306 tidb_user false false <nil> false UTC true 0 30s 1m0s 0 false false 0}
[2020/04/16 04:10:44] [info] binlogsyncer.go:377 begin to sync binlog from GTID set 22863285-e03a-11e9-a452-ac1f6b403494:1-8454425,704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890944214,1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422
[2020/04/16 04:10:44] [info] binlogsyncer.go:776 rotate to (mysql-bin.006834, 4)
syncer.log (44.7 KB)
1、报错信息显示为:
可能是等锁超时出现,建议再查看下 tidb 的 log ,找到具体的信息。另外,需要确认下,除 syncer 外,是否还有其他业务再对该表数据进行操作~
2、请尝试重启 syncer 任务,重启后,再次观察
3、自 3.0.8 版本开始默认开启悲观锁:
https://pingcap.com/docs-cn/v3.0/reference/transactions/transaction-pessimistic/#tidb-悲观事务模型