syncer 同步失败导致的PD 集群synchronize leader error

* 【TiDB 版本】:V3.0.12

* 【问题描述】: 您好,我早上配置syncer集群同步mysql的数据,然后遇到同步任务失败,无法重启任务的状况后,多次尝试重启,均失败,然后按照你们提供给我的修改悲观锁,修改完了也无法开启任务同步,就重启集群,导致的下面的集群PD LEADER选举失败。帮忙看一下

这是pd.log:

[2020/04/16 05:57:33.746 +00:00] [INFO] [leader.go:418] ["server enable region storage"]
[2020/04/16 05:57:33.747 +00:00] [ERROR] [client.go:103] ["server failed to establish sync stream with leader"] [server=pd_db] [leader=pd] [error=EOF]
[2020/04/16 05:57:33.827 +00:00] [INFO] [leader.go:356] ["leader is deleted"]
[2020/04/16 05:57:34.747 +00:00] [INFO] [leader.go:111] ["leader changed, try to campaign leader"]
[2020/04/16 05:57:34.748 +00:00] [INFO] [leader.go:117] ["skip campaign leader and check later"] [server-name=pd_db] [etcd-leader-id=7063057977773903482]
[2020/04/16 05:57:34.949 +00:00] [INFO] [leader.go:109] ["start watch leader"] [leader="name:\"pd\" member_id:7063057977773903482 peer_urls:\"http://10.1.0.12:2380\" client_urls:\"http://10.1.0.12:2379\" "]
[2020/04/16 05:57:34.950 +00:00] [INFO] [leader.go:418] ["server enable region storage"]
[2020/04/16 05:57:34.950 +00:00] [INFO] [client.go:107] ["server starts to synchronize with leader"] [server=pd_db] [leader=pd] [request-index=1333400]
[2020/04/16 05:59:19.767 +00:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/04/16 05:59:19.767 +00:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/04/16 05:59:19.767 +00:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 10.1.0.21:2379->10.1.0.20:44090: use of closed network connection"]
[2020/04/16 05:59:22.768 +00:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/04/16 05:59:22.769 +00:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/04/16 05:59:22.769 +00:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 10.1.0.21:2379->10.1.0.5:49062: use of closed network connection"]
[2020/04/16 06:01:31.769 +00:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/04/16 06:01:31.769 +00:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/04/16 06:01:31.769 +00:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 10.1.0.21:2379->10.1.0.20:44100: use of closed network connection"]


 这是开启同步任务的及日志:
[root@db mydumper]# /bin/syncer -config config.toml --enable-gtid
[2020/04/16 06:05:31] [info] binlogsyncer.go:144 create BinlogSyncer with config {101 mysql 10.1.0.34 3306 tidb_user    false false <nil> false UTC true 0 30s 1m0s 0 false false 0}
[2020/04/16 06:05:31] [info] binlogsyncer.go:377 begin to sync binlog from GTID set 704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890944214,1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425
[2020/04/16 06:05:31] [info] binlogsyncer.go:776 rotate to (mysql-bin.006834, 4)

   syncer.log:
2020/04/16 06:05:31 printer.go:52: [info] Welcome to syncer
2020/04/16 06:05:31 printer.go:53: [info] Release Version: v1.0.0-78-g6aea485
2020/04/16 06:05:31 printer.go:54: [info] Git Commit Hash: 6aea4851bb0c6e599c64b5c952ce257863c21586
2020/04/16 06:05:31 printer.go:55: [info] Git Branch: master
2020/04/16 06:05:31 printer.go:56: [info] UTC Build Time: 2019-12-18 04:25:45
2020/04/16 06:05:31 printer.go:57: [info] Go Version: go version go1.13 linux/amd64
2020/04/16 06:05:31 main.go:55: [info] config: {"log-level":"info","log-file":"syncer.log","log-rotate":"day","status-addr":":8271","server-id":101,"meta":"/data2/mydumper/syncer.meta","persistent-dir":"","flavor":"mysql","worker-count":42,"batch":1000000,"max-retry":100,"replicate-do-table":null,"replicate-do-db":["infinity_boe","warning_data_boe"],"replicate-ignore-table":null,"replicate-ignore-db":null,"skip-ddls":[],"skip-dmls":null,"route-rules":null,"from":{"host":"10.1.0.34","user":"tidb_user","port":3306,"max-allowed-packet":67108864,"enable-tidb-txn-auto-retry":""},"to":{"host":"10.1.0.5","user":"root","port":4000,"max-allowed-packet":67108864,"enable-tidb-txn-auto-retry":"auto"},"enable-gtid":true,"auto-fix-gtid":false,"disable-detect":false,"safe-mode":false,"config-file":"config.toml","stop-on-ddl":false,"execute-ddl-timeout":"3h","execute-dml-timeout":"1m","execute-queue-length":5000,"enable-ansi-quotes":false,"timezone":"UTC"}
2020/04/16 06:05:31 metrics.go:115: [info] listening on :8271 for status and metrics report.
2020/04/16 06:05:31 syncer.go:981: [info] begin to sync binlog from GTID set 1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425,704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890944214
2020/04/16 06:05:31 syncer.go:967: [info] [syncer] last slave connection id 15880710
2020/04/16 06:05:31 meta.go:135: [info] save position to file, binlog-name:mysql-bin.006834 binlog-pos:660249670 binlog-gtid:704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890944214,1961de6e-db49-11e9-INTO `infinity_boe`.`if_user_city` (`id`,`date`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b205c`,`b205a`,`b206`,`b206c`,`food`,`wood`,`iron`,`miths`,`steel`,`b206a`,`tst`,`countp`,`b301`,`crystal`,`scrystal`,`b207`,`b302`,`sfood`,`swood`,`siron`,`smiths`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id` = ? LIMIT 1; REPLACE INTO `infinity_boe`.`if_user_city_u` (`id`,`date`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b205c`,`b205a`,`b206`,`b206c`,`food`,`wood`,`iron`,`miths`,`steel`,`b206a`,`tst`,`countp`,`b301`,`crystal`,`scrystal`,`b207`,`b302`,`sfood`,`swood`,`siron`,`smiths`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); DELETE FROM `infinity_boe`.`if_user_city_u` WHERE `id` = ? LIMIT 1; REPLACE INTO `infinity_boe`.`if_user_city_u` (`id`,`date`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b205c`,`b205a`,`b206`,`b206c`,`food`,`wood`,`iron`,`miths`,`steel`,`b206a`,`tst`,`countp`,`b301`,`crystal`,`scrystal`,`b207`,`b302`,`sfood`,`swood`,`siron`,`smiths`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); DELETE FROM `infinity_boe`.`if_user_city_u` WHERE `id` = ? LIMIT 1; REPLACE INTO `infinity_boe`.`if_user_city_u` (`id`,`date`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b205c`,`b205a`,`b206`,`b206c`,`food`,`wood`,`iron`,`miths`,`steel`,`b206a`,`tst`,`countp`,`b301`,`crystal`,`scrystal`,`b207`,`b302`,`sfood`,`swood`,`siron`,`smiths`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); DELETE FROM `infinity_boe`.`if_user_city_u` WHERE `id` = ? LIMIT 1; REPLACE INTO `infinity_boe`.`if_user_city_u` (`id`,`date`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b205c`,`b205a`,`b206`,`b206c`,`food`,`wood`,`iron`,`miths`,`steel`,`b206a`,`tst`,`countp`,`b301`,`crystal`,`scrystal`,`b207`,`b302`,`sfood`,`swood`,`siron`,`smiths`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); DELETE FROM `infinity_boe`.`if_user_city_u` WHERE `id` = ? LIMIT 1; REPLACE INTO `infinity_boe`.`if_user_city_u` (`id`,`date`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b20 0 0 20 10 20 10 9 16 18 16 13 12 17 10 12 5 12 20 5 13 10 5 10 15 5 11 13 8 10 13 7 12017293 3453186 36738 55365 60754 11 2020-04-10 21:31:39 23 30 0 0 0 0 11989718 3263675 33118 29060] [590860406 2020-04-10] [590860406 2020-04-10 1586553326 31 5843153 1 1 0 0 14 13 1 13 13 0 0 0 0 0 14 13 14 13 6 14 14 14 13 14 14 4 14 8 3 14 3 8 14 1 14 0 0 0 13 5 4 13 1 103099 1 19659 0 38897 13 2020-04-10 21:31:39 24 30 0 0 0 0 0 1 19659 0] [2379928016 2020-04-10 99 2327251 55 100101 1 {"USE_BACKPACK_ITEM": 1}] [2379928022 2020-04-10 99 1606852 21 100101 1 {"USE_BACKPACK_ITEM": 1}] [2379928075 2020-04-10 99 5352490 119 100101 1 {"USE_BACKPACK_ITEM": 1}] [2379928098 2020-04-10 99 3865445 85 100101 1 {"USE_BACKPACK_ITEM": 1}] [2379928107 2020-04-10 99 5853347 130 100101 1 {"USE_BACKPACK_ITEM": 1}] [2379928187 2020-04-10 99 1337138 31 100103 1 {"USE_BACKPACK_ITEM": 1}] [2379928198 2020-04-10 99 2231462 52 100103 2 {"USE_BACKPACK_ITEM": 2}] [2379928261 2020-04-10 99 299162 23 100103 1 {"USE_BACKPACK_ITEM": 1}] [2379928286 2020-04-10 99 3078427 70 100103 1 {"USE_BACKPACK_ITEM": 1}]][error]Error 9007: Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]
2020/04/16 06:05:31 syncer.go:516: [fatal] [error rows event] Error 9007: Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]
/home/jenkins/agent/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/db.go:156: 
/home/jenkins/agent/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/db.go:117: 

这是TIDB的日志:
tidb.log
[2020/04/16 06:05:31.413 +00:00] [INFO] [server.go:367] ["new connection"] [conn=59] [remoteAddr=10.1.0.21:42140]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.12:20172] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.26:20171] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.26:20173] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.26:20172] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.12:20172] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.21:20172] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.12:20173] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.21:20172] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [session.go:435] ["can not retry txn"] [conn=38] [label=general] [error="[kv:9007]Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]"] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=false] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2020/04/16 06:05:31.675 +00:00] [WARN] [session.go:461] ["commit failed"] [conn=38] ["finished txn"="Txn{state=invalid}"] [error="[kv:9007]Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [session.go:987] ["run statement error"] [conn=38] [schemaVersion=2238] [error="previous statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928286,'2020-04-10',99,3078427,70,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');: [kv:9007]Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]"] [errorVerbose="[kv:9007]Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]\nprevious statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928286,'2020-04-10',99,3078427,70,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');"] [session="{\n  \"currDBName\": \"\",\n  \"id\": 38,\n  \"status\": 2,\n  \"strictMode\": false,\n  \"user\": {\n    \"Username\": \"root\",\n    \"Hostname\": \"10.1.0.21\",\n    \"CurrentUser\": false,\n    \"AuthUsername\": \"root\",\n    \"AuthHostname\": \"%\"\n  }\n}"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [conn.go:682] ["dispatch error"] [conn=38] [connInfo="id:38, addr:10.1.0.21:42098 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql=COMMIT] [err="[kv:9007]Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]\nprevious statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928286,'2020-04-10',99,3078427,70,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');"]
[2020/04/16 06:05:31.686 +00:00] [INFO] [2pc.go:1119] ["2PC clean up done"] [conn=38] [txnStartTS=416027018926751853]
[2020/04/16 06:05:31.699 +00:00] [WARN] [conn.go:658] ["read packet failed, close this connection"] [conn=23] [error="read tcp 10.1.0.5:4000->10.1.0.21:42070: read: connection reset by peer"]
[2020/04/16 06:05:31.699 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=50]
[2020/04/16 06:05:31.699 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=23]
[2020/04/16 06:05:31.699 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=38]
[2020/04/16 06:05:31.699 +00:00] [WARN] [conn.go:658] ["read packet failed, close this connection"] [conn=37] [error="read tcp 10.1.0.5:4000->10.1.0.21:42096: read: connection reset by peer"]
[2020/04/16 06:05:31.699 +00:00] [WARN] [conn.go:658] ["read packet failed, close this connection"] [conn=32] [error="read tcp 10.1.0.5:4000->10.1.0.21:42086: read: connection reset by peer"]
[2020/04/16 06:05:31.699 +00:00] [WARN] [conn.go:658] ["read packet failed, close this connection"] [[2020/04/16 06:05:31.731 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=35]
[2020/04/16 06:05:31.743 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=36]
[2020/04/16 06:05:31.765 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=46]
[2020/04/16 06:05:31.819 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.21:20172] [cause="context canceled"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.26:20171] [cause="context canceled"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [session.go:435] ["can not retry txn"] [conn=26] [label=general] [error="[kv:9007]Write conflict, txnStartTS=416027018926751858, conflictStartTS=416027018939858948, conflictCommitTS=416027018966073345, key={tableID=47940, handle=2396775430} primary={tableID=41880, indexID=1, indexValues={5824401, }} [try again later]"] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=false] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2020/04/16 06:05:31.819 +00:00] [WARN] [session.go:461] ["commit failed"] [conn=26] ["finished txn"="Txn{state=invalid}"] [error="[kv:9007]Write conflict, txnStartTS=416027018926751858, conflictStartTS=416027018939858948, conflictCommitTS=416027018966073345, key={tableID=47940, handle=2396775430} primary={tableID=41880, indexID=1, indexValues={5824401, }} [try again later]"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [session.go:987] ["run statement error"] [conn=26] [schemaVersion=2238] [error="previous statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928318,'2020-04-10',99,4576340,101,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');: [kv:9007]Write conflict, txnStartTS=416027018926751858, conflictStartTS=416027018939858948, conflictCommitTS=416027018966073345, key={tableID=47940, handle=2396775430} primary={tableID=41880, indexID=1, indexValues={5824401, }} [try again later]"] [errorVerbose="[kv:9007]Write conflict, txnStartTS=416027018926751858, conflictStartTS=416027018939858948, conflictCommitTS=416027018966073345, key={tableID=47940, handle=2396775430} primary={tableID=41880, indexID=1, indexValues={5824401, }} [try again later]\nprevious statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928318,'2020-04-10',99,4576340,101,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');"] [session="{\n  \"currDBName\": \"\",\n  \"id\": 26,\n  \"status\": 2,\n  \"strictMode\": false,\n  \"user\": {\n    \"Username\": \"root\",\n    \"Hostname\": \"10.1.0.21\",\n    \"CurrentUser\": false,\n    \"AuthUsername\": \"root\",\n    \"AuthHostname\": \"%\"\n  }\n}"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [client_batch.go:577] ["send request is cancelled"] [to=10.1.0.26:20172] [cause="context canceled"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.21:20172] [cause="context canceled"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [conn.go:682] ["dispatch error"] [conn=26] [connInfo="id:26, addr:10.1.0.21:42074 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql=COMMIT] [err="[kv:9007]Write conflict, txnStartTS=416027018926751858, conflictStartTS=416027018939858948, conflictCommitTS=416027018966073345, key={tableID=47940, handle=2396775430} primary={tableID=41880, indexID=1, indexValues={5824401, }} [try again later]\nprevious statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928318,'2020-04-10',99,4576340,101,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [session.go:435] ["can not retry txn"] [conn=53] [label=general] [error="[kv:9007]Write conflict, txnStartTS=416027018926751851, conflictStartTS=416027018926751853, conflictCommitTS=416027018926751853, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 3078427, 100103, }} primary={tableID=41880, indexID=1, indexValues={5826503, }} [try again later]"] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=false] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2020/04/16 06:05:31.819 +00:00] [WARN] [session.go:461] ["commit failed"] [conn=53] ["finished txn"="Txn{state=invalid}"] [error="[kv:9007]Write conflict, txnStartTS=416027018926751851, conflictStartTS=416027018926751853, conflictCommitTS=416027018926751853, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 3078427, 100103, }} primary={tableID=41880, indexID=1, indexValues={5826503, }} [try again later]"]
[2020/04/16 06:05:31.819 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=26]
[2020/04/16 06:05:31.819 +00:00] [WARN] [session.go:987] ["run statement error"] [conn=53] [schemaVersion=2238] [error="previous statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928277,'2020-04-10',99,4361397,96,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');: [kv:9007]Write conflict, txnStartTS=416027018926751851, conflictStartTS=416027018926751853, conflictCommitTS=416027018926751853, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 3078427, 100103, }} primary={tableID=41880, indexID=1, indexValues={5826503, }} [try again later]"] [errorVerbose="[kv:9007]Write conflict, txnStartTS=416027018926751851, conflictStartTS=416027018926751853, conflictCommitTS=416027018926751853, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 3078427, 100103, }} primary={tableID=41880, indexID=1, indexValues={5826503, }} [try again later]\nprevious statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928277,'2020-04-10',99,4361397,96,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');"] [session="{\n  \"currDBName\": \"\",\n  \"id\": 53,\n  \"status\": 2,\n  \"strictMode\": false,\n  \"user\": {\n    \"Username\": \"root\",\n    \"Hostname\": \"10.1.0.21\",\n    \"CurrentUser\": false,\n    \"AuthUsername\": \"root\",\n    \"AuthHostname\": \"%\"\n  }\n}"]
[2020/04/16 06:05:31.820 +00:00] [WARN] [conn.go:682] ["dispatch error"] [conn=53] [connInfo="id:53, addr:10.1.0.21:42127 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql=COMMIT] [err="[kv:9007]Write conflict, txnStartTS=416027018926751851, conflictStartTS=416027018926751853, conflictCommitTS=416027018926751853, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 3078427, 100103, }} primary={tableID=41880, indexID=1, indexValues={5826503, }} [try again later]\nprevious statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928277,'2020-04-10',99,4361397,96,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');"]
[2020/04/16 06:05:31.820 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=53]
[2020/04/16 06:05:31.827 +00:00] [INFO] [2pc.go:1119] ["2PC clean up done"] [conn=26] [txnStartTS=416027018926751858]
[2020/04/16 06:05:31.828 +00:00] [INFO] [2pc.go:1119] ["2PC clean up done"] [conn=53] [txnStartTS=416027018926751851]
2020/04/16 06:05:59.145 terror.go:357: [error] EOF
github.com/pingcap/errors.AddStack
    /home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/errors.go:174
github.com/pingcap/errors.Trace
    /home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/juju_adaptor.go:15
github.com/pingcap/tidb/server.(*packetIO).readOnePacket
    /home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/packetio.go:80
github.com/pingcap/tidb/server.(*packetIO).readPacket
    /home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/packetio.go:105
github.com/pingcap/tidb/server.(*clientConn).readPacket
    /home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/conn.go:265
github.com/pingcap/tidb/server.(*clientConn).readOptionalSSLRequestAndHandshakeResponse
    /home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/conn.go:471
github.com/pingcap/tidb/server.(*clientConn).handshake
    /home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/conn.go:172
github.com/pingcap/tidb/server.(*Server).onConn
    /home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/server.go:345
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1357
2020/04/16 06:06:29.144 terror.go:357: [error] EOF
github.com/pingcap/errors.AddStack
    /home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/errors.go:174
github.com/pingcap/errors.Trace

tidb_stderr.log :

{"level":"warn","ts":"2020-04-16T05:42:55.472Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-e045b21a-fae3-40e5-ac3d-b
74a18671b5f/10.1.13.12:2379","attempt":99,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
{"level":"warn","ts":"2020-04-16T05:56:43.802Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-a9046073-c502-48b9-bd27-4
0c3f5cfa89f/10.1.13.12:2379","attempt":0,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
{"level":"warn","ts":"2020-04-16T05:56:43.802Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-a9046073-c502-48b9-bd27-4
0c3f5cfa89f/10.1.13.12:2379","attempt":1,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}


tikv.log:
	[2020/04/16 06:16:45.118 +00:00] [INFO] [pd.rs:566] ["try to transfer leader"] [to_peer="id: 540425 store_id: 9024"] [from_peer="id: 726427 store_id: 8159"] [region_id=540424]
[2020/04/16 06:16:45.130 +00:00] [INFO] [peer.rs:1825] ["transfer leader"] [peer="id: 540425 store_id: 9024"] [peer_id=726427] [region_id=540424]
[2020/04/16 06:16:45.130 +00:00] [INFO] [raft.rs:1335] ["[region 540424] 726427 [term 230] starts to transfer leadership to 540425"]
[2020/04/16 06:16:45.130 +00:00] [INFO] [raft.rs:1345] ["[region 540424] 726427 sends MsgTimeoutNow to 540425 immediately as 540425 already has up-to-date log"]
[2020/04/16 06:16:45.300 +00:00] [INFO] [raft.rs:964] ["[region 557899] 726678 [term: 226] received a MsgRequestVote message with higher term from 557900 [term: 227]"]
[2020/04/16 06:16:45.300 +00:00] [INFO] [raft.rs:763] ["[region 557899] 726678 became follower at term 227"]
[2020/04/16 06:16:45.300 +00:00] [INFO] [raft.rs:1148] ["[region 557899] 726678 [logterm: 226, index: 230, vote: 0] cast MsgRequestVote for 557900 [logterm: 226, index: 230] at term 227"]

日志收集.txt (51.8 KB)

1、使用 pd-ctl memeber ,查看 pd 节点的状态

2、使用 pd-ctl store 查看 tikv 状态,从 TiKV 上面的日志看,没有异常信息

3、确认下 tidb server 的状态

另外,改为乐观模型后, syncer 的报错日志有写冲突,需要确认下,同步的目标表,除 syncer 外,还有其他业务在写吗?

+ member
{
  "header": {
    "cluster_id": 6810255259461000908
  },
  "members": [
    {
      "name": "pd",
      "member_id": 7063057977773903482,
      "peer_urls": [
        "http://10.1.0.12:2380"
      ],
      "client_urls": [
        "http://10.1.0.12:2379"
      ]
    },
    {
      "name": "pd_db",
      "member_id": 9878740079948586593,
      "peer_urls": [
        "http://10.1.0.21:2380"
      ],
      "client_urls": [
        "http://10.1.0.21:2379"
      ]
    },
    {
      "name": "pd_db",
      "member_id": 11688234528480141613,
      "peer_urls": [
        "http://10.1.0.26:2380"
      ],
      "client_urls": [
        "http://10.1.0.26:2379"
      ]
    }
  ],
  "leader": {
    "name": "pd",
    "member_id": 7063057977773903482,
    "peer_urls": [
      "http://10.1.0.12:2380"
    ],
    "client_urls": [
      "http://10.1.0.12:2379"
    ]
  },
  "etcd_leader": {
    "name": "pd",
    "member_id": 7063057977773903482,
    "peer_urls": [
      "http://10.1.0.12:2380"
    ],
    "client_urls": [
      "http://10.1.0.12:2379"
    ]
  }
}

2、
+ store
{
  "count": 9,
  "stores": [
    {
      "store": {
        "id": 9023,
        "address": "10.1.0.12:20173",
        "labels": [
          {
            "key": "host",
            "value": "tidb3"
          }
        ],
        "version": "3.0.12",
        "state_name": "Up"
      },
      "status": {
        "capacity": "1.862TiB",
        "available": "1.66TiB",
        "leader_count": 11899,
        "leader_weight": 1,
        "leader_score": 363782,
        "leader_size": 363782,
        "region_count": 30015,
        "region_weight": 1,
        "region_score": 952259,
        "region_size": 952259,
        "start_ts": "2020-04-16T05:57:37Z",
        "last_heartbeat_ts": "2020-04-16T06:40:47.706860444Z",
        "uptime": "43m10.706860444s"
      }
    },
    {
      "store": {
        "id": 1,
        "address": "10.1.0.12:20171",
        "labels": [
          {
            "key": "host",
            "value": "tidb3"
          }
        ],
        "version": "3.0.12",
        "state_name": "Up"
      },
      "status": {
        "capacity": "1.862TiB",
        "available": "1.658TiB",
        "leader_count": 8741,
        "leader_weight": 1,
        "leader_score": 364618,
        "leader_size": 364618,
        "region_count": 25342,
        "region_weight": 1,
        "region_score": 1062150,
        "region_size": 1062150,
        "start_ts": "2020-04-16T05:57:36Z",
        "last_heartbeat_ts": "2020-04-16T06:40:47.268329979Z",
        "uptime": "43m11.268329979s"
      }
    },
    {
      "store": {
        "id": 4,
        "address": "10.1.0.21:20171",
        "labels": [
          {
            "key": "host",
            "value": "tidb4"
          }
        ],
        "version": "3.0.12",
        "state_name": "Up"
      },
      "status": {
        "capacity": "1.861TiB",
        "available": "1.639TiB",
        "leader_count": 7647,
        "leader_weight": 1,
        "leader_score": 364584,
        "leader_size": 364584,
        "region_count": 25663,
        "region_weight": 1,
        "region_score": 1143855,
        "region_size": 1143855,
        "start_ts": "2020-04-16T05:57:37Z",
        "last_heartbeat_ts": "2020-04-16T06:40:47.521310166Z",
        "uptime": "43m10.521310166s"
      }
    },
    {
      "store": {
        "id": 5,
        "address": "10.1.0.26:20171",
        "labels": [
          {
            "key": "host",
            "value": "tidb5"
          }
        ],
        "version": "3.0.12",
        "state_name": "Up"
      },
      "status": {
        "capacity": "1.861TiB",
        "available": "1.676TiB",
        "leader_count": 11335,
        "leader_weight": 1,
        "leader_score": 363543,
        "leader_size": 363543,
        "region_count": 26415,
        "region_weight": 1,
        "region_score": 856654,
        "region_size": 856654,
        "start_ts": "2020-04-16T05:57:38Z",
        "last_heartbeat_ts": "2020-04-16T06:40:48.475126829Z",
        "uptime": "43m10.475126829s"
      }
    },
    {
      "store": {
        "id": 8157,
        "address": "10.1.0.12:20172",
        "labels": [
          {
            "key": "host",
            "value": "tidb3"
          }
        ],
        "version": "3.0.12",
        "state_name": "Up"
      },
      "status": {
        "capacity": "1.862TiB",
        "available": "1.659TiB",
        "leader_count": 8939,
        "leader_weight": 1,
        "leader_score": 364624,
        "leader_size": 364624,
        "region_count": 25964,
        "region_weight": 1,
        "region_score": 1049933,
        "region_size": 1049933,
        "start_ts": "2020-04-16T05:57:36Z",
        "last_heartbeat_ts": "2020-04-16T06:40:47.125536473Z",
        "uptime": "43m11.125536473s"
      }
    },
    {
      "store": {
        "id": 8158,
        "address": "10.1.0.21:20172",
        "labels": [
          {
            "key": "host",
            "value": "tidb4"
          }
        ],
        "version": "3.0.12",
        "state_name": "Up"
      },
      "status": {
        "capacity": "1.861TiB",
        "available": "1.645TiB",
        "leader_count": 7970,
        "leader_weight": 1,
        "leader_score": 364627,
        "leader_size": 364627,
        "region_count": 26724,
        "region_weight": 1,
        "region_score": 1143517,
        "region_size": 1143517,
        "start_ts": "2020-04-16T05:57:37Z",
        "last_heartbeat_ts": "2020-04-16T06:40:47.544248261Z",
        "uptime": "43m10.544248261s"
      }
    },
    {
      "store": {
        "id": 8159,
        "address": "10.1.0.26:20172",
        "labels": [
          {
            "key": "host",
            "value": "tidb5"
          }
        ],
        "version": "3.0.12",
        "state_name": "Up"
      },
      "status": {
        "capacity": "1.861TiB",
        "available": "1.65TiB",
        "leader_count": 8193,
        "leader_weight": 1,
        "leader_score": 364518,
        "leader_size": 364518,
        "region_count": 26068,
        "region_weight": 1,
        "region_score": 1101251,
        "region_size": 1101251,
        "start_ts": "2020-04-16T05:57:37Z",
        "last_heartbeat_ts": "2020-04-16T06:40:47.852420731Z",
        "uptime": "43m10.852420731s"
      }
    },
    {
      "store": {
        "id": 9024,
        "address": "10.1.0.21:20173",
        "labels": [
          {
            "key": "host",
            "value": "tidb4"
          }
        ],
        "version": "3.0.12",
        "state_name": "Up"
      },
      "status": {
        "capacity": "1.861TiB",
        "available": "1023GiB",
        "leader_count": 7330,
        "leader_weight": 1,
        "leader_score": 149520,
        "leader_size": 149520,
        "region_count": 28934,
        "region_weight": 1,
        "region_score": 776970,
        "region_size": 776970,
        "start_ts": "2020-04-16T05:57:41Z",
        "last_heartbeat_ts": "2020-04-16T06:40:45.51350006Z",
        "uptime": "43m4.51350006s"
      }
    },
    {
      "store": {
        "id": 9025,
        "address": "10.1.0.26:20173",
        "labels": [
          {
            "key": "host",
            "value": "tidb5"
          }
        ],
        "version": "3.0.12",
        "state_name": "Up"
      },
      "status": {
        "capacity": "1.861TiB",
        "available": "1.652TiB",
        "leader_count": 9267,
        "leader_weight": 1,
        "leader_score": 364526,
        "leader_size": 364526,
        "region_count": 28838,
        "region_weight": 1,
        "region_score": 1106437,
        "region_size": 1106437,
        "start_ts": "2020-04-16T05:57:37Z",
        "last_heartbeat_ts": "2020-04-16T06:40:48.2125762Z",
        "uptime": "43m11.2125762s"
      }
    }
  ]
}

3、没有,这个只有syncer的同步任务,没有其他的业务

2台TIDB 服务器,一台服务器1个实例。 3台tikv 服务器,一台tikv 服务器部署3个tikv 实例。然后PD集群3个也是一起部署在tikv 3台服务器上

[tidb_servers]
10.1.0.5
10.1.0.20

[tikv_servers]
TiKV1-1 ansible_host=10.1.0.12 deploy_dir=/data/tidb/deploy tikv_port=20171 tikv_status_port=20181 labels="host=tidb3"
TiKV1-2 ansible_host=10.1.0.12 deploy_dir=/data1/tidb/deploy tikv_port=20172 tikv_status_port=20182 labels="host=tidb3"
TiKV1-3 ansible_host=10.1.0.12 deploy_dir=/data2/tidb/deploy tikv_port=20173 tikv_status_port=20183 labels="host=tidb3"

TiKV2-1 ansible_host=10.1.0.21 deploy_dir=/data/tidb/deploy tikv_port=20171 tikv_status_port=20181 labels="host=tidb4"
TiKV2-2 ansible_host=10.1.0.21 deploy_dir=/data1/tidb/deploy tikv_port=20172 tikv_status_port=20182 labels="host=tidb4"
TiKV2-3 ansible_host=10.1.0.21 deploy_dir=/data2/tidb/deploy tikv_port=20173 tikv_status_port=20183 labels="host=tidb4"

TiKV3-1 ansible_host=10.1.0.26 deploy_dir=/data/tidb/deploy tikv_port=20171 tikv_status_port=20181 labels="host=tidb5"
TiKV3-2 ansible_host=10.1.0.26 deploy_dir=/data1/tidb/deploy tikv_port=20172 tikv_status_port=20182 labels="host=tidb5"
TiKV3-3 ansible_host=10.1.0.26 deploy_dir=/data2/tidb/deploy tikv_port=20173 tikv_status_port=20183 labels="host=tidb5"


[pd_servers]
10.1.0.12
10.1.0.21
10.1.0.26

1、请再次使用 pd-ctl health 看下 pd 节点的状态,这个报警,应该不会影响 pd 的正常使用~

2、syncer 无法同步是指进程无法启动还是说,启动后出现 write conflict 报错?

3、如果允许,请上传用于 syncer 同步的 tidb server 较为完整的日志

4、请确认下当前的事务模式:show variables like ‘%txn_mode%’ 以及 tidb_disable_txn_auto_retry 参数的值

1、+ health 
[
  {
    "name": "pd",
    "member_id": 7063057977773903482,
    "client_urls": [
      "http://10.1.0.12:2379"
    ],
    "health": true
  },
  {
    "name": "pd_db1",
    "member_id": 9878740079948586593,
    "client_urls": [
      "http://10.1.0.21:2379"
    ],
    "health": true
  },
  {
    "name": "pd_db2",
    "member_id": 11688234528480141613,
    "client_urls": [
      "http://10.1.0.26:2379"
    ],
    "health": true
  }
]

2、无法启动,启动会报错:
[root@db mydumper]# /bin/syncer -config config.toml --enable-gtid
/bin/syncer -config config.toml --enable-gtid
[2020/04/16 07:10:06] [info] binlogsyncer.go:144 create BinlogSyncer with config {101 mysql 10.1.0.3 3306 tidb_user    false false <nil> false UTC true 0 30s 1m0s 0 false false 0}
[2020/04/16 07:10:06] [info] binlogsyncer.go:377 begin to sync binlog from GTID set 1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425,704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890944214
[2020/04/16 07:10:06] [info] binlogsyncer.go:776 rotate to (mysql-bin.006834, 4)


more syncer.log
2020/04/16 07:10:06 printer.go:52: [info] Welcome to syncer
2020/04/16 07:10:06 printer.go:53: [info] Release Version: v1.0.0-78-g6aea485
2020/04/16 07:10:06 printer.go:54: [info] Git Commit Hash: 6aea4851bb0c6e599c64b5c952ce257863c21586
2020/04/16 07:10:06 printer.go:55: [info] Git Branch: master
2020/04/16 07:10:06 printer.go:56: [info] UTC Build Time: 2019-12-18 04:25:45
2020/04/16 07:10:06 printer.go:57: [info] Go Version: go version go1.13 linux/amd64
2020/04/16 07:10:06 main.go:55: [info] config: {"log-level":"info","log-file":"syncer.log","log-rotate":"day","status-addr":":8271","server-id":101,"meta":"/data2/mydumper/syncer.meta","
persistent-dir":"","flavor":"mysql","worker-count":42,"batch":1000000,"max-retry":100,"replicate-do-table":null,"replicate-do-db":["infinity_boe","warning_data_boe"],"replicate-ignore-ta
ble":null,"replicate-ignore-db":null,"skip-ddls":[],"skip-dmls":null,"route-rules":null,"from":{"host":"10.1.13.34","user":"tidb_user","port":3306,"max-allowed-packet":67108864,"enable-t
idb-txn-auto-retry":""},"to":{"host":"10.1.13.5","user":"root","port":4000,"max-allowed-packet":67108864,"enable-tidb-txn-auto-retry":"auto"},"enable-gtid":true,"auto-fix-gtid":false,"di
sable-detect":false,"safe-mode":false,"config-file":"config.toml","stop-on-ddl":false,"execute-ddl-timeout":"3h","execute-dml-timeout":"1m","execute-queue-length":5000,"enable-ansi-quote
s":false,"timezone":"UTC"}
2020/04/16 07:10:06 metrics.go:115: [info] listening on :8271 for status and metrics report.
2020/04/16 07:10:06 syncer.go:981: [info] begin to sync binlog from GTID set 1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425,704aa4cd-eaa
b-11e9-a391-ac1f6b41525a:1-890944214
2020/04/16 07:10:06 syncer.go:967: [info] [syncer] last slave connection id 15881567
2020/04/16 07:10:06 meta.go:135: [info] save position to file, binlog-name:mysql-bin.006834 binlog-pos:660249670 binlog-gtid:704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890944214,1961de6e-db4
9-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425
2020/04/16 07:10:06 db.go:155: [error] exec sqls[[DELETE FROM `infinity_boe`.`if_user_city` WHERE `id` = ? AND `date` = ? LIMIT 1; REPLACE INTO `infinity_boe`.`if_user_city` (`id`,`date`
,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,
`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b205c`,`b205a`,`b206`,`b206c`,`food`,`wood`,`iron`,`miths`
,`steel`,`b206a`,`tst`,`countp`,`b301`,`crystal`,`scrystal`,`b207`,`b302`,`sfood`,`swood`,`siron`,`smiths`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,
?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,
?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_it
em_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`g
oodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); RE
PLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn
` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,
`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `in
finity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`
,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES
 (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`i
f_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,
`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,
?,?);]] commit failed Error 9007: Write conflict, txnStartTS=416028034760966213, conflictStartTS=416028034760966217, conflictCommitTS=416028034787180546, key={tableID=47940, handle=23967
75438} primary={tableID=42382, indexID=1, indexValues={590861867, 1848186687314722816, }} [try again later]
2020/04/16 07:10:06 db.go:116: [error] [exec][sql][DELETE FROM `infinity_boe`.`if_user_city` WHERE `id` = ? AND `date` = ? LIMIT 1; REPLACE INTO `infinity_boe`.`if_user_city` (`id`,`date
`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`
,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b205c`,`b205a`,`b206`,`b206c`,`food`,`wood`,`iron`,`miths
`,`steel`,`b206a`,`tst`,`countp`,`b301`,`crystal`,`scrystal`,`b207`,`b302`,`sfood`,`swood`,`siron`,`smiths`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?
,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?
,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_i
tem_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`
goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); R
EPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_war
n` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`
,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `i
nfinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date
`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUE
S (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`
if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`
,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?
,?,?);][args][[590861867 2020-04-10] [590861867 2020-04-10 1586553326 31 5856638 1 1 1 0 11 10 10 7 9 0 0 0 0 0 11 0 9 9 8 10 10 10 10 10 10 8 10 9 7 10 9 8 10 2 10 0 0 0 10 3 10 9 6 165
0111 1036539 762525 0 54320 7 2020-04-10 21:31:39 24 30 0 0 0 0 664911 738277 586864 0] [2379928049 2020-04-10 99 5943205 132 100101 1 {"USE_BACKPACK_ITEM": 1}] [2379928240 2020-04-10 99
 5541281 123 100103 1 {"USE_BACKPACK_ITEM": 1}] [2379928254 2020-04-10 99 2601749 61 100103 1 {"USE_BACKPACK_ITEM": 1}] [2379928280 2020-04-10 99 5690160 126 100103 1 {"USE_BACKPACK_ITEM
": 1}] [2379928322 2020-04-10 99 3006206 55 100103 1 {"USE_BACKPACK_ITEM": 1}] [2379928335 2020-04-10 99 2532295 59 100105 1 {"KILL_MONSTER": 1}] [2379928441 2020-04-10 99 416720 11 1001
05 1 {"KILL_MONSTER": 1}] [2379928447 2020-04-10 99 1464113 34 100105 1 {"GET_ALL_MAIL_EXTRA": 1}] [2379928496 2020-04-10 99 5633473 125 100105 1 {"KILL_MONSTER": 1}] [2379928524 2020-04
-10 99 5398262 120 100105 1 {"KILL_MONSTER": 1}] [2379928541 2020-04-10 99 3837367 84 100105 1 {"KILL_MONSTER": 1}] [2379928549 2020-04-10 99 372224 9 100105 1 {"KILL_MONSTER": 1}] [2379
928550 2020-04-10 99 2873681 53 100105 1 {"KILL_MONSTER": 1}] [2379928556 2020-04-10 99 1478146 34 100105 1 {"KILL_MONSTER": 1}] [2379928590 2020-04-10 99 4742512 31 100105 1 {"KILL_MONS
TER": 1}] [2379928673 2020-04-10 99 1872699 44 100105 1 {"KILL_MONSTER": 1}]][error]Error 9007: Write conflict, txnStartTS=416028034760966213, conflictStartTS=416028034760966217, conflic
tCommitTS=416028034787180546, key={tableID=47940, handle=2396775438} primary={tableID=42382, indexID=1, indexValues={590861867, 1848186687314722816, }} [try again later]
2020/04/16 07:10:06 syncer.go:516: [fatal] [error rows event] Error 9007: Write conflict, txnStartTS=416028034760966213, conflictStartTS=416028034760966217, conflictCommitTS=416028034787
180546, key={tableID=47940, handle=2396775438} primary={tableID=42382, indexID=1, indexValues={590861867, 1848186687314722816, }} [try again later]
/home/jenkins/agent/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/db.go:156: 
/home/jenkins/agent/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/db.go:117: 

MySQL [(none)]> show variables like '%txn_mode%' ;
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| tidb_txn_mode |       |
+---------------+-------+
1 row in set (0.01 sec)

MySQL [(none)]> 
MySQL [(none)]> show variables like '%tidb_disable_txn_auto_retry%';
+-----------------------------+-------+
| Variable_name               | Value |
+-----------------------------+-------+
| tidb_disable_txn_auto_retry | 1     |
+-----------------------------+-------+
1 row in set (0.00 sec)

出现冲突,可能是在导入时下游导入的速度比较慢,然后 syncer 会进行重试,写冲突持续出现,建议调小并发再观察下:

worker-count = 42 ----> 16

batch = 1000 ----> 100

也不行,还是继续上面那个错误

这个和事物的大小有关系没?

我改了这两个值也是继续报错:

worker-count = 16
batch = 1
  

  020/04/16 07:49:07 printer.go:52: [info] Welcome to syncer
    2020/04/16 07:49:07 printer.go:53: [info] Release Version: v1.0.0-78-g6aea485
    2020/04/16 07:49:07 printer.go:54: [info] Git Commit Hash: 6aea4851bb0c6e599c64b5c952ce257863c21586
    2020/04/16 07:49:07 printer.go:55: [info] Git Branch: master
    2020/04/16 07:49:07 printer.go:56: [info] UTC Build Time: 2019-12-18 04:25:45
    2020/04/16 07:49:07 printer.go:57: [info] Go Version: go version go1.13 linux/amd64
    2020/04/16 07:49:07 main.go:55: [info] config: {"log-level":"info","log-file":"syncer.log","log-rotate":"day","status-addr":":8271","server-id":101,"meta":"/data2/mydumper/syncer.meta","persistent-dir":"","flavor":"mysql","worker-count":16,"batch":1,"max-retry":100,"replicate-do-table":null,"replicate-do-db":["infinity_boe","warning_data_boe"],"replicate-ignore-table":null,"replicate-ignore-db":null,"skip-ddls":[],"skip-dmls":null,"route-rules":null,"from":{"host":"10.1.13.34","user":"tidb_user","port":3306,"max-allowed-packet":67108864,"enable-tidb-txn-auto-retry":""},"to":{"host":"10.1.13.5","user":"root","port":4000,"max-allowed-packet":67108864,"enable-tidb-txn-auto-retry":"auto"},"enable-gtid":true,"auto-fix-gtid":false,"disable-detect":false,"safe-mode":false,"config-file":"config.toml","stop-on-ddl":false,"execute-ddl-timeout":"3h","execute-dml-timeout":"1m","execute-queue-length":5000,"enable-ansi-quotes":false,"timezone":"UTC"}
    2020/04/16 07:49:07 metrics.go:115: [info] listening on :8271 for status and metrics report.
    2020/04/16 07:49:07 syncer.go:981: [info] begin to sync binlog from GTID set 704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890944214,1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425
    2020/04/16 07:49:07 syncer.go:967: [info] [syncer] last slave connection id 15882086
    2020/04/16 07:49:07 meta.go:135: [info] save position to file, binlog-name:mysql-bin.006834 binlog-pos:660249670 binlog-gtid:704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890944214,1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425
    2020/04/16 07:49:12 db.go:155: [error] exec sqls[[REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?);]] commit failed Error 9007: Write conflict, txnStartTS=416028649803218976, conflictStartTS=416028649803218969, conflictCommitTS=416028649803218981, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 577124, 100105, }} primary={tableID=47940, indexID=1, indexValues={2379929113, 1848186687314722816, }} [try again later]
    2020/04/16 07:49:12 db.go:116: [error] [exec][sql][REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (?,?,?,?,?,?,?,?);][args][[2379929125 2020-04-10 99 5141375 114 100105 1 {"KILL_MONSTER": 1}]][error]Error 9007: Write conflict, txnStartTS=416028649803218976, conflictStartTS=416028649803218969, conflictCommitTS=416028649803218981, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 577124, 100105, }} primary={tableID=47940, indexID=1, indexValues={2379929113, 1848186687314722816, }} [try again later]
    2020/04/16 07:49:12 syncer.go:507: [fatal] [error rows event] Error 9007: Write conflict, txnStartTS=416028649803218976, conflictStartTS=416028649803218969, conflictCommitTS=416028649803218981, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 577124, 100105, }} primary={tableID=47940, indexID=1, indexValues={2379929113, 1848186687314722816, }} [try again later]
    /home/jenkins/agent/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/db.go:156: 
    /home/jenkins/agent/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/db.go:117:

好的,咱们这套环境部署监控了吧,方便导出下 tidb 和 tikv-details 整个监控面板的信息吗?导出方式如下:

1、chrome 安装这个插件 https://chrome.google.com/webstore/detail/full-page-screen-capture/fdpohaocaechififmbbbbbknoalclacl

2、鼠标焦点置于 Dashboard 上,按 ?可显示所有快捷键,先按 d 再按 E 可将所有 Rows 的 Panels 打开,需等待一段时间待页面加载完成。

3、使用这个 full-page-screen-capture 插件进行截屏保存

2 步骤不能输入?

dE 输入到哪里呢

老师,我还是不懂怎么导出来,现在没办法继续做同步任务,这个项目我已经研究了一个季度了,不能在这个关键节点掉链子,能否帮忙一下

现在是想确认下下游是否还有事务在重试:

1、可以将 tidb 监控面板的 transaction 子面板,以及 kv error 面板截图上传下也可以~

2、请在下游两个 tidb server 执行下,show processlist 看下结果哈~

MySQL [infinity_boe]> show processlist
    -> ;
+------+------+-----------+--------------+---------+------+-------+------------------+
| Id   | User | Host      | db           | Command | Time | State | Info             |
+------+------+-----------+--------------+---------+------+-------+------------------+
|  461 | root | 127.0.0.1 | infinity_boe | Query   |    0 | 2     | show processlist |
+------+------+-----------+--------------+---------+------+-------+------------------+
1 row in set (0.00 sec)

image image image image image image image image image image

CREATE TABLE `if_add_item_source_warn` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '自增id',
  `date` date NOT NULL DEFAULT '0000-00-00' COMMENT '日期',
  `hour` int(10) NOT NULL DEFAULT '88' COMMENT '小时,99表示全天',
  `uid` bigint(20) NOT NULL DEFAULT '0' ,
  `kid` int(11) NOT NULL DEFAULT '0' ,
  `goodsid` int(11) NOT NULL DEFAULT '0',
  `num` bigint(20) NOT NULL DEFAULT '0',
  `data` varchar(2000) NOT NULL DEFAULT '',
   PRIMARY KEY (`id`,`date`),
  UNIQUE KEY `idx_unique_key` (`date`,`hour`,`uid`,`goodsid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin AUTO_INCREMENT=2415833799 COMMENT='玩家获取物品来源预警'
PARTITION BY RANGE COLUMNS(date) (
  PARTITION p20200328 VALUES LESS THAN ("2020-03-29"),
  PARTITION p20200329 VALUES LESS THAN ("2020-03-30"),
  PARTITION p20200330 VALUES LESS THAN ("2020-03-31"),
  PARTITION p20200331 VALUES LESS THAN ("2020-04-01"),
  PARTITION p20200401 VALUES LESS THAN ("2020-04-02"),
  PARTITION p20200402 VALUES LESS THAN ("2020-04-03"),
  PARTITION p20200403 VALUES LESS THAN ("2020-04-04"),
  PARTITION p20200404 VALUES LESS THAN ("2020-04-05"),
  PARTITION p20200405 VALUES LESS THAN ("2020-04-06"),
  PARTITION p20200406 VALUES LESS THAN ("2020-04-07"),
  PARTITION p20200407 VALUES LESS THAN ("2020-04-08"),
  PARTITION p20200408 VALUES LESS THAN ("2020-04-09"),
  PARTITION p20200409 VALUES LESS THAN ("2020-04-10"),
  PARTITION p20200410 VALUES LESS THAN ("2020-04-11"),
  PARTITION p20200411 VALUES LESS THAN ("2020-04-12"),
  PARTITION p20200412 VALUES LESS THAN ("2020-04-13"),
  PARTITION p20200413 VALUES LESS THAN ("2020-04-14"),
  PARTITION p20200414 VALUES LESS THAN ("2020-04-15"),
  PARTITION p20200415 VALUES LESS THAN ("2020-04-16"),
  PARTITION p20200416 VALUES LESS THAN ("2020-04-17"),
  PARTITION p20200417 VALUES LESS THAN ("2020-04-18"),
  PARTITION p20200418 VALUES LESS THAN ("2020-04-19"),
  PARTITION p20200419 VALUES LESS THAN ("2020-04-20"),
  PARTITION p20200420 VALUES LESS THAN ("2020-04-21"),
  PARTITION p20200421 VALUES LESS THAN ("2020-04-22"),
  PARTITION p20200422 VALUES LESS THAN ("2020-04-23"),
  PARTITION p20200423 VALUES LESS THAN ("2020-04-24"),
  PARTITION p20200424 VALUES LESS THAN ("2020-04-25"),
  PARTITION p20200425 VALUES LESS THAN ("2020-04-26"),
  PARTITION p20200426 VALUES LESS THAN ("2020-04-27"),
  PARTITION p20200427 VALUES LESS THAN ("2020-04-28"),
  PARTITION p20200428 VALUES LESS THAN ("2020-04-29"),
  PARTITION p20200429 VALUES LESS THAN ("2020-04-30"),
  PARTITION p20200430 VALUES LESS THAN ("2020-05-01"),
  PARTITION p20200501 VALUES LESS THAN ("2020-05-02"),
  PARTITION p20200502 VALUES LESS THAN ("2020-05-03"),
  PARTITION p20200503 VALUES LESS THAN ("2020-05-04"),
  PARTITION p20200504 VALUES LESS THAN ("2020-05-05"),
  PARTITION p20200505 VALUES LESS THAN ("2020-05-06"),
  PARTITION p20200506 VALUES LESS THAN ("2020-05-07"),
  PARTITION p20200507 VALUES LESS THAN ("2020-05-08"),
  PARTITION p20200508 VALUES LESS THAN ("2020-05-09"),
  PARTITION p20200509 VALUES LESS THAN ("2020-05-10"),
  PARTITION p20200510 VALUES LESS THAN ("2020-05-11"))

tidb的sql_mode 和mysql 的是一致的。

MySQL [(none)]> select @@sql_mode; ±-----------------------+ | @@sql_mode | ±-----------------------+ | NO_ENGINE_SUBSTITUTION | ±-----------------------+

  1. 麻烦调整 worker-count = 1 进行尝试。看会不会出现问题
  2. 如果继续出现。麻烦上传一下出现时间段的 TiDB 的日志,以及 dm-worker 的日志。