* 【TiDB 版本】：V3.0.12
* 【问题描述】： 您好，我早上配置syncer集群同步mysql的数据，然后遇到同步任务失败，无法重启任务的状况后，多次尝试重启，均失败，然后按照你们提供给我的修改悲观锁，修改完了也无法开启任务同步，就重启集群，导致的下面的集群PD LEADER选举失败。帮忙看一下
这是pd.log:
[2020/04/16 05:57:33.746 +00:00] [INFO] [leader.go:418] ["server enable region storage"]
[2020/04/16 05:57:33.747 +00:00] [ERROR] [client.go:103] ["server failed to establish sync stream with leader"] [server=pd_db] [leader=pd] [error=EOF]
[2020/04/16 05:57:33.827 +00:00] [INFO] [leader.go:356] ["leader is deleted"]
[2020/04/16 05:57:34.747 +00:00] [INFO] [leader.go:111] ["leader changed, try to campaign leader"]
[2020/04/16 05:57:34.748 +00:00] [INFO] [leader.go:117] ["skip campaign leader and check later"] [server-name=pd_db] [etcd-leader-id=7063057977773903482]
[2020/04/16 05:57:34.949 +00:00] [INFO] [leader.go:109] ["start watch leader"] [leader="name:\"pd\" member_id:7063057977773903482 peer_urls:\"http://10.1.0.12:2380\" client_urls:\"http://10.1.0.12:2379\" "]
[2020/04/16 05:57:34.950 +00:00] [INFO] [leader.go:418] ["server enable region storage"]
[2020/04/16 05:57:34.950 +00:00] [INFO] [client.go:107] ["server starts to synchronize with leader"] [server=pd_db] [leader=pd] [request-index=1333400]
[2020/04/16 05:59:19.767 +00:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/04/16 05:59:19.767 +00:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/04/16 05:59:19.767 +00:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 10.1.0.21:2379->10.1.0.20:44090: use of closed network connection"]
[2020/04/16 05:59:22.768 +00:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/04/16 05:59:22.769 +00:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/04/16 05:59:22.769 +00:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 10.1.0.21:2379->10.1.0.5:49062: use of closed network connection"]
[2020/04/16 06:01:31.769 +00:00] [ERROR] [grpclog.go:75] ["transport: Got too many pings from the client, closing the connection."]
[2020/04/16 06:01:31.769 +00:00] [ERROR] [grpclog.go:75] ["transport: loopyWriter.run returning. Err: transport: Connection closing"]
[2020/04/16 06:01:31.769 +00:00] [WARN] [grpclog.go:60] ["transport: http2Server.HandleStreams failed to read frame: read tcp 10.1.0.21:2379->10.1.0.20:44100: use of closed network connection"]
这是开启同步任务的及日志：
[root@db mydumper]# /bin/syncer -config config.toml --enable-gtid
[2020/04/16 06:05:31] [info] binlogsyncer.go:144 create BinlogSyncer with config {101 mysql 10.1.0.34 3306 tidb_user false false <nil> false UTC true 0 30s 1m0s 0 false false 0}
[2020/04/16 06:05:31] [info] binlogsyncer.go:377 begin to sync binlog from GTID set 704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890944214,1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425
[2020/04/16 06:05:31] [info] binlogsyncer.go:776 rotate to (mysql-bin.006834, 4)
syncer.log:
2020/04/16 06:05:31 printer.go:52: [info] Welcome to syncer
2020/04/16 06:05:31 printer.go:53: [info] Release Version: v1.0.0-78-g6aea485
2020/04/16 06:05:31 printer.go:54: [info] Git Commit Hash: 6aea4851bb0c6e599c64b5c952ce257863c21586
2020/04/16 06:05:31 printer.go:55: [info] Git Branch: master
2020/04/16 06:05:31 printer.go:56: [info] UTC Build Time: 2019-12-18 04:25:45
2020/04/16 06:05:31 printer.go:57: [info] Go Version: go version go1.13 linux/amd64
2020/04/16 06:05:31 main.go:55: [info] config: {"log-level":"info","log-file":"syncer.log","log-rotate":"day","status-addr":":8271","server-id":101,"meta":"/data2/mydumper/syncer.meta","persistent-dir":"","flavor":"mysql","worker-count":42,"batch":1000000,"max-retry":100,"replicate-do-table":null,"replicate-do-db":["infinity_boe","warning_data_boe"],"replicate-ignore-table":null,"replicate-ignore-db":null,"skip-ddls":[],"skip-dmls":null,"route-rules":null,"from":{"host":"10.1.0.34","user":"tidb_user","port":3306,"max-allowed-packet":67108864,"enable-tidb-txn-auto-retry":""},"to":{"host":"10.1.0.5","user":"root","port":4000,"max-allowed-packet":67108864,"enable-tidb-txn-auto-retry":"auto"},"enable-gtid":true,"auto-fix-gtid":false,"disable-detect":false,"safe-mode":false,"config-file":"config.toml","stop-on-ddl":false,"execute-ddl-timeout":"3h","execute-dml-timeout":"1m","execute-queue-length":5000,"enable-ansi-quotes":false,"timezone":"UTC"}
2020/04/16 06:05:31 metrics.go:115: [info] listening on :8271 for status and metrics report.
2020/04/16 06:05:31 syncer.go:981: [info] begin to sync binlog from GTID set 1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425,704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890944214
2020/04/16 06:05:31 syncer.go:967: [info] [syncer] last slave connection id 15880710
2020/04/16 06:05:31 meta.go:135: [info] save position to file, binlog-name:mysql-bin.006834 binlog-pos:660249670 binlog-gtid:704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-890944214,1961de6e-db49-11e9-INTO `infinity_boe`.`if_user_city` (`id`,`date`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b205c`,`b205a`,`b206`,`b206c`,`food`,`wood`,`iron`,`miths`,`steel`,`b206a`,`tst`,`countp`,`b301`,`crystal`,`scrystal`,`b207`,`b302`,`sfood`,`swood`,`siron`,`smiths`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id` = ? LIMIT 1; REPLACE INTO `infinity_boe`.`if_user_city_u` (`id`,`date`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b205c`,`b205a`,`b206`,`b206c`,`food`,`wood`,`iron`,`miths`,`steel`,`b206a`,`tst`,`countp`,`b301`,`crystal`,`scrystal`,`b207`,`b302`,`sfood`,`swood`,`siron`,`smiths`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); DELETE FROM `infinity_boe`.`if_user_city_u` WHERE `id` = ? LIMIT 1; REPLACE INTO `infinity_boe`.`if_user_city_u` (`id`,`date`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b205c`,`b205a`,`b206`,`b206c`,`food`,`wood`,`iron`,`miths`,`steel`,`b206a`,`tst`,`countp`,`b301`,`crystal`,`scrystal`,`b207`,`b302`,`sfood`,`swood`,`siron`,`smiths`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); DELETE FROM `infinity_boe`.`if_user_city_u` WHERE `id` = ? LIMIT 1; REPLACE INTO `infinity_boe`.`if_user_city_u` (`id`,`date`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b205c`,`b205a`,`b206`,`b206c`,`food`,`wood`,`iron`,`miths`,`steel`,`b206a`,`tst`,`countp`,`b301`,`crystal`,`scrystal`,`b207`,`b302`,`sfood`,`swood`,`siron`,`smiths`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); DELETE FROM `infinity_boe`.`if_user_city_u` WHERE `id` = ? LIMIT 1; REPLACE INTO `infinity_boe`.`if_user_city_u` (`id`,`date`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b205c`,`b205a`,`b206`,`b206c`,`food`,`wood`,`iron`,`miths`,`steel`,`b206a`,`tst`,`countp`,`b301`,`crystal`,`scrystal`,`b207`,`b302`,`sfood`,`swood`,`siron`,`smiths`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); DELETE FROM `infinity_boe`.`if_user_city_u` WHERE `id` = ? LIMIT 1; REPLACE INTO `infinity_boe`.`if_user_city_u` (`id`,`date`,`curhour`,`game`,`uid`,`is1220`,`is1221`,`is1222`,`is1223`,`b101`,`b102`,`b103`,`b104`,`b105`,`b106`,`b107`,`b108`,`b109`,`b110`,`b111`,`b112`,`b113`,`b114`,`b115`,`b116`,`b117`,`b118`,`b119`,`b120`,`b121`,`b122`,`b201`,`b201c`,`b201a`,`b202`,`b202c`,`b202a`,`b203`,`b203c`,`b203a`,`b204`,`b204c`,`b204a`,`b205`,`b20 0 0 20 10 20 10 9 16 18 16 13 12 17 10 12 5 12 20 5 13 10 5 10 15 5 11 13 8 10 13 7 12017293 3453186 36738 55365 60754 11 2020-04-10 21:31:39 23 30 0 0 0 0 11989718 3263675 33118 29060] [590860406 2020-04-10] [590860406 2020-04-10 1586553326 31 5843153 1 1 0 0 14 13 1 13 13 0 0 0 0 0 14 13 14 13 6 14 14 14 13 14 14 4 14 8 3 14 3 8 14 1 14 0 0 0 13 5 4 13 1 103099 1 19659 0 38897 13 2020-04-10 21:31:39 24 30 0 0 0 0 0 1 19659 0] [2379928016 2020-04-10 99 2327251 55 100101 1 {"USE_BACKPACK_ITEM": 1}] [2379928022 2020-04-10 99 1606852 21 100101 1 {"USE_BACKPACK_ITEM": 1}] [2379928075 2020-04-10 99 5352490 119 100101 1 {"USE_BACKPACK_ITEM": 1}] [2379928098 2020-04-10 99 3865445 85 100101 1 {"USE_BACKPACK_ITEM": 1}] [2379928107 2020-04-10 99 5853347 130 100101 1 {"USE_BACKPACK_ITEM": 1}] [2379928187 2020-04-10 99 1337138 31 100103 1 {"USE_BACKPACK_ITEM": 1}] [2379928198 2020-04-10 99 2231462 52 100103 2 {"USE_BACKPACK_ITEM": 2}] [2379928261 2020-04-10 99 299162 23 100103 1 {"USE_BACKPACK_ITEM": 1}] [2379928286 2020-04-10 99 3078427 70 100103 1 {"USE_BACKPACK_ITEM": 1}]][error]Error 9007: Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]
2020/04/16 06:05:31 syncer.go:516: [fatal] [error rows event] Error 9007: Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]
/home/jenkins/agent/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/db.go:156:
/home/jenkins/agent/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/syncer/db.go:117:
这是TIDB的日志：
tidb.log
[2020/04/16 06:05:31.413 +00:00] [INFO] [server.go:367] ["new connection"] [conn=59] [remoteAddr=10.1.0.21:42140]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.12:20172] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.26:20171] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.26:20173] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.26:20172] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.12:20172] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.21:20172] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.12:20173] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.21:20172] [cause="context canceled"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [session.go:435] ["can not retry txn"] [conn=38] [label=general] [error="[kv:9007]Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]"] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=false] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2020/04/16 06:05:31.675 +00:00] [WARN] [session.go:461] ["commit failed"] [conn=38] ["finished txn"="Txn{state=invalid}"] [error="[kv:9007]Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [session.go:987] ["run statement error"] [conn=38] [schemaVersion=2238] [error="previous statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928286,'2020-04-10',99,3078427,70,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');: [kv:9007]Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]"] [errorVerbose="[kv:9007]Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]\
previous statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928286,'2020-04-10',99,3078427,70,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');"] [session="{\
\"currDBName\": \"\",\
\"id\": 38,\
\"status\": 2,\
\"strictMode\": false,\
\"user\": {\
\"Username\": \"root\",\
\"Hostname\": \"10.1.0.21\",\
\"CurrentUser\": false,\
\"AuthUsername\": \"root\",\
\"AuthHostname\": \"%\"\
}\
}"]
[2020/04/16 06:05:31.675 +00:00] [WARN] [conn.go:682] ["dispatch error"] [conn=38] [connInfo="id:38, addr:10.1.0.21:42098 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql=COMMIT] [err="[kv:9007]Write conflict, txnStartTS=416027018926751853, conflictStartTS=416027018926751859, conflictCommitTS=416027018952966166, key={tableID=47940, indexID=1, indexValues={2379928261, 1848186687314722816, }} primary={tableID=41880, indexID=1, indexValues={5822015, }} [try again later]\
previous statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928286,'2020-04-10',99,3078427,70,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');"]
[2020/04/16 06:05:31.686 +00:00] [INFO] [2pc.go:1119] ["2PC clean up done"] [conn=38] [txnStartTS=416027018926751853]
[2020/04/16 06:05:31.699 +00:00] [WARN] [conn.go:658] ["read packet failed, close this connection"] [conn=23] [error="read tcp 10.1.0.5:4000->10.1.0.21:42070: read: connection reset by peer"]
[2020/04/16 06:05:31.699 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=50]
[2020/04/16 06:05:31.699 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=23]
[2020/04/16 06:05:31.699 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=38]
[2020/04/16 06:05:31.699 +00:00] [WARN] [conn.go:658] ["read packet failed, close this connection"] [conn=37] [error="read tcp 10.1.0.5:4000->10.1.0.21:42096: read: connection reset by peer"]
[2020/04/16 06:05:31.699 +00:00] [WARN] [conn.go:658] ["read packet failed, close this connection"] [conn=32] [error="read tcp 10.1.0.5:4000->10.1.0.21:42086: read: connection reset by peer"]
[2020/04/16 06:05:31.699 +00:00] [WARN] [conn.go:658] ["read packet failed, close this connection"] [[2020/04/16 06:05:31.731 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=35]
[2020/04/16 06:05:31.743 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=36]
[2020/04/16 06:05:31.765 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=46]
[2020/04/16 06:05:31.819 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.21:20172] [cause="context canceled"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.26:20171] [cause="context canceled"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [session.go:435] ["can not retry txn"] [conn=26] [label=general] [error="[kv:9007]Write conflict, txnStartTS=416027018926751858, conflictStartTS=416027018939858948, conflictCommitTS=416027018966073345, key={tableID=47940, handle=2396775430} primary={tableID=41880, indexID=1, indexValues={5824401, }} [try again later]"] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=false] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2020/04/16 06:05:31.819 +00:00] [WARN] [session.go:461] ["commit failed"] [conn=26] ["finished txn"="Txn{state=invalid}"] [error="[kv:9007]Write conflict, txnStartTS=416027018926751858, conflictStartTS=416027018939858948, conflictCommitTS=416027018966073345, key={tableID=47940, handle=2396775430} primary={tableID=41880, indexID=1, indexValues={5824401, }} [try again later]"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [session.go:987] ["run statement error"] [conn=26] [schemaVersion=2238] [error="previous statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928318,'2020-04-10',99,4576340,101,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');: [kv:9007]Write conflict, txnStartTS=416027018926751858, conflictStartTS=416027018939858948, conflictCommitTS=416027018966073345, key={tableID=47940, handle=2396775430} primary={tableID=41880, indexID=1, indexValues={5824401, }} [try again later]"] [errorVerbose="[kv:9007]Write conflict, txnStartTS=416027018926751858, conflictStartTS=416027018939858948, conflictCommitTS=416027018966073345, key={tableID=47940, handle=2396775430} primary={tableID=41880, indexID=1, indexValues={5824401, }} [try again later]\
previous statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928318,'2020-04-10',99,4576340,101,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');"] [session="{\
\"currDBName\": \"\",\
\"id\": 26,\
\"status\": 2,\
\"strictMode\": false,\
\"user\": {\
\"Username\": \"root\",\
\"Hostname\": \"10.1.0.21\",\
\"CurrentUser\": false,\
\"AuthUsername\": \"root\",\
\"AuthHostname\": \"%\"\
}\
}"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [client_batch.go:577] ["send request is cancelled"] [to=10.1.0.26:20172] [cause="context canceled"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [client_batch.go:590] ["wait response is cancelled"] [to=10.1.0.21:20172] [cause="context canceled"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [conn.go:682] ["dispatch error"] [conn=26] [connInfo="id:26, addr:10.1.0.21:42074 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql=COMMIT] [err="[kv:9007]Write conflict, txnStartTS=416027018926751858, conflictStartTS=416027018939858948, conflictCommitTS=416027018966073345, key={tableID=47940, handle=2396775430} primary={tableID=41880, indexID=1, indexValues={5824401, }} [try again later]\
previous statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928318,'2020-04-10',99,4576340,101,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');"]
[2020/04/16 06:05:31.819 +00:00] [WARN] [session.go:435] ["can not retry txn"] [conn=53] [label=general] [error="[kv:9007]Write conflict, txnStartTS=416027018926751851, conflictStartTS=416027018926751853, conflictCommitTS=416027018926751853, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 3078427, 100103, }} primary={tableID=41880, indexID=1, indexValues={5826503, }} [try again later]"] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=false] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2020/04/16 06:05:31.819 +00:00] [WARN] [session.go:461] ["commit failed"] [conn=53] ["finished txn"="Txn{state=invalid}"] [error="[kv:9007]Write conflict, txnStartTS=416027018926751851, conflictStartTS=416027018926751853, conflictCommitTS=416027018926751853, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 3078427, 100103, }} primary={tableID=41880, indexID=1, indexValues={5826503, }} [try again later]"]
[2020/04/16 06:05:31.819 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=26]
[2020/04/16 06:05:31.819 +00:00] [WARN] [session.go:987] ["run statement error"] [conn=53] [schemaVersion=2238] [error="previous statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928277,'2020-04-10',99,4361397,96,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');: [kv:9007]Write conflict, txnStartTS=416027018926751851, conflictStartTS=416027018926751853, conflictCommitTS=416027018926751853, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 3078427, 100103, }} primary={tableID=41880, indexID=1, indexValues={5826503, }} [try again later]"] [errorVerbose="[kv:9007]Write conflict, txnStartTS=416027018926751851, conflictStartTS=416027018926751853, conflictCommitTS=416027018926751853, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 3078427, 100103, }} primary={tableID=41880, indexID=1, indexValues={5826503, }} [try again later]\
previous statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928277,'2020-04-10',99,4361397,96,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');"] [session="{\
\"currDBName\": \"\",\
\"id\": 53,\
\"status\": 2,\
\"strictMode\": false,\
\"user\": {\
\"Username\": \"root\",\
\"Hostname\": \"10.1.0.21\",\
\"CurrentUser\": false,\
\"AuthUsername\": \"root\",\
\"AuthHostname\": \"%\"\
}\
}"]
[2020/04/16 06:05:31.820 +00:00] [WARN] [conn.go:682] ["dispatch error"] [conn=53] [connInfo="id:53, addr:10.1.0.21:42127 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql=COMMIT] [err="[kv:9007]Write conflict, txnStartTS=416027018926751851, conflictStartTS=416027018926751853, conflictCommitTS=416027018926751853, key={tableID=47940, indexID=2, indexValues={1848186687314722816, 99, 3078427, 100103, }} primary={tableID=41880, indexID=1, indexValues={5826503, }} [try again later]\
previous statement: REPLACE INTO `infinity_boe`.`if_add_item_source_warn` (`id`,`date`,`hour`,`uid`,`kid`,`goodsid`,`num`,`data`) VALUES (2379928277,'2020-04-10',99,4361397,96,100103,1,'{\\\"USE_BACKPACK_ITEM\\\": 1}');"]
[2020/04/16 06:05:31.820 +00:00] [INFO] [server.go:370] ["connection closed"] [conn=53]
[2020/04/16 06:05:31.827 +00:00] [INFO] [2pc.go:1119] ["2PC clean up done"] [conn=26] [txnStartTS=416027018926751858]
[2020/04/16 06:05:31.828 +00:00] [INFO] [2pc.go:1119] ["2PC clean up done"] [conn=53] [txnStartTS=416027018926751851]
2020/04/16 06:05:59.145 terror.go:357: [error] EOF
github.com/pingcap/errors.AddStack
/home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/errors.go:174
github.com/pingcap/errors.Trace
/home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/juju_adaptor.go:15
github.com/pingcap/tidb/server.(*packetIO).readOnePacket
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/packetio.go:80
github.com/pingcap/tidb/server.(*packetIO).readPacket
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/packetio.go:105
github.com/pingcap/tidb/server.(*clientConn).readPacket
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/conn.go:265
github.com/pingcap/tidb/server.(*clientConn).readOptionalSSLRequestAndHandshakeResponse
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/conn.go:471
github.com/pingcap/tidb/server.(*clientConn).handshake
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/conn.go:172
github.com/pingcap/tidb/server.(*Server).onConn
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/server.go:345
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
2020/04/16 06:06:29.144 terror.go:357: [error] EOF
github.com/pingcap/errors.AddStack
/home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/errors.go:174
github.com/pingcap/errors.Trace
tidb_stderr.log :
{"level":"warn","ts":"2020-04-16T05:42:55.472Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-e045b21a-fae3-40e5-ac3d-b
74a18671b5f/10.1.13.12:2379","attempt":99,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
{"level":"warn","ts":"2020-04-16T05:56:43.802Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-a9046073-c502-48b9-bd27-4
0c3f5cfa89f/10.1.13.12:2379","attempt":0,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
{"level":"warn","ts":"2020-04-16T05:56:43.802Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-a9046073-c502-48b9-bd27-4
0c3f5cfa89f/10.1.13.12:2379","attempt":1,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
tikv.log:
[2020/04/16 06:16:45.118 +00:00] [INFO] [pd.rs:566] ["try to transfer leader"] [to_peer="id: 540425 store_id: 9024"] [from_peer="id: 726427 store_id: 8159"] [region_id=540424]
[2020/04/16 06:16:45.130 +00:00] [INFO] [peer.rs:1825] ["transfer leader"] [peer="id: 540425 store_id: 9024"] [peer_id=726427] [region_id=540424]
[2020/04/16 06:16:45.130 +00:00] [INFO] [raft.rs:1335] ["[region 540424] 726427 [term 230] starts to transfer leadership to 540425"]
[2020/04/16 06:16:45.130 +00:00] [INFO] [raft.rs:1345] ["[region 540424] 726427 sends MsgTimeoutNow to 540425 immediately as 540425 already has up-to-date log"]
[2020/04/16 06:16:45.300 +00:00] [INFO] [raft.rs:964] ["[region 557899] 726678 [term: 226] received a MsgRequestVote message with higher term from 557900 [term: 227]"]
[2020/04/16 06:16:45.300 +00:00] [INFO] [raft.rs:763] ["[region 557899] 726678 became follower at term 227"]
[2020/04/16 06:16:45.300 +00:00] [INFO] [raft.rs:1148] ["[region 557899] 726678 [logterm: 226, index: 230, vote: 0] cast MsgRequestVote for 557900 [logterm: 226, index: 230] at term 227"]
