tidb 当连接超过几十，tidb就会卡死无法连接

🌏 互助交流区
1

tidb的版本是5.2 ；
sysbench 测试过程中，thread 设置为16，通过navicat就无法连接tidb服务。

image
image1005×1505 166 KB

image

2

mysql shell手工连下看看，如果可以那就是ui工具的问题。

3

TiDB日志有报错记录吗

4

不太像是，我通过程序去联，也会因为超时无法连接。

5

tidb_stderr.log 里没有，tidb.log有报错，
[2022/08/25 10:36:35.048 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error="[server:1045]Access denied for user ‘root’@‘16.16.16.65’ (using password: NO)"] [stack="github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/sr

c/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516"]
这是因为16.16.16.65之前是drainer，我配置了binlog，感觉并发太低了，我先把pump和drainer暂停了，所以会有这个报错。

6

[2022/08/25 00:22:37.237 +08:00] [ERROR] [analyze.go:177] [“save table stats to storage failed”] [error=“previous statement: update mysql.stats_histograms set last_analyze_pos = _binary’\u0003\ufffd\0\0\0\0}\u0018\ufffd’ where table_id = 596821 and is_index = 1 and hist_id = 1: tikv aborts txn: Error(Txn(Error(Mvcc(Error(PessimisticLockNotFound { start_ts: TimeStamp(435514798432321537), key: [116, 128, 0, 0, 0, 0, 0, 0, 25, 95, 105, 128, 0, 0, 0, 0, 0, 0, 1, 3, 128, 0, 0, 0, 0, 9, 27, 85, 3, 128, 0, 0, 0, 0, 0, 0, 0, 3, 128, 0, 0, 0, 0, 0, 0, 2, 3, 128, 0, 0, 0, 0, 0, 0, 247] })))))”]
[2022/08/25 00:22:37.237 +08:00] [ERROR] [update.go:1191] ["[stats] auto analyze failed"] [sql=“analyze table sbtest.sbtest2 index k_2“] [cost_time=17m29.807513151s] [error=“previous statement: update mysql.stats_histograms set last_analyze_pos = _binary’\u0003\ufffd\0\0\0\0}\u0018\ufffd’ where table_id = 596821 and is_index = 1 and hist_id = 1: tikv aborts txn: Error(Txn(Error(Mvcc(Error(PessimisticLockNotFound { start_ts: TimeStamp(435514798432321537), key: [116, 128, 0, 0, 0, 0, 0, 0, 25, 95, 105, 128, 0, 0, 0, 0, 0, 0, 1, 3, 128, 0, 0, 0, 0, 9, 27, 85, 3, 128, 0, 0, 0, 0, 0, 0, 0, 3, 128, 0, 0, 0, 0, 0, 0, 2, 3, 128, 0, 0, 0, 0, 0, 0, 247] })))))”]
[2022/08/25 00:39:26.633 +08:00] [ERROR] [binloginfo.go:251] [“write binlog failed”] [binlog_type=Prewrite] [binlog_start_ts=435515337361063937] [binlog_commit_ts=0] [error=“write binlog failed, the last error no avaliable pump to write binlog”]
[2022/08/25 00:39:26.633 +08:00] [ERROR] [binloginfo.go:257] [“write binlog fail but error ignored”]
[2022/08/25 00:52:08.627 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 16.16.16.47:4000->16.16.16.38:37028: write: broken pipe”] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*packetIO).writePacket\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/packetio.go:174\ngithub.com/pingcap/tidb/server.(*clientConn).writePacket\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:406\ngithub.com/pingcap/tidb/server.(*clientConn).writeError\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1501\ngithub.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1124\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:08.627 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“connection was bad”] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1125\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:11.629 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 16.16.16.47:4000->16.16.16.38:37054: write: broken pipe”] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*packetIO).writePacket\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/packetio.go:174\ngithub.com/pingcap/tidb/server.(*clientConn).writePacket\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:406\ngithub.com/pingcap/tidb/server.(*clientConn).writeError\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1501\ngithub.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1124\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:11.629 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“connection was bad”] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1125\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:16.213 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 16.16.16.47:4000->16.16.16.38:37012: write: broken pipe”] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*packetIO).writePacket\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/packetio.go:174\ngithub.com/pingcap/tidb/server.(*clientConn).writePacket\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:406\ngithub.com/pingcap/tidb/server.(*clientConn).writeError\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1501\ngithub.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1124\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:16.213 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“connection was bad”] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1125\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:25.208 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 16.16.16.47:4000->16.16.16.38:37068: write: broken pipe”] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*packetIO).writePacket\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/packetio.go:174\ngithub.com/pingcap/tidb/server.(*clientConn).writePacket\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:406\ngithub.com/pingcap/tidb/server.(*clientConn).writeError\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1501\ngithub.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1124\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:25.208 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“connection was bad”] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1125\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:28.333 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 16.16.16.47:4000->16.16.16.38:37042: write: broken pipe”] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*packetIO).writePacket\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/packetio.go:174\ngithub.com/pingcap/tidb/server.(*clientConn).writePacket\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:406\ngithub.com/pingcap/tidb/server.(*clientConn).writeError\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1501\ngithub.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1124\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:28.334 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“connection was bad”] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1125\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:31.160 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“write tcp 16.16.16.47:4000->16.16.16.38:37006: write: broken pipe”] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*packetIO).writePacket\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/packetio.go:174\ngithub.com/pingcap/tidb/server.(*clientConn).writePacket\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:406\ngithub.com/pingcap/tidb/server.(*clientConn).writeError\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1501\ngithub.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1124\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 00:52:31.160 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“connection was bad”] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1125\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:548”]
[2022/08/25 01:18:10.403 +08:00] [ERROR] [domain.go:1041] [“LoadSysVarCacheLoop loop watch channel closed”]
[2022/08/25 01:18:10.413 +08:00] [ERROR] [domain.go:978] [“load privilege loop watch channel closed”]
[2022/08/25 01:18:11.527 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=EOF] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/08/25 01:18:12.022 +08:00] [ERROR] [client.go:786] [”[pd] getTS error”] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.027 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.035 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.039 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.040 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.043 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.047 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.050 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.405 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.409 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.433 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.436 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.443 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.446 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.455 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.458 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.488 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.491 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.524 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.527 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.539 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.543 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.544 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.548 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.717 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.721 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.771 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.775 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.784 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.784 +08:00] [ERROR] [pd.go:236] [“updateTS error”] [txnScope=global] [error=“rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster”]
[2022/08/25 01:18:12.787 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.792 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.795 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.816 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.820 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.913 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.916 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.926 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.930 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:12.958 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:18:12.960 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:18:13.169 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:20:14.419 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:20:14.423 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:20:14.423 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:20:14.426 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:20:14.524 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:20:14.526 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:20:14.527 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:20:14.529 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:20:14.784 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:20:14.784 +08:00] [ERROR] [pd.go:236] [“updateTS error”] [txnScope=global] [error=“rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster”]
[2022/08/25 01:20:14.786 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:20:14.801 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:20:14.803 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:20:14.819 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:20:14.821 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:20:14.858 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:20:14.861 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:20:15.395 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:20:16.280 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=EOF] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/08/25 01:21:17.727 +08:00] [ERROR] [http_status.go:472] [“start status/rpc server error”] [error=“accept tcp [::]:10080: use of closed network connection”]
[2022/08/25 01:21:17.727 +08:00] [ERROR] [http_status.go:467] [“http server error”] [error=“http: Server closed”]
[2022/08/25 01:21:17.727 +08:00] [ERROR] [http_status.go:462] [“grpc server error”] [error=“mux: listener closed”]
[2022/08/25 01:21:19.042 +08:00] [ERROR] [client.go:752] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]
[2022/08/25 01:23:37.236 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=EOF] [stack=“github.com/pingcap/tidb/parser/terror.Log\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ github.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:516”]
[2022/08/25 01:23:38.352 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:23:38.359 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:23:38.359 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:23:38.362 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:23:38.678 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:23:38.682 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:23:38.732 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:23:38.735 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:23:39.288 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:23:39.289 +08:00] [ERROR] [pd.go:236] [“updateTS error”] [txnScope=global] [error=“rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster”]
[2022/08/25 01:23:39.292 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://16.16.16.56:2379 http://16.16.16.57:2379 http://16.16.16.58:2379] error"]
[2022/08/25 01:23:39.338 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"]
[2022/08/25 01:25:25.453 +08:00] [ERROR] [http_status.go:472] [“start status/rpc server error”] [error=“accept tcp [::]:10080: use of closed network connection”]
[2022/08/25 01:25:25.453 +08:00] [ERROR] [http_status.go:462] [“grpc server error”] [error=“mux: listener closed”]
[2022/08/25 01:25:25.453 +08:00] [ERROR] [http_status.go:467] [“http server error”] [error=“http: Server closed”]
[2022/08/25 01:25:26.755 +08:00] [ERROR] [client.go:752] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]

7

image
image1365×596 21.2 KB