pd不知明原因挂掉

🪐 TiDB 技术问题
1

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】6.2
【遇到的问题】pd不知明原因挂掉
【复现路径】无操作
【问题现象及影响】
服务器部分连接中断

【附件】
tidb_stderr.log
{“level”:“warn”,“ts”:“2022-09-28T16:23:02.963+0800”,“logger”:“etcd-client”,“caller”:“v3@v3.5.2/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“etcd-endpoints://0xc000d0e000/172.21.0.14:2379”,“attempt”:0,“error”:“rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout”}

{“level”:“warn”,“ts”:“2022-09-28T16:23:04.963+0800”,“logger”:“etcd-client”,“caller”:“v3@v3.5.2/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“etcd-endpoints://0xc000d0e000/172.21.0.14:2379”,“attempt”:1,“error”:“rpc error: code = DeadlineExceeded desc = context deadline exceeded”}

{“level”:“warn”,“ts”:“2022-09-28T16:23:10.964+0800”,“logger”:“etcd-client”,“caller”:“v3@v3.5.2/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“etcd-endpoints://0xc000d0e000/172.21.0.14:2379”,“attempt”:0,“error”:“rpc error: code = DeadlineExceeded desc = context deadline exceeded”}

{“level”:“warn”,“ts”:“2022-09-28T16:23:16.964+0800”,“logger”:“etcd-client”,“caller”:“v3@v3.5.2/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“etcd-endpoints://0xc000d0e000/172.21.0.14:2379”,“attempt”:0,“error”:“rpc error: code = DeadlineExceeded desc = context deadline exceeded”}
pd_stderr.log
[2022/09/28 16:21:58.000 +08:00] [WARN] [retry_interceptor.go:62] [“retrying of unary invoker failed”] [target=endpoint://client-a0b85822-3db7-41ab-a2f9-123e14c288e8/172.21.0.14:2379] [attempt=0] [error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded”]

[2022/09/28 16:23:06.755 +08:00] [WARN] [retry_interceptor.go:62] [“retrying of unary invoker failed”] [target=endpoint://client-a0b85822-3db7-41ab-a2f9-123e14c288e8/172.21.0.14:2379] [attempt=0] [error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded”]

[2022/09/28 16:23:06.759 +08:00] [WARN] [retry_interceptor.go:62] [“retrying of unary invoker failed”] [target=endpoint://client-a0b85822-3db7-41ab-a2f9-123e14c288e8/172.21.0.14:2379] [attempt=0] [error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded”]

[2022/09/28 16:21:50.048 +08:00] [WARN] [retry_interceptor.go:62] [“retrying of unary invoker failed”] [target=endpoint://client-a0b85822-3db7-41ab-a2f9-123e14c288e8/172.21.0.14:2379] [attempt=0] [error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded”]

[2022/09/28 16:23:06.992 +08:00] [WARN] [retry_interceptor.go:62] [“retrying of unary invoker failed”] [target=endpoint://client-a0b85822-3db7-41ab-a2f9-123e14c288e8/172.21.0.14:2379] [attempt=0] [error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded”]

[2022/09/28 16:23:10.065 +08:00] [WARN] [retry_interceptor.go:62] [“retrying of unary invoker failed”] [target=endpoint://client-a0b85822-3db7-41ab-a2f9-123e14c288e8/172.21.0.14:2379] [attempt=0] [error=“rpc error: code = Unavailable desc = etcdserver: leader changed”]

[2022/09/28 16:23:17.028 +08:00] [WARN] [retry_interceptor.go:62] [“retrying of unary invoker failed”] [target=endpoint://client-a0b85822-3db7-41ab-a2f9-123e14c288e8/172.21.0.14:2379] [attempt=1] [error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded”]

[2022/09/28 16:23:20.182 +08:00] [WARN] [retry_interceptor.go:62] [“retrying of unary invoker failed”] [target=endpoint://client-a0b85822-3db7-41ab-a2f9-123e14c288e8/172.21.0.14:2379] [attempt=0] [error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded”]

不知道啥原因导致的

2

pd的日志发一下，看起来是网络问题，先检查一下网络

3

[2022/09/28 16:23:10.081 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=5e5482d8c84e1d91] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=0a9282d8c82d36af] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=44.354µs] [request=“header:<ID:9985187191419172187 > lease_revoke:id:5e5482d8c8512356”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=5e5482d8c84a26c5] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=5e5482d8c8512356] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=39.925µs] [request=“header:<ID:9985187191419172188 > lease_revoke:id:5e5482d8c84a26c2”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=31.62µs] [request=“header:<ID:9985187191419172189 > lease_revoke:id:1dd582d8c84f797b”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=16.881µs] [request=“header:<ID:9985187191419172190 > lease_revoke:id:1dd582d8c84f7971”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=5e5482d8c84a26c2] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=1dd582d8c84f797b] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=2.885µs] [request=“header:<ID:9985187191419172191 > put:<key:"/topology/tidb/172.21.0.4:4000/ttl" value_size:19 lease:6797201605192868870 >”] [response=] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=27.812µs] [request=“header:<ID:11373140301580791609 > lease_revoke:id:1dd582d8c835e4bf”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=1dd582d8c84f7971] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=24.056µs] [request=“header:<ID:9985187191419172192 > lease_revoke:id:0a9282d8c8316d5b”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.081 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=0a9282d8c8316d5b] [error=“lease not found”]

[2022/09/28 16:23:10.082 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=26.972µs] [request=“header:<ID:9985187191419172193 > lease_revoke:id:0a9282d8c8316d5d”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.082 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=0a9282d8c8316d5d] [error=“lease not found”]

[2022/09/28 16:23:10.082 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=23.364µs] [request=“header:<ID:9985187191419172194 > lease_revoke:id:5e5482d8c84e1d89”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.082 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=5e5482d8c84e1d89] [error=“lease not found”]

[2022/09/28 16:23:10.082 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=15.159µs] [request=“header:<ID:9985187191419172198 > lease_revoke:id:5e5482d8c83c6406”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.082 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=29.035µs] [request=“header:<ID:9985187191419172199 > lease_revoke:id:5e5482d8c84a26be”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.083 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=5e5482d8c83c6406] [error=“lease not found”]

[2022/09/28 16:23:10.083 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=38.502µs] [request=“header:<ID:9985187191419172200 > lease_revoke:id:5e5482d8c853f552”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.083 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=5e5482d8c84a26be] [error=“lease not found”]

[2022/09/28 16:23:10.083 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=21.59µs] [request=“header:<ID:9985187191419172201 > lease_revoke:id:5e5482d8c853f560”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.083 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=5e5482d8c853f552] [error=“lease not found”]

[2022/09/28 16:23:10.083 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=36.589µs] [request=“header:<ID:9985187191419172202 > lease_revoke:id:5e5482d8c853f555”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.083 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=5e5482d8c853f560] [error=“lease not found”]

[2022/09/28 16:23:10.083 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=23.495µs] [request=“header:<ID:9985187191419172203 > lease_revoke:id:1dd582d8c8524e92”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:10.083 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=1dd582d8c8524e92] [error=“lease not found”]

[2022/09/28 16:23:10.083 +08:00] [WARN] [server.go:1102] [“failed to revoke lease”] [lease-id=5e5482d8c853f555] [error=“lease not found”]

[2022/09/28 16:23:10.115 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=5.34µs] [request=“header:<ID:9985187191419172206 > put:<key:"/topology/tidb/172.21.0.4:4000/ttl" value_size:19 lease:6797201605192868870 >”] [response=] [error=“lease not found”]

[2022/09/28 16:23:10.148 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=3.547µs] [request=“header:<ID:11373140301580791611 > put:<key:"/topology/tidb/172.21.0.4:4000/ttl" value_size:19 lease:6797201605192868870 >”] [response=] [error=“lease not found”]

[2022/09/28 16:23:10.180 +08:00] [INFO] [cluster.go:389] [“metrics are reset”]

[2022/09/28 16:23:10.180 +08:00] [INFO] [cluster.go:391] [“metrics collection job has been stopped”]

[2022/09/28 16:23:10.181 +08:00] [INFO] [cluster.go:470] [“raftcluster is stopped”]

[2022/09/28 16:23:10.181 +08:00] [INFO] [tso.go:405] [“reset the timestamp in memory”]

[2022/09/28 16:23:11.544 +08:00] [WARN] [probing_status.go:70] [“prober detected unhealthy status”] [round-tripper-name=ROUND_TRIPPER_RAFT_MESSAGE] [remote-peer-id=ae7e6304d9ba5e54] [rtt=686.041µs] [error=“dial tcp 172.21.0.8:2380: i/o timeout”]

[2022/09/28 16:23:11.561 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=60.674µs] [request=“header:<ID:11373140301580791614 > txn:<compare:<target:CREATE key:"/tidb/telemetry/owner/5e5482d8c83919f7" create_revision:0 > success:<request_put:<key:"/tidb/telemetry/owner/5e5482d8c83919f7" value_size:36 lease:6797201605192653303 >> failure:<>>”] [response=] [error=“lease not found”]

[2022/09/28 16:23:11.561 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=40.626µs] [request=“header:<ID:9985187191419172209 > txn:<compare:<target:CREATE key:"/tidb/bindinfo/owner/1dd582d8c835e4bd" create_revision:0 > success:<request_put:<key:"/tidb/bindinfo/owner/1dd582d8c835e4bd" value_size:36 lease:2149768264722801853 >> failure:<request_range:<key:"/tidb/bindinfo/owner/1dd582d8c835e4bd" > >>”] [response=] [error=“lease not found”]

[2022/09/28 16:23:11.564 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=50.647µs] [request=“header:<ID:11373140301580791615 > lease_revoke:id:5e5482d8c83919f7”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:11.564 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=44.553µs] [request=“header:<ID:11373140301580791616 > lease_revoke:id:1dd582d8c835e4bd”] [response=size:31] [error=“lease not found”]

[2022/09/28 16:23:11.591 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=4.137µs] [request=“header:<ID:9985187191419172215 > put:<key:"/topology/tidb/172.21.0.4:4000/ttl" value_size:19 lease:6797201605192868870 >”] [response=] [error=“lease not found”]

[2022/09/28 16:23:13.622 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=4.459µs] [request=“header:<ID:11373140301580791626 > put:<key:"/topology/tidb/172.21.0.5:4000/ttl" value_size:19 lease:6797201605193770686 >”] [response=] [error=“lease not found”]

[2022/09/28 16:23:14.444 +08:00] [INFO] [stream.go:250] [“set message encoder”] [from=b0dbc9ae56478a92] [to=b0dbc9ae56478a92] [stream-type=“stream Message”]

[2022/09/28 16:23:14.444 +08:00] [INFO] [peer_status.go:51] [“peer became active”] [peer-id=ae7e6304d9ba5e54]

[2022/09/28 16:23:14.444 +08:00] [WARN] [stream.go:277] [“established TCP streaming connection with remote peer”] [stream-writer-type=“stream Message”] [local-member-id=b0dbc9ae56478a92] [remote-peer-id=ae7e6304d9ba5e54]

[2022/09/28 16:23:14.500 +08:00] [INFO] [stream.go:425] [“established TCP streaming connection with remote peer”] [stream-reader-type=“stream MsgApp v2”] [local-member-id=b0dbc9ae56478a92] [remote-peer-id=ae7e6304d9ba5e54]

[2022/09/28 16:23:14.532 +08:00] [INFO] [stream.go:250] [“set message encoder”] [from=b0dbc9ae56478a92] [to=b0dbc9ae56478a92] [stream-type=“stream MsgApp v2”]

[2022/09/28 16:23:14.532 +08:00] [WARN] [stream.go:277] [“established TCP streaming connection with remote peer”] [stream-writer-type=“stream MsgApp v2”] [local-member-id=b0dbc9ae56478a92] [remote-peer-id=ae7e6304d9ba5e54]

[2022/09/28 16:23:14.539 +08:00] [INFO] [raft.go:978] [“b0dbc9ae56478a92 [logterm: 7, index: 47782817, vote: 1089c682c7149dd5] rejected MsgPreVote from ae7e6304d9ba5e54 [logterm: 6, index: 47782708] at term 7”]

[2022/09/28 16:23:14.539 +08:00] [INFO] [raft.go:844] [“b0dbc9ae56478a92 [logterm: 7, index: 47782817, vote: 1089c682c7149dd5] ignored MsgPreVote from ae7e6304d9ba5e54 [logterm: 7, index: 47782709] at term 7: lease is not expired (remaining ticks: 5)”]

[2022/09/28 16:23:14.548 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=7.224µs] [request=“header:<ID:6797201605195765791 > put:<key:"/topology/tidb/172.21.0.5:4000/ttl" value_size:19 lease:6797201605193770686 >”] [response=] [error=“lease not found”]

[2022/09/28 16:23:14.649 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=6.142µs] [request=“header:<ID:6797201605195765801 > put:<key:"/topology/tidb/172.21.0.5:4000/ttl" value_size:19 lease:6797201605193770686 >”] [response=] [error=“lease not found”]

[2022/09/28 16:23:14.680 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=3.036µs] [request=“header:<ID:9985187191419172220 > put:<key:"/topology/tidb/172.21.0.5:4000/ttl" value_size:19 lease:6797201605193770686 >”] [response=] [error=“lease not found”]

[2022/09/28 16:23:14.712 +08:00] [WARN] [util.go:121] [“failed to apply request”] [took=3.757µs] [request=“header:<ID:11373140301580791628 > put:<key:"/topology/tidb/172.21.0.5:4000/ttl" value_size:19 lease:6797201605193770686 >”] [response=] [error=“lease not found”]

[2022/09/28 16:23:14.857 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=2.000082567s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/topology/tidb/" range_end:"/topology/tidb0" "] [response=] [error=“context canceled”]

[2022/09/28 16:23:14.857 +08:00] [INFO] [trace.go:145] [“trace[154602650] range”] [detail=“{range_begin:/topology/tidb/; range_end:/topology/tidb0; }”] [duration=2.000151438s] [start=2022/09/28 16:23:12.857 +08:00] [end=2022/09/28 16:23:14.857 +08:00] [steps=“["trace[154602650] ‘agreement among raft nodes before linearized reading’ (duration: 2.000098989s)"]”]

[2022/09/28 16:23:14.929 +08:00] [WARN] [grpclog.go:60] [“transport: http2Server.HandleStreams failed to read frame: read tcp 172.21.0.14:2379->172.21.0.8:50494: read: connection reset by peer”]
[2022/09/28 16:23:16.041 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=5.000206274s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/tidb/store/gcworker/saved_safe_point" "] [response=] [error=“context canceled”]

[2022/09/28 16:23:16.042 +08:00] [INFO] [trace.go:145] [“trace[743756693] range”] [detail=“{range_begin:/tidb/store/gcworker/saved_safe_point; range_end:; }”] [duration=5.0002992s] [start=2022/09/28 16:23:11.041 +08:00] [end=2022/09/28 16:23:16.042 +08:00] [steps=“["trace[743756693] ‘agreement among raft nodes before linearized reading’ (duration: 5.000216293s)"]”]

[2022/09/28 16:23:16.237 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=5.000403615s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/tidb/store/gcworker/saved_safe_point" "] [response=] [error=“context canceled”]

[2022/09/28 16:23:16.237 +08:00] [INFO] [trace.go:145] [“trace[110478184] range”] [detail=“{range_begin:/tidb/store/gcworker/saved_safe_point; range_end:; }”] [duration=5.000469028s] [start=2022/09/28 16:23:11.236 +08:00] [end=2022/09/28 16:23:16.237 +08:00] [steps=“["trace[110478184] ‘agreement among raft nodes before linearized reading’ (duration: 5.000412823s)"]”]

[2022/09/28 16:23:16.964 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=4.999986121s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/tidb/store/gcworker/saved_safe_point" "] [response=] [error=“context canceled”]

[2022/09/28 16:23:16.964 +08:00] [INFO] [trace.go:145] [“trace[187868401] range”] [detail=“{range_begin:/tidb/store/gcworker/saved_safe_point; range_end:; }”] [duration=5.000066182s] [start=2022/09/28 16:23:11.964 +08:00] [end=2022/09/28 16:23:16.964 +08:00] [steps=“["trace[187868401] ‘agreement among raft nodes before linearized reading’ (duration: 4.999998094s)"]”]

[2022/09/28 16:23:17.028 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=6.937127677s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/pd/7068527063746750139/config" "] [response=] [error=“context deadline exceeded”]

[2022/09/28 16:23:17.028 +08:00] [INFO] [trace.go:145] [“trace[809723980] range”] [detail=“{range_begin:/pd/7068527063746750139/config; range_end:; }”] [duration=6.937215091s] [start=2022/09/28 16:23:10.091 +08:00] [end=2022/09/28 16:23:17.028 +08:00] [steps=“["trace[809723980] ‘agreement among raft nodes before linearized reading’ (duration: 6.937139508s)"]”]

[2022/09/28 16:23:17.028 +08:00] [WARN] [etcdutil.go:121] [“kv gets too slow”] [request-key=/pd/7068527063746750139/config] [cost=10.000133505s] [error=“context deadline exceeded”]

[2022/09/28 16:23:17.028 +08:00] [ERROR] [etcdutil.go:126] [“load from etcd meet error”] [key=/pd/7068527063746750139/config] [error=“[PD:etcd:ErrEtcdKVGet]context deadline exceeded: context deadline exceeded”]

[2022/09/28 16:23:17.028 +08:00] [WARN] [manager.go:101] [“failed to reload persist options”]

[2022/09/28 16:23:20.182 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=10.000344281s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/pd/7068527063746750139/leader" "] [response=] [error=“context deadline exceeded”]

[2022/09/28 16:23:20.182 +08:00] [WARN] [etcdutil.go:121] [“kv gets too slow”] [request-key=/pd/7068527063746750139/leader] [cost=10.000858248s] [error=“context deadline exceeded”]

[2022/09/28 16:23:20.182 +08:00] [INFO] [trace.go:145] [“trace[261177145] range”] [detail=“{range_begin:/pd/7068527063746750139/leader; range_end:; }”] [duration=10.000431594s] [start=2022/09/28 16:23:10.182 +08:00] [end=2022/09/28 16:23:20.182 +08:00] [steps=“["trace[261177145] ‘agreement among raft nodes before linearized reading’ (duration: 10.000355161s)"]”]

[2022/09/28 16:23:20.182 +08:00] [ERROR] [etcdutil.go:126] [“load from etcd meet error”] [key=/pd/7068527063746750139/leader] [error=“[PD:etcd:ErrEtcdKVGet]context deadline exceeded: context deadline exceeded”]

[2022/09/28 16:23:20.182 +08:00] [ERROR] [member.go:167] [“getting pd leader meets error”] [error=“[PD:etcd:ErrEtcdKVGet]context deadline exceeded: context deadline exceeded”]

[2022/09/28 16:23:21.065 +08:00] [WARN] [v3_server.go:746] [“timed out waiting for read index response (local node might have slow network)”] [timeout=11s]

[2022/09/28 16:23:21.065 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=11.299996594s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/topology/tidb/" range_end:"/topology/tidb0" "] [response=] [error=“etcdserver: request timed out”]

[2022/09/28 16:23:21.065 +08:00] [INFO] [trace.go:145] [“trace[1792108143] range”] [detail=“{range_begin:/topology/tidb/; range_end:/topology/tidb0; }”] [duration=11.300055395s] [start=2022/09/28 16:23:09.765 +08:00] [end=2022/09/28 16:23:21.065 +08:00] [steps=“["trace[1792108143] ‘agreement among raft nodes before linearized reading’ (duration: 11.300009868s)"]”]

[2022/09/28 16:23:21.067 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=4.038381697s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/pd/7068527063746750139/config" "] [response=“range_response_count:1 size:3721”]

[2022/09/28 16:23:21.067 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=6.449138685s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/tidb/telemetry/owner/" range_end:"/tidb/telemetry/owner0" limit:1 sort_order:DESCEND sort_target:CREATE max_create_revision:14808025 "] [response=“range_response_count:0 size:9”]

[2022/09/28 16:23:21.067 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=9.49992742s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/tidb/telemetry/owner/" range_end:"/tidb/telemetry/owner0" limit:1 sort_order:DESCEND sort_target:CREATE max_create_revision:16326087 "] [response=“range_response_count:0 size:9”]

4

这是当时的部分日志

5

3个pd都是相同的配置都是同一个云服务商，其他pd服务没有这个问题

6

说不定网络稳定就好了…:joy:

7

应该是选主的问题

8

天天出现呀，有啥解决办法吗

9

你买的云服务，肯定要咨询云服务商了，为啥网络那么不稳定…

10

这玩意有啥可量化的指标吗，找运营商也要有证据呀

11

运营商不给你提供监控的信息么，

那内部只能看网络是否有断流的情况，可以通过 prometheus 试试…