9002 tikv server timeout 问题

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:

【TiDB 版本】
V5.01

【问题描述】
TiDB集群在k8s上搭建完毕,初始化正常,可以正常连接;
运行一段时间后,发现无法通过nvicat客户端连接Tidb,报错:9002 tikv server timeout

查看TiKV日志,报错日志如下:

[2021/05/21 09:00:32.889 +00:00] [INFO] [util.rs:536] [“connecting to PD endpoint”] [endpoints=http://tidb-pd-2.tidb-pd-peer.tidb-admin.svc:2379]
[2021/05/21 09:00:33.317 +00:00] [ERROR] [util.rs:416] [“request failed, retry”] [err_code=KV:PD:gRPC] [err=“Grpc(RpcFailure(RpcStatus { status: 14-UNAVAILABLE, details: Some(“failed to connect to all addresses”) }))”]
[2021/05/21 09:00:33.317 +00:00] [INFO] [util.rs:536] [“connecting to PD endpoint”] [endpoints=http://tidb-pd-0.tidb-pd-peer.tidb-admin.svc:2379]
[2021/05/21 09:00:33.872 +00:00] [ERROR] [util.rs:416] [“request failed, retry”] [err_code=KV:PD:gRPC] [err=“Grpc(RpcFailure(RpcStatus { status: 14-UNAVAILABLE, details: Some(“failed to connect to all addresses”) }))”]
[2021/05/21 09:00:33.872 +00:00] [INFO] [util.rs:536] [“connecting to PD endpoint”] [endpoints=http://tidb-pd-0.tidb-pd-peer.tidb-admin.svc:2379]
[2021/05/21 09:00:33.882 +00:00] [ERROR] [util.rs:584] [“connect failed”] [error=“Grpc(RpcFailure(RpcStatus { status: 4-DEADLINE_EXCEEDED, details: Some(“Deadline Exceeded”) }))”] [endpoints=http://tidb-pd-2.tidb-pd-peer.tidb-admin.svc:2379]
[2021/05/21 09:00:33.887 +00:00] [ERROR] [util.rs:584] [“connect failed”] [error=“Grpc(RpcFailure(RpcStatus { status: 4-DEADLINE_EXCEEDED, details: Some(“Deadline Exceeded”) }))”] [endpoints=http://tidb-pd-0.tidb-pd-peer.tidb-admin.svc:2379]


若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。