tikv 狂刷日志，有连接本地pd timeout,麻烦问一下是什么原因

🪐 TiDB 技术问题 部署&运维管理
1

[2020/05/20 15:50:33.676 +08:00] [INFO] [util.rs:397] [“connecting to PD endpoint”] [endpoints=127.0.0.1:2379] [2020/05/20 15:50:33.677 +08:00] [INFO] [subchannel.cc:878] [“Connect failed: {“created”:”@1589961033.676903126",“description”:“Failed to connect to remote host: OS Error”,“errno”:111,“file”:"/rust/registry/src/gi thub.com-1ecc6299db9ec823/grpcio-sys-0.4.7/grpc/src/core/lib/iomgr/tcp_client_posix.cc",“file_line”:207,“os_error”:“Connection refused”,“syscall”:“connect”,“target_address”:“ipv4:127.0.0.1:2379”}"] [2020/05/20 15:50:33.677 +08:00] [INFO] [subchannel.cc:760] [“Subchannel 0x7f5041e33a00: Retry in 999 milliseconds”] [2020/05/20 15:50:33.677 +08:00] [INFO] [util.rs:357] [“PD failed to respond”] [err=“Grpc(RpcFailure(RpcStatus { status: Unavailable, details: Some(“Connect Failed”) }))”] [endpoints=127.0.0.1:2379]

2
  1. 上传下 tikv 报错发生前的一小时日志和报错的一段日志
  2. 上传/var/log/message系统日志，tikv 报错时的当天日志
  3. tidb 版本是什么？ 什么环境？是单机的吗？ 看到连接的是127.0.0.1的ip？
  4. pd-ctl 命令，反馈下 member 和 store 状态，多谢。
3

请确认一下 pd 是否正常运行？

4

message.log (1006.9 KB) tikv.log (8.0 KB) pd_member.log (1.1 KB) pd_store.log (1.9 KB)

tidb: 5.7.25-TiDB-v3.0.7
服务环境：zone
集群版

5

[2020/05/21 11:15:57.454 +08:00] [WARN] [util.go:144] [“apply request took too long”] [took=1.151530753s] [expected-duration=100ms] [prefix="read-only range “] [request=“key:”/tidb-binlog/v1/pumps/tidb001.stg.piaoshen.host:8250” "] [response=“range_response_count:1 size:274”] [] 2020/05/21 11:21:04.719 log.go:86: [warning] wal: [sync duration of 3.992022469s, expected less than 1s] [2020/05/21 11:21:04.725 +08:00] [WARN] [util.go:144] [“apply request took too long”] [took=1.956833505s] [expected-duration=100ms] [prefix="read-only range “] [request=“key:”/tidb-binlog/v1/pumps/tidb002.stg.piaoshen.host:8250” "] [response=“range_response_count:1 size:274”] [] [2020/05/21 11:30:45.082 +08:00] [INFO] [index.go:190] [“compact tree index”] [revision=22396841] [2020/05/21 11:30:45.102 +08:00] [INFO] [kvstore_compaction.go:57] [“finished scheduled compaction”] [compact-revision=22396841] [took=19.754775ms] [2020/05/21 11:32:10.085 +08:00] [WARN] [util.go:144] [“apply request took too long”] [took=227.404973ms] [expected-duration=100ms] [prefix="read-only range “] [request=“key:”/tidb-binlog/v1/pumps/tidb001.stg.piaoshen.host:8250” "] [response=“range_response_count:1 size:274”] [] 2020/05/21 11:43:27.328 log.go:86: [warning] wal: [sync duration of 1.633052988s, expected less than 1s] [2020/05/21 11:43:27.332 +08:00] [WARN] [util.go:144] [“apply request took too long”] [took=1.580954502s] [expected-duration=100ms] [prefix="read-only range “] [request=“key:”/tidb-binlog/v1/pumps/tidb002.stg.piaoshen.host:8250” "] [response=“range_response_count:1 size:274”] [] [2020/05/21 12:00:44.535 +08:00] [WARN] [util.go:144] [“apply request took too long”] [took=2.086751166s] [expected-duration=100ms] [prefix="read-only range “] [request=“key:”/tidb-binlog/v1/pumps/tidb002.stg.piaoshen.host:8250” "] [response=“range_response_count:1 size:274”] [] [2020/05/21 12:00:44.537 +08:00] [WARN] [util.go:144] [“apply request took too long”] [took=2.078428764s] [expected-duration=100ms] [prefix="read-only range “] [request=“key:”/tidb-binlog/v1/pumps/tidb003.stg.piaoshen.host:8250” "] [response=“range_response_count:1 size:274”] []

pd 服务是正常，但是会报request too long

6

请反馈 over-view ， pd ， detail-tikv 的监控信息。

看起来PD 反映比较慢。可以先检查下，PD 服务器资源是否有打满

(1)、chrome 安装这个插件https://chrome.google.com/webstore/detail/full-page-screen-capture/fdpohaocaechififmbbbbbknoalclacl

(2)、鼠标焦点置于 Dashboard 上，按 ?可显示所有快捷键，先按 d 再按 E 可将所有 Rows 的 Panels 打开，需等待一段时间待页面加载完成。

(3)、使用这个 full-page-screen-capture 插件进行截屏保存