如何排查三个 kv 只有一个 kv 负载很高的问题呢？

🪐 TiDB 技术问题 部署&运维管理
1

整体负载情况

image
image852×379 116 KB

tidb.log 日志查看到的

[2019/08/20 12:49:34.228 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:919.732706ms txnStartTS:410590005716582401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:914"]
[2019/08/20 12:49:38.459 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:348.905422ms txnStartTS:410590007289446401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:336"]
[2019/08/20 12:49:39.049 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:589.799082ms txnStartTS:410590007289446401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:582"]
[2019/08/20 12:49:42.426 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:1.315736362s txnStartTS:410590008075878401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:1307"]
[2019/08/20 12:49:42.726 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:300.21055ms txnStartTS:410590008075878401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:295"]
[2019/08/20 12:49:44.694 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:583.099712ms txnStartTS:410590008862310401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:575"]
[2019/08/20 12:49:45.231 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:537.125202ms txnStartTS:410590008862310401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:533"]
[2019/08/20 12:49:47.638 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:528.016477ms txnStartTS:410590009648742401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:525"]
[2019/08/20 12:49:47.959 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:320.673644ms txnStartTS:410590009648742401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:309"]
[2019/08/20 12:49:50.742 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:632.108121ms txnStartTS:410590010435174401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:624"]
[2019/08/20 12:49:51.466 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:723.620033ms txnStartTS:410590010435174401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:714"]

这个问题如何查呢？

2

可以看下是否是热点导致的只有一个 kv 节点负载高，另外只提供日志无法判断原因，可以根据监控排查一下问题。

如果是 3.0 版本，可以从监控页面的 trouble-shooting 中查看读写热点；

3

如何查看热点数据呢？

4

kv的日志

2019/08/20 13:17:32.323 ERRO endpoint.rs:441: Locked(primary_lock: "t200\000\000\000\000\000\004267_r200\000\000\000\000\001i\024" lock_version: 410590445292748808 key: "t200\000\000\000\000\000\004357_r200\000\000\000\000\003360370" lock_ttl: 3014)
2019/08/20 13:17:32.412 ERRO endpoint.rs:441: Locked(primary_lock: "t200\000\000\000\000\000\004267_r200\000\000\000\000\001i\024" lock_version: 410590445292748808 key: "t200\000\000\000\000\000\004357_r200\000\000\000\000\003360370" lock_ttl: 3014)
2019/08/20 13:17:32.549 ERRO endpoint.rs:441: Locked(primary_lock: "t200\000\000\000\000\000\004267_r200\000\000\000\000\001i\024" lock_version: 410590445292748808 key: "t200\000\000\000\000\000\004305_r200\000\000\000\000\000350(" lock_ttl: 3014)
2019/08/20 13:17:33.023 ERRO endpoint.rs:441: Locked(primary_lock: "t200\000\000\000\000\000\004267_r200\000\000\000\000\000215)" lock_version: 410590445292748811 key: "t200\000\000\000\000\000\004357_r200\000\000\000\000\003360370" lock_ttl: 3010)
2019/08/20 13:17:33.107 ERRO endpoint.rs:441: Locked(primary_lock: "t200\000\000\000\000\000\004267_r200\000\000\000\000\000215)" lock_version: 410590445292748811 key: "t200\000\000\000\000\000\004357_r200\000\000\000\000\003360370" lock_ttl: 3010)
2019/08/20 13:17:33.186 ERRO endpoint.rs:441: Locked(primary_lock: "t200\000\000\000\000\000\004267_r200\000\000\000\000\000215)" lock_version: 410590445292748811 key: "t200\000\000\000\000\000\004357_r200\000\000\000\000\003360370" lock_ttl: 3010)
2019/08/20 13:17:33.221 ERRO endpoint.rs:441: Locked(primary_lock: "t200\000\000\000\000\000\004267_r200\000\000\000\000\000215)" lock_version: 410590445292748811 key: "t200\000\000\000\000\000\004357_r200\000\000\000\000\003360370" lock_ttl: 3010)
2019/08/20 13:17:33.473 ERRO endpoint.rs:441: Locked(primary_lock: "t200\000\000\000\000\000\004267_r200\000\000\000\000\001h370" lock_version: 410590445633536002 key: "t200\000\000\000\000\000\004357_r200\000\000\000\000\003360370" lock_ttl: 5729)
2019/08/20 13:17:33.502 ERRO endpoint.rs:441: Locked(primary_lock: "t200\000\000\000\000\000\004267_r200\000\000\000\000\001h370" lock_version: 410590445633536002 key: "t200\000\000\000\000\000\004357_r200\000\000\000\000\003360370" lock_ttl: 5729)
2019/08/20 13:17:33.516 ERRO endpoint.rs:441: Locked(primary_lock: "t200\000\000\000\000\000\004267_r200\000\000\000\000\001h370" lock_version: 410590445633536002 key: "t200\000\000\000\000\000\004357_r200\000\000\000\000\003360370" lock_ttl: 5729)
2019/08/20 13:17:33.589 ERRO endpoint.rs:441: Locked(primary_lock: "t200\000\000\000\000\000\004267_r200\000\000\000\000\001h370" lock_version: 410590445633536002 key: "t200\000\000\000\000\000\004357_r200\000\000\000\000\003360370" lock_ttl: 5729)
2019/08/20 13:17:33.648 ERRO endpoint.rs:441: Locked(primary_lock: "t200\000\000\000\000\000\004267_r200\000\000\000\000\001h370" lock_version: 410590445633536002 key: "t200\000\000\000\000\000\004357_r200\000\000\000\000\003360370" lock_ttl: 5729)
5

请按照提问模版提供版本，如是 3.0 版本可以从 trouble shooting 查看是否存在读写热点；其他版本可在 TiKV 监控面板查看对应监控信息