如何排查三个 kv 只有一个 kv 负载很高的问题呢?

整体负载情况

tidb.log 日志查看到的

[2019/08/20 12:49:34.228 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:919.732706ms txnStartTS:410590005716582401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:914"]
[2019/08/20 12:49:38.459 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:348.905422ms txnStartTS:410590007289446401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:336"]
[2019/08/20 12:49:39.049 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:589.799082ms txnStartTS:410590007289446401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:582"]
[2019/08/20 12:49:42.426 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:1.315736362s txnStartTS:410590008075878401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:1307"]
[2019/08/20 12:49:42.726 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:300.21055ms txnStartTS:410590008075878401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:295"]
[2019/08/20 12:49:44.694 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:583.099712ms txnStartTS:410590008862310401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:575"]
[2019/08/20 12:49:45.231 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:537.125202ms txnStartTS:410590008862310401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:533"]
[2019/08/20 12:49:47.638 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:528.016477ms txnStartTS:410590009648742401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:525"]
[2019/08/20 12:49:47.959 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:320.673644ms txnStartTS:410590009648742401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:309"]
[2019/08/20 12:49:50.742 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:632.108121ms txnStartTS:410590010435174401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:624"]
[2019/08/20 12:49:51.466 +08:00] [INFO] [coprocessor.go:723] ["[TIME_COP_WAIT] resp_time:723.620033ms txnStartTS:410590010435174401 region_id:24 store_addr:10.0.0.4:20160 kv_wait_ms:714"]

这个问题如何查呢?

可以看下是否是热点导致的只有一个 kv 节点负载高,另外只提供日志无法判断原因,可以根据监控排查一下问题。

如果是 3.0 版本,可以从监控页面的 trouble-shooting 中查看读写热点;

如何查看热点数据呢?

kv的日志

2019/08/20 13:17:32.323 ERRO endpoint.rs:441: Locked(primary_lock: "t200000000000004267_r2000000000001i24" lock_version: 410590445292748808 key: "t200000000000004357_r2000000000003360370" lock_ttl: 3014)
2019/08/20 13:17:32.412 ERRO endpoint.rs:441: Locked(primary_lock: "t200000000000004267_r2000000000001i24" lock_version: 410590445292748808 key: "t200000000000004357_r2000000000003360370" lock_ttl: 3014)
2019/08/20 13:17:32.549 ERRO endpoint.rs:441: Locked(primary_lock: "t200000000000004267_r2000000000001i24" lock_version: 410590445292748808 key: "t200000000000004305_r2000000000000350(" lock_ttl: 3014)
2019/08/20 13:17:33.023 ERRO endpoint.rs:441: Locked(primary_lock: "t200000000000004267_r2000000000000215)" lock_version: 410590445292748811 key: "t200000000000004357_r2000000000003360370" lock_ttl: 3010)
2019/08/20 13:17:33.107 ERRO endpoint.rs:441: Locked(primary_lock: "t200000000000004267_r2000000000000215)" lock_version: 410590445292748811 key: "t200000000000004357_r2000000000003360370" lock_ttl: 3010)
2019/08/20 13:17:33.186 ERRO endpoint.rs:441: Locked(primary_lock: "t200000000000004267_r2000000000000215)" lock_version: 410590445292748811 key: "t200000000000004357_r2000000000003360370" lock_ttl: 3010)
2019/08/20 13:17:33.221 ERRO endpoint.rs:441: Locked(primary_lock: "t200000000000004267_r2000000000000215)" lock_version: 410590445292748811 key: "t200000000000004357_r2000000000003360370" lock_ttl: 3010)
2019/08/20 13:17:33.473 ERRO endpoint.rs:441: Locked(primary_lock: "t200000000000004267_r2000000000001h370" lock_version: 410590445633536002 key: "t200000000000004357_r2000000000003360370" lock_ttl: 5729)
2019/08/20 13:17:33.502 ERRO endpoint.rs:441: Locked(primary_lock: "t200000000000004267_r2000000000001h370" lock_version: 410590445633536002 key: "t200000000000004357_r2000000000003360370" lock_ttl: 5729)
2019/08/20 13:17:33.516 ERRO endpoint.rs:441: Locked(primary_lock: "t200000000000004267_r2000000000001h370" lock_version: 410590445633536002 key: "t200000000000004357_r2000000000003360370" lock_ttl: 5729)
2019/08/20 13:17:33.589 ERRO endpoint.rs:441: Locked(primary_lock: "t200000000000004267_r2000000000001h370" lock_version: 410590445633536002 key: "t200000000000004357_r2000000000003360370" lock_ttl: 5729)
2019/08/20 13:17:33.648 ERRO endpoint.rs:441: Locked(primary_lock: "t200000000000004267_r2000000000001h370" lock_version: 410590445633536002 key: "t200000000000004357_r2000000000003360370" lock_ttl: 5729)

请按照提问模版提供版本,如是 3.0 版本可以从 trouble shooting 查看是否存在读写热点;其他版本可在 TiKV 监控面板查看对应监控信息