建表慢

🪐 TiDB 技术问题 性能调优
122

:joy: 等大佬来解答了
tikv那几个指标的延迟都太高了

  • kv_pessimistic_lock：对 key 加悲观锁，防止其他事务修改
  • kv_check_txn_status：检查事务的状态
  • kv_commit：2PC 的第二阶段，提交 prewrite 阶段写入的数据
    image
123

你这是真的测试结果吗read 的iops只有297？
fio测试iops读写都需要测试 -rw=randwrite

image
image770×211 10.1 KB

124

这个测试的吞吐量

125

这个是我之前测试的阿里云的essd磁盘和本地磁盘的性能

image
image838×414 8.33 KB

126

我用的阿里提供的方法，测试结果差不多

127

最开始我也是用的essd，并发量上来后这个磁盘扛不住，全换成本地盘了，性能差距太大，关键本地盘还便宜

128

夜里低峰期的时候 ，操作也慢，这与磁盘就没有关系了吧

129

其实我们想表达的是，目前你的存储磁盘I/O不足以支持你现在的业务体量了；
本身磁盘没问题，但I/O很可能到了上限；
如果可以的话，可以申请 裸金属搭配Nvme，通过扩缩容的方式把tidb集群平滑迁移到裸金属上，看看性能是不是有改善

130

磁盘的IOPS与吞吐量都没有到上限，使用的很认底

131

生产环境，不是那么好迁的

132

[2024/03/20 14:14:09.246 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=122232161]
[2024/03/20 14:14:09.246 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=122232163] [region_id=122232161]
[2024/03/20 14:14:10.072 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=104738827]
[2024/03/20 14:14:10.072 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=104738829] [region_id=104738827]
[2024/03/20 14:14:10.165 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=96489380]
[2024/03/20 14:14:10.165 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=96489383] [region_id=96489380]
[2024/03/20 14:14:10.478 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=103209962]
[2024/03/20 14:14:10.478 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=103209965] [region_id=103209962]
[2024/03/20 14:14:11.327 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=98427848]
[2024/03/20 14:14:11.327 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=98427849] [region_id=98427848]
[2024/03/20 14:14:12.030 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=92873751]
[2024/03/20 14:14:12.030 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=92873752] [region_id=92873751]
[2024/03/20 14:14:15.658 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=120992152]
[2024/03/20 14:14:15.658 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=120992154] [region_id=120992152]
[2024/03/20 14:14:16.839 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=118837112]
[2024/03/20 14:14:16.839 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=118837113] [region_id=118837112]
[2024/03/20 14:14:17.237 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=98555903]
[2024/03/20 14:14:17.237 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=98555905] [region_id=98555903]
[2024/03/20 14:14:18.368 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=97418302]
[2024/03/20 14:14:18.368 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=145129189] [region_id=97418302]
[2024/03/20 14:14:19.190 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=94352464]
[2024/03/20 14:14:19.190 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=106159357] [region_id=94352464]
[2024/03/20 14:14:19.820 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=92592456]
[2024/03/20 14:14:19.820 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=122462590] [region_id=92592456]
[2024/03/20 14:14:20.070 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=133589164]
[2024/03/20 14:14:20.070 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=133589166] [region_id=133589164]
[2024/03/20 14:14:21.227 +08:00] [ERROR] [split_observer.rs:142] [“failed to handle split req”] [err=“"no valid key found for split."”] [region_id=104373387]
[2024/03/20 14:14:21.227 +08:00] [WARN] [peer.rs:4339] [“skip proposal”] [error_code=KV:Raftstore:Coprocessor] [err=“Coprocessor(Other("[components/raftstore/src/coprocessor/split_observer.rs:147]: no valid key found for split."))”] [peer_id=104373390] [region_id=104373387]
这个报错有影响吗

133

https://docs.pingcap.com/zh/tidb/stable/clinic-introduction
试试这个，现在貌似还是免费期

134

[2024/03/20 15:30:17.155 +08:00] [WARN] [endpoint.rs:782] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 89020294, leader may Some(id: 119643143 store_id: 92768299)" not_leader { region_id: 89020294 leader { id: 119643143 store_id: 92768299 } }”]
[2024/03/20 15:30:17.160 +08:00] [WARN] [endpoint.rs:782] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 89020294, leader may Some(id: 119643143 store_id: 92768299)" not_leader { region_id: 89020294 leader { id: 119643143 store_id: 92768299 } }”]
[2024/03/20 15:30:17.174 +08:00] [WARN] [endpoint.rs:782] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 87301562, leader may Some(id: 92893779 store_id: 92768299)" not_leader { region_id: 87301562 leader { id: 92893779 store_id: 92768299 } }”]
[2024/03/20 15:30:17.174 +08:00] [WARN] [endpoint.rs:782] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 87301562, leader may Some(id: 92893779 store_id: 92768299)" not_leader { region_id: 87301562 leader { id: 92893779 store_id: 92768299 } }”]
[2024/03/20 15:30:17.219 +08:00] [WARN] [endpoint.rs:782] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 87301562, leader may Some(id: 92893779 store_id: 92768299)" not_leader { region_id: 87301562 leader { id: 92893779 store_id: 92768299 } }”]
[2024/03/20 15:30:17.223 +08:00] [WARN] [endpoint.rs:782] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 89020294, leader may Some(id: 119643143 store_id: 92768299)" not_leader { region_id: 89020294 leader { id: 119643143 store_id: 92768299 } }”]

135

这个报错有没有关系啊

136

查询正常，就写入慢

137

删表也慢，没救了

138

image
image1843×767 155 KB

看这个IO也是正常的啊

139

image
image1854×334 65.9 KB

看这块监控，好像是有问题

140

你这个咋都是秒级别

141

我看了我们的在100ms以下

C9MHRScvgE
C9MHRScvgE1210×387 71.8 KB