3.7.1 Metrics that DBAs should notice(运维中的关键监控)

课程名称:Metrics that DBAs should notice(运维中的关键监控)

学习时长:30分钟

课程收获:运维中的关键监控

课程内容:

SYSTem监控指标注意项:
CPU使用率
如果超过80%,将可能达到系统瓶颈
CPU负载
这个值应该小于CPU的总核数
内存可用率
TiKV nodes: memory usage<60%
TiDB nodes: 20% free memory
网络传输
网络流量不要打满网卡
IO使用率
如果超过80%,将可能达到系统瓶颈

TIDB监控指标注意项:
Query Summary
A) Duration: 对于OLTP的负载99%的延迟都应该地域100ms
B) Slow Query:不应该有太多的慢查询
C) Ideal CPS: 判断延迟出现在数据库端还是客户端
Server
Get token duration:better < 1ms, 或者检查token-limit配置值是否大于总的连接数
Executor
A) Parse duration: better < 10ms
B) Compile duration: better < 30ms
KV Error
A) Lock Resolve OPS: better < 500 for expired and not expired or too many conflicts,太多锁冲突建议用悲观锁
B) KV Backoff OPS: better< 500 for txnLockFast and txnLock
PD Client
PD TSO .99 wait duration: better<5ms



TIkv监控指标注意项:
TiKV
Cluster
Region: better < 50K ,region太多的话,心跳和raft状态机的开销过大,可以通过region merge或者调节hibernate region降低开销
gPRC
.99 gRPC message duration: better < 100ms,这个延迟越低越好
Thread CPU
A) Raft store CPU: better < 75%*raftstore.store-pool-size
B) Async apply CPU: better <75%*raftstore.apply-pool-size
C) Schduler worker CPU: better <80%*storage.scheduler-worker-pool-size
D) gRPC poll CPU: better <80%*server.grpc-concurrency
E) Unified read pool CPU: better <80%*readpool.unified.max-thread-count
F) Storage ReadPool CPU: better <80%*readpool.storage.normal-concurrency
Raft IO
A) Append log duration: 99 latency better < 10ms
B) Apply log duration: 99 latency better < 30ms
C) Commit log duration: 99 latency better < 30ms
D) Also should notice the .999 latency for above metrics
Raft propose
A) Propose wait duration: 99 latency better <20ms
B) Apply wait duration: 99 latency better < 50ms
C) Also should notice the .999 latency for above metrics
Errors
Server is busy: better there is no busy error


pd监控指标注意项:
etcd
99% WAL fsync duration: better < 5ms
Heartbeat
99% Region heartbeat latency: better < 5ms
Dashboard

DASHBOARD监控指标:

学习过程中遇到的问题或延伸思考:

  • 问题 1:
  • 问题 2:
  • 延伸思考 1:
  • 延伸思考 2:

学习过程中参考的其他资料

同学你好,感谢参与 TiDB 4.0 课程的学习!

本篇笔记逻辑清晰、内容丰富,被评选为优质笔记,将额外获得 20 积分,并在 「TiDB 培训」分类下获得“置顶”权益,积分兑换规则将于近期开放,敬请关注!

期待您继续产出优质内容!