突然出现大量慢SQL,INSERT INTO mysql.stats_histograms

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】6.5.3
【遇到的问题:问题现象及影响】
突然出现大量慢sql

【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面

【附件:截图/日志/监控】

大佬们有知道什么原因吗?

执行计划贴出来看看

id task estRows operator info actRows execution info memory disk
Insert_1 root 0 N/A 0 time:1.85ms, loops:1, prepare: 47µs, check_insert: {total_time: 1.8ms, mem_insert_time: 292.6µs, prefetch: 1.51ms, rpc:{BatchGet:{num_rpc:2, total_time:1.43ms}, tikv_wall_time: 721.3µs, scan_detail: {total_process_keys: 26, total_process_keys_size: 2080, total_keys: 26, get_snapshot_time: 39.5µs, rocksdb: {block: {cache_hit_count: 108}}}}}, commit_txn: {prewrite:2.8s, get_commit_ts:237.1µs, commit:2.3s, slowest_prewrite_rpc: {total: 2.800s, region_id: 116045, store: 10.18.67.47:20160, tikv_wall_time: 2.8s, scan_detail: {get_snapshot_time: 15.9µs, rocksdb: {block: {cache_hit_count: 69}}}, write_detail: {store_batch_wait: 934.2ms, propose_send_wait: 0s, persist_log: {total: 1.87s, write_leader_wait: 642.4ms, sync_log: 1.22s, write_memtable: 6.21µs}, commit_log: 1.87s, apply_batch_wait: 41.1µs, apply: {total:231.5µs, mutex_lock: 0s, write_leader_wait: 0s, write_wal: 41.1µs, write_memtable: 54.8µs}}}, commit_primary_rpc: {total: 2.299s, region_id: 116045, store: 10.18.67.47:20160, tikv_wall_time: 2.3s, scan_detail: {get_snapshot_time: 21.2µs, rocksdb: {block: {}}}, write_detail: {store_batch_wait: 1.12s, propose_send_wait: 0s, persist_log: {total: 1.18s, write_leader_wait: 543.6ms, sync_log: 633.2ms, write_memtable: 5.02µs}, commit_log: 1.18s, apply_batch_wait: 34.5µs, apply: {total:220µs, mutex_lock: 0s, write_leader_wait: 0s, write_wal: 34.5µs, write_memtable: 76.8µs}}}, region_num:1, write_keys:13, write_byte:1170} 11.9 KB N/A

这个表是更新统计信息吗?有做过表分析?

没做过,就正常运行的

mysql.stats_histograms 存储了关于表的统计信息(如索引和数据列的不同值数量以及NULL值数量等),这些统计信息可以帮助优化SQL查询的性能

这种应该还好 可能是锁导致的

不是,其他也就算了,commit要7.5秒啊?集群当时是不是都快挂了?应该是io完全挂死才能出现commit需要7秒多吧?

有段时间IO高

还是磁盘性能不太好,有影响

mysql.stats_histograms这个表是状态直方图?为什么会有这个的插入啊?

内部统计信息影响正常使用了?

而且看你的执行计划详情,基本时间都卡在刷盘上了,还是先看下磁盘是不是有问题了,磁盘性能不好也不至于刷个盘要这么长时间吧,应该还是出故障了。。。

大佬,这该怎么检测?麻烦指导下 :pray:

磁盘是什么盘啊 不会是机械盘吧

到对应的磁盘上测试下io的情况
time dd if=/dev/zero of=test bs=8k count=100000 oflag=direct

可以看看监控tidb-nova-prod-Disk-Performance,这个监控项,看看io情况