慢SQL优化

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。

  • 【TiDB 版本】:v4.0.0
  • 【问题描述】:有一条数据量不是很大的sql,查询慢

您好,有一条慢SQL帮忙看看能不能调优,数据量大小: 3614176,查询时间:7s~12s

  1. sql语句
SELECT
	id,
	SUM(djsl),
	SUM(djje) 
FROM
	tablea 
WHERE
	time BETWEEN 201701 AND 201709
	GROUP BY id
  1. 表健康检查:
show stats_healthy

为100

  1. 建表语句
show create table tablea;

+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table               | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tablea | CREATE TABLE `tablea` (
  `id` varchar(30) DEFAULT NULL,
  `name` varchar(132) DEFAULT NULL,
  `djsl` bigint(20) DEFAULT NULL,
  `djje` decimal(18,2) DEFAULT NULL,
  `numa` decimal(18,2) DEFAULT NULL,
  `numb` decimal(18,2) DEFAULT NULL,
  `numc` decimal(18,2) DEFAULT NULL,
  `numd` decimal(18,2) DEFAULT NULL,
  `time` int(11) DEFAULT NULL,
  KEY `idx_table_id` (`id`),
  KEY `idx_table_time` (`time`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin/*!90000 SHARD_ROW_ID_BITS=4 PRE_SPLIT_REGIONS=3 */
PARTITION BY RANGE ( `` ) (
  PARTITION `p201701` VALUES LESS THAN (201702),
  PARTITION `p201702` VALUES LESS THAN (201703),
  PARTITION `p201703` VALUES LESS THAN (201704),
  PARTITION `p201704` VALUES LESS THAN (201705),
  PARTITION `p201705` VALUES LESS THAN (201706),
  PARTITION `p201706` VALUES LESS THAN (201707),
  PARTITION `p201707` VALUES LESS THAN (201708),
  PARTITION `p201708` VALUES LESS THAN (201709),
  PARTITION `p201709` VALUES LESS THAN (201710),
  PARTITION `p201710` VALUES LESS THAN (201711),
  PARTITION `p201711` VALUES LESS THAN (201712),
  PARTITION `p201712` VALUES LESS THAN (201801)
) |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  1. 分区情况(按time分区)
SELECT PARTITION_NAME,TABLE_ROWS,PARTITION_EXPRESSION,PARTITION_DESCRIPTION FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_NAME = 'tablea';

+----------------+------------+----------------------+-----------------------+
| PARTITION_NAME | TABLE_ROWS | PARTITION_EXPRESSION | PARTITION_DESCRIPTION |
+----------------+------------+----------------------+-----------------------+
| p201701        |     525995 | `time`               | 201702                |
| p201702        |     486064 | `time`               | 201703                |
| p201703        |          0 | `time`               | 201704                |
| p201704        |     524834 | `time`               | 201705                |
| p201705        |     555750 | `time`               | 201706                |
| p201706        |          0 | `time`               | 201707                |
| p201707        |     728309 | `time`               | 201708                |
| p201708        |          0 | `time`               | 201709                |
| p201709        |     793224 | `time`               | 201710                |
| p201710        |          0 | `time`               | 201711                |
| p201711        |          0 | `time`               | 201712                |
| p201712        |          0 | `time`               | 201801                |
+----------------+------------+----------------------+-----------------------+
  1. explain 和 explian analyze情况
    explain.txt (47.4 KB)
  1. 查看执行计划,每个分区先聚合计算后,再汇总计算,执行计划上感觉没问题,每个分区的计算时间都在3s. 之后所有分区再聚合再消耗3s,总共消耗了6s多。
  2. 稍等我们再查下是否有其他参数可以调整,多谢。

好的,谢谢

  1. 调整参数 set @@tidb_distsql_scan_concurrency = 80; 试试
  2. 开启 Super batch 功能,这个功能如果要开启,请先升级到最新版本,多谢。

https://docs.pingcap.com/zh/tidb/stable/tune-tiflash-performance#tidb-相关参数调优

我试了一下,没有任何变化。
我又试了给分区去掉,1.3s查询出结果

但业务上,数据不断按月汇总到表里,后期还是需要分区。请问,能解决这个问题吗

super batch 也开了吗?

分区表里,开启了

我后来试的无分区表,没有开启

  1. 是的,只需要分区表开启即可。
  2. 请检查系统资源是否满足条件,是否有瓶颈,多谢。

@etcxy Hi,麻烦提供更多信息帮助确认性能慢的问题:

  1. tiflash 部署节点的硬件配置

  2. 在分区表情况下,设置 tidb_distsql_scan_concurrency 和 打开 Super batch 之后的 explain analyze 结果。

  3. 先跑几次查询,然后导出 Grafana 监控的 Node_exporter 页面(TiFlash 节点) 和 TiFlash-Summary 页面。如果导出完整页面不方便,可以截图

    • Node_exporter 的 CPU 和 Disk 监控
    • TiFlash-Summary 的 Coprocesser, Storage, 以及 Raft
  4. 如果业务允许,可以考虑升级到 v4.0.5 看看是否有帮助,这个版本相对于 v4.0.0 有较大优化,比如优化 super batch,以及分区表性能。

Grafana 监控完整页面导出方法
1、chrome 安装这个插件https://chrome.google.com/webstore/detail/full-page-screen-capture/fdpohaocaechififmbbbbbknoalclacl
2、鼠标焦点置于左下角 Dashboard 上,按 ?可显示所有快捷键,先按 d 再按 E 可将所有 Rows 的 Panels 打开,需等待一段时间待页面加载完成。
3、使用这个 full-page-screen-capture 插件进行截屏保存

有些 partition 是在 tiflash 上查的,有些是在 tikv 上查的。tiflash 最慢 3s 左右,而 tikv 很快。
建议用 hint 强制在 tikv 上执行试试。
https://docs.pingcap.com/zh/tidb/stable/optimizer-hints#read_from_storagetiflasht1_name--tl_name--tikvt2_name--tl_name-

好的,谢谢

感谢反馈,新问题麻烦您创建新帖子

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。