慢查优化,不走索引

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】
【复现路径】做过哪些操作出现的问题
【遇到的问题:问题现象及影响】
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【附件:截图/日志/监控】
查询某用户是否存在风险表里

SELECT risk_value AS risk_t 
FROM user_risk  
WHERE risk_value = (
    SELECT mobile 
    FROM user 
    WHERE id = 12345
) 
AND risk_item = 1 
AND risk_type = 1 
AND is_deleted = 0 
AND status = 1 

表结构

CREATE TABLE `user_risk` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键id',
  ...
  `risk_item` smallint(2) NOT NULL COMMENT '风险项',
  `risk_type` smallint(2) NOT NULL COMMENT '风险类型',
  `risk_value` varchar(128) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT '风险项对应的值,混合型',
  PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */,
  ...
  KEY `risk_value` (`risk_item`,`risk_type`,`risk_value`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci AUTO_INCREMENT=8536500 COMMENT='风险表'


CREATE TABLE `user` (
  `id` bigint(20) NOT NULL COMMENT '账户id',
  `mobile` varchar(32)  DEFAULT '' COMMENT '手机号',
  ...
  PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */,
  ...
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin COMMENT='账户表'

执行以下查询,可以用到复合索引

SELECT risk_value AS risk_t 
FROM user_risk  
WHERE risk_value = (
    SELECT mobile 
    FROM user 
    WHERE id = 12345
) 
AND risk_item = 1 
AND risk_type = 1 
+-----------------------------+------------+-----------+---------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------+
| id                          | estRows    | task      | access object                                                                                           | operator info                                                          |
+-----------------------------+------------+-----------+---------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------+
| Projection_11               | 2433100.99 | root      |                                                                                                         | user_risk.risk_value                                 |
| └─IndexReader_14            | 2433100.99 | root      |                                                                                                         | index:Selection_13                                                     |
|   └─Selection_13            | 2433100.99 | cop[tikv] |                                                                                                         | eq(user_risk.risk_value, "xxx") |
|     └─IndexRangeScan_12     | 3041376.23 | cop[tikv] | table:user_risk, index:idx_risk_value(risk_item, risk_type, risk_value) | range:[1 1,1 1], keep order:false                                      |
+-----------------------------+------------+-----------+---------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------+

如果加上is_deleted=0 and status=1,则全表扫描


+----------------------------+------------+-----------+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                         | estRows    | task      | access object       | operator info                                                                                                                                                                                                                                |
+----------------------------+------------+-----------+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection_11              | 2351569.96 | root      |                     | user_risk.risk_value                                                                                                                                                                                                       |
| └─TableReader_14           | 2351569.96 | root      |                     | data:Selection_13                                                                                                                                                                                                                            |
|   └─Selection_13           | 2351569.96 | cop[tikv] |                     | eq(user_risk.risk_value, "xxx"), eq(user_risk.is_deleted, 0), eq(user_risk.risk_item, 1), eq(user_risk.risk_type, 1), eq(user_risk.status, 1) |
|     └─TableFullScan_12     | 8450909.00 | cop[tikv] | table:user_risk | keep order:false                                                                                                                                                                                                                             |
+----------------------------+------------+-----------+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

这种如何优化?按业务需求,执行select id from user where id=12345 and exist(select * from user_risk where user_risk=mobile and …)更符合语义,但是更慢。

子查询,是不是可以改成关联查询

  1. 子查询改关联查询
  2. user 表对 mobile 字段加上索引
  3. user_risk 表 对 risk_value加上索引
  4. 加上is_deleted=0 and status=1 不走索引,是因为 index:idx_risk_value(risk_item, risk_type, risk_value) 索引 优化器觉得代价太大
1 个赞

如果走idx_risk_value索引确实很好的话,那就直接hint绑定执行计划就行了。

SELECT /*+ USE_INDEX(user_risk,idx_risk_value) */ risk_value AS risk_t 
FROM user_risk  
WHERE risk_value = (
    SELECT mobile 
    FROM user 
    WHERE id = 12345
) 
AND risk_item = 1 
AND risk_type = 1 
AND is_deleted = 0 
AND status = 1

这就是个单表查询啊

这个索引没用上 risk_value 列,我没看出来为啥

这个查询能用上,加上is_deleted和status=1就不可以了
只要不是索引覆盖,select *也会不走复合索引,走全表扫描

优化器觉得那个索引太低效就不走了呀

看这里,没有用上 risk_value 列哦

调整一下idx_risk_value (risk_item,risk_type,risk_value)索引的顺序再试试

第一个能走索引是因为不用回表吧,数据量这么大,回表代价高吧

mysql> explain  select * from user_risk where risk_value='xxxxx' AND risk_item = 1  and risk_type = 1 and is_deleted = 0  and status = 1;
+--------------------------------+---------+-----------+---------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+
| id                             | estRows | task      | access object                                                                                           | operator info                                                                           |
+--------------------------------+---------+-----------+---------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+
| IndexLookUp_15                 | 1.06    | root      |                                                                                                         |                                                                                         |
| ├─IndexRangeScan_12(Build)     | 1.10    | cop[tikv] | table:user_risk, index:idx_risk_value(risk_item, risk_type, risk_value) | range:[1 1 "xxx",1 1 "xxx"], keep order:false |
| └─Selection_14(Probe)          | 1.06    | cop[tikv] |                                                                                                         | eq(user_risk.is_deleted, 0), eq(user_risk.status, 1)        |
|   └─TableRowIDScan_13          | 1.10    | cop[tikv] | table:user_risk                                                                                     | keep order:false                                                                        |
+--------------------------------+---------+-----------+---------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+
4 rows in set (0.01 sec)

这种就可以使用索引,即使是回表也用到了索引,一旦risk_value=(子查询)或risk_value in (子查询)就违背常理的慢

你这estrows差距也太大了吧,预估就一条数据,回表代价肯定低

请执行explain analyze

select * from user_risk where risk_value = (子查询) AND risk_item = 1 and risk_type = 1 and is_deleted = 0 and status = 1;

+-----------------------------+------------+---------+-----------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| id                          | estRows    | actRows | task      | access object       | execution info                                                                                                                                                                                                                                                                                                                                               | operator info                                                                                                                                                        | memory  | disk    |
+-----------------------------+------------+---------+-----------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
| HashJoin_9                  | 2351569.96 | 1       | root      |                     | time:2.18s, loops:2, build_hash_table:{total:1.96ms, fetch:1.95ms, build:5.6µs}, probe:{concurrency:5, total:10.9s, max:2.18s, probe:438.6ms, fetch:10.5s}                                                                                                                                                                                                   | semi join, equal:[eq(user_risk.risk_value, user.mobile)]                                                                          | 25.7 KB | 0 Bytes |
| ├─Point_Get_21(Build)       | 1.00       | 1       | root      | table:user       | time:1.91ms, loops:2, Get:{num_rpc:1, total_time:1.87ms}, total_process_time: 1ms, tikv_wall_time: 1.68ms, scan_detail: {total_process_keys: 1, total_process_keys_size: 783, total_keys: 1, get_snapshot_time: 5.23µs, rocksdb: {block: {cache_hit_count: 12, read_count: 2, read_byte: 45.1 KB, read_time: 1.25ms}}}                                       | handle:210415010079291504                                                                                                                                            | N/A     | N/A     |
| └─TableReader_12(Probe)     | 2939462.44 | 2865758 | root      |                     | time:2.15s, loops:2819, cop_task: {num: 253, max: 1.23s, min: 1.47ms, avg: 91.9ms, p95: 301.2ms, max_proc_keys: 426539, p95_proc_keys: 109536, tot_proc: 21.4s, tot_wait: 385ms, rpc_num: 253, rpc_time: 23.3s, copr_cache_hit_ratio: 0.00, distsql_concurrency: 15}                                                                                         | data:Selection_11                                                                                                                                                    | 20.1 MB | N/A     |
|   └─Selection_11            | 2939462.44 | 2865758 | cop[tikv] |                     | tikv_task:{proc max:1.22s, min:0s, avg: 79.9ms, p80:129ms, p95:262ms, iters:9273, tasks:253}, scan_detail: {total_process_keys: 8450908, total_process_keys_size: 1666484336, total_keys: 8457484, get_snapshot_time: 129.5ms, rocksdb: {key_skipped_count: 8457547, block: {cache_hit_count: 2865, read_count: 27259, read_byte: 477.4 MB, read_time: 2s}}} | eq(user_risk.is_deleted, 0), eq(user_risk.risk_item, 1), eq(user_risk.risk_type, 1), eq(user_risk.status, 1) | N/A     | N/A     |
|     └─TableFullScan_10      | 8450909.00 | 8450908 | cop[tikv] | table:user_risk | tikv_task:{proc max:1.2s, min:0s, avg: 77.7ms, p80:126ms, p95:255ms, iters:9273, tasks:253}                                                                                                                                                                                                                                                                  | keep order:false                                                                                                                                                     | N/A     | N/A     |
+-----------------------------+------------+---------+-----------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+
5 rows in set (2.19 sec)

select * from user_risk where risk_value = “xxx” AND risk_item = 1 and risk_type = 1 and is_deleted = 0 and status = 1;

+--------------------------------+---------+---------+-----------+---------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+---------+------+
| id                             | estRows | actRows | task      | access object                                                                                           | execution info                                                                                                                                                                                                                                                                                                                                                                                                                                       | operator info                                                                           | memory  | disk |
+--------------------------------+---------+---------+-----------+---------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+---------+------+
| IndexLookUp_15                 | 1.06    | 1       | root      |                                                                                                         | time:5.1ms, loops:2, index_task: {total_time: 4.19ms, fetch_handle: 4.19ms, build: 592ns, wait: 675ns}, table_task: {total_time: 826.7µs, num: 1, concurrency: 5}, next: {wait_index: 4.32ms, wait_table_lookup_build: 0s, wait_table_lookup_resp: 759.1µs}                                                                                                                                                                                          |                                                                                         | 17.2 KB | N/A  |
| ├─IndexRangeScan_12(Build)     | 1.10    | 3       | cop[tikv] | table:user_risk, index:idx_risk_value(risk_item, risk_type, risk_value) | time:4.18ms, loops:3, cop_task: {num: 1, max: 1.05ms, proc_keys: 3, rpc_num: 2, rpc_time: 1.62ms, copr_cache_hit_ratio: 0.00, distsql_concurrency: 15}, backoff{regionMiss: 2ms}, tikv_task:{time:1ms, loops:1}, scan_detail: {total_process_keys: 3, total_process_keys_size: 504, total_keys: 4, get_snapshot_time: 7.29µs, rocksdb: {key_skipped_count: 3, block: {cache_hit_count: 9, read_count: 1, read_byte: 27.9 KB, read_time: 449.5µs}}}   | range:[1 1 "xxx",1 1 "xxx"], keep order:false | N/A     | N/A  |
| └─Selection_14(Probe)          | 1.06    | 1       | cop[tikv] |                                                                                                         | time:741.8µs, loops:2, cop_task: {num: 1, max: 699.4µs, proc_keys: 3, rpc_num: 1, rpc_time: 684.7µs, copr_cache_hit_ratio: 0.00, distsql_concurrency: 15}, tikv_task:{time:1ms, loops:1}, scan_detail: {total_process_keys: 3, total_process_keys_size: 540, total_keys: 3, get_snapshot_time: 7.2µs, rocksdb: {block: {cache_hit_count: 24, read_count: 2, read_byte: 41.2 KB, read_time: 21.4µs}}}                                                 | eq(user_risk.is_deleted, 0), eq(user_risk.status, 1)        | N/A     | N/A  |
|   └─TableRowIDScan_13          | 1.10    | 3       | cop[tikv] | table:user_risk                                                                                     | tikv_task:{time:1ms, loops:1}                                                                                                                                                                                                                                                                                                                                                                                                                        | keep order:false                                                                        | N/A     | N/A  |
+--------------------------------+---------+---------+-----------+---------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+---------+------+
4 rows in set (0.01 sec)

子查询的结果与外表执行了hash join,这个如何关闭

你这应该有问题,你发 plan replayer 给大家看下

同不懂啊,等大佬答疑解惑

结贴了:字段排序规则不同,导致无法使用索引,我之前就发类似的问题。辛苦各位了,感谢感谢

+-----------------+----------------+--------------------+
| TABLE_NAME      | COLUMN_NAME    | COLLATION_NAME     |
+-----------------+----------------+--------------------+
| user            | mobile         | utf8mb4_bin        |
| user_risk       | risk_value     | utf8mb4_unicode_ci |
+-----------------+----------------+--------------------+

遇到过类似问题,建表时没有指定排序字符集,后续增加字段时,字段的排序字符集都是乱的