like模糊查询如何优化

【 TiDB 使用环境】生产环境
【 TiDB 版本】6.5.1
sql:


	id                            	task     	estRows	operator info                                                                                                                                                                                                                                                                                                                	actRows	execution info                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       	memory  	disk
	Projection_8                  	root     	500    	mall_qianchuan.qc_material.id, mall_qianchuan.qc_material.name, mall_qianchuan.qc_material.url, mall_qianchuan.qc_material.cover_url, mall_qianchuan.qc_material.duration, mall_qianchuan.qc_material.height, mall_qianchuan.qc_material.width, mall_qianchuan.qc_material.image_mode, mall_qianchuan.qc_material.inefficient	1      	time:7.87s, loops:2, Concurrency:OFF                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 	159.9 KB	N/A
	└─Limit_14                    	root     	500    	offset:0, count:500                                                                                                                                                                                                                                                                                                          	1      	time:7.87s, loops:2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  	N/A     	N/A
	  └─IndexLookUp_35            	root     	500    	                                                                                                                                                                                                                                                                                                                             	1      	time:7.87s, loops:2, index_task: {total_time: 4.89s, fetch_handle: 324ms, build: 84.6ms, wait: 4.49s}, table_task: {total_time: 32.6s, num: 24, concurrency: 5}, next: {wait_index: 34.7ms, wait_table_lookup_build: 1.45ms, wait_table_lookup_resp: 7.83s}                                                                                                                                                                                                                                                                                                                                                                                          	3.63 MB 	N/A
	    ├─IndexRangeScan_32(Build)	cop[tikv]	8093.04	table:qc_material, index:merchant_id(merchant_id, create_time), range:[1402519596895268865,1402519596895268865], keep order:true, desc                                                                                                                                                                                       	395797 	time:294ms, loops:393, cop_task: {num: 16, max: 973.8ms, min: 5.63ms, avg: 178.8ms, p95: 973.8ms, max_proc_keys: 50144, p95_proc_keys: 50144, tot_proc: 2.71s, tot_wait: 45ms, rpc_num: 16, rpc_time: 2.86s, copr_cache_hit_ratio: 0.00, distsql_concurrency: 15}, tikv_task:{proc max:967ms, min:1ms, avg: 169.5ms, p80:272ms, p95:967ms, iters:450, tasks:16}, scan_detail: {total_process_keys: 395797, total_process_keys_size: 21768835, total_keys: 395813, get_snapshot_time: 409.8µs, rocksdb: {delete_skipped_count: 2, key_skipped_count: 395814, block: {cache_hit_count: 206, read_count: 313, read_byte: 7.76 MB, read_time: 175.3ms}}}	N/A     	N/A
	    └─Selection_34(Probe)     	cop[tikv]	500    	eq(mall_qianchuan.qc_material.del_flag, 0), eq(mall_qianchuan.qc_material.inefficient, 0), eq(mall_qianchuan.qc_material.material_type, "VIDEO"), like(mall_qianchuan.qc_material.name, "%李红宇8.15素1_衍生_1%", 92)                                                                                                  	1      	time:32.4s, loops:25, cop_task: {num: 307, max: 3.6s, min: 472µs, avg: 419.5ms, p95: 1.46s, max_proc_keys: 5204, p95_proc_keys: 3264, tot_proc: 1m56.9s, tot_wait: 10.2s, rpc_num: 307, rpc_time: 2m8.8s, copr_cache_hit_ratio: 0.00, distsql_concurrency: 15}, tikv_task:{proc max:3.59s, min:0s, avg: 381.2ms, p80:694ms, p95:1.39s, iters:1511, tasks:307}, scan_detail: {total_process_keys: 395797, total_process_keys_size: 258454850, total_keys: 395814, get_snapshot_time: 1.59s, rocksdb: {key_skipped_count: 41, block: {cache_hit_count: 4829056, read_count: 7475, read_byte: 95.0 MB, read_time: 1.04s}}}                             	N/A     	N/A
	      └─TableRowIDScan_33     	cop[tikv]	8093.04	table:qc_material, keep order:false                                                                                                                                                                                                                                                                                          	395797 	tikv_task:{proc max:3.59s, min:0s, avg: 381.1ms, p80:693ms, p95:1.39s, iters:1511, tasks:307}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        	N/A     	N/A

like的前导%,从索引结构上没办法解决
build其他索引吧

你现在只用了一个merchan_id字段命中的索引,过滤出395797条数据,后面加上了inefficient、del_flag、material_type和name过滤之后只剩下1条数据。你搞个索引把merchan_id、inefficient、del_flag、material_type字段都带上。这样就不需要大量回表了

我感觉好像没啥用,因为merchan_id、inefficient、del_flag、material_type这几个条件过滤出来的数据就有几万条,只有like模糊那个条件最终才能筛选出来

前%没法直接用索引,如果全索引扫描like % 把主键查出来再用其他条件来过滤可能会好一点

放es分词处理啥的,tp好像没有好办法

这样走不了索引

只能改sql了

like如果走索引,只会走左匹配。
最好把create 语句导出来一下,才能综合评价如何优化。

目前用搜索引擎把。分词索引 :thinking:,索引格式变成tableid_columnid_分词:rowid。 :thinking:,好像可行 :joy:就是不知道哪位大佬能搞

‌建立联合索引并使用索引覆盖‌:通过为相关字段建立联合索引,并利用索引覆盖技术,避免回表操作,提高查询效率‌。不知道这样子优化有没有用。

你可以试试以下两种方法:
1、merchan_id、inefficient、del_flag、material_type、del_flag、name建个联合索引,看看过滤效果如何;
2、如果过滤效果不好,可以给这个表建TiFlash副本,让其走TiFlash MPP;
还有一个,看着你name里存储的字符串有点像是拼接起来的,这里是否可以考虑分成多个字段存储,这样就可以考虑建索引用于过滤。

用例存

木的办法,tidb不支持全文索引。。。导入到es里面玩吧

name放到索引里,至少可以减少回表

全模糊没有什么太好的办法

1 str% 正常建索引;
2, %str 尝试建倒序索引;
3,%str% 如果有规律,尝试用substr建索引;如果没有规律,使用tiflash试试。

1 个赞


+------------------------------------+----------+-----------+--------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                                 | estRows  | task      | access object                                                                                                                  | operator info                                                                                                                                                                                                                                                                                                                 |
+------------------------------------+----------+-----------+--------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Projection_8                       | 500.00   | root      |                                                                                                                                | mall_qianchuan.qc_material.id, mall_qianchuan.qc_material.name, mall_qianchuan.qc_material.url, mall_qianchuan.qc_material.cover_url, mall_qianchuan.qc_material.duration, mall_qianchuan.qc_material.height, mall_qianchuan.qc_material.width, mall_qianchuan.qc_material.image_mode, mall_qianchuan.qc_material.inefficient |
| └─TopN_10                          | 500.00   | root      |                                                                                                                                | mall_qianchuan.qc_material.create_time:desc, offset:0, count:500                                                                                                                                                                                                                                                              |
|   └─IndexLookUp_24                 | 500.00   | root      |                                                                                                                                |                                                                                                                                                                                                                                                                                                                               |
|     ├─TopN_23(Build)               | 500.00   | cop[tikv] |                                                                                                                                | mall_qianchuan.qc_material.create_time:desc, offset:0, count:500                                                                                                                                                                                                                                                              |
|     │ └─Selection_22               | 37914.80 | cop[tikv] |                                                                                                                                | like(mall_qianchuan.qc_material.name, "%oppoa93%", 92)                                                                                                                                                                                                                                                                        |
|     │   └─IndexRangeScan_20        | 47393.50 | cop[tikv] | table:qc_material, index:idx_mid_mtype_inff_del_name_crt(merchant_id, material_type, inefficient, del_flag, name, create_time) | range:[1714530771000516609 "VIDEO" 0 0,1714530771000516609 "VIDEO" 0 0], keep order:false                                                                                                                                                                                                                                     |
|     └─TableRowIDScan_21(Probe)     | 500.00   | cop[tikv] | table:qc_material                                                                                                              | keep order:false                                                                                                                                                                                                                                                                                                              |
+------------------------------------+----------+-----------+--------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
7 rows in set (0.00 sec)

可以的,过滤完之后需要排序的数据还是比较多,索引可以再加上create_time