【TiDB 使用环境】生产环境
【TiDB 版本】8.5.1
【问题复现路径】
【遇到的问题：问题现象及影响】
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【复制黏贴 ERROR 报错的日志】
【其他附件：截图/日志/监控】
今天突然有几个sql执行时间很长，平时都是几毫秒，然后dashboard查看执行计划，里面有很多rpc_errors报错，这种报错一般是什么情况导致的，我查看了当时节点的CPU，内存情况都正常，执行计划太长了，这里截取了两部分，求大佬帮忙分析
| │ │ │ │ │ └─Projection_158(Probe) | 20.00 | 195211.30 | 20 | root | | time:33s, loops:2, Concurrency:OFF | testvoc.t_voc_internationalcompetitivereview.id, testvoc.t_voc_internationalcompetitivereview.platform, testvoc.t_voc_internationalcompetitivereview.site, testvoc.t_voc_internationalcompetitivereview.testsku, testvoc.t_voc_internationalcompetitivereview.asin, testvoc.t_voc_internationalcompetitivereview.username, testvoc.t_voc_internationalcompetitivereview.score, testvoc.t_voc_internationalcompetitivereview.title, testvoc.t_voc_internationalcompetitivereview.chinesetitle, testvoc.t_voc_internationalcompetitivereview.commentdetails, testvoc.t_voc_internationalcompetitivereview.chinesecommentdetails, testvoc.t_voc_internationalcompetitivereview.commentdate, testvoc.t_voc_internationalcompetitivereview.commentsourcesite, testvoc.t_voc_internationalcompetitivereview.commentlikescount, testvoc.t_voc_internationalcompetitivereview.authenticateduser, testvoc.t_voc_internationalcompetitivereview.classifyby, testvoc.t_voc_internationalcompetitivereview.lastby, testvoc.t_voc_internationalcompetitivereview.lastdate, testvoc.t_voc_internationalcompetitivereview.commentaddress | 24.9 KB | N/A |
| │ │ │ │ │ └─TopN_100 | 20.00 | 195173.38 | 20 | root | | time:33s, loops:2 | Column#145:desc, testvoc.t_voc_internationalcompetitivereview.id, offset:0, count:20 | 270.4 KB | 0 Bytes |
| │ │ │ │ │ └─Projection_159 | 20.00 | 179171.62 | 627 | root | | time:33s, loops:3, Concurrency:OFF | testvoc.t_voc_internationalcompetitivereview.id, testvoc.t_voc_internationalcompetitivereview.platform, testvoc.t_voc_internationalcompetitivereview.site, testvoc.t_voc_internationalcompetitivereview.testsku, testvoc.t_voc_internationalcompetitivereview.asin, testvoc.t_voc_internationalcompetitivereview.username, testvoc.t_voc_internationalcompetitivereview.score, testvoc.t_voc_internationalcompetitivereview.title, testvoc.t_voc_internationalcompetitivereview.chinesetitle, testvoc.t_voc_internationalcompetitivereview.commentdetails, testvoc.t_voc_internationalcompetitivereview.chinesecommentdetails, testvoc.t_voc_internationalcompetitivereview.commentdate, testvoc.t_voc_internationalcompetitivereview.commentsourcesite, testvoc.t_voc_internationalcompetitivereview.commentlikescount, testvoc.t_voc_internationalcompetitivereview.authenticateduser, testvoc.t_voc_internationalcompetitivereview.classifyby, testvoc.t_voc_internationalcompetitivereview.lastby, testvoc.t_voc_internationalcompetitivereview.lastdate, testvoc.t_voc_internationalcompetitivereview.commentaddress, date_format(testvoc.t_voc_internationalcompetitivereview.commentdate, %Y-%m-%d %H:%i:%s)->Column#145 | 592.5 KB | N/A |
| │ │ │ │ │ └─TableReader_105 | 20.00 | 178934.09 | 627 | root | partition:all | time:33s, loops:3, cop_task: {num: 194, max: 8.41s, min: 1.17ms, avg: 933.4ms, p95: 8.41s, max_proc_keys: 367, p95_proc_keys: 121, tot_proc: 52ms, tot_wait: 17.4ms, copr_cache_hit_ratio: 0.00, build_task_duration: 233.8?s, max_distsql_concurrency: 15}, rpc_info:{Cop:{num_rpc:224, total_time:6m8.2s}, rpc_errors:{not_leader:26, rpc error: code = Unavailable desc = error reading from server: read tcp 10.12.1.107:53874->10.12.1.105:20160: read: connection timed out:4}}, backoff{regionMiss: 2ms, tikvRPC: 69ms} | data:TopN_104 | 66.0 KB | N/A |
| │ │ │ │ │ └─TopN_104 | 20.00 | 2467636.99 | 627 | cop[tikv] | | tikv_task:{proc max:2ms, min:0s, avg: 247.4?s, p80:1ms, p95:1ms, iters:194, tasks:194}, scan_detail: {total_process_keys: 3085, total_process_keys_size: 2575997, total_keys: 3279, get_snapshot_time: 7.13ms, rocksdb: {key_skipped_count: 5828, block: {cache_hit_count: 1185}}}, time_detail: {total_process_time: 52ms, total_suspend_time: 27.2?s, total_wait_time: 17.4ms, total_kv_read_wall_time: 37ms, tikv_wall_time: 1m2.8s} | date_format(testvoc.t_voc_internationalcompetitivereview.commentdate, "%Y-%m-%d %H:%i:%s"):desc, testvoc.t_voc_internationalcompetitivereview.id, offset:0, count:20 | N/A | N/A |
| │ │ │ │ │ └─TableFullScan_103 | 3085.00 | 1637443.40 | 3085 | cop[tikv] | table:T1 | tikv_task:{proc max:2ms, min:0s, avg: 190.7?s, p80:0s, p95:1ms, iters:194, tasks:194} | keep order:false | N/A | N/A |
| └─HashJoin_149(Probe) | 1923.00 | 893585.61 | 1923 | root | | time:21.7s, loops:6, build_hash_table:{total:17.1s, fetch:17.1s, build:341.8?s}, probe:{concurrency:5, total:1m48.3s, max:21.7s, probe:3.53ms, fetch and wait:1m48.3s} | inner join, equal:[eq(testvoc.t_voc_cqelabel.skuid, testvoc.t_voc_skulist.id)] | 137.8 KB | 0 Bytes |
| ├─TableReader_152(Build) | 1923.00 | 66757.00 | 1923 | root | | time:17.1s, loops:3, cop_task: {num: 4, max: 8.57s, min: 1.38s, avg: 4.27s, p95: 8.57s, max_proc_keys: 992, p95_proc_keys: 992, tot_proc: 2.63ms, tot_wait: 512.6?s, copr_cache_hit_ratio: 0.00, build_task_duration: 1.68?s, max_distsql_concurrency: 1}, rpc_info:{Cop:{num_rpc:4, total_time:17.1s}} | data:Selection_151 | 49.8 KB | N/A |
| │ └─Selection_151 | 1923.00 | 637430.35 | 1923 | cop[tikv] | | tikv_task:{proc max:1ms, min:0s, avg: 500?s, p80:1ms, p95:1ms, iters:16, tasks:4}, scan_detail: {total_process_keys: 1925, total_process_keys_size: 187464, total_keys: 1930, get_snapshot_time: 247.6?s, rocksdb: {key_skipped_count: 1926, block: {cache_hit_count: 31}}}, time_detail: {total_process_time: 2.63ms, total_wait_time: 512.6?s, total_kv_read_wall_time: 2ms, tikv_wall_time: 4.91s} | eq(testvoc.t_voc_cqelabel.forbiddenstatus, 0) | N/A | N/A |
| │ └─TableFullScan_150 | 1925.00 | 541372.85 | 1925 | cop[tikv] | table:T1 | tikv_task:{proc max:1ms, min:0s, avg: 500?s, p80:1ms, p95:1ms, iters:16, tasks:4} | keep order:false | N/A | N/A |
| └─IndexReader_154(Probe) | 15268.00 | 310286.89 | 15268 | root | | time:21.7s, loops:17, cop_task: {num: 7, max: 8.41s, min: 2.51ms, avg: 1.69s, p95: 8.41s, max_proc_keys: 992, p95_proc_keys: 992, tot_proc: 1.59ms, tot_wait: 1.74ms, copr_cache_hit_ratio: 0.57, build_task_duration: 7.63?s, max_distsql_concurrency: 1}, rpc_info:{Cop:{num_rpc:9, total_time:21.7s}, rpc_errors:{not_leader:2}}, backoff{regionMiss: 2ms} | index:IndexFullScan_153 | 179.7 KB | N/A |
| └─IndexFullScan_153 | 15268.00 | 2978316.74 | 15268 | cop[tikv] | table:T2, index:AK_Key_Number(Number) | tikv_task:{proc max:2ms, min:0s, avg: 1.43ms, p80:2ms, p95:2ms, iters:42, tasks:7}, scan_detail: {total_process_keys: 1696, total_process_keys_size: 101042, total_keys: 1699, get_snapshot_time: 1.31ms, rocksdb: {key_skipped_count: 1696, block: {cache_hit_count: 15}}}, time_detail: {total_process_time: 1.59ms, total_wait_time: 1.74ms, total_kv_read_wall_time: 2ms, tikv_wall_time: 4.58s} | keep order:false | N/A | N/A |
执行计划.txt (188.3 KB)
