GC调整遇到问题

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:

【TiDB 版本】4.0.12

【问题描述】将tikv_gc_life_time从96H调整到48H后,GC监控页数据全部没了,
集群告警TiKV_scheduler_command_duration_seconds
调整GC流控
tiup ctl:v5.0.1 tikv --host=10.254.212.16:20160 modify-tikv-config -n gc.max_write_bytes_per_sec -v 10MB
后情况依旧没有好转,tikv_gc_last_run_time依旧停留在多日之前


若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

有么有详细日志。

gc leader 对应的 tidb-server的日志
[2021/04/27 13:34:46.630 +08:00] [ERROR] [gc_worker.go:567] ["[gc worker] delete range returns an error"] [uuid=5e40b4970e8000f] [error="[tikv:9005]Region is unavailable"] [2021/04/27 13:34:46.630 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="[tikv:9005]Region is unavailable"] [2021/04/27 13:37:22.984 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="[tikv:9005]Region is unavailable"] [2021/04/27 13:40:09.065 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="[tikv:9002]TiKV server timeout"] [2021/04/27 13:40:09.065 +08:00] [ERROR] [gc_worker.go:192] ["[gc worker] runGCJob"] [error="[tikv:9005]Region is unavailable"] [2021/04/27 13:41:03.121 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="tidb server shutting down"] [2021/04/27 13:41:03.121 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="session pool closed"] [2021/04/27 14:09:33.657 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="[tikv:9002]TiKV server timeout"] [2021/04/27 14:11:17.091 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="[tikv:9005]Region is unavailable"] [2021/04/27 14:13:34.357 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="[tikv:9002]TiKV server timeout"] [2021/04/27 14:15:53.935 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="[tikv:9005]Region is unavailable"] [2021/04/27 14:18:20.249 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="[tikv:9002]TiKV server timeout"] [2021/04/27 14:20:37.518 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="[tikv:9002]TiKV server timeout"] [2021/04/27 14:22:52.103 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="[tikv:9002]TiKV server timeout"] [2021/04/27 14:25:21.358 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="[tikv:9005]Region is unavailable"] [2021/04/27 14:27:37.072 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="tidb server shutting down"] [2021/04/27 14:27:37.073 +08:00] [WARN] [gc_worker.go:218] ["[gc worker] check leader"] [error="tidb server shutting down"] [2021/04/29 15:41:09.457 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:42:09.461 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:43:09.454 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:44:09.461 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:45:09.448 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:46:09.457 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:47:09.447 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:48:09.448 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:49:09.447 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:50:09.447 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:51:09.460 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:52:09.452 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:53:09.455 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:54:09.446 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:55:09.461 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"] [2021/04/29 15:56:09.453 +08:00] [WARN] [gc_worker.go:225] ["[gc worker] leader tick"] [error="time: missing unit in duration 48"]

执行下 select VARIABLE_NAME, VARIABLE_VALUE from mysql.tidb where VARIABLE_NAME like "tikv_gc%"; 看下 GC 相关配置


看下是不是配置的单位有问题

太尴尬了 抱歉,没有仔细检查。:joy:

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。