drop 大表空间未回收gc 时间不变

🪐 TiDB 技术问题 性能调优
1

为提高效率，提问时请提供以下信息，问题描述清晰可优先响应。

  • 【TiDB 版本】：Release Version: v3.0.2
  • 【问题描述】：1、2020-11-30 12:11:44 执行drop 大表后发现空间未回收
    2、查看gc 如下
    | tikv_gc_leader_uuid | 5bbe04616500008 |
    | tikv_gc_leader_desc | host:tidb-pd3, pid:26129, start at 2019-12-12 11:05:28.223028091 +0800 CST m=+0.101313748 |
    | tikv_gc_leader_lease | 20201130-17:03:29 +0800 |
    | tikv_gc_run_interval | 10m0s |
    | tikv_gc_life_time | 10m |
    | tikv_gc_last_run_time | 20201130-12:28:30 +0800 |
    | tikv_gc_safe_point | 20201130-12:18:30 +0800 |
    | tikv_gc_concurrency | 3 |
    | tikv_gc_enable | true |
    | tikv_gc_auto_concurrency | true |
    | tikv_gc_mode | distributed |

3、查看tidb 日志

[2020/11/30 12:10:31.080 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 12:11:31.046 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 12:12:30.353 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 12:13:30.256 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 12:14:30.364 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 12:15:30.362 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 12:16:30.324 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 12:16:37.566 +08:00] [INFO] [gc_worker.go:794] ["[gc worker] finish resolve locks"] [uuid=5bbe04616500008] [safePoint=421184533931491328] [regions=295093]
[2020/11/30 12:17:30.487 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 12:18:17.831 +08:00] [INFO] [gc_worker.go:582] ["[gc worker] start delete"] [uuid=5bbe04616500008] [ranges=0]
[2020/11/30 12:18:17.831 +08:00] [INFO] [gc_worker.go:601] ["[gc worker] finish delete ranges"] [uuid=5bbe04616500008] [“num of ranges”=0] [“cost time”=627ns]
[2020/11/30 12:18:18.131 +08:00] [INFO] [gc_worker.go:624] ["[gc worker] start redo-delete ranges"] [uuid=5bbe04616500008] [“num of ranges”=0]
[2020/11/30 12:18:18.131 +08:00] [INFO] [gc_worker.go:643] ["[gc worker] finish redo-delete ranges"] [uuid=5bbe04616500008] [“num of ranges”=0] [“cost time”=457ns]
[2020/11/30 12:18:18.978 +08:00] [INFO] [gc_worker.go:921] ["[gc worker] sent safe point to PD"] [uuid=5bbe04616500008] [“safe point”=421184533931491328]
[2020/11/30 12:18:31.679 +08:00] [INFO] [gc_worker.go:263] ["[gc worker] another gc job has just finished, skipped."] [“leaderTick on “=5bbe04616500008]
[2020/11/30 12:28:31.265 +08:00] [INFO] [gc_worker.go:277] [”[gc worker] starts the whole job”] [uuid=5bbe04616500008] [safePoint=421189362663292928] [concurrency=6]
[2020/11/30 12:28:31.265 +08:00] [INFO] [gc_worker.go:773] ["[gc worker] start resolve locks"] [uuid=5bbe04616500008] [safePoint=421189362663292928] [concurrency=6]

若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出打印结果，请务必全选并复制粘贴上传。

2

[2020/11/30 17:37:30.281 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 17:38:29.749 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 17:39:30.401 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 17:40:29.760 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 17:41:30.491 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 17:42:30.566 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]
[2020/11/30 17:43:30.319 +08:00] [INFO] [gc_worker.go:246] ["[gc worker] there’s already a gc job running, skipped"] [“leaderTick on”=5bbe04616500008]

可能是这个问题，请教大佬们如何销解决？

3

[2020/11/30 12:28:31.265 +08:00] [INFO] [gc_worker.go:773] ["[gc worker] start resolve locks"] [uuid=5bbe04616500008] [safePoint=421189362663292928] [concurrency=6]

这条 log 显示 新的 GC work 已经开始，safe point 是 11-30 12:18:30

17点 43 分 log 显示 GC worker 正在进行 可以再关注下

tikv_gc_safe_point 是否已经推进

4

空间已回收，谢谢！ 没想到要这么久:grinning:

5

看日志 涉及的 region count 有 29w 只多 不知是否集群写入很大。还是之前 GC lift time 进行过调整。可以关注下 tikv detail 中 gc 相关监控

https://docs.pingcap.com/zh/tidb/stable/grafana-tikv-dashboard#gc

image
image1482×1042 160 KB