select或者set操作都报错: ERROR 9006 (HY000): GC life time is shorter than transaction duration, transaction starts at 2023-10-31 08:55:09.646 +0000 UTC, GC safe point is 2140-03-23 09:32:10.849 +0000 UTC

Zz_zZ · 2023 年10 月 31 日 09:27

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】TiDB-v5.2.2
【复现路径】暂无法在测试环境复现
【遇到的问题：问题现象及影响】:
执行select set等sql时都报错 9006 ，未找到论坛内类似9006问题，由于无法set 也无法修改tidb_gc_life_time变量
打开tidb的debug日志显示：

[2023/10/31 09:25:36.411 +00:00] [DEBUG] [ddl_worker.go:179] ["[ddl] wait to check DDL status again"] [worker="worker 2, tp add index"] [interval=1s]
[2023/10/31 09:25:36.412 +00:00] [DEBUG] [ddl_worker.go:179] ["[ddl] wait to check DDL status again"] [worker="worker 1, tp general"] [interval=1s]
[2023/10/31 09:25:36.412 +00:00] [DEBUG] [ddl.go:220] ["[ddl] check whether is the DDL owner"] [isOwner=true] [selfID=03f10a9b-0eb2-497c-9cc8-6ae9ff12db2a]
[2023/10/31 09:25:36.412 +00:00] [DEBUG] [ddl.go:220] ["[ddl] check whether is the DDL owner"] [isOwner=true] [selfID=03f10a9b-0eb2-497c-9cc8-6ae9ff12db2a]
[2023/10/31 09:25:36.412 +00:00] [DEBUG] [txn.go:431] ["[kv] rollback txn"] [txnStartTS=445315635319930881]
[2023/10/31 09:25:36.412 +00:00] [WARN] [ddl_worker.go:199] ["[ddl] handle DDL job failed"] [worker="worker 2, tp add index"] [error="[tikv:9006]GC life time is shorter than transaction duration, transaction starts at 2023-10-31 09:25:36.395 +0000 UTC, GC safe point is 21
40-03-23 09:32:10.849 +0000 UTC"]

【资源配置】:
cpu: 2
memory: 8Gi

pd.log (43.9 KB)
tidb.log (10.6 MB)
tikv.log (15.6 MB)

Soysauce520 · 2023 年10 月 31 日 10:00

监控中gc进程时间是正常走的吗

Miracle · 2023 年10 月 31 日 10:57

gc现在是多少？

大飞哥online · 2023 年10 月 31 日 11:14

看下，是不是有DDL hang住了

Zz_zZ · 2023 年10 月 31 日 12:04

感谢关注～

这个是dashboard的情况

这个是metrics接口查询的顺时监控
metric.log (446.7 KB)

Zz_zZ · 2023 年10 月 31 日 12:15

感谢关注～

这个是grafana dashbaord的情况

这些指标我不太了解，看起来job的延时分布好像都在延时很高的部分

# HELP tidb_ddl_deploy_syncer_duration_seconds Bucketed histogram of processing time (s) of deploy syncer
# TYPE tidb_ddl_deploy_syncer_duration_seconds histogram
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="0.001"} 0
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="0.002"} 0
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="0.004"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="0.008"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="0.016"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="0.032"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="0.064"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="0.128"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="0.256"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="0.512"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="1.024"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="2.048"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="4.096"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="8.192"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="16.384"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="32.768"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="65.536"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="131.072"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="262.144"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="524.288"} 1
tidb_ddl_deploy_syncer_duration_seconds_bucket{result="ok",type="init",le="+Inf"} 1
tidb_ddl_deploy_syncer_duration_seconds_sum{result="ok",type="init"} 0.002846399
tidb_ddl_deploy_syncer_duration_seconds_count{result="ok",type="init"} 1
# HELP tidb_ddl_update_self_ver_duration_seconds Bucketed histogram of processing time (s) of update self version
# TYPE tidb_ddl_update_self_ver_duration_seconds histogram
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="0.001"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="0.002"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="0.004"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="0.008"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="0.016"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="0.032"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="0.064"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="0.128"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="0.256"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="0.512"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="1.024"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="2.048"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="4.096"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="8.192"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="16.384"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="32.768"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="65.536"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="131.072"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="262.144"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="524.288"} 1
tidb_ddl_update_self_ver_duration_seconds_bucket{result="ok",le="+Inf"} 1
tidb_ddl_update_self_ver_duration_seconds_sum{result="ok"} 0.000376232
tidb_ddl_update_self_ver_duration_seconds_count{result="ok"} 1
# HELP tidb_ddl_worker_operation_total Counter of creating ddl/worker and isowner.
# TYPE tidb_ddl_worker_operation_total counter
tidb_ddl_worker_operation_total{type="create_ddl_instance"} 1
tidb_ddl_worker_operation_total{type="create_ddl_worker 1, tp general"} 1
tidb_ddl_worker_operation_total{type="create_ddl_worker 2, tp add index"} 1
tidb_ddl_worker_operation_total{type="owner_v5.2.2"} 13258
tidb_ddl_worker_operation_total{type="start_clean_work"} 1
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="0.0005"} 7026
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="0.001"} 13168
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="0.002"} 13221
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="0.004"} 13237
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="0.008"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="0.016"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="0.032"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="0.064"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="0.128"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="0.256"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="0.512"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="1.024"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="2.048"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="4.096"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="8.192"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="16.384"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="32.768"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="65.536"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="131.072"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="262.144"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="524.288"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="1048.576"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="2097.152"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="4194.304"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="8388.608"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="16777.216"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="33554.432"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="67108.864"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="134217.728"} 13240
tidb_meta_operation_duration_seconds_bucket{result="err",type="get_ddl_job",le="+Inf"} 13240
tidb_meta_operation_duration_seconds_sum{result="err",type="get_ddl_job"} 6.707527507000024
tidb_meta_operation_duration_seconds_count{result="err",type="get_ddl_job"} 13240
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="0.0005"} 16
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="0.001"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="0.002"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="0.004"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="0.008"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="0.016"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="0.032"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="0.064"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="0.128"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="0.256"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="0.512"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="1.024"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="2.048"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="4.096"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="8.192"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="16.384"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="32.768"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="65.536"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="131.072"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="262.144"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="524.288"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="1048.576"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="2097.152"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="4194.304"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="8388.608"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="16777.216"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="33554.432"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="67108.864"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="134217.728"} 18
tidb_meta_operation_duration_seconds_bucket{result="ok",type="get_ddl_job",le="+Inf"} 18
tidb_meta_operation_duration_seconds_sum{result="ok",type="get_ddl_job"} 0.007464113
tidb_meta_operation_duration_seconds_count{result="ok",type="get_ddl_job"} 18
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="0.0005"} 0
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="0.001"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="0.002"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="0.004"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="0.008"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="0.016"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="0.032"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="0.064"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="0.128"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="0.256"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="0.512"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="1.024"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="2.048"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="4.096"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="8.192"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="16.384"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="32.768"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="65.536"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="131.072"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="262.144"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="524.288"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="1048.576"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39",le="+Inf"} 1
tidb_owner_new_session_duration_seconds_sum{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39"} 0.000842373
tidb_owner_new_session_duration_seconds_count{result="ok",type="[ddl-syncer] /tidb/ddl/all_schema_versions/72a55cc4-fa94-4dd2-9ed6-b7630eebcd39"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="0.0005"} 0
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="0.001"} 0
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="0.002"} 0
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="0.004"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="0.008"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="0.016"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="0.032"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="0.064"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="0.128"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="0.256"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="0.512"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="1.024"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="2.048"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="4.096"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="8.192"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="16.384"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="32.768"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="65.536"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="131.072"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="262.144"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="524.288"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="1048.576"} 1
tidb_owner_new_session_duration_seconds_bucket{result="ok",type="[ddl] /tidb/ddl/fg/owner",le="+Inf"} 1
tidb_owner_new_session_duration_seconds_sum{result="ok",type="[ddl] /tidb/ddl/fg/owner"} 0.002533479
tidb_owner_new_session_duration_seconds_count{result="ok",type="[ddl] /tidb/ddl/fg/owner"} 1

Zz_zZ · 2023 年10 月 31 日 12:17

参考文章收集gc相关信息【SOP 系列 25】GC 常见问题排查

gc参数配置，select报错获取不到tidb表

MySQL [(none)]> select * from mysql.tidb;
ERROR 9006 (HY000): GC life time is shorter than transaction duration, transaction starts at 2023-10-31 12:09:23.145 +0000 UTC, GC safe point is 2140-03-23 09:32:10.849 +0000 UTC
MySQL [(none)]>
MySQL [(none)]>
MySQL [(none)]>
MySQL [(none)]> show variables like '%gc%';
+------------------------+--------+
| Variable_name          | Value  |
+------------------------+--------+
| tidb_gc_concurrency    | -1     |
| tidb_gc_enable         | ON     |
| tidb_gc_life_time      | 10m0s  |
| tidb_gc_run_interval   | 10m0s  |
| tidb_gc_scan_lock_mode | LEGACY |
+------------------------+--------+
5 rows in set (0.007 sec)

查看tidb日志中gc-worker日志

[2023/10/31 10:16:26.140 +00:00] [INFO] [gc_worker.go:197] ["[gc worker] start"] [uuid=62e14f023bc0002]
[2023/10/31 10:16:26.144 +00:00] [DEBUG] [gc_worker.go:1826] ["[gc worker] load kv"] [key=tikv_gc_leader_uuid] [value=629bd7ea8d80002]
[2023/10/31 10:16:26.144 +00:00] [DEBUG] [gc_worker.go:1684] ["[gc worker] got leader"] [uuid=629bd7ea8d80002]
[2023/10/31 10:16:26.146 +00:00] [WARN] [gc_worker.go:276] ["[gc worker] check leader"] [error="inconsistent index PRIMARY handle count 1 isn't equal to value count 0"]
[2023/10/31 10:17:26.148 +00:00] [WARN] [gc_worker.go:276] ["[gc worker] check leader"] [error="[tikv:9006]GC life time is shorter than transaction duration, transaction starts at 2023-10-31 10:17:26.145 +0000 UTC, GC safe point is 2140-03-23 09:32:10.849 +0000 UTC"]
[2023/10/31 10:18:26.148 +00:00] [WARN] [gc_worker.go:276] ["[gc worker] check leader"] [error="[tikv:9006]GC life time is shorter than transaction duration, transaction starts at 2023-10-31 10:18:26.146 +0000 UTC, GC safe point is 2140-03-23 09:32:10.849 +0000 UTC"]

查看service-gc-safepoint

{
  "service_gc_safe_points": [
    {
      "service_id": "gc_worker",
      "expired_at": 9223372036854775807,
      "safe_point": 1408180297622880256
    }
  ],
  "gc_safe_point": 1408180297622880256
}

Miracle · 2023 年10 月 31 日 12:43

这个gc safe point的时间是不是有点穿越了。。。

大飞哥online · 2023 年10 月 31 日 13:04

集群时间同步看看正常不

Zz_zZ · 2023 年11 月 1 日 02:03

生产环境是单机环境，可能存在时间不同步的情况，明天去现场了，我再检查下系统日志看下时间ntpdate操作

Zz_zZ · 2023 年11 月 1 日 02:04

我把pd tikv数据拷贝回测试环境，测试环境的时间是正常的，怀疑可能是之前某个时间不一致

有猫万事足 · 2023 年11 月 1 日 02:23

select tidb_parse_tso(1408180297622880256) -- 2140-03-23 17:32:10.849000

你的税确实是收到2140年去了。

TiDBer_小阿飞 · 2023 年11 月 1 日 02:32

哈哈！还有117年就到了，也不远，也就下下辈子的事情

Zz_zZ · 2023 年11 月 1 日 05:29

这个tso 专栏 - PD的时钟服务——TSO | TiDB 社区像这种情况时间超前的话，怎么恢复呀，不能强制修改吧

如果修改系统时间到2140年它会恢复吗，恢复正常后再改回正确时间，这种时间穿越 pd会跪的更彻底吧😂
要么就是记录pd的clusertid 通过https://docs.pingcap.com/zh/tidb/stable/pd-recover#第-3-步使用-pd-recover-修复元数据这个文档强制重建pd？

有猫万事足 · 2023 年11 月 1 日 05:34

我也没有找到修改safepoint的方法。
如果超出了，可以肯定的是后续的查询修改这些应该都不成功。以前的数据应该还在，但是为了把safepoint调回来，可能需要重建集群了，重新导入数据了。

你的第二条是个值得尝试的方法，但是我没有实际做过。所以不清楚后果是如何的。

Zz_zZ · 2023 年11 月 1 日 08:04

测试环境尝试pd-recover修复元数据方案，初步测试成功了

获取pd-id

cat pd.log |grep "init cluster id"
[2023/11/01 07:35:58.449 +00:00] [INFO] [server.go:351] ["init cluster id"] [cluster-id=7275302868813208333]

停掉pd tikv tidb然后删除pd的数据

rm -rf /mnt/locals/tidb-pd/volume0/*

启动新的pd tikv 时 pd重新初始化，tikv报错

["failed to bootstrap node id: \"[src/server/node.rs:236]: cluster ID mismatch, local 7275302868813208333 != remote 7296392347708071649, you are trying to connect to another cluster, please reconnect to the correct PD\""]

使用pd-recover 恢复cluster-id

wget https://download.pingcap.org/tidb-community-toolkit-v5.2.2-linux-amd64.tar.gz
tar zxf tidb-community-toolkit-v5.2.2-linux-amd64.tar.gz
cd  tidb-community-toolkit-v5.2.2-linux-amd64
./bin/pd-recover -endpoints http://127.0.0.1:2379 -cluster-id 7275302868813208333 -alloc-id 10000

然后重启pd和tikv都正常启动，查看pd的safe-point

./pd-ctl service-gc-safepoint
{
  "service_gc_safe_points": [],
  "gc_safe_point": 0
}

进入tidb执行select查询 ,不再报错9006

 mysql -h 127.0.0.1 -P 4000 -u root
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 403
Server version: 5.7.25-TiDB-v5.2.2 TiDB Server (Apache License 2.0) Community Edition, MySQL 5.7 compatible

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> select * from mysql.tidb;
+--------------------------+---------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+
| VARIABLE_NAME            | VARIABLE_VALUE                                                                                    | COMMENT                                                                                     |
+--------------------------+---------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+
| bootstrapped             | True                                                                                              | Bootstrap flag. Do not delete.                                                              |
| tidb_server_version      | 72                                                                                                | Bootstrap version. Do not delete.                                                           |
| system_tz                | UTC                                                                                               | TiDB Global System Timezone.                                                                |
| new_collation_enabled    | False                                                                                             | If the new collations are enabled. Do not edit it.                                          |
| tikv_gc_leader_uuid      | 629bd7ea8d80002                                                                                   | Current GC worker leader UUID. (DO NOT EDIT)                                                |
| tikv_gc_leader_desc      | host:tidb-default-tidb-0, pid:1, start at 2023-09-07 11:20:55.101415326 +0000 UTC m=+42.568797401 | Host name and pid of current GC leader. (DO NOT EDIT)                                       |
| tikv_gc_enable           | true                                                                                              | Current GC enable status                                                                    |
| tikv_gc_run_interval     | 10m0s                                                                                             | GC run interval, at least 10m, in Go format.                                                |
| tikv_gc_life_time        | 10m0s                                                                                             | All versions within life time will not be collected by GC, at least 10m, in Go format.      |
| tikv_gc_auto_concurrency | true                                                                                              | Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used |
| tikv_gc_scan_lock_mode   | legacy                                                                                            | Mode of scanning locks, "physical" or "legacy"                                              |
| tikv_gc_mode             | distributed                                                                                       | Mode of GC, "central" or "distributed"                                                      |
+--------------------------+---------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+
12 rows in set (0.002 sec)

但是最后有一个疑问 pd启动等待多个gc 10分钟后这个safe-point仍然为空，这个何时会触发更新？

system · 2023 年12 月 31 日 08:04

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。