5.0.4 升级到7.1.5 后ddl执行失败

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】
【复现路径】做过哪些操作出现的问题
【遇到的问题:问题现象及影响】
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【附件:截图/日志/监控】
从 5.0.4 升级到7.1.5 后ddl执行失败

MySQL [(none)]> admin show ddl jobs;
+--------+---------+------------------+---------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+------------+
| JOB_ID | DB_NAME | TABLE_NAME       | JOB_TYPE      | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | CREATE_TIME         | START_TIME          | END_TIME            | STATE
     |
+--------+---------+------------------+---------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+------------+
|   3482 | mysql   | tables_priv      | modify column | public       |         3 |       11 |         0 | 2024-09-06 01:29:30 | 2024-09-06 01:29:30 | NULL                | done
     |
|   3483 | mysql   | stats_buckets    | modify column | none         |         3 |       25 |         0 | 2024-09-06 01:29:30 | NULL                | NULL                | cancelling |
|   3485 | test    | t1               | create table  | none         |         1 |     3484 |         0 | 2024-09-06 14:27:51 | NULL                | NULL                | cancelling |
|   3487 | test    | t2               | create table  | none         |         1 |     3486 |         0 | 2024-09-06 15:14:09 | NULL                | NULL                | queueing   |
|   3481 | mysql   | load_data_jobs   | create table  | public       |         3 |     3480 |         0 | 2024-09-06 01:29:30 | 2024-09-06 01:29:30 | 2024-09-06 01:29:30 | synced     |
|   3479 | mysql   | tidb_global_task | create table  | none         |         3 |     3478 |         0 | 2024-09-06 01:29:30 | 2024-09-06 01:29:30 | 2024-09-06 01:29:30 | cancelled  |
|   3477 | mysql   | columns_priv     | modify column | public       |         3 |       13 |         0 | 2024-09-06 01:29:30 | 2024-09-06 01:29:30 | 2024-09-06 01:29:30 | synced     |
|   3476 | mysql   | tidb_global_task | create table  | public       |         3 |     3474 |         0 | 2024-09-06 01:29:30 | 2024-09-06 01:29:30 | 2024-09-06 01:29:30 | synced     |
|   3475 | mysql   | tidb_mdl_view    | create view   | none         |         3 |     3471 |         0 | 2024-09-06 01:29:30 | 2024-09-06 01:29:30 | 2024-09-06 01:29:30 | cancelled  |
|   3473 | mysql   | tidb_mdl_view    | create view   | public       |         3 |     3472 |         0 | 2024-09-06 01:29:29 | 2024-09-06 01:29:30 | 2024-09-06 01:29:30 | synced     |
|   3470 | mysql   | stats_histograms | modify column | public       |         3 |       23 |         0 | 2024-09-06 01:29:29 | 2024-09-06 01:29:29 | 2024-09-06 01:29:30 | synced     |
|   3469 | mysql   | tidb_mdl_view    | create view   | public       |         3 |     3468 |         0 | 2024-09-06 01:29:29 | 2024-09-06 01:29:29 | 2024-09-06 01:29:29 | synced     |
|   3467 | mysql   | columns_priv     | modify column | public       |         3 |       13 |         0 | 2024-09-06 01:29:29 | 2024-09-06 01:29:29 | 2024-09-06 01:29:29 | synced     |
|   3466 | mysql   | tidb_mdl_view    | create view   | public       |         3 |     3465 |         0 | 2024-09-06 01:29:29 | 2024-09-06 01:29:29 | 2024-09-06 01:29:29 | synced     |
+--------+---------+------------------+---------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+------------+

尝试过切换ddl owner 但未修复。
日志报错
dd7e-493a-8f10-35372ca05fc0] [currentVer=0] [latestVer=3794]
[2024/09/06 03:53:19.046 +08:00] [INFO] [syncer.go:379] [“[ddl] syncer check all versions, someone is not synced, continue checking”] [ddl=/tidb/ddl/all_schema_versions/1960c880-dd7e-493a-8f10-35372ca05fc0] [currentVer=0] [latestVer=3794]
[2024/09/06 03:53:20.122 +08:00] [INFO] [syncer.go:379] [“[ddl] syncer check all versions, someone is not synced, continue checking”] [ddl=/tidb/ddl/all_schema_versions/1960c880-dd7e-493a-8f10-35372ca05fc0] [currentVer=0] [latestVer=3794]
[2024/09/06 03:53:21.196 +08:00] [INFO] [syncer.go:379] [“[ddl] syncer check all versions, someone is not synced, continue checking”] [ddl=/tidb/ddl/all_schema_versions/1960c880-dd7e-493a-8f10-35372ca05fc0] [currentVer=0] [latestVer=3794]
[2024/09/06 03:53:22.265 +08:00] [INFO] [syncer.go:379] [“[ddl] syncer check all versions, someone is not synced, continue checking”] [ddl=/tidb/ddl/all_schema_versions/1960c880-dd7e-493a-8f10-35372ca05fc0] [currentVer=0] [latestVer=3794]
[2024/09/06 03:53:23.378 +08:00] [INFO] [syncer.go:379] [“[ddl] syncer check all versions, someone is not synced, continue checking”] [ddl=/tidb/ddl/all_schema_versions/1960c880-dd7e-493a-8f10-35372ca05fc0] [currentVer=0] [latestVer=3794]

3482 这个job是done状态 ,还没完成schema的变更同步。 看下mysql.tidb_mdl_view 有阻塞信息没

[2024/09/06 01:19:21.089 +08:00] [WARN] [backoff.go:172] [“regionMiss backoffer.maxSleep 40000ms is exceeded, errors:\ntikv disk full: store_id:47953057 reason:"propose failed:
tikv disk full, cmd diskFullOpt=AllowedOnAlmostFull, leader diskUsage=AlreadyFull" ctx: region ID: 8465778522, meta: id:8465778522 start_key:"t\200\000\000\000\000\000\014\307" region_epoch:<conf_ver:109926 version:34697 > peers:<id:8482728204 store_id:18 > peers:<id:8482735266 store_id:47953057 > peers:<id:8482740545 store_id:1218144925 > ,
peer: id:8482735266 store_id:47953057 , addr: 10.52.40.50:20162, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv at 2024-09-06T01:19:06.080188296+08:00\ntikv disk full: store_id:47953057 reason:"propose failed: tikv disk full, cmd diskFullOpt=AllowedOnAlmostFull, leader diskUsage=AlreadyFull" ctx: region ID: 8465778522, meta: id:8465778522 start_key:"t\200\000\000\000\000\000\014\307" region_epoch:<conf_ver:109926 version:34697 > peers:<id:8482728204 store_id:18 > peers:<id:8482735266 store_id:47953057 > peers:<id:8482740545 store_id:1218144925 > , peer: id:8482735266 store_id:47953057 , addr: 10.52.40.50:20162, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv at 2024-09-06T01:19:11.08154653+08:00\ntikv disk full: store_id:47953057 reason:"propose failed: tikv disk full, cmd diskFullOpt=AllowedOnAlmostFull, leader diskUsage=AlreadyFull" ctx: region ID: 8465778522, meta: id:8465778522 start_key:"t\200\000\000\000\000\000\014\307" region_epoch:<conf_ver:109926 version:34697 > peers:<id:8482728204 store_id:18 > peers:<id:8482735266 store_id:47953057 > peers:<id:8482740545 store_id:1218144925 > , peer: id:8482735266 store_id:47953057 , addr: 10.52.40.50:20162, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv at 2024-09-06T01:19:16.084795197+08:00\ntotal-backoff-times: 11, backoff-detail: tikvDiskFull:11\nlongest sleep type: tikvDiskFull, time: 42500ms”]
[2024/09/06 01:19:21.089 +08:00] [WARN] [split_region.go:126] [“[ddl] split table region failed”] [error=“tikv disk full”]

这个kv磁盘爆了?

应该是磁盘满了

errors:\ntikv disk full: store_id:47953057 reason:"propose failed:
tikv disk full, cmd diskFullOpt=AllowedOnAlmostFull,

看着是空间不够了

看日志tikv disk full,磁盘爆了,加加加,加起来


磁盘满了吧 看下日志有多少,是否可以通过清理日志解决,否则就得想办法加磁盘了

磁盘满了,有监控告警的重要性啊

哈哈 重启tidb-server 自动恢复

跟磁盘满 没有一定关系

此话题已在最后回复的 7 天后被自动关闭。不再允许新回复。