TiDM 同步任务,DDL执行阻塞,wait latest schema version changed

【 TiDB 使用环境】生产环境
【 TiDB 版本】7.1.0
【复现路径】做过哪些操作出现的问题

【遇到的问题:问题现象及影响】
DM同步上游MySQL执行的DDL时候,阻塞
alter table table_a add index idx_taxpayer_number(CH_NUMBER);

[2024/05/09 10:52:50.361 +08:00] [INFO] [ddl_worker.go:1204] ["[ddl] wait latest schema version changed(get the metadata lock if tidb_enable_metadata_lock is true)"] [ver=821386] ["take time"=12m57.081431959s] [job="ID:1610598, Type:add index, State:running, SchemaState:delete only, SchemaID:6313, TableID:7240, RowCount:0, ArgLen:6, start time: 2024-05-09 10:39:53.158 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0, UniqueWarnings:0"]
[2024/05/09 10:52:50.369 +08:00] [INFO] [ddl_worker.go:980] ["[ddl] run DDL job"] [worker="worker 5, tp add index"] [job="ID:1610598, Type:add index, State:running, SchemaState:delete only, SchemaID:6313, TableID:7240, RowCount:0, ArgLen:0, start time: 2024-05-09 10:39:53.158 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0, UniqueWarnings:0"]
[2024/05/09 10:52:50.385 +08:00] [INFO] [domain.go:240] ["diff load InfoSchema success"] [currentSchemaVersion=821386] [neededSchemaVersion=821387] ["start time"=2.597749ms] [gotSchemaVersion=821387] [phyTblIDs="[7240]"] [actionTypes="[7]"]
[2024/05/09 10:52:50.390 +08:00] [INFO] [domain.go:833] ["mdl gets lock, update to owner"] [jobID=1610598] [version=821387]
[2024/05/09 10:52:50.433 +08:00] [INFO] [ddl_worker.go:1204] ["[ddl] wait latest schema version changed(get the metadata lock if tidb_enable_metadata_lock is true)"] [ver=821387] ["take time"=52.453221ms] [job="ID:1610598, Type:add index, State:running, SchemaState:write only, SchemaID:6313, TableID:7240, RowCount:0, ArgLen:6, start time: 2024-05-09 10:39:53.158 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0, UniqueWarnings:0"]
[2024/05/09 10:52:50.441 +08:00] [INFO] [ddl_worker.go:980] ["[ddl] run DDL job"] [worker="worker 4, tp add index"] [job="ID:1610598, Type:add index, State:running, SchemaState:write only, SchemaID:6313, TableID:7240, RowCount:0, ArgLen:0, start time: 2024-05-09 10:39:53.158 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0, UniqueWarnings:0"]

代码位置

【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【附件:截图/日志/监控】

tidb 的 DDL 是动态的切换过程,需要让子弹飞一会…

除非确定 DDL 的 operation 被 lock了,或者 Dead,没到下一步

有两种办法:

  1. DM手动操作跳过这个 DDL 步骤
  2. DDL Job 手动Stop,在找空闲的时间,手工的处理这个 DDL 的过程

7.1.0 有些小bug,建议升级到 最新的小版本,7.5.1…

不敢升级呀,上次从5.X升级到7.1.0发现好几个不兼容的问题

写错了,7.1.5,

大版本有一些变化的(主要是新特性,肯定会有一些影响了),小版本都是修复bug为主了

小版本升级可以直接冲!

元数据锁查了吗,https://docs.pingcap.com/zh/tidb/v6.5/metadata-lock#元数据锁的可观测性

完全查不动,information_schema.ddl_jobs表太大了,连个索引都没有