【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】V7.6.0
【复现路径】做过哪些操作出现的问题
【遇到的问题：问题现象及影响】
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【附件：截图/日志/监控】
DM同步告警DM数据中台生产同步数据错误!!!
alertname: DM_pending_DDL
status: firing
cluster: datamg_dm_cluster
task: ms_member_prd_s5_cynosdb
instance: 10.8.94.3:8261
level: critical
msg: DDL pending exceed 1h
starttime: 2024-11-05 08:49:53
查看DM同步任务 “result”: true,
“msg”: “”,
“sourceStatus”: {
“source”: “s_ms_member_prd_5”,
“worker”: “dm-10.8.94.102-8280”,
“result”: null,
“relayStatus”: null
},
“subTaskStatus”: [
{
“name”: “ms_member_prd_s5_cynosdb”,
“stage”: “Running”,
“unit”: “Sync”,
“result”: null,
“unresolvedDDLLockID”: “”,
“sync”: {
“totalEvents”: “87715”,
“totalTps”: “0”,
“recentTps”: “0”,
“masterBinlog”: “(mysql-bin.002133, 146207655)”,
“masterBinlogGtid”: “082ba7f6-0353-11e9-b888-6c92bf6298d6:1-1315626032,135efe6a-0353-11e9-b888-6c92bf62e276:1-69,5e191e78-4bcf-11ee-bb1b-0c42a1209f9a:1-175909705,840f2810-fe53-11ed-971b-0c42a149c351:1-183853645,92475832-8ff0-11ed-a2cb-b026284353c0:1-224598306,b0e7a1b2-a84a-11ee-a5be-0c42a172a76a:1-22097579,e468fbc9-b54a-11ee-8dd7-0c42a164302d:1-735606835,f179477d-be95-11eb-884d-b8599f24f968:1-20323”,
“syncerBinlog”: “(mysql-bin.002133, 146121488)”,
“syncerBinlogGtid”: “082ba7f6-0353-11e9-b888-6c92bf6298d6:1-1315626032,135efe6a-0353-11e9-b888-6c92bf62e276:1-69,5e191e78-4bcf-11ee-bb1b-0c42a1209f9a:1-175909705,840f2810-fe53-11ed-971b-0c42a149c351:1-183853645,92475832-8ff0-11ed-a2cb-b026284353c0:1-224598306,b0e7a1b2-a84a-11ee-a5be-0c42a172a76a:1-22097579,e468fbc9-b54a-11ee-8dd7-0c42a164302d:1-735606649,f179477d-be95-11eb-884d-b8599f24f968:1-20323”,
“blockingDDLs”: [
],
“unresolvedGroups”: [
{
“target”: “member_prd.external_card_wechat”,
“DDLs”: [
],
“firstLocation”: “”,
“synced”: [
],
“unsynced”: [
“this DM-worker doesn’t receive any shard DDL of this group”
查看共享DDL锁，确实存在，但是不能解锁 {
“ID”: “ms_member_prd_s5_cynosdb-member_prd.external_card_wechat”,
“task”: “ms_member_prd_s5_cynosdb”,
“mode”: “pessimistic”,
“owner”: “s_ms_member_prd_5”,
“DDLs”: [
“ALTER TABLE member_prd.external_card_wechat ADD COLUMN migration TINYINT(1) NOT NULL DEFAULT ‘1’ COMMENT ‘迁移来源 1:open_account_binding,2:official_account,0:member_wechat’ AFTER subscribe”
],
“synced”: [
“s_ms_member_prd_5”
],
“unsynced”: [
]
}
]
}
» shard-ddl-lock unlock ms_member_prd_s5_cynosdb-member_prd.external_card_wechat
{
“result”: false,
“msg”: “[code=38024:class=dm-master:scope=internal:level=high], Message: the owner s_ms_member_prd_5 of the lock ms_member_prd_s5_cynosdb-member_prd.external_card_wechat has not done the operation”
}
这个怎么处理呢？
1 个赞
同步是正常的，强制解除DDL锁后，就没告警了
1 个赞
这种情况么？