DLL操作卡死

🪐 TiDB 技术问题 性能调优
1

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：
【 TiDB 使用环境】
生产环境，用于数据仓库

【概述】 DDL操作卡死，admin cancel ddl jobs xxx,报error: [admin:6]This job:6136158 is almost finished, can’t be cancelled now

【背景】 truncate table

【现象】 ddl卡死

【问题】 当前遇到的问题

【业务影响】 无法正常运行

【TiDB 版本】 3.1

[tidb@bi-mysql_30_186 tidb-ansible]$ curl http://172.29.29.12:10080/info/all
{
“servers_num”: 3,
“owner_id”: “47dff6dc-c74a-4b3d-8041-997e036cf9a3”,
“is_all_server_version_consistent”: true,
“all_servers_info”: {
“47dff6dc-c74a-4b3d-8041-997e036cf9a3”: {
“version”: “5.7.25-TiDB-v3.0.9”,
“git_hash”: “bf155a7773d8978e7f80e2c7055c46da73c8bcc3”,
“ddl_id”: “47dff6dc-c74a-4b3d-8041-997e036cf9a3”,
“ip”: “172.29.29.12”,
“listening_port”: 4000,
“status_port”: 10080,
“lease”: “45s”,
“binlog_status”: “Off”
},
“50eed940-d177-4da1-a4c4-335648a64bcd”: {
“version”: “5.7.25-TiDB-v3.0.9”,
“git_hash”: “bf155a7773d8978e7f80e2c7055c46da73c8bcc3”,
“ddl_id”: “50eed940-d177-4da1-a4c4-335648a64bcd”,
“ip”: “172.29.29.16”,
“listening_port”: 4000,
“status_port”: 10080,
“lease”: “45s”,
“binlog_status”: “Off”
},
“5f896284-7e9a-480e-9d39-b7889f93b56c”: {
“version”: “5.7.25-TiDB-v3.0.9”,
“git_hash”: “bf155a7773d8978e7f80e2c7055c46da73c8bcc3”,
“ddl_id”: “5f896284-7e9a-480e-9d39-b7889f93b56c”,
“ip”: “172.30.30.185”,
“listening_port”: 4000,
“status_port”: 10080,
“lease”: “45s”,
“binlog_status”: “Off”
}
}

2

[2022/03/17 17:05:15.980 +08:00] [INFO] [ddl_worker.go:502] ["[ddl] run DDL job"] [worker=“worker 1, tp general”] [job=“ID:6136158, Type:drop table, State:running, SchemaState:delete only, SchemaID:8256, TableID:6135924, RowCount:0, ArgLen:0, start time: 2022-03-17 09:24:42.715 +0800 CST, Err:[ddl:101]invalid table state none, ErrCount:24157, SnapshotVersion:0”]
[2022/03/17 17:05:15.983 +08:00] [ERROR] [ddl_worker.go:588] ["[ddl] run DDL job error"] [worker=“worker 1, tp general”] [error="[ddl:101]invalid table state none"] [errorVerbose="[ddl:101]invalid table state none\ngithub.com/pingcap/errors.AddStack\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/pkg/mod/github.com/pingcap/errors@v0.11.4/errors.go:174\ github.com/pingcap/parser/terror.(*Error).GenWithStack\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20200108075733-73763c7a9133/terror/terror.go:231\ngithub.com/pingcap/tidb/ddl.onDropTableOrView\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/table.go:179\ github.com/pingcap/tidb/ddl.(*worker).runDDLJob\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:531\ngithub.com/pingcap/tidb/ddl.(*worker).handleDDLJobQueue.func1.1\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:413\ngithub.com/pingcap/tidb/util.WithRecovery\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/util/misc.go:91\ github.com/pingcap/tidb/ddl.(*worker).handleDDLJobQueue.func1\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:412\ngithub.com/pingcap/tidb/kv.RunInNewTxn\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/kv/txn.go:50\ github.com/pingcap/tidb/ddl.(*worker).handleDDLJobQueue\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:375\ngithub.com/pingcap/tidb/ddl.(*worker).start\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:140\ngithub.com/pingcap/tidb/ddl.(*ddl).start.func1\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/ddl.go:443\ngithub.com/pingcap/tidb/util.WithRecovery\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/util/misc.go:91\ runtime.goexit\ \t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack=“github.com/pingcap/tidb/ddl.(*worker).runDDLJob\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:588\ngithub.com/pingcap/tidb/ddl.(*worker).handleDDLJobQueue.func1.1\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:413\ngithub.com/pingcap/tidb/util.WithRecovery\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/util/misc.go:91\ github.com/pingcap/tidb/ddl.(*worker).handleDDLJobQueue.func1\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:412\ngithub.com/pingcap/tidb/kv.RunInNewTxn\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/kv/txn.go:50\ github.com/pingcap/tidb/ddl.(*worker).handleDDLJobQueue\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:375\ngithub.com/pingcap/tidb/ddl.(*worker).start\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:140\ngithub.com/pingcap/tidb/ddl.(*ddl).start.func1\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/ddl/ddl.go:443\ngithub.com/pingcap/tidb/util.WithRecovery\ \t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb/util/misc.go:91”]

3

同问，这里的almost判断的标准是什么？

4

重建了5.4的tidb，然后重建了数仓，这个问题无法解决

5

可以参考下这个 FAQ 排查一下，另外你的版本是 3.1 不推荐使用哈，建议升级一下。

6

该主题在最后一个回复创建后60天后自动关闭。不再允许新的回复。