为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。
- 【TiDB 版本】: docker-compose版本,DM v1.0.3版本(Ansible部署)
- 【问题描述】:
问题
问题一
start-task tasks/task_merge.yml
{
"result": false,
"msg": "[code=38008:class=dm-master:scope=internal:level=high] fetch config of worker 172.26.8.82:8262: grpc request error: rpc error: code = DeadlineExceeded desc = context deadline exceeded\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/dm/master/workerrpc.callRPC\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/master/workerrpc/rawgrpc.go:124\
github.com/pingcap/dm/dm/master/workerrpc.(*GRPCClient).SendRequest\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/master/workerrpc/rawgrpc.go:64\
github.com/pingcap/dm/dm/master.(*Server).getWorkerConfigs.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/master/server.go:1756\
github.com/pingcap/dm/dm/master.(*AgentPool).Emit\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/master/agent_pool.go:117\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
"workers": [
]
}
问题二
query-status task_merge
{
"result": true,
"msg": "",
"workers": [
{
"result": true,
"worker": "172.26.8.83:8262",
"msg": "",
"subTaskStatus": [
{
"name": "task_merge",
"stage": "Paused",
"unit": "Sync",
"result": {
"isCanceled": false,
"errors": [
{
"Type": "UnknownError",
"msg": "[code=11029:class=functional:scope=internal:level=high] get stat for relay log /home/tidb/deploy133/relay_log/dfcf069c-976c-11e6-9ebe-305a3a5a429f.000001/mysql-bin.000103: stat /home/tidb/deploy133/relay_log/dfcf069c-976c-11e6-9ebe-305a3a5a429f.000001/mysql-bin.000103: no such file or directory\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).checkRelayPos\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:98\
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).StartSync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:122\
github.com/pingcap/dm/syncer.(*localBinlogReader).generateStreamer\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:102\
github.com/pingcap/dm/syncer.(*Syncer).Run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1079\
github.com/pingcap/dm/syncer.(*Syncer).Process\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:598\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
"error": null
}
],
"detail": null
},
"unresolvedDDLLockID": "",
"sync": {
"totalEvents": "0",
"totalTps": "0",
"recentTps": "0",
"masterBinlog": "(mysql-bin.000001, 27353658)",
"masterBinlogGtid": "4f51782c-b270-11e7-93f6-5254002147c0:508480007-509375779:513595970-521566909,adb758e5-081a-11e9-8c70-525400dfdd97:1-4,c1bc3c0e-fc5e-11e8-a45c-5254007af3b6:3-743071983",
"syncerBinlog": "(mysql-bin|000001.000103, 521435566)",
"syncerBinlogGtid": "",
"blockingDDLs": [
],
"unresolvedGroups": [
],
"synced": false
}
}
],
"relayStatus": {
"masterBinlog": "(mysql-bin.000001, 27353658)",
"masterBinlogGtid": "4f51782c-b270-11e7-93f6-5254002147c0:508480007-509375779:513595970-521566909,adb758e5-081a-11e9-8c70-525400dfdd97:1-4,c1bc3c0e-fc5e-11e8-a45c-5254007af3b6:3-743071983",
"relaySubDir": "dfcf069c-976c-11e6-9ebe-305a3a5a429f.000001",
"relayBinlog": "(mysql-bin.000001, 27353658)",
"relayBinlogGtid": "",
"relayCatchUpMaster": true,
"stage": "Running",
"result": null
},
"sourceID": "replica-133"
},
{
"result": true,
"worker": "172.26.8.83:8263",
"msg": "",
"subTaskStatus": [
{
"name": "task_merge",
"stage": "Paused",
"unit": "Sync",
"result": {
"isCanceled": false,
"errors": [
{
"Type": "UnknownError",
"msg": "[code=11029:class=functional:scope=internal:level=high] get stat for relay log /home/tidb/deploy135/relay_log/54d11464-77ee-11e7-b052-9c5c8e7240e6.000001/mysql-bin.000257: stat /home/tidb/deploy135/relay_log/54d11464-77ee-11e7-b052-9c5c8e7240e6.000001/mysql-bin.000257: no such file or directory\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).checkRelayPos\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:98\
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).StartSync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:122\
github.com/pingcap/dm/syncer.(*localBinlogReader).generateStreamer\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:102\
github.com/pingcap/dm/syncer.(*Syncer).Run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1079\
github.com/pingcap/dm/syncer.(*Syncer).Process\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:598\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
"error": null
}
],
"detail": null
},
"unresolvedDDLLockID": "",
"sync": {
"totalEvents": "0",
"totalTps": "0",
"recentTps": "0",
"masterBinlog": "(mysql-bin.000001, 43313276)",
"masterBinlogGtid": "07efafff-80cd-11e8-a628-52540038247b:104-109,4f51782c-b270-11e7-93f6-5254002147c0:1149040627-1183411505,54d11464-77ee-11e7-b052-9c5c8e7240e6:1-74940,8614aa74-6670-11e9-803d-5254005f2010:1-7,f5633113-666f-11e9-aeac-52540009d31a:1-287611299",
"syncerBinlog": "(mysql-bin|000001.000257, 474543098)",
"syncerBinlogGtid": "",
"blockingDDLs": [
],
"unresolvedGroups": [
],
"synced": false
}
}
],
"relayStatus": {
"masterBinlog": "(mysql-bin.000001, 43313276)",
"masterBinlogGtid": "54d11464-77ee-11e7-b052-9c5c8e7240e6:1-74940,8614aa74-6670-11e9-803d-5254005f2010:1-7,f5633113-666f-11e9-aeac-52540009d31a:1-287611299,07efafff-80cd-11e8-a628-52540038247b:104-109,4f51782c-b270-11e7-93f6-5254002147c0:1149040627-1183411505",
"relaySubDir": "54d11464-77ee-11e7-b052-9c5c8e7240e6.000001",
"relayBinlog": "(mysql-bin.000001, 43313276)",
"relayBinlogGtid": "",
"relayCatchUpMaster": true,
"stage": "Running",
"result": null
},
"sourceID": "replica-135"
}
]
}
配置
inventory.ini
[dm_master_servers]
dm_master ansible_host=172.26.8.81
[dm_worker_servers]
dm-worker82_132 ansible_host=172.26.8.82 source_id="replica-132" mysql_host=172.23.0.132 mysql_user=root mysql_password='jFSwbMR6kC78GQnRffyrb0Cx1646E5hccOVc' mysql_port=5688
dm-worker83_133 ansible_host=172.26.8.83 source_id="replica-133" mysql_host=172.23.0.133 mysql_user=root mysql_password='jFSwbMR6kC78GQnRffyrb0Cx1646E5hccOVc' mysql_port=3306 deploy_dir=/home/tidb/deploy133 dm_worker_port=8262
dm-worker83_135 ansible_host=172.26.8.83 source_id="replica-135" mysql_host=172.23.0.135 mysql_user=root mysql_password='jFSwbMR6kC78GQnRffyrb0Cx1646E5hccOVc' mysql_port=5688 deploy_dir=/home/tidb/deploy135 dm_worker_port=8263
[dm_portal_servers]
dm_portal ansible_host=172.26.8.81
[prometheus_servers]
prometheus ansible_host=172.26.8.81
[grafana_servers]
grafana ansible_host=172.26.8.81
[alertmanager_servers]
alertmanager ansible_host=172.26.8.81
[all:vars]
cluster_name = test-cluster
ansible_user = tidb
dm_version = v1.0.3
deploy_dir = /home/tidb/deploy
grafana_admin_user = "admin"
grafana_admin_password = "admin"
hosts.ini
[servers]
172.26.8.81
172.26.8.82
172.26.8.83
[all:vars]
username = tidb
task_merge.yml
name: task_merge
task-mode: all
is-sharding: true
ignore-checking-items: ["auto_increment_ID"]
target-database:
host: "172.24.28.1"
port: 4000
user: "root"
password: ""
mysql-instances:
-
source-id: "replica-132"
black-white-list: "bw_list"
-
source-id: "replica-133"
black-white-list: "bw_list"
-
source-id: "replica-135"
black-white-list: "bw_list"
black-white-list:
bw_list:
do-dbs: ["tidb_task"]
做过的努力
- 停止任务,停止DM集群,清除所有dm-master&dm-worker的deploy目录,重新deploy&start DM集群,还是有这个问题。
- telnet 172.26.8.82 8262 没问题。