DM-work报错[ERROR] [subtask.go:419] ["wait condition"] [subtask=test5] [error="[code=40067:class=dm-worker:scope=internal:level=high] wait relay catchup timeout, loader end binlog pos: (mysql-bin.000005, 18712), relay binlog pos: (, 4)"]

为提高效率,提问时请尽量提供详细背景信息,问题描述清晰可优先响应。以下信息点请尽量提供:

  • 系统版本 & kernel 版本】centos7
  • TiDB 版本】v3.0.1
  • 磁盘型号】sata
  • 集群节点分布】3pd 3tikv 2tidb k8s部署
  • 数据量 & region 数量 & 副本数】测试dm用的,数据量不大
  • 问题描述(我做了什么)】 dm安装是通过ansible的方式 创建一个同步任务通过web-ui,任务报错
name: test5
task-mode: all
is-sharding: false
target-database:
  host: tidb.xxx.xx
  port: 3306
  user: root
  password: xxx
mysql-instances:
- source-id: mysql-replica-01
  meta:
    binlog-name: mysql-bin.0000001
    binlog-pos: 4
  filter-rules: []
  route-rules:
  - mysql-replica-01.route_rules.1
  - mysql-replica-01.route_rules.2
  black-white-list: mysql-replica-01.bw_list.1
  mydumper-config-name: mysql-replica-01.dump
routes:
  mysql-replica-01.route_rules.1:
    schema-pattern: dm_test
    table-pattern: ""
    target-schema: dm_test
    target-table: ""
  mysql-replica-01.route_rules.2:
    schema-pattern: dm_test
    table-pattern: t1
    target-schema: dm_test
    target-table: t1
filters: {}
black-white-list:
  mysql-replica-01.bw_list.1:
    do-tables:
    - db-name: dm_test
      tbl-name: t1
    do-dbs: []
    ignore-tables: []
    ignore-dbs: []
mydumpers:
  mysql-replica-01.dump:
    mydumper-path: bin/mydumper
    threads: 4
    chunk-filesize: 64
    skip-tz-utc: true
    extra-args: -T dm_test.t1
  • 关键词
» query-status test5
{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "10.125.0.34:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "test5",
                    "stage": "Paused",
                    "unit": "Load",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] file dm_test.t1.sql: execute statement failed: commit: Error 1062: Duplicate entry '1' for key 'PRIMARY'
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:164
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:174
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:149
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:180
github.com/pingcap/dm/loader.(*DBConn).executeSQL
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:144
github.com/pingcap/dm/loader.(*Worker).run.func1
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:152
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1337",
                                "error": null
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "load": {
                        "finishedBytes": "0",
                        "totalBytes": "110",
                        "progress": "0.00 %",
                        "metaBinlog": "(mysql-bin.000005, 64313)"
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000005, 65063)",
                "masterBinlogGtid": "561d10a3-7614-11e9-8f94-246e96757168:1-1223137",
                "relaySubDir": "561d10a3-7614-11e9-8f94-246e96757168.000001",
                "relayBinlog": "(, 4)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": false,
                "stage": "Paused",
                "result": {
                    "isCanceled": false,
                    "errors": [
                        {
                            "Type": "UnknownError",
                            "msg": "[code=30015:class=relay-unit:scope=upstream:level=high] TCPReader get event: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/binlog/reader.(*TCPReader).GetEvent
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/binlog/reader/tcp.go:151
github.com/pingcap/dm/relay/reader.(*reader).GetEvent
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/reader/reader.go:144
github.com/pingcap/dm/relay.(*Relay).handleEvents
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:378
github.com/pingcap/dm/relay.(*Relay).process
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:292
github.com/pingcap/dm/relay.(*Relay).Process
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:191
github.com/pingcap/dm/dm/worker.(*realRelayHolder).run
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:164
github.com/pingcap/dm/dm/worker.(*realRelayHolder).Start.func1
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:140
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1337",
                            "error": null
                        }
                    ],
                    "detail": null
                }
            },
            "sourceID": "mysql-replica-01"
        }
    ]
}

dm-work报的错

[2019/11/13 11:48:46.103 +08:00] [WARN] [task_checker.go:348] ["task can't auto resume"] [component="task checker"] [task=test5] ["paused duration"=14m55.000061086s]
[2019/11/13 11:48:49.204 +08:00] [ERROR] [subtask.go:419] ["wait condition"] [subtask=test4] [error="[code=40067:class=dm-worker:scope=internal:level=high] wait relay catchup timeout, loader end binlog pos: (mysql-bin.000005, 18712), relay binlog pos: (, 4)"]
[2019/11/13 11:48:49.204 +08:00] [INFO] [worker.go:883] ["end to execute operation"] [component="worker controller"] ["oplog ID"=149] [error="[code=40067:class=dm-worker:scope=internal:level=high] wait relay catchup timeout, loader end binlog pos: (mysql-bin.000005, 18712), relay binlog pos: (, 4)"]

您好: 1. dm使用的哪个版本? 2. mysql是否有主从配置,是使用主库还是从库同步? mysql是否有配置gtid? 3. 将inventory.ini 的配置也发送下,多谢.

你好,请问 worker 上 relay log 配置开启了么?看报错是全量导完后,relay log 的 pos 没有追上。

你好,怎么看开启了没哦

# Worker Configuration.

server-id = 101
source-id = "mysql-replica-01"
flavor = "mysql"
enable-gtid = true
relay-binlog-name = ""
relay-binlog-gtid = "561d10a3-7614-11e9-8f94-246e96757168:1-1221235"
#charset of DSN of source mysql/mariadb instance
charset = ""
meta-dir = ""

[from]
host = "10.125.0.49"
user = "test"
password = "ndzYxh+L/P6hXPTV5ZT+u0B9xziKiNAmTw=="
port = 3306

#relay log purge strategy
[purge]
interval = 3600
expires = 0
remain-space = 15

dm版本:v1.1.0-alpha-29-gb95de78 mysql在有主从, 使用的主库同步的,是 gtid

## DM modules
[dm_master_servers]
dm_master ansible_host=10.125.0.21

[dm_worker_servers]

dm-worker1 ansible_host=10.125.0.34 source_id="mysql-replica-01" server_id=101 enable_gtid=true relay_binlog_gtid="561d10a3-7614-11e9-8f94-246e96757168:1-1221235" mysql_host=10.125.0.49 mysql_user=test mysql_password='ndzYxh+L/P6hXPTV5ZT+u0B9xziKiNAmTw==' mysql_port=3306

dm-worker2 ansible_host=10.125.0.45 source_id="mysql-replica-02" server_id=102 enable_gtid=true relay_binlog_gtid="561d10a3-7614-11e9-8f94-246e96757168:1-1221235" mysql_host=10.125.0.49 mysql_user=test mysql_password='ndzYxh+L/P6hXPTV5ZT+u0B9xziKiNAmTw==' mysql_port=3306

[dm_portal_servers]
dm_portal ansible_host=10.125.0.21

## Monitoring modules
[prometheus_servers]
prometheus ansible_host=10.125.0.21

[grafana_servers]
grafana ansible_host=10.125.0.21

[alertmanager_servers]
alertmanager ansible_host=10.125.0.21

## Global variables
[all:vars]
cluster_name = test-cluster

ansible_user = tidb

dm_version = latest

deploy_dir = /home/tidb/deploy

grafana_admin_user = "admin"
grafana_admin_password = "admin"

不好意思,回复慢了

你好: 这里感觉配置的没问题,请将dm-master,dm-worker, *.task 文件都反馈下,dm的报错日志完整上传下,多谢

现在又做了一次task,同步dm-test库到tidb 因为看dm-master中有描述 “metadata will not be used. for Full mode, incremental sync will never occur; for All mode, the meta dumped by MyDumper will be used” 所以用了增量的方式,貌似还是不行

  • 这个是上游库
    image
  • 下面是下游库
    image test5-task.yaml
name: test5
task-mode: incremental
is-sharding: false
target-database:
  host: tidb.xxx
  port: 3306
  user: root
  password: h46xBDWEdjCj+uy5kJxeMUAB4z4KJDaJG2U=
mysql-instances:
- source-id: mysql-replica-01
  meta:
    binlog-name: mysql-bin.000005
    binlog-pos: 72150
  filter-rules: []
  route-rules:
  - mysql-replica-1.route_rules.1
  - mysql-replica-1.route_rules.2
  black-white-list: mysql-replica-1.bw_list.1
  mydumper-config-name: mysql-replica-1.dump
routes:
  mysql-replica-1.route_rules.1:
    schema-pattern: dm_test
    table-pattern: ""
    target-schema: dm_test
    target-table: ""
  mysql-replica-1.route_rules.2:
    schema-pattern: dm_test
    table-pattern: t1
    target-schema: dm_test
    target-table: t1
filters: {}
black-white-list:
  mysql-replica-1.bw_list.1:
    do-tables:
    - db-name: dm_test
      tbl-name: t1
    do-dbs: []
    ignore-tables:
    - db-name: drone
      tbl-name: builds
    - db-name: drone
      tbl-name: cron
    - db-name: drone
      tbl-name: logs
    - db-name: drone
      tbl-name: migrations
    - db-name: drone
      tbl-name: nodes
    - db-name: drone
      tbl-name: perms
    - db-name: drone
      tbl-name: repos
    - db-name: drone
      tbl-name: secrets
    - db-name: drone
      tbl-name: stages
    - db-name: drone
      tbl-name: stages_unfinished
    - db-name: drone
      tbl-name: steps
    - db-name: drone
      tbl-name: users
    - db-name: grafana
      tbl-name: alert
    - db-name: grafana
      tbl-name: alert_notification
    - db-name: grafana
      tbl-name: alert_notification_state
    - db-name: grafana
      tbl-name: annotation
    - db-name: grafana
      tbl-name: annotation_tag
    - db-name: grafana
      tbl-name: api_key
    - db-name: grafana
      tbl-name: cache_data
    - db-name: grafana
      tbl-name: dashboard
    - db-name: grafana
      tbl-name: dashboard_acl
    - db-name: grafana
      tbl-name: dashboard_provisioning
    - db-name: grafana
      tbl-name: dashboard_snapshot
    - db-name: grafana
      tbl-name: dashboard_tag
    - db-name: grafana
      tbl-name: dashboard_version
    - db-name: grafana
      tbl-name: data_source
    - db-name: grafana
      tbl-name: login_attempt
    - db-name: grafana
      tbl-name: migration_log
    - db-name: grafana
      tbl-name: org
    - db-name: grafana
      tbl-name: org_user
    - db-name: grafana
      tbl-name: playlist
    - db-name: grafana
      tbl-name: playlist_item
    - db-name: grafana
      tbl-name: plugin_setting
    - db-name: grafana
      tbl-name: preferences
    - db-name: grafana
      tbl-name: quota
    - db-name: grafana
      tbl-name: server_lock
    - db-name: grafana
      tbl-name: session
    - db-name: grafana
      tbl-name: star
    - db-name: grafana
      tbl-name: tag
    - db-name: grafana
      tbl-name: team
    - db-name: grafana
      tbl-name: team_member
    - db-name: grafana
      tbl-name: temp_user
    - db-name: grafana
      tbl-name: test_data
    - db-name: grafana
      tbl-name: user
    - db-name: grafana
      tbl-name: user_auth
    - db-name: grafana
      tbl-name: user_auth_token
    ignore-dbs: []
mydumpers:
  mysql-replica-1.dump:
    mydumper-path: bin/mydumper
    threads: 4
    chunk-filesize: 64
    skip-tz-utc: true
    extra-args: -T dm_test.t1

通过dmctl执行任务后,查询报错

» start-task /tmp/test5-task.yaml
{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "10.125.0.34:8262",
            "msg": ""
        }
    ]
}
» query-status test5
{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "10.125.0.34:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "test5",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "UnknownError",
                                "msg": "[code=11025:class=functional:scope=internal:level=high] parse relay dir /home/tidb/deploy/relay_log/561d10a3-7614-11e9-8f94-246e96757168.000001 with pos (mysql-bin.000005, 72150): base file mysql-bin.000005 in directory /home/tidb/deploy/relay_log/561d10a3-7614-11e9-8f94-246e96757168.000001 not found
github.com/pingcap/dm/pkg/terror.(*Error).Generate
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:232
github.com/pingcap/dm/pkg/streamer.CollectBinlogFilesCmp
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/file.go:78
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).parseDirAsPossible
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:181
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).parseRelay
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:138
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).StartSync.func1
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:111
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1337",
                                "error": null
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "0",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.000005, 72523)",
                        "masterBinlogGtid": "561d10a3-7614-11e9-8f94-246e96757168:1-1223157",
                        "syncerBinlog": "(mysql-bin.000005, 72150)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000005, 72523)",
                "masterBinlogGtid": "561d10a3-7614-11e9-8f94-246e96757168:1-1223157",
                "relaySubDir": "561d10a3-7614-11e9-8f94-246e96757168.000001",
                "relayBinlog": "(, 4)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": false,
                "stage": "Paused",
                "result": {
                    "isCanceled": false,
                    "errors": [
                        {
                            "Type": "UnknownError",
                            "msg": "[code=30015:class=relay-unit:scope=upstream:level=high] TCPReader get event: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/binlog/reader.(*TCPReader).GetEvent
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/binlog/reader/tcp.go:151
github.com/pingcap/dm/relay/reader.(*reader).GetEvent
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/reader/reader.go:144
github.com/pingcap/dm/relay.(*Relay).handleEvents
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:378
github.com/pingcap/dm/relay.(*Relay).process
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:292
github.com/pingcap/dm/relay.(*Relay).Process
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:191
github.com/pingcap/dm/dm/worker.(*realRelayHolder).run
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:164
github.com/pingcap/dm/dm/worker.(*realRelayHolder).Start.func1
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:140
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1337",
                            "error": null
                        }
                    ],
                    "detail": null
                }
            },
            "sourceID": "mysql-replica-01"
        }
    ]
}

dm-master日志

[2019/11/13 14:56:38.017 +08:00] [INFO] [server.go:228] [payload="task:"name: test5\ntask-mode: incremental\nis-sharding: false\ntarget-database:\n  host: tidb.xxx\n  port: 3306\n  user: root\n  password: h46xBDWEdjCj+uy5kJxeMUAB4z4KJDaJG2U=\nmysql-instances:\n- source-id: mysql-replica-1\n  meta:\n    binlog-name: mysql-bin.000005\n    binlog-pos: 72150\n  filter-rules: []\n  route-rules:\n  - mysql-replica-1.route_rules.1\n  - mysql-replica-1.route_rules.2\n  black-white-list: mysql-replica-1.bw_list.1\n  mydumper-config-name: mysql-replica-1.dump\nroutes:\n  mysql-replica-1.route_rules.1:\n    schema-pattern: dm_test\n    table-pattern: \"\"\n    target-schema: dm_test\n    target-table: \"\"\n  mysql-replica-1.route_rules.2:\n    schema-pattern: dm_test\n    table-pattern: t1\n    target-schema: dm_test\n    target-table: t1\nfilters: {}\nblack-white-list:\n  mysql-replica-1.bw_list.1:\n    do-tables:\n    - db-name: dm_test\n      tbl-name: t1\n    do-dbs: []\n    ignore-tables:\n    - db-name: drone\n      tbl-name: builds\n    - db-name: drone\n      tbl-name: cron\n    - db-name: drone\n      tbl-name: logs\n    - db-name: drone\n      tbl-name: migrations\n    - db-name: drone\n      tbl-name: nodes\n    - db-name: drone\n      tbl-name: perms\n    - db-name: drone\n      tbl-name: repos\n    - db-name: drone\n      tbl-name: secrets\n    - db-name: drone\n      tbl-name: stages\n    - db-name: drone\n      tbl-name: stages_unfinished\n    - db-name: drone\n      tbl-name: steps\n    - db-name: drone\n      tbl-name: users\n    - db-name: grafana\n      tbl-name: alert\n    - db-name: grafana\n      tbl-name: alert_notification\n    - db-name: grafana\n      tbl-name: alert_notification_state\n    - db-name: grafana\n      tbl-name: annotation\n    - db-name: grafana\n      tbl-name: annotation_tag\n    - db-name: grafana\n      tbl-name: api_key\n    - db-name: grafana\n      tbl-name: cache_data\n    - db-name: grafana\n      tbl-name: dashboard\n    - db-name: grafana\n      tbl-name: dashboard_acl\n    - db-name: grafana\n      tbl-name: dashboard_provisioning\n    - db-name: grafana\n      tbl-name: dashboard_snapshot\n    - db-name: grafana\n      tbl-name: dashboard_tag\n    - db-name: grafana\n      tbl-name: dashboard_version\n    - db-name: grafana\n      tbl-name: data_source\n    - db-name: grafana\n      tbl-name: login_attempt\n    - db-name: grafana\n      tbl-name: migration_log\n    - db-name: grafana\n      tbl-name: org\n    - db-name: grafana\n      tbl-name: org_user\n    - db-name: grafana\n      tbl-name: playlist\n    - db-name: grafana\n      tbl-name: playlist_item\n    - db-name: grafana\n      tbl-name: plugin_setting\n    - db-name: grafana\n      tbl-name: preferences\n    - db-name: grafana\n      tbl-name: quota\n    - db-name: grafana\n      tbl-name: server_lock\n    - db-name: grafana\n      tbl-name: session\n    - db-name: grafana\n      tbl-name: star\n    - db-name: grafana\n      tbl-name: tag\n    - db-name: grafana\n      tbl-name: team\n    - db-name: grafana\n      tbl-name: team_member\n    - db-name: grafana\n      tbl-name: temp_user\n    - db-name: grafana\n      tbl-name: test_data\n    - db-name: grafana\n      tbl-name: user\n    - db-name: grafana\n      tbl-name: user_auth\n    - db-name: grafana\n      tbl-name: user_auth_token\n    ignore-dbs: []\nmydumpers:\n  mysql-replica-1.dump:\n    mydumper-path: bin/mydumper\n    threads: 4\n    chunk-filesize: 64\n    skip-tz-utc: true\n    extra-args: -T dm_test.t1\n" "] [request=StartTask]
[2019/11/13 14:57:27.718 +08:00] [INFO] [server.go:228] [payload="task:"name: test5\ntask-mode: incremental\nis-sharding: false\ntarget-database:\n  host: tidb.xxx\n  port: 3306\n  user: root\n  password: h46xBDWEdjCj+uy5kJxeMUAB4z4KJDaJG2U=\nmysql-instances:\n- source-id: mysql-replica-01\n  meta:\n    binlog-name: mysql-bin.000005\n    binlog-pos: 72150\n  filter-rules: []\n  route-rules:\n  - mysql-replica-1.route_rules.1\n  - mysql-replica-1.route_rules.2\n  black-white-list: mysql-replica-1.bw_list.1\n  mydumper-config-name: mysql-replica-1.dump\nroutes:\n  mysql-replica-1.route_rules.1:\n    schema-pattern: dm_test\n    table-pattern: \"\"\n    target-schema: dm_test\n    target-table: \"\"\n  mysql-replica-1.route_rules.2:\n    schema-pattern: dm_test\n    table-pattern: t1\n    target-schema: dm_test\n    target-table: t1\nfilters: {}\nblack-white-list:\n  mysql-replica-1.bw_list.1:\n    do-tables:\n    - db-name: dm_test\n      tbl-name: t1\n    do-dbs: []\n    ignore-tables:\n    - db-name: drone\n      tbl-name: builds\n    - db-name: drone\n      tbl-name: cron\n    - db-name: drone\n      tbl-name: logs\n    - db-name: drone\n      tbl-name: migrations\n    - db-name: drone\n      tbl-name: nodes\n    - db-name: drone\n      tbl-name: perms\n    - db-name: drone\n      tbl-name: repos\n    - db-name: drone\n      tbl-name: secrets\n    - db-name: drone\n      tbl-name: stages\n    - db-name: drone\n      tbl-name: stages_unfinished\n    - db-name: drone\n      tbl-name: steps\n    - db-name: drone\n      tbl-name: users\n    - db-name: grafana\n      tbl-name: alert\n    - db-name: grafana\n      tbl-name: alert_notification\n    - db-name: grafana\n      tbl-name: alert_notification_state\n    - db-name: grafana\n      tbl-name: annotation\n    - db-name: grafana\n      tbl-name: annotation_tag\n    - db-name: grafana\n      tbl-name: api_key\n    - db-name: grafana\n      tbl-name: cache_data\n    - db-name: grafana\n      tbl-name: dashboard\n    - db-name: grafana\n      tbl-name: dashboard_acl\n    - db-name: grafana\n      tbl-name: dashboard_provisioning\n    - db-name: grafana\n      tbl-name: dashboard_snapshot\n    - db-name: grafana\n      tbl-name: dashboard_tag\n    - db-name: grafana\n      tbl-name: dashboard_version\n    - db-name: grafana\n      tbl-name: data_source\n    - db-name: grafana\n      tbl-name: login_attempt\n    - db-name: grafana\n      tbl-name: migration_log\n    - db-name: grafana\n      tbl-name: org\n    - db-name: grafana\n      tbl-name: org_user\n    - db-name: grafana\n      tbl-name: playlist\n    - db-name: grafana\n      tbl-name: playlist_item\n    - db-name: grafana\n      tbl-name: plugin_setting\n    - db-name: grafana\n      tbl-name: preferences\n    - db-name: grafana\n      tbl-name: quota\n    - db-name: grafana\n      tbl-name: server_lock\n    - db-name: grafana\n      tbl-name: session\n    - db-name: grafana\n      tbl-name: star\n    - db-name: grafana\n      tbl-name: tag\n    - db-name: grafana\n      tbl-name: team\n    - db-name: grafana\n      tbl-name: team_member\n    - db-name: grafana\n      tbl-name: temp_user\n    - db-name: grafana\n      tbl-name: test_data\n    - db-name: grafana\n      tbl-name: user\n    - db-name: grafana\n      tbl-name: user_auth\n    - db-name: grafana\n      tbl-name: user_auth_token\n    ignore-dbs: []\nmydumpers:\n  mysql-replica-1.dump:\n    mydumper-path: bin/mydumper\n    threads: 4\n    chunk-filesize: 64\n    skip-tz-utc: true\n    extra-args: -T dm_test.t1\n" "] [request=StartTask]
[2019/11/13 14:57:27.727 +08:00] [INFO] [common.go:109] ["no tables need to sync"] [schema=drone]
[2019/11/13 14:57:27.728 +08:00] [INFO] [common.go:109] ["no tables need to sync"] [schema=grafana]
[2019/11/13 14:57:27.728 +08:00] [INFO] [checker.go:225] ["
************ task test5 checking items ************
mysql_version
mysql_binlog_enable
mysql_binlog_format
mysql_binlog_row_image
source db dump privilege chcker
source db replication privilege chcker
table structure compatibility check
************ task test5 checking items ************"] [unit="task check"]
[2019/11/13 14:57:27.735 +08:00] [INFO] [server.go:237] ["task name"=test5] [task="name: test5
task-mode: incremental
is-sharding: false
ignore-checking-items: []
meta-schema: dm_meta
remove-meta: false
disable-heartbeat: true
enable-heartbeat: false
heartbeat-update-interval: 1
heartbeat-report-interval: 10
timezone: ""
case-sensitive: false
target-database:
  host: tidb.xxx
  port: 3306
  user: root
  password: h46xBDWEdjCj+uy5kJxeMUAB4z4KJDaJG2U=
  max-allowed-packet: null
mysql-instances:
- source-id: mysql-replica-01
  meta:
    binlog-name: mysql-bin.000005
    binlog-pos: 72150
  filter-rules: []
  column-mapping-rules: []
  route-rules:
  - mysql-replica-1.route_rules.1
  - mysql-replica-1.route_rules.2
  black-white-list: mysql-replica-1.bw_list.1
  mydumper-config-name: mysql-replica-1.dump
  mydumper:
    mydumper-path: bin/mydumper
    threads: 4
    chunk-filesize: 64
    skip-tz-utc: true
    extra-args: -T dm_test.t1
  mydumper-thread: 0
  loader-config-name: ""
  loader:
    pool-size: 16
    dir: ./dumped_data
  loader-thread: 0
  syncer-config-name: ""
  syncer:
    meta-file: ""
    worker-count: 16
    batch: 100
    max-retry: 0
    auto-fix-gtid: false
    enable-gtid: false
    disable-detect: false
    safe-mode: false
    enable-ansi-quotes: false
  syncer-thread: 0
online-ddl-scheme: ""
routes:
  mysql-replica-1.route_rules.1:
    schema-pattern: dm_test
    table-pattern: ""
    target-schema: dm_test
    target-table: ""
  mysql-replica-1.route_rules.2:
    schema-pattern: dm_test
    table-pattern: t1
    target-schema: dm_test
    target-table: t1
filters: {}
column-mappings: {}
black-white-list:
  mysql-replica-1.bw_list.1:
    do-tables:
    - db-name: dm_test
      tbl-name: t1
    do-dbs: []
    ignore-tables:
    - db-name: drone
      tbl-name: builds
    - db-name: drone
      tbl-name: cron
    - db-name: drone
      tbl-name: logs
    - db-name: drone
      tbl-name: migrations
    - db-name: drone
      tbl-name: nodes
    - db-name: drone
      tbl-name: perms
    - db-name: drone
      tbl-name: repos
    - db-name: drone
      tbl-name: secrets
    - db-name: drone
      tbl-name: stages
    - db-name: drone
      tbl-name: stages_unfinished
    - db-name: drone
      tbl-name: steps
    - db-name: drone
      tbl-name: users
    - db-name: grafana
      tbl-name: alert
    - db-name: grafana
      tbl-name: alert_notification
    - db-name: grafana
      tbl-name: alert_notification_state
    - db-name: grafana
      tbl-name: annotation
    - db-name: grafana
      tbl-name: annotation_tag
    - db-name: grafana
      tbl-name: api_key
    - db-name: grafana
      tbl-name: cache_data
    - db-name: grafana
      tbl-name: dashboard
    - db-name: grafana
      tbl-name: dashboard_acl
    - db-name: grafana
      tbl-name: dashboard_provisioning
    - db-name: grafana
      tbl-name: dashboard_snapshot
    - db-name: grafana
      tbl-name: dashboard_tag
    - db-name: grafana
      tbl-name: dashboard_version
    - db-name: grafana
      tbl-name: data_source
    - db-name: grafana
      tbl-name: login_attempt
    - db-name: grafana
      tbl-name: migration_log
    - db-name: grafana
      tbl-name: org
    - db-name: grafana
      tbl-name: org_user
    - db-name: grafana
      tbl-name: playlist
    - db-name: grafana
      tbl-name: playlist_item
    - db-name: grafana
      tbl-name: plugin_setting
    - db-name: grafana
      tbl-name: preferences
    - db-name: grafana
      tbl-name: quota
    - db-name: grafana
      tbl-name: server_lock
    - db-name: grafana
      tbl-name: session
    - db-name: grafana
      tbl-name: star
    - db-name: grafana
      tbl-name: tag
    - db-name: grafana
      tbl-name: team
    - db-name: grafana
      tbl-name: team_member
    - db-name: grafana
      tbl-name: temp_user
    - db-name: grafana
      tbl-name: test_data
    - db-name: grafana
      tbl-name: user
    - db-name: grafana
      tbl-name: user_auth
    - db-name: grafana
      tbl-name: user_auth_token
    ignore-dbs: []
mydumpers:
  mysql-replica-1.dump:
    mydumper-path: bin/mydumper
    threads: 4
    chunk-filesize: 64
    skip-tz-utc: true
    extra-args: -T dm_test.t1
loaders: {}
syncers: {}
"] [request=StartTask]
[2019/11/13 14:57:27.742 +08:00] [INFO] [server.go:1913] ["wait op log result"] [task=test5] ["operation log ID"=166] [result="meta:<result:true > log:<id:166 task:<op:Start name:"test5" task:"is-sharding = false\nonline-ddl-scheme = \"\"\ncase-sensitive = false\nname = \"test5\"\nmode = \"incremental\"\nsource-id = \"mysql-replica-01\"\nserver-id = 101\nflavor = \"mysql\"\nmeta-schema = \"dm_meta\"\nremove-meta = false\ndisable-heartbeat = true\nheartbeat-update-interval = 1\nheartbeat-report-interval = 10\nenable-heartbeat = false\ntimezone = \"\"\nbinlog-type = \"local\"\nrelay-dir = \"/home/tidb/deploy/relay_log\"\nfilter-rules = []\nmapping-rule = []\nmydumper-path = \"bin/mydumper\"\nthreads = 4\nchunk-filesize = 64\nskip-tz-utc = true\nextra-args = \"-T dm_test.t1\"\npool-size = 16\ndir = \"./dumped_data.test5\"\nmeta-file = \"\"\nworker-count = 16\nbatch = 100\nmax-retry = 0\nauto-fix-gtid = false\nenable-gtid = true\ndisable-detect = false\nsafe-mode = false\nenable-ansi-quotes = false\nlog-level = \"info\"\nlog-file = \"/home/tidb/deploy/log/dm-worker.log\"\nlog-rotate = \"\"\npprof-addr = \"\"\nstatus-addr = \"\"\n\n[meta]\n  BinLogName = \"mysql-bin.000005\"\n  BinLogPos = 72150\n\n[from]\n  host = \"10.125.0.49\"\n  port = 3306\n  user = \"test\"\n  password = \"ndzYxh+L/P6hXPTV5ZT+u0B9xziKiNAmTw==\"\n  max-allowed-packet = 67108864\n\n[to]\n  host = \"tidb.xxx\"\n  port = 3306\n  user = \"root\"\n  password = \"h46xBDWEdjCj+uy5kJxeMUAB4z4KJDaJG2U=\"\n  max-allowed-packet = 67108864\n\n[[route-rules]]\n  schema-pattern = \"dm_test\"\n  table-pattern = \"\"\n  target-schema = \"dm_test\"\n  target-table = \"\"\n\n[[route-rules]]\n  schema-pattern = \"dm_test\"\n  table-pattern = \"t1\"\n  target-schema = \"dm_test\"\n  target-table = \"t1\"\n\n[black-white-list]\n  do-dbs = []\n  ignore-dbs = []\n\n  [[black-white-list.do-tables]]\n    db-name = \"dm_test\"\n    tbl-name = \"t1\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"drone\"\n    tbl-name = \"builds\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"drone\"\n    tbl-name = \"cron\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"drone\"\n    tbl-name = \"logs\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"drone\"\n    tbl-name = \"migrations\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"drone\"\n    tbl-name = \"nodes\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"drone\"\n    tbl-name = \"perms\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"drone\"\n    tbl-name = \"repos\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"drone\"\n    tbl-name = \"secrets\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"drone\"\n    tbl-name = \"stages\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"drone\"\n    tbl-name = \"stages_unfinished\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"drone\"\n    tbl-name = \"steps\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"drone\"\n    tbl-name = \"users\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"alert\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"alert_notification\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"alert_notification_state\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"annotation\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"annotation_tag\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"api_key\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"cache_data\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"dashboard\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"dashboard_acl\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"dashboard_provisioning\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"dashboard_snapshot\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"dashboard_tag\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"dashboard_version\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"data_source\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"login_attempt\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"migration_log\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"org\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"org_user\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"playlist\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"playlist_item\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"plugin_setting\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"preferences\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"quota\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"server_lock\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"session\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"star\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"tag\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"team\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"team_member\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"temp_user\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"test_data\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"user\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"user_auth\"\n\n  [[black-white-list.ignore-tables]]\n    db-name = \"grafana\"\n    tbl-name = \"user_auth_token\"\n" > ts:1573628247743165775 > "]
[2019/11/13 14:57:28.743 +08:00] [INFO] [server.go:1041] ["update workers of task"] [task=test5] [workers="[10.125.0.34:8262]"]
[2019/11/13 14:57:37.993 +08:00] [INFO] [server.go:497] [payload="name:"test5" "] [request=QueryStatus]

dm-work.log dm-work (58.6 KB)

您好: 1. 根据报错:你配置的这个位点信息,没有找到,你改为all的方式,不要指定位点信息,尝试自动拉取吧.重启任务 binlog-name: mysql-bin.000005 binlog-pos: 72150 2. 查看下你的relay log目录下自动拉取binglog文件成功了吗?是否拉取了最新的binlog文件

改成all,看日期binglog文件没有拉取到哦 image
dm-work日志

[2019/11/13 15:32:13.613 +08:00] [ERROR] [db.go:172] ["execute statements failed after retry"] [task=test5] [unit=load] [component="remote checkpoint"] [queries="[INSERT INTO `dm_meta`.`test5_loader_checkpoint` (`id`, `filename`, `cp_schema`, `cp_table`, `offset`, `end_pos`) VALUES(?,?,?,?,?,?)]"] [arguments="[[mysql-replica-01 dm_test.t1.sql dm_test t1 0 110]]"] [error="[code=10006:class=database:scope=not-set:level=high] execute statement failed: commit: Error 1062: Duplicate entry 'mysql-replica-01-dm_test.t1.sql' for key 'uk_id_f'"]
[2019/11/13 15:32:13.613 +08:00] [INFO] [checkpoint.go:294] ["checkpoint record already exists, skip it."] [task=test5] [unit=load] [component="remote checkpoint"] [id=mysql-replica-01] [filename=dm_test.t1.sql]
[2019/11/13 15:32:13.649 +08:00] [INFO] [loader.go:252] ["data are scanned finished."] [task=test5] [unit=load] ["worker ID"=0] ["data file"=dumped_data.test5/dm_test.t1.sql]
[2019/11/13 15:32:13.665 +08:00] [ERROR] [db.go:172] ["execute statements failed after retry"] [task=test5] [unit=load] ["worker ID"=0] [queries="[USE `dm_test`; INSERT INTO `t1` VALUES
(1),
(2),
(3),
(4); UPDATE `dm_meta`.`test5_loader_checkpoint` SET `offset`=110 WHERE `id` ='mysql-replica-01' AND `filename`='dm_test.t1.sql';]"] [arguments="[]"] [error="[code=10006:class=database:scope=not-set:level=high] execute statement failed: commit: Error 1062: Duplicate entry '1' for key 'PRIMARY'"]
[2019/11/13 15:32:13.665 +08:00] [INFO] [loader.go:201] ["finish to restore dump sql file"] [task=test5] [unit=load] ["worker ID"=0] ["data file"=dumped_data.test5/dm_test.t1.sql]
[2019/11/13 15:32:13.665 +08:00] [INFO] [loader.go:1119] ["all data files have been finished"] [task=test5] [unit=load] ["cost time"=76.342451ms]
[2019/11/13 15:32:13.665 +08:00] [INFO] [status.go:75] ["progress status of load"] [task=test5] [unit=load] [finished_bytes=0] [total_bytes=110] [total_file_count=3] [progress="0.00 %"]
[2019/11/13 15:32:13.665 +08:00] [INFO] [subtask.go:233] ["unit process returned"] [subtask=test5] [unit=Load] [stage=Paused] [status="{"totalBytes":110,"progress":"0.00 %","metaBinlog":"(mysql-bin.000005, 64313)"}"]
[2019/11/13 15:32:13.665 +08:00] [ERROR] [subtask.go:252] ["unit process error"] [subtask=test5] [unit=Load] ["error information"="{"Type":1,"msg":"[code=10006:class=database:scope=downstream:level=high] file dm_test.t1.sql: execute statement failed: commit: Error 1062: Duplicate entry '1' for key 'PRIMARY'\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:164\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:174\ngithub.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:149\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:180\ngithub.com/pingcap/dm/loader.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:144\ngithub.com/pingcap/dm/loader.(*Worker).run.func1\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:152\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337","error":{"ErrCode":10006,"ErrClass":1,"ErrScope":2,"ErrLevel":3,"Message":"file dm_test.t1.sql: execute statement failed: commit: Error 1062: Duplicate entry '1' for key 'PRIMARY'","RawCause":"Error 1062: Duplicate entry '1' for key 'PRIMARY'"}}"]
[2019/11/13 15:32:14.244 +08:00] [INFO] [server.go:227] [request=QueryTaskOperation] [payload="name:"test5" logID:186 "]
[2019/11/13 15:32:16.103 +08:00] [WARN] [task_checker.go:351] ["task can't auto resume"] [component="task checker"] [task=test5]

您好: 1. 这个是在测试对吧? 如果库里的数据没有用,都删掉,再看下是否能够成功,当前看报主键错误。 2. 或者改为syncer,但是要指定到对的位点信息才行。 3. cd 56xxxx-.00001 , 看下目录下的relay log 文件生成了吗?

表删了,同步是成功了 但是还是在报错

query-task test5

{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "10.125.0.34:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "test5",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "0",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.000005, 74761)",
                        "masterBinlogGtid": "561d10a3-7614-11e9-8f94-246e96757168:1-1223163",
                        "syncerBinlog": "(, 4)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000005, 74761)",
                "masterBinlogGtid": "561d10a3-7614-11e9-8f94-246e96757168:1-1223163",
                "relaySubDir": "561d10a3-7614-11e9-8f94-246e96757168.000001",
                "relayBinlog": "(, 4)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": false,
                "stage": "Paused",
                "result": {
                    "isCanceled": false,
                    "errors": [
                        {
                            "Type": "UnknownError",
                            "msg": "[code=30015:class=relay-unit:scope=upstream:level=high] TCPReader get event: ERROR 1236 (HY000): The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/binlog/reader.(*TCPReader).GetEvent
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/binlog/reader/tcp.go:151
github.com/pingcap/dm/relay/reader.(*reader).GetEvent
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/reader/reader.go:144
github.com/pingcap/dm/relay.(*Relay).handleEvents
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:378
github.com/pingcap/dm/relay.(*Relay).process
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:292
github.com/pingcap/dm/relay.(*Relay).Process
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/relay/relay.go:191
github.com/pingcap/dm/dm/worker.(*realRelayHolder).run
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:164
github.com/pingcap/dm/dm/worker.(*realRelayHolder).Start.func1
	/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/relay.go:140
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1337",
                            "error": null
                        }
                    ],
                    "detail": null
                }
            },
            "sourceID": "mysql-replica-01"
        }
    ]
}

dm-work

[2019/11/13 16:03:49.274 +08:00] [ERROR] [subtask.go:183] ["wait condition"] [subtask=test5] [error="[code=40067:class=dm-worker:scope=internal:level=high] wait relay catchup timeout, loader end binlog pos: (mysql-bin.000005, 64313), relay binlog pos: (, 4)"]
[2019/11/13 16:03:51.103 +08:00] [WARN] [task_checker.go:354] ["backoff skip auto resume task"] [component="task checker"] [task=test5] [latestResumeTime=2019/11/13 16:02:21.103 +08:00] [duration=2m16.455801104s]
[2019/11/13 16:03:56.103 +08:00] [WARN] [task_checker.go:354] ["backoff skip auto resume task"] [component="task checker"] [task=test5] [latestResumeTime=2019/11/13 16:02:21.103 +08:00] [duration=3m50.492712013s]
[2019/11/13 16:03:57.103 +08:00] [ERROR] [subtask.go:419] ["wait condition"] [subtask=test5] [error="[code=40067:class=dm-worker:scope=internal:level=high] wait relay catchup timeout, loader end binlog pos: (mysql-bin.000005, 64313), relay binlog pos: (, 4)"]
[2019/11/13 16:03:57.103 +08:00] [INFO] [worker.go:883] ["end to execute operation"] [component="worker controller"] ["oplog ID"=192] [error="[code=40067:class=dm-worker:scope=internal:level=high] wait relay catchup timeout, loader end binlog pos: (mysql-bin.000005, 64313), relay binlog pos: (, 4)"]
[2019/11/13 16:03:57.202 +08:00] [INFO] [worker.go:792] ["start to execute operation"] [component="worker controller"] [oplog="{"id":193,"task":{"op":6,"name":"test5"},"ts":1573631941103556898}"]
[2019/11/13 16:03:57.202 +08:00] [INFO] [worker.go:879] ["auto_resume sub task"] [component="worker controller"] [task=test5]
[2019/11/13 16:03:57.202 +08:00] [INFO] [subtask.go:624] ["wait condition between two units"] [subtask=test5] ["previous unit"=Load] [unit=Sync]

每次新建同步任务都要指定最新的binlog-name, binlog-pos么

看下 dm-worker 是否拉取上游的 binlog ? relay.meta 中的记录当前是什么情况 ?

image

看起来是因为 dm-worker 没有从上游拉取 binlog 导致,和 task 同步任务如何配置没有关系。 可以按照以下方式恢复

  • 停掉 dm-worker
  • 修改 relay.meta 中的内容,需要从上游哪个 GTID 位点开始拉取
  • 启动 dm-worker

另外 v1.1.0-alpha-29-gb95de78 ,alpha 版本不是稳定版,目前 DM 版本为 1.0.x 推荐使用 1.0.2 版本。

2赞

好像还是不行 我换个稳定版安装下试试吧 谢谢各位pingcap的大佬啦

好的,换个稳定版本试下,启动 dm-worker 后,会去从上游拉取 binlog,拉取正常后,再启动 task 同步任务。如果还有其他问题,可以贴出来继续跟进。辛苦。

换了稳定版本,测试task已经成功啦

:+1::+1::+1: