数据同步问题

nimeijian · 2020 年1 月 15 日 03:18

为提高效率，提问时请提供以下信息，问题描述清晰可优先响应。

【TiDB 版本】： docker-compose版本，DM v1.0.3版本（Ansible部署）
【问题描述】：

问题

问题一

start-task tasks/task_merge.yml
{
    "result": false,
    "msg": "[code=38008:class=dm-master:scope=internal:level=high] fetch config of worker 172.26.8.82:8262: grpc request error: rpc error: code = DeadlineExceeded desc = context deadline exceeded\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/dm/master/workerrpc.callRPC\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/master/workerrpc/rawgrpc.go:124\
github.com/pingcap/dm/dm/master/workerrpc.(*GRPCClient).SendRequest\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/master/workerrpc/rawgrpc.go:64\
github.com/pingcap/dm/dm/master.(*Server).getWorkerConfigs.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/master/server.go:1756\
github.com/pingcap/dm/dm/master.(*AgentPool).Emit\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/master/agent_pool.go:117\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
    "workers": [
    ]
}

问题二

query-status task_merge
{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "172.26.8.83:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "task_merge",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "UnknownError",
                                "msg": "[code=11029:class=functional:scope=internal:level=high] get stat for relay log /home/tidb/deploy133/relay_log/dfcf069c-976c-11e6-9ebe-305a3a5a429f.000001/mysql-bin.000103: stat /home/tidb/deploy133/relay_log/dfcf069c-976c-11e6-9ebe-305a3a5a429f.000001/mysql-bin.000103: no such file or directory\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).checkRelayPos\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:98\
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).StartSync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:122\
github.com/pingcap/dm/syncer.(*localBinlogReader).generateStreamer\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:102\
github.com/pingcap/dm/syncer.(*Syncer).Run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1079\
github.com/pingcap/dm/syncer.(*Syncer).Process\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:598\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "0",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.000001, 27353658)",
                        "masterBinlogGtid": "4f51782c-b270-11e7-93f6-5254002147c0:508480007-509375779:513595970-521566909,adb758e5-081a-11e9-8c70-525400dfdd97:1-4,c1bc3c0e-fc5e-11e8-a45c-5254007af3b6:3-743071983",
                        "syncerBinlog": "(mysql-bin|000001.000103, 521435566)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000001, 27353658)",
                "masterBinlogGtid": "4f51782c-b270-11e7-93f6-5254002147c0:508480007-509375779:513595970-521566909,adb758e5-081a-11e9-8c70-525400dfdd97:1-4,c1bc3c0e-fc5e-11e8-a45c-5254007af3b6:3-743071983",
                "relaySubDir": "dfcf069c-976c-11e6-9ebe-305a3a5a429f.000001",
                "relayBinlog": "(mysql-bin.000001, 27353658)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "replica-133"
        },
        {
            "result": true,
            "worker": "172.26.8.83:8263",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "task_merge",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "UnknownError",
                                "msg": "[code=11029:class=functional:scope=internal:level=high] get stat for relay log /home/tidb/deploy135/relay_log/54d11464-77ee-11e7-b052-9c5c8e7240e6.000001/mysql-bin.000257: stat /home/tidb/deploy135/relay_log/54d11464-77ee-11e7-b052-9c5c8e7240e6.000001/mysql-bin.000257: no such file or directory\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).checkRelayPos\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:98\
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).StartSync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:122\
github.com/pingcap/dm/syncer.(*localBinlogReader).generateStreamer\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:102\
github.com/pingcap/dm/syncer.(*Syncer).Run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1079\
github.com/pingcap/dm/syncer.(*Syncer).Process\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:598\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "0",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.000001, 43313276)",
                        "masterBinlogGtid": "07efafff-80cd-11e8-a628-52540038247b:104-109,4f51782c-b270-11e7-93f6-5254002147c0:1149040627-1183411505,54d11464-77ee-11e7-b052-9c5c8e7240e6:1-74940,8614aa74-6670-11e9-803d-5254005f2010:1-7,f5633113-666f-11e9-aeac-52540009d31a:1-287611299",
                        "syncerBinlog": "(mysql-bin|000001.000257, 474543098)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000001, 43313276)",
                "masterBinlogGtid": "54d11464-77ee-11e7-b052-9c5c8e7240e6:1-74940,8614aa74-6670-11e9-803d-5254005f2010:1-7,f5633113-666f-11e9-aeac-52540009d31a:1-287611299,07efafff-80cd-11e8-a628-52540038247b:104-109,4f51782c-b270-11e7-93f6-5254002147c0:1149040627-1183411505",
                "relaySubDir": "54d11464-77ee-11e7-b052-9c5c8e7240e6.000001",
                "relayBinlog": "(mysql-bin.000001, 43313276)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "replica-135"
        }
    ]
}

配置

inventory.ini

[dm_master_servers]
dm_master ansible_host=172.26.8.81

[dm_worker_servers]
dm-worker82_132 ansible_host=172.26.8.82 source_id="replica-132" mysql_host=172.23.0.132 mysql_user=root mysql_password='jFSwbMR6kC78GQnRffyrb0Cx1646E5hccOVc' mysql_port=5688

dm-worker83_133 ansible_host=172.26.8.83 source_id="replica-133" mysql_host=172.23.0.133 mysql_user=root mysql_password='jFSwbMR6kC78GQnRffyrb0Cx1646E5hccOVc' mysql_port=3306 deploy_dir=/home/tidb/deploy133 dm_worker_port=8262

dm-worker83_135 ansible_host=172.26.8.83 source_id="replica-135" mysql_host=172.23.0.135 mysql_user=root mysql_password='jFSwbMR6kC78GQnRffyrb0Cx1646E5hccOVc' mysql_port=5688 deploy_dir=/home/tidb/deploy135 dm_worker_port=8263

[dm_portal_servers]
dm_portal ansible_host=172.26.8.81

[prometheus_servers]
prometheus ansible_host=172.26.8.81

[grafana_servers]
grafana ansible_host=172.26.8.81

[alertmanager_servers]
alertmanager ansible_host=172.26.8.81

[all:vars]
cluster_name = test-cluster

ansible_user = tidb

dm_version = v1.0.3

deploy_dir = /home/tidb/deploy

grafana_admin_user = "admin"
grafana_admin_password = "admin"

hosts.ini

[servers]
172.26.8.81
172.26.8.82
172.26.8.83

[all:vars]
username = tidb

task_merge.yml

name: task_merge
task-mode: all
is-sharding: true
ignore-checking-items: ["auto_increment_ID"]

target-database:
  host: "172.24.28.1"
  port: 4000
  user: "root"
  password: ""

mysql-instances:
  -
    source-id: "replica-132"
    black-white-list:  "bw_list"
  -
    source-id: "replica-133"
    black-white-list:  "bw_list"
  -
    source-id: "replica-135"
    black-white-list: "bw_list"

black-white-list:
  bw_list:
    do-dbs: ["tidb_task"]

做过的努力

停止任务，停止DM集群，清除所有dm-master&dm-worker的deploy目录，重新deploy&start DM集群，还是有这个问题。
telnet 172.26.8.82 8262 没问题。

GangShen · 2020 年1 月 15 日 03:28

检查一下这个文件是否存在， /home/tidb/deploy133/relay_log/dfcf069c-976c-11e6-9ebe-305a3a5a429f.000001/mysql-bin.000103

如果可能是 relay log 拉取有问题，可以检查一下 relay log 拉取的情况：

上游数据库连接是否成功
目录是否有写入权限

nimeijian · 2020 年1 月 15 日 03:44

没有这个文件，很奇怪，为什么会去找一个并不存在的binlog文件，是哪里出错了？生成binlog与找binlog有出入？

上游数据库是可以连接的，虽然通过 dmctl -encrypt 命令生成的密码每次都在变，但原始密码确实是正确的。

dm-worker82_132:

drwxr-xr-x 8 tidb tidb    86 1月  15 11:12 deploy

dm-worker83_133/135:

drwxr-xr-x 9 tidb tidb   108 1月  15 11:13 deploy133
drwxr-xr-x 9 tidb tidb   108 1月  15 11:13 deploy135

GangShen · 2020 年1 月 15 日 05:04

关于第一个问题：

因为是 docker 方式部署，所以可以排查一下 docker 网络的问题

关于第二个问题：
看下 /home/tidb/deploy133/relay_log 这个目录下有生成多个 uuid 吗

nimeijian · 2020 年1 月 15 日 07:25

我docker-compose restart了一下，这个问题没有了，非常感谢！但第二个问题还在。

为了排除是 单节点上多个 DM-worker 实例的问题，我把3个worker部署到3台机子上了，就是说现在是单节点上单个 DM-Worker 实例的集群拓扑结构了，但跑着跑着还是就会出现下面的问题：

 query-status task_merge
{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "172.26.8.39:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "task_merge",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "UnknownError",
                                "msg": "[code=11029:class=functional:scope=internal:level=high] get stat for relay log /home/tidb/deploy/relay_log/54d11464-77ee-11e7-b052-9c5c8e7240e6.000001/mysql-bin.000257: stat /home/tidb/deploy/relay_log/54d11464-77ee-11e7-b052-9c5c8e7240e6.000001/mysql-bin.000257: no such file or directory\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).checkRelayPos\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:98\
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).StartSync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:122\
github.com/pingcap/dm/syncer.(*localBinlogReader).generateStreamer\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:102\
github.com/pingcap/dm/syncer.(*Syncer).Run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1079\
github.com/pingcap/dm/syncer.(*Syncer).Process\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:598\
github.com/pingcap/dm/syncer.(*Syncer).Resume\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:2306\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "0",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.000001, 121614583)",
                        "masterBinlogGtid": "07efafff-80cd-11e8-a628-52540038247b:104-109,4f51782c-b270-11e7-93f6-5254002147c0:1149040627-1183411505,54d11464-77ee-11e7-b052-9c5c8e7240e6:1-93980,8614aa74-6670-11e9-803d-5254005f2010:1-7,f5633113-666f-11e9-aeac-52540009d31a:1-287611299",
                        "syncerBinlog": "(mysql-bin|000001.000257, 474543098)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000001, 121614583)",
                "masterBinlogGtid": "54d11464-77ee-11e7-b052-9c5c8e7240e6:1-93980,8614aa74-6670-11e9-803d-5254005f2010:1-7,f5633113-666f-11e9-aeac-52540009d31a:1-287611299,07efafff-80cd-11e8-a628-52540038247b:104-109,4f51782c-b270-11e7-93f6-5254002147c0:1149040627-1183411505",
                "relaySubDir": "54d11464-77ee-11e7-b052-9c5c8e7240e6.000001",
                "relayBinlog": "(mysql-bin.000001, 121614583)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "replica-135"
        },
        {
            "result": true,
            "worker": "172.26.8.82:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "task_merge",
                    "stage": "Running",
                    "unit": "Load",
                    "result": null,
                    "unresolvedDDLLockID": "",
                    "load": {
                        "finishedBytes": "215552514",
                        "totalBytes": "240641592",
                        "progress": "89.57 %",
                        "metaBinlog": "(mysql-bin.000001, 154)"
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000001, 154)",
                "masterBinlogGtid": "4f51782c-b270-11e7-93f6-5254002147c0:1149040627-1367253991,7a5c0260-65a6-11e9-a7f0-525400804425:1-493574901,07efafff-80cd-11e8-a628-52540038247b:104-182,486d4dbd-666f-11e9-afe9-525400b94bd8:1-18",
                "relaySubDir": "9500e33f-f4c0-11e6-95b5-2c56dc4e770e.000001",
                "relayBinlog": "(mysql-bin.000001, 154)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "replica-132"
        },
        {
            "result": true,
            "worker": "172.26.8.83:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "task_merge",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "UnknownError",
                                "msg": "[code=11029:class=functional:scope=internal:level=high] get stat for relay log /home/tidb/deploy/relay_log/dfcf069c-976c-11e6-9ebe-305a3a5a429f.000001/mysql-bin.000103: stat /home/tidb/deploy/relay_log/dfcf069c-976c-11e6-9ebe-305a3a5a429f.000001/mysql-bin.000103: no such file or directory\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).checkRelayPos\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:98\
github.com/pingcap/dm/pkg/streamer.(*BinlogReader).StartSync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/streamer/reader.go:122\
github.com/pingcap/dm/syncer.(*localBinlogReader).generateStreamer\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:102\
github.com/pingcap/dm/syncer.(*Syncer).Run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1079\
github.com/pingcap/dm/syncer.(*Syncer).Process\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:598\
github.com/pingcap/dm/syncer.(*Syncer).Resume\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:2306\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "0",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.000001, 107563849)",
                        "masterBinlogGtid": "4f51782c-b270-11e7-93f6-5254002147c0:508480007-509375779:513595970-521566909,adb758e5-081a-11e9-8c70-525400dfdd97:1-4,c1bc3c0e-fc5e-11e8-a45c-5254007af3b6:3-743071983",
                        "syncerBinlog": "(mysql-bin|000001.000103, 521435566)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000001, 107563849)",
                "masterBinlogGtid": "4f51782c-b270-11e7-93f6-5254002147c0:508480007-509375779:513595970-521566909,adb758e5-081a-11e9-8c70-525400dfdd97:1-4,c1bc3c0e-fc5e-11e8-a45c-5254007af3b6:3-743071983",
                "relaySubDir": "dfcf069c-976c-11e6-9ebe-305a3a5a429f.000001",
                "relayBinlog": "(mysql-bin.000001, 107563849)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "replica-133"
        }
    ]
}

以下是139机子的relay_log目录结构：
tidb@yzf-ThinkCentre-M910t-D335:~/deploy/relay_log$ tree

.
├── 54d11464-77ee-11e7-b052-9c5c8e7240e6.000001
│   ├── mysql-bin.000001
│   └── relay.meta
└── server-uuid.index

server-uuid.index：

54d11464-77ee-11e7-b052-9c5c8e7240e6.000001

relay.meta：

binlog-name = "mysql-bin.000001"
binlog-pos = 121890965
binlog-gtid = ""

dm-worker.log：
dm-worker.log (110.3 KB)

疑问：

为什么跑着跑着 binlog 就会有问题呢？如何校正呢？

GangShen · 2020 年1 月 15 日 07:36

有设置 relay log 保留策略吗，可以看下是不是因为 relay log 被清理掉了

nimeijian · 2020 年1 月 15 日 07:38

没有额外设置什么保留策略，用的默认的，我 deploy 之前会手动删除 worker 的 deploy 目录，防止有脏数据，其它的没有操作过。

小王同学 · 2020 年1 月 15 日 07:45

检查下下游数据库是否存在 dm_meta 库，可以看下相关同步任务的 checkpoint 位点。猜测该位点信息是从 checkpoint 点中获取的，是否有修改过？

nimeijian · 2020 年1 月 15 日 07:58

有这个库，是从 checkpoint 中获取的，没有修改过这个dm_meta 库，但修改过数据库名，原来叫task，后来叫tidb_task，但 task_merge_syncer_checkpoint 表里 cp_schema 字段还显示 task，我把dm_meta 库清掉，重新deploy试一次？

小王同学 · 2020 年1 月 15 日 08:50

请问下是上游修改了库名吗？看起来

这个报错目录不存在也是因为 checkpoint 点有这个信息。可以看下这两条记录的 mvcc 版本，大概是什么时间插入的？ tidb/docs/tidb_http_api.md at master · pingcap/tidb · GitHub

如果需要快速恢复的话，可以把 dm_meta 清空，下游已导入数据清空，恢复同步任务，则重新开始 dump 并 load sync 流程。

nimeijian · 2020 年1 月 15 日 09:23

好的，已经可以同步了，非常感谢！我是3个库合成1个库，第1个库的数据已经全部同步过去了，但从第二个库开始，会出现主键重复，因为我们有的表数据需要合并，但有的表数据其实只要一份上游库里的就够了，就像数据字典表，每个库里的表数据都相同，对于这种数据可以让相同值的数据覆盖掉或者不同步过去吗，而不是因为主键值相等就暂停同步吗？因为一个库里的表可能比较多，这样我就需要一张表一张表去看哪些表要合，哪些表只要同步一份就够了。

GangShen · 2020 年1 月 15 日 09:42

上游两个库的表如果是有相同数据的话，load 阶段没办法避免，load 阶段的时候都是 insert 插入；需要手工处理；

sync 阶段可以把 safe-mode 打开，这样 insert 就会转成 replace into，覆盖掉

nimeijian · 2020 年1 月 30 日 06:55

load 阶段能否解析 Mydumper 后的数据并将 insert 转成 replace into？

或者更好的情况是在主键冲突时提供多种策略，比如：

整个同步过程pause，待手动解决后继续
报错，但继续同步
同主键记录覆盖
同主键记录不再同步

TiDB这边提供多种策略，具体使用什么策略由用户自己决定？

小王同学 · 2020 年1 月 30 日 08:47

不能。dump 和 load 是自动衔接的，并且 load 是将 insert 语句插入到数据中。同步出现主键冲突是上游存在冲突数据导致，建议去排查上游，目前主键冲突是需要手动解决。 sync 阶段可以开启 safe-mode。