DM迁移报错,看日志已经数据库连接已经断开连接了

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。

  • 【TiDB 版本】:4.0.0
  • 【问题描述】: DM合表合库之后报错。只是显示连接不上去。

报错部分

» query-status
{
    "result": true,
    "msg": "",
    "tasks": [
        {
            "taskName": "xue",
            "taskStatus": "Error - Some error occurred in subtask",
            "workers": [
                "10.204.54.85:8262"
            ]
        }
    ]
}

task配置

name: xue # global unique
task-mode: all  # full/incremental/all
is-sharding: true

target-database:
  host: "10.204.54.78"
  port: 4000
  user: "root"
  password: ""

mysql-instances:             # one or more source database, config more source database for sharding merge
  -
    source-id: "mysql-replica-01" # unique in all instances, used as id when save checkpoints, configs, etc.
    route-rules: ["user-route-rule"]

    black-white-list:  "instance"
    column-mapping-rules: ["cm001","cm002","cm003"]
    # number of threads to use for dump data
    mydumper-thread: 4

    # number of threads to use for load data
    loader-thread: 16

    # number of threads to use for replicate binlog
    syncer-thread: 16


black-white-list:
  instance:
    do-dbs: ["uic2_*"]        # white list of upstream schemas needs to be replicated, regular expression (starts with ~) is supported
    ignore-tables:
    - db-name: "uic2_*"
      tbl-name: "~s_uic*"


routes:
  user-route-rule:
    schema-pattern: "uic2_*"
    target-schema: "uic"


column-mappings:
  cm001:
    schema-pattern: "uic2_*"
    table-pattern: "uic_user_passive_associated"
    expression: "partition id"
    source-column: "id"
    target-column: "id"
    arguments: ["1","uic2","uic_user_passive_associated","_"]
  cm002:
    schema-pattern: "uic2_*"
    table-pattern: "uic_user_face_compare"
    expression: "partition id"
    source-column: "id"
    target-column: "id"
    arguments: ["1","uic2","uic_user_passive_associated","_"]
  cm003:
    schema-pattern: "uic2_*"
    table-pattern: "uic_user_active_associated"
    expression: "partition id"
    source-column: "id"
    target-column: "id"
    arguments: ["1","uic2","uic_user_passive_associated","_"]

报错日志
dm-worker.log (3.6 MB)

你好,

  1. 提供下 dm 版本
  2. query-status xue 返回下。

dm:1.0.5

» query-status xue
{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "10.204.54.85:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "xue",
                    "stage": "Paused",
                    "unit": "Load",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "UnknownError",
                                "msg": "[code=34004:class=load-unit:scope=internal:level=high] restore data file (uic2_7.uic_user_face_compare.sql) failed: file dumped_data.xue/uic2_7.uic_user_face_compare.sql: mapping row data [295324168918953984 295266455908282368 http://appmgr-1251779293.file.myqcloud.com/faceImage/2018-10/2952664559082823681539931469410.jpg 87.00 2018-10-19 14:44:31 丰巢APP 2018-10-19 14:44:31 2019-07-31 21:19:24 {\\\"images\\\":[{\\\"score\\\":86.87,\\\"time\\\":1564579164168,\\\"url\\\":\\\"http://appmgr-1251779293.file.myqcloud.com/faceImage/2019-07/2952664559082823681564579160641.jpg\\\"}]} 1 0  http://appmgr-1251779293.file.myqcloud.com/faceImage/2019-07/2952664559082823681564579160641.jpg 86.87 2019-07-31 21:19:24 6] for table \u0026{sourceSchema:uic2_7 sourceTable:uic_user_face_compare targetSchema:uic targetTable:uic_user_face_compare columnNameList:[id user_id url score opt_tm source create_time update_time face_images status is_del remark face_url face_score face_time face_source] insertHeadStmt:INSERT INTO `uic_user_face_compare` VALUES}: uic_user_passive_associated_ is not the prefix of uic_user_face_compare not valid\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/loader.parseRowValues\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/convert_data.go:162\
github.com/pingcap/dm/loader.parseInsertStmt\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/convert_data.go:85\
github.com/pingcap/dm/loader.reassemble\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/convert_data.go:297\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:304\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_2.uic_open_user_detail.sql) failed: initialize checkpoint: execute statement failed: begin: context canceled\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_3.uic_account_index.sql) failed: initialize checkpoint: execute statement failed: begin: driver: bad connection\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_8.uic_open_user_detail.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_4.uic_wechat_index.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_2.uic_account_index.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_1.uic_account_index.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_7.uic_user_active_associated.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_1.uic_unionid_index.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_4.uic_account_index.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_1.uic_user_face_store.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_4.uic_user_active_associated.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_8.uic_user_level.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_7.uic_wechat_index.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_3.uic_user_active_associated.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "UnknownError",
                                "msg": "[code=10006:class=database:scope=downstream:level=high] restore data file (uic2_6.uic_user.sql) failed: initialize checkpoint: execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:149\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:210\
github.com/pingcap/dm/loader.(*DBConn).executeSQL.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:150\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:216\
github.com/pingcap/dm/loader.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/db.go:145\
github.com/pingcap/dm/loader.(*RemoteCheckPoint).Init\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/checkpoint.go:301\
github.com/pingcap/dm/loader.(*Worker).dispatchSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:257\
github.com/pingcap/dm/loader.(*Worker).restoreDataFile\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:218\
github.com/pingcap/dm/loader.(*Worker).run\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:206\
github.com/pingcap/dm/loader.(*Loader).initAndStartWorkerPool.func1\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:778\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "load": {
                        "finishedBytes": "181835",
                        "totalBytes": "29767740",
                        "progress": "0.61 %",
                        "metaBinlog": "(uic-mysql-bin.000484, 2107039)"
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(uic-mysql-bin.000484, 2145092)",
                "masterBinlogGtid": "e85cd2a9-eeba-11e6-a793-005056b40da0:1-220978678",
                "relaySubDir": "e85cd2a9-eeba-11e6-a793-005056b40da0.000001",
                "relayBinlog": "(uic-mysql-bin.000484, 2145092)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "mysql-replica-01"
        }
    ]
}

你好,

子任务 xue 目前处于 load 阶段,已完成 0.61 %

报错信息为

execute statement failed: begin: sql: connection is already closed

辛苦提供下 tidb.log 看下是否有相关报错

提取出来的错误日志
logs-tidb_10_204_54_78_4000.tar (138.5 KB)

完整的日志
logs-tidb_10_204_54_78_4000_all.tar (282.5 KB)

看起来是tidb 写check_point写不进去?

[2020/06/16 17:59:04.126 +08:00] [INFO] [tidb.go:200] ["rollbackTxn for ddl/autocommit failed"]
[2020/06/16 17:59:04.126 +08:00] [WARN] [session.go:1044] ["run statement failed"] [conn=677619] [schemaVersion=4461] [error="[schema:1050]Table 'uic.uic_open_index' already exists"] [session="{\
  \"currDBName\": \"uic\",\
  \"id\": 677619,\
  \"status\": 2,\
  \"strictMode\": true,\
  \"user\": {\
    \"Username\": \"root\",\
    \"Hostname\": \"10.204.54.85\",\
    \"CurrentUser\": false,\
    \"AuthUsername\": \"root\",\
    \"AuthHostname\": \"%\"\
  }\
}"]
[2020/06/16 17:59:04.126 +08:00] [ERROR] [conn.go:728] ["command dispatched failed"] [conn=677619] [connInfo="id:677619, addr:10.204.54.85:53812 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="CREATE TABLE `uic_open_index` (`open_id` varchar(64) COLLATE utf8mb4_bin NOT NULL COMMENT '开放平台openId',`user_id` bigint(20) NOT NULL COMMENT '用户id',`type` tinyint(4) NOT NULL DEFAULT '1' COMMENT '接入的开放平台类型:1微信公众号/2支付宝服务窗/3京东服务号',PRIMARY KEY (`open_id`,`user_id`,`type`),KEY `idx_open_id_type` (`open_id`,`type`) USING BTREE) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin COMMENT='开放平台open_idd和user_id的映射表';"] [txn_mode=OPTIMISTIC] [err="[schema:1050]Table 'uic.uic_open_index' already exists\
github.com/pingcap/errors.AddStack\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\
github.com/pingcap/parser/terror.(*Error).GenWithStackByArgs\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20200427031542-879c7bd4f27d/terror/terror.go:243\
github.com/pingcap/tidb/ddl.(*ddl).CreateTableWithInfo\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/ddl_api.go:1516\
github.com/pingcap/tidb/ddl.(*ddl).CreateTable\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/ddl_api.go:1498\
github.com/pingcap/tidb/executor.(*DDLExec).executeCreateTable\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/ddl.go:195\
github.com/pingcap/tidb/executor.(*DDLExec).Next\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/ddl.go:92\
github.com/pingcap/tidb/executor.Next\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/executor.go:249\
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/adapter.go:507\
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/adapter.go:389\
github.com/pingcap/tidb/executor.(*ExecStmt).Exec\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/adapter.go:357\
github.com/pingcap/tidb/session.runStmt\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/session/tidb.go:281\
github.com/pingcap/tidb/session.(*session).executeStatement\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/session/session.go:1041\
github.com/pingcap/tidb/session.(*session).execute\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/session/session.go:1159\
github.com/pingcap/tidb/session.(*session).Execute\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/session/session.go:1089\
github.com/pingcap/tidb/server.(*TiDBContext).Execute\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/driver_tidb.go:248\
github.com/pingcap/tidb/server.(*clientConn).handleQuery\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/conn.go:1265\
github.com/pingcap/tidb/server.(*clientConn).dispatch\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/conn.go:899\
github.com/pingcap/tidb/server.(*clientConn).Run\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/conn.go:713\
github.com/pingcap/tidb/server.(*Server).onConn\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/server.go:415\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
[2020/06/16 17:59:04.128 +08:00] [INFO] [session.go:2126] ["CRUCIAL OPERATION"] [conn=677619] [schemaVersion=4461] [cur_db=uic] [sql="CREATE TABLE `uic_account_index` (`account` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL COMMENT '账户',`user_id` bigint(20) NOT NULL COMMENT 'uic_user_id',PRIMARY KEY (`account`)) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='账户id索引表';"] [user=root@10.204.54.85]
[2020/06/16 17:59:04.128 +08:00] [INFO] [session.go:1418] ["NewTxn() inside a transaction auto commit"] [conn=677619] [schemaVersion=4461] [txnStartTS=417412295983104008]
[2020/06/16 17:59:04.128 +08:00] [INFO] [tidb.go:200] ["rollbackTxn for ddl/autocommit failed"]
[2020/06/16 17:59:04.128 +08:00] [WARN] [session.go:1044] ["run statement failed"] [conn=677619] [schemaVersion=4461] [error="[schema:1050]Table 'uic.uic_account_index' already exists"] [session="{\
  \"currDBName\": \"uic\",\
  \"id\": 677619,\
  \"status\": 2,\
  \"strictMode\": true,\
  \"user\": {\
    \"Username\": \"root\",\
    \"Hostname\": \"10.204.54.85\",\
    \"CurrentUser\": false,\
    \"AuthUsername\": \"root\",\
    \"AuthHostname\": \"%\"\
  }\
}"]
[2020/06/16 17:59:04.128 +08:00] [ERROR] [conn.go:728] ["command dispatched failed"] [conn=677619] [connInfo="id:677619, addr:10.204.54.85:53812 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="CREATE TABLE `uic_account_index` (`account` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL COMMENT '账户',`user_id` bigint(20) NOT NULL COMMENT 'uic_user_id',PRIMARY KEY (`account`)) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='账户id索引表';"] [txn_mode=OPTIMISTIC] [err="[schema:1050]Table 'uic.uic_account_index' already exists\
github.com/pingcap/errors.AddStack\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\
github.com/pingcap/parser/terror.(*Error).GenWithStackByArgs\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20200427031542-879c7bd4f27d/terror/terror.go:243\
github.com/pingcap/tidb/ddl.(*ddl).CreateTableWithInfo\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/ddl_api.go:1516\
github.com/pingcap/tidb/ddl.(*ddl).CreateTable\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/ddl_api.go:1498\
github.com/pingcap/tidb/executor.(*DDLExec).executeCreateTable\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/ddl.go:195\
github.com/pingcap/tidb/executor.(*DDLExec).Next\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/ddl.go:92\
github.com/pingcap/tidb/executor.Next\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/executor.go:249\
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/adapter.go:507\
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/adapter.go:389\
github.com/pingcap/tidb/executor.(*ExecStmt).Exec\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/adapter.go:357\
github.com/pingcap/tidb/session.runStmt\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/session/tidb.go:281\
github.com/pingcap/tidb/session.(*session).executeStatement\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/session/session.go:1041\
github.com/pingcap/tidb/session.(*session).execute\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/session/session.go:1159\
github.com/pingcap/tidb/session.(*session).Execute\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/session/session.go:1089\
github.com/pingcap/tidb/server.(*TiDBContext).Execute\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/driver_tidb.go:248\
github.com/pingcap/tidb/server.(*clientConn).handleQuery\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/conn.go:1265\
github.com/pingcap/tidb/server.(*clientConn).dispatch\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/conn.go:899\
github.com/pingcap/tidb/server.(*clientConn).Run\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/conn.go:713\
github.com/pingcap/tidb/server.(*Server).onConn\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/server.go:415\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
[2020/06/16 17:59:04.130 +08:00] [INFO] [session.go:2126] ["CRUCIAL OPERATION"] [conn=677619] [schemaVersion=4461] [cur_db=uic] [sql="CREATE TABLE `uic_wechat_index` (`wx_id` varchar(150) DEFAULT '' COMMENT '微信openid',`user_id` bigint(20) NOT NULL COMMENT 'uic_user_id',KEY `idx_wx_id` (`wx_id`) USING BTREE) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='微信id索引表';"] [user=root@10.204.54.85]
[2020/06/16 17:59:04.130 +08:00] [INFO] [session.go:1418] ["NewTxn() inside a transaction auto commit"] [conn=677619] [schemaVersion=4461] [txnStartTS=417412295983104011]
[2020/06/16 17:59:04.130 +08:00] [INFO] [tidb.go:200] ["rollbackTxn for ddl/autocommit failed"]
[2020/06/16 17:59:04.130 +08:00] [WARN] [session.go:1044] ["run statement failed"] [conn=677619] [schemaVersion=4461] [error="[schema:1050]Table 'uic.uic_wechat_index' already exists"] [session="{\
  \"currDBName\": \"uic\",\
  \"id\": 677619,\
  \"status\": 2,\
  \"strictMode\": true,\
  \"user\": {\
    \"Username\": \"root\",\
    \"Hostname\": \"10.204.54.85\",\
    \"CurrentUser\": false,\
    \"AuthUsername\": \"root\",\
    \"AuthHostname\": \"%\"\
  }\
}"]
[2020/06/16 17:59:04.130 +08:00] [ERROR] [conn.go:728] ["command dispatched failed"] [conn=677619] [connInfo="id:677619, addr:10.204.54.85:53812 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="CREATE TABLE `uic_wechat_index` (`wx_id` varchar(150) DEFAULT '' COMMENT '微信openid',`user_id` bigint(20) NOT NULL COMMENT 'uic_user_id',KEY `idx_wx_id` (`wx_id`) USING BTREE) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='微信id索引表';"] [txn_mode=OPTIMISTIC] [err="[schema:1050]Table 'uic.uic_wechat_index' already exists\
github.com/pingcap/errors.AddStack\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\
github.com/pingcap/parser/terror.(*Error).GenWithStackByArgs\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20200427031542-879c7bd4f27d/terror/terror.go:243\
github.com/pingcap/tidb/ddl.(*ddl).CreateTableWithInfo\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/ddl_api.go:1516\
github.com/pingcap/tidb/ddl.(*ddl).CreateTable\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/ddl_api.go:1498\
github.com/pingcap/tidb/executor.(*DDLExec).executeCreateTable\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/ddl.go:195\
github.com/pingcap/tidb/executor.(*DDLExec).Next\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/ddl.go:92\
github.com/pingcap/tidb/executor.Next\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/executor.go:249\
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/adapter.go:507\
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/adapter.go:389\
github.com/pingcap/tidb/executor.(*ExecStmt).Exec\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/executor/adapter.go:357\
github.com/pingcap/tidb/session.runStmt\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/session/tidb.go:281\
github.com/pingcap/tidb/session.(*session).executeStatement\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/session/session.go:1041\
github.com/pingcap/tidb/session.(*session).execute\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/session/session.go:1159\
github.com/pingcap/tidb/session.(*session).Execute\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/session/session.go:1089\
github.com/pingcap/tidb/server.(*TiDBContext).Execute\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/driver_tidb.go:248\
github.com/pingcap/tidb/server.(*clientConn).handleQuery\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/conn.go:1265\
github.com/pingcap/tidb/server.(*clientConn).dispatch\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/conn.go:899\
github.com/pingcap/tidb/server.(*clientConn).Run\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/conn.go:713\
github.com/pingcap/tidb/server.(*Server).onConn\
\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/server/server.go:415\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
[2020/06/16 17:59:04.131 +08:00] [INFO] [server.go:391] ["connection closed"] [conn=677619]
[2020/06/16 17:59:04.161 +08:00] [WARN] [session.go:474] ["can not retry txn"] [conn=677618] [label=general] [error="[kv:1062]Duplicate entry 'mysql-replica-01-uic2_2.uic_user_active_associated.sql' for key 'uk_id_f'"] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=false] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2020/06/16 17:59:04.161 +08:00] [WARN] [session.go:490] ["commit failed"] [conn=677618] ["finished txn"="Txn{state=invalid}"] [error="[kv:1062]Duplicate entry 'mysql-replica-01-uic2_2.uic_user_active_associated.sql' for key 'uk_id_f'"]
[2020/06/16 17:59:04.161 +08:00] [ERROR] [conn.go:728] ["command dispatched failed"] [conn=677618] [connInfo="id:677618, addr:10.204.54.85:53810 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql=COMMIT] [txn_mode=OPTIMISTIC] [err="previous statement: INSERT INTO `dm_meta`.`xue_loader_checkpoint` (`id`, `filename`, `cp_schema`, `cp_table`, `offset`, `end_pos`) VALUES('mysql-replica-01','uic2_2.uic_user_active_associated.sql','uic2_2','uic_user_active_associated',0,2949): [kv:1062]Duplicate entry 'mysql-replica-01-uic2_2.uic_user_active_associated.sql' for key 'uk_id_f'"]

你好

tidb 日志中大都 error 为 table already exists,

可以尝试通过以下步骤看是否可以恢复。

  1. stop-task xue
  2. 删除下游中与 dump 冲突的数据,譬如截图中的 uic.uic_wechat_index
  3. 删除 dm-meta 中 load 相关的表
  4. start-task

这是合表操作,报表已存在不是很正常么。不过我看他老在retry。数据的话uic_wechat_index 这张表里没数据呢。
刚才已经unsafe_cleaup清空了整个md重新弄过。
也删除了dm_meta库。
还删除了目标库上的所有表。

dumper出来的表结构是预期中的表结构和数据,都是预期中的


但是写入dm_meta的数据是残缺不全的。

查出来了,是合表的时候有唯一键冲突。暂时屏蔽掉该表就正常了

[2020/06/16 18:57:17.630 +08:00] [ERROR] [db.go:176] ["execute statements failed after retry"] [task=xue2] [unit=load] ["worker ID"=3] [queries="[USE `uic`; INSERT INTO `uic_user_verified_info` VALUES\
(267129000466345984,'431081198912311350','罗涛',1,'汉','/2019-11-05/35082652833182515215729330675871.jpg','/2019-11-05/35082652833182515215729330841552.jpg','1989-12-31 00:00:00','2016-12-02 00:00:00','2036-12-02 23:59:59','深圳市公安局宝安分局','广东省深圳市龙华新区东环二路2号',1,1,'','350826528331825152','2019-11-05 13:51:44','350826528331825152','2019-11-05 13:51:45',1,''),\
(298884867540680704,'360321199201087028','刘倩',NULL,'','','',NULL,NULL,NULL,NULL,NULL,1,1,'','175391136548945920','2018-10-29 10:33:28','175391136548945920','2018-10-29 10:33:28',0,''),\
(302272188285747200,'452122199111023038','苏常宇',1,'壮','/2020-05-27/28664943473173708815905619218841.jpg','/2020-05-27/28664943473173708815905619327322.jpg','1991-11-02 00:00:00','2013-03-08 00:00:00','2023-03-08 23:59:59','横县公安局','广西横县平马镇平马街5号',1,1,'','286649434731737088','2020-05-27 14:45:38','286649434731737088','2020-05-27 14:4..."] [arguments="[]"] [error="[code=10006:class=database:scope=not-set:level=high] execute statement failed: commit: Error 1062: Duplicate entry '420922198902060013' for key 'card_no'"]

:call_me_hand:

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。