sql-skip not working

elvizlai · 2020 年7 月 10 日 13:48

DM2

Release Version: v2.0.0-beta.2
Git Commit Hash: e1c5376b4d3d31c353c6431d3eb9e91dc0608105
Git Branch: master
UTC Build Time: 2020-06-19 02:26:16
Go Version: go version go1.13 linux/amd64

"msg": "current location position: (mysql-bin.000010, 730462614), gtid-set: 08e024dd-22db-11ea-837a-00163e04dda5:1-3918708: encountered incompatible DDL in TiDB:\
\tplease confirm your DDL statement is correct and needed.\
\tfor TiDB compatible DDL, please see the docs:\
\t  English version: https://pingcap.com/docs/dev/reference/mysql-compatibility/#ddl\
\t  Chinese version: https://pingcap.com/docs-cn/dev/reference/mysql-compatibility/#ddl\
\tif the DDL is not needed, you can use a filter rule with \"*\" schema-pattern to ignore it.\
\t : parse statement: line 1 column 28 near \"error='Value 2464021540749.000000 is too small or too large.' where itemid=31598\" %!!(MISSING)(EXTRA string=update item_rtdata set error='Value 2464021540749.000000 is too small or too large.' where itemid=31598)"

When error occurred, try using sql-skip cmd to skip, but still not working.

sql-skip -s mysql-replica-01 --binlog-pos=mysql-bin.000010:730462614 test

来了老弟 · 2020 年7 月 13 日 01:02

目前 dm 2.0 还是 master 版本，建议先使用 dm v1.0.6 来进行同步任务，等到 2.0 GA 后在进行线上使用，

当前问题请确认在执行完 sql-skip 之后同步暂停的 pos 点是否已经变化？并非当前指定的 mysql-bin.000010:730462614。请告知

确认下执行完 sql-skip 之后 dm-worker log 中是否有相关记录呢。可以上传一下

sql-skip 的使用文档：
https://docs.pingcap.com/zh/tidb-data-migration/stable/skip-or-replace-abnormal-sql-statements/

elvizlai · 2020 年7 月 13 日 01:26

主要是部分数据库名存在问题，形如：abc-xyz (中划线，不是下划线)。DM1 对应的 mydumper 本身无法支持，遵循了 github 的 issue 使用 DM2 做迁移测试。

来了老弟 · 2020 年7 月 13 日 01:44

中划线等表名格式问题还可以在 mydumper 的 extra-args 中指定 -x 或者 --regex 方式处理，使用 dm 1.0.6 同步试下

elvizlai · 2020 年7 月 13 日 02:01

通过 regex 确实可以解决数据库名称问题。

比如只处理两个数据库中的所有表，参见以下 task 配置

task

---
name: testx
task-mode: all

target-database:
  host: "127.0.0.1"
  port: 4000
  user: "root"
  password: ""

routes:
  route-rule-1:
    schema-pattern: "abc-workorder"
    target-schema: "dev_abc_workorder"
  route-rule-2:
    schema-pattern: "abc_mall_serviceTrade"
    target-schema: "dev_abc_mall_trade"

mysql-instances:
  - source-id: "mysql-replica-01"
    route-rules: ["route-rule-1", "route-rule-2"]
    mydumper-config-name: "global"
    mydumper-thread: 4
    loader-thread: 16
    syncer-thread: 16

mydumpers:
  global:
    # default s=1000000
    extra-args: "-s 100000 -x '^(abc-workorder|abc_mall_serviceTrade)$' --no-locks"

但是执行一会后报错

{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "127.0.0.1:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "testx",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "UnknownError",
                                "msg": "",
                                "error": {
                                    "ErrCode": 10005,
                                    "ErrClass": 1,
                                    "ErrScope": 2,
                                    "ErrLevel": 3,
                                    "Message": "current pos (mysql-bin|000001.000011, 556541211): query statement failed: SHOW COLUMNS FROM `abc-systemctl`.`sys_user_access`: Error 1146: Table 'abc-systemctl.sys_user_access' doesn't exist",
                                    "RawCause": "Error 1146: Table 'abc-systemctl.sys_user_access' doesn't exist"
                                }
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "0",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.000011, 557596831)",
                        "masterBinlogGtid": "08e024dd-22db-11ea-837a-00163e04dda5:1-3918708",
                        "syncerBinlog": "(mysql-bin.000011, 556540807)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000011, 557596831)",
                "masterBinlogGtid": "08e024dd-22db-11ea-837a-00163e04dda5:1-3918708",
                "relaySubDir": "0652de9c-ddd6-11e9-8f03-00163e00241b.000001",
                "relayBinlog": "(mysql-bin.000011, 557596831)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "mysql-replica-01"
        }
    ]
}

尝试跳过，但无效（看起来不像是 binlog 错误）

sql-skip -w 127.0.0.1:8262 --binlog-pos=mysql-bin|000001.000011:556541211 testx
resume-task testx

所有数据清除（db_meta, relay_log）等
然后重新执行任务报错

{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "127.0.0.1:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "testx",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "UnknownError",
                                "msg": "",
                                "error": {
                                    "ErrCode": 36014,
                                    "ErrClass": 11,
                                    "ErrScope": 3,
                                    "ErrLevel": 3,
                                    "Message": "current pos (mysql-bin|000001.000011, 560220437): query insert into history (itemid,clock,ns,value) values (33178,1594609498,138348513,0.050000),(10078,1594609498,138405431,0.000000),(31798,1594609498,139342296,0.000000),(30418,1594609498,157258671,0.135387),(30478,1594609498,157724065,0.000000),(30538,1594609498,161597569,1297.693433),(33118,1594609498,162150819,0.000000),(31618,1594609498,177130351,40167.000000): only support ROW format binlog, unexpected DML statement found in query event",
                                    "RawCause": ""
                                }
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "0",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.000011, 560236982)",
                        "masterBinlogGtid": "08e024dd-22db-11ea-837a-00163e04dda5:1-3918708",
                        "syncerBinlog": "(mysql-bin.000011, 560219850)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000011, 560236982)",
                "masterBinlogGtid": "08e024dd-22db-11ea-837a-00163e04dda5:1-3918708",
                "relaySubDir": "0652de9c-ddd6-11e9-8f03-00163e00241b.000001",
                "relayBinlog": "(mysql-bin.000011, 560236982)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "mysql-replica-01"
        }
    ]
}

该相关错误依然无法跳过。

来了老弟 · 2020 年7 月 13 日 03:12

这个报错是正常的吧，因为没有复制该表，通过 binlog event 跳过这个报错是下。

这个问题可以看下 dm-worker log 中是否有信息，先 stop.yml -l dm-worker02，在删除 relay log 目录，在 start.yml -l dm-worker02，防止 relay log 从 dm-wroker 缓存中在读取出来。

elvizlai · 2020 年7 月 13 日 03:23

跳过无效。
按文档提示，先 skip 然后 resume，还是报这个错误（看起来就是 skip 无效，但实际上master/worker都有跳过的日志）。

另：上文中的 regex 好像写错了。
用 mydumper 命令直接执行 -x '^(fsk-workorder|fsk_mall_serviceTrade)$' 是不能备份出来的，将 $去掉就可以。

来了老弟 · 2020 年7 月 13 日 03:42

https://docs.pingcap.com/zh/tidb-data-migration/stable/feature-overview#binlog-event-filter

elvizlai · 2020 年7 月 13 日 04:56

{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "127.0.0.1:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "testx",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "UnknownError",
                                "msg": "",
                                "error": {
                                    "ErrCode": 36014,
                                    "ErrClass": 11,
                                    "ErrScope": 3,
                                    "ErrLevel": 3,
                                    "Message": "current pos (mysql-bin|000001.000011, 563512734): query insert into history (itemid,clock,ns,value) values (30545,1594610825,619296162,0.002084),(29165,1594610825,631770238,0.000000),(30485,1594610825,638184812,0.084674),(33185,1594610825,638661342,98.755741),(33210,1594610825,638661342,1.244259),(33605,1594610825,638779336,5.291456),(33125,1594610825,639247050,0.000000),(33245,1594610825,642376694,0.000000),(32885,1594610825,642678786,1.109648): only support ROW format binlog, unexpected DML statement found in query event",
                                    "RawCause": ""
                                }
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "0",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.000011, 580672689)",
                        "masterBinlogGtid": "08e024dd-22db-11ea-837a-00163e04dda5:1-3918708",
                        "syncerBinlog": "(mysql-bin.000011, 563512115)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000011, 580672689)",
                "masterBinlogGtid": "08e024dd-22db-11ea-837a-00163e04dda5:1-3918708",
                "relaySubDir": "0652de9c-ddd6-11e9-8f03-00163e00241b.000001",
                "relayBinlog": "(mysql-bin.000011, 580672689)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "mysql-replica-01"
        }
    ]
}

这类 insert 也需要 events 跳过？我 MySQL 配置的 ROW format，为何还是提示 format 问题。

show global variables like '%binlog_format%'

result:
binlog_format,ROW

来了老弟 · 2020 年7 月 13 日 05:48

hi，请返回下以下命令结果，

select variable_name, variable_value from information_schema.global_variables where variable_name in ('binlog_format','binlog_row_image');

elvizlai · 2020 年7 月 13 日 06:39

The given command is not executable on our server:

[HY000][3167] The 'INFORMATION_SCHEMA.GLOBAL_VARIABLES' feature is disabled; see the documentation for 'show_compatibility_56'

Try using command instead: show global variables like 'binlog_%'

来了老弟 · 2020 年7 月 13 日 06:43

sorry 以上命令是在 mysql 中执行，请问当前 mysql 版本时多少呢

elvizlai · 2020 年7 月 13 日 06:56

SELECT version(); 5.7.17-log

我开启 set @@global.show_compatibility_56=ON; 之后，以上命令就可以执行了。

移除 TiDB 中的 dm_meta，重新搭建了一套 DM1，重新开始同步，提示以下信息，其实很奇怪，Table ‘zabbix.history’ doesn’t exist"

zabbix.history 本来就不是一个 table，而是 db 跟 table 的组合，这块感觉是使用正则后导致的？（其实这个数据库.表的组合在数据库里是真实存在的）

{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "127.0.0.1:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "testx",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "UnknownError",
                                "msg": "",
                                "error": {
                                    "ErrCode": 10005,
                                    "ErrClass": 1,
                                    "ErrScope": 2,
                                    "ErrLevel": 3,
                                    "Message": "current pos (mysql-bin|000001.000012, 4180741): query statement failed: SHOW COLUMNS FROM `zabbix`.`history`: Error 1146: Table 'zabbix.history' doesn't exist",
                                    "RawCause": "Error 1146: Table 'zabbix.history' doesn't exist"
                                }
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "0",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.000012, 5022580)",
                        "masterBinlogGtid": "08e024dd-22db-11ea-837a-00163e04dda5:1-3918708",
                        "syncerBinlog": "(mysql-bin.000012, 4180236)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000012, 5022580)",
                "masterBinlogGtid": "08e024dd-22db-11ea-837a-00163e04dda5:1-3918708",
                "relaySubDir": "0652de9c-ddd6-11e9-8f03-00163e00241b.000001",
                "relayBinlog": "(mysql-bin.000012, 5022580)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "mysql-replica-01"
        }
    ]
}

来了老弟 · 2020 年7 月 13 日 07:16

过滤规则是什么样子的呢。

elvizlai · 2020 年7 月 13 日 07:18

---
name: testx
task-mode: all

target-database:
  host: "127.0.0.1"
  port: 4000
  user: "root"
  password: ""

routes:
  route-rule-1:
    schema-pattern: "abc-workorder"
    target-schema: "dev_abc_workorder"
  route-rule-2:
    schema-pattern: "abc_mall_serviceTrade"
    target-schema: "dev_abc_mall_trade"

mysql-instances:
  - source-id: "mysql-replica-01"
    route-rules: ["route-rule-1", "route-rule-2"]
    mydumper-config-name: "global"
    mydumper-thread: 4
    loader-thread: 16
    syncer-thread: 16

mydumpers:
  global:
    # default s=1000000
    extra-args: "-s 100000 -x '^(abc-workorder|abc_mall_serviceTrade)' --no-locks"

task 任务参见如上

来了老弟 · 2020 年7 月 13 日 08:30

mydumpers: 中是过滤了 dumpe 时的条件，建议添加 black write list 增加 sync 阶段的同步。譬如我只 dumpe test 数据库的信息，但是没有设置黑白名单，那 sync 阶段会同步所有的 binlog 譬如会同步 test2 数据库的 binlog 那此时 dm 就会报错。

https://docs.pingcap.com/zh/tidb-data-migration/stable/task-configuration-file-full

elvizlai · 2020 年7 月 13 日 08:58

你这个解决方案又绕回来了。

历史原因，我们的数据库名称用 xyz-abc 的命名。写到 black-white-list 中就会直接报错了，所有采用了正则的方案。

来了老弟 · 2020 年7 月 13 日 09:04

mydumpers 部分的使用方式不变，在 black-white-list 加一下这边看下，mydumpers 的参数生效等级 > black-write-list。

elvizlai · 2020 年7 月 13 日 10:17

依然跳过不了

{
    "op": "Resume",
    "result": true,
    "msg": "",
    "workers": [
        {
            "meta": {
                "result": true,
                "worker": "127.0.0.1:8262",
                "msg": ""
            },
            "op": "Resume",
            "logID": "11"
        }
    ]
}
» query-status testx
{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "127.0.0.1:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "testx",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "UnknownError",
                                "msg": "",
                                "error": {
                                    "ErrCode": 11006,
                                    "ErrClass": 2,
                                    "ErrScope": 3,
                                    "ErrLevel": 3,
                                    "Message": "current pos (mysql-bin|000001.000012, 53304437): encountered incompatible DDL in TiDB:\
\tplease confirm your DDL statement is correct and needed.\
\tfor TiDB compatible DDL, please see the docs:\
\t  English version: https://pingcap.com/docs/dev/reference/mysql-compatibility/#ddl\
\t  Chinese version: https://pingcap.com/docs-cn/dev/reference/mysql-compatibility/#ddl\
\tif the DDL is not needed, you can use a filter rule with \"*\" schema-pattern to ignore it.\
\t : parse statement: [parser:1115]Unknown character set: 'utf32'%!!(MISSING)(EXTRA string=ALTER TABLE `fsk_erp_trade`.`et_device_order_round` \r\
MODIFY COLUMN `id` varchar(50) CHARACTER SET utf32 NOT NULL COMMENT '主键' FIRST,\r\
MODIFY COLUMN `round_detail` varchar(1000) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL COMMENT '查房详情' AFTER `round_type`)",
                                    "RawCause": "[parser:1115]Unknown character set: 'utf32'"
                                }
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "30785",
                        "totalTps": "11",
                        "recentTps": "1",
                        "masterBinlog": "(mysql-bin.000012, 54790690)",
                        "masterBinlogGtid": "08e024dd-22db-11ea-837a-00163e04dda5:1-3918708",
                        "syncerBinlog": "(mysql-bin|000001.000012, 53304003)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.000012, 54790690)",
                "masterBinlogGtid": "08e024dd-22db-11ea-837a-00163e04dda5:1-3918708",
                "relaySubDir": "0652de9c-ddd6-11e9-8f03-00163e00241b.000001",
                "relayBinlog": "(mysql-bin.000012, 54790690)",
                "relayBinlogGtid": "",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "mysql-replica-01"
        }
    ]
}

难道是 pos 用错了？这类数据库编码错误类的，也只有个别库需要一次性处理。这类无法预知。

尝试使用一下代码跳过：

sql-skip -w 127.0.0.1:8262 --binlog-pos=mysql-bin|000001.000012:53304437 testx
resume-task testx

master log

[2020/07/13 18:16:14.083 +08:00] [INFO] [server.go:738] ["task name"=testx] [payload="name:\"testx\" binlogPos:\"mysql-bin|000001.000012:53304437\" worker:\"127.0.0.1:8262\" "] [request=HandleSQLs]
[2020/07/13 18:16:16.633 +08:00] [INFO] [server.go:323] [payload="op:Resume name:\"testx\" "] [request=OperateTask]
[2020/07/13 18:16:16.634 +08:00] [INFO] [server.go:1913] ["wait op log result"] [task=testx] [worker=127.0.0.1:8262] ["operation log ID"=15] [result="meta:<result:true > log:<id:15 task:<op:Resume name:\"testx\" > ts:1594635376633535417 > "]

worker log

[2020/07/13 18:16:07.407 +08:00] [WARN] [task_checker.go:358] ["backoff skip auto resume task"] [component="task checker"] [task=testx] [latestResumeTime=2020/07/13 18:14:02.407 +08:00] [duration=5m0s]
[2020/07/13 18:16:12.407 +08:00] [WARN] [task_checker.go:358] ["backoff skip auto resume task"] [component="task checker"] [task=testx] [latestResumeTime=2020/07/13 18:14:02.407 +08:00] [duration=5m0s]
[2020/07/13 18:16:14.083 +08:00] [INFO] [server.go:346] [request=HandleSQLs] [payload="name:\"testx\" binlogPos:\"mysql-bin|000001.000012:53304437\" "]
[2020/07/13 18:16:14.083 +08:00] [WARN] [operator.go:133] ["overwrite operator"] [task=testx] [unit="binlog replication"] ["old operator"="uuid: 7b675dad-5116-467f-869a-022d1d94ca50, pos: (mysql-bin|000001.000012, 53304437), op: SKIP, args: "] ["new operator"="uuid: 5028d478-c40f-406b-8a20-65b23700b29c, pos: (mysql-bin|000001.000012, 53304437), op: SKIP, args: "]
[2020/07/13 18:16:14.083 +08:00] [INFO] [operator.go:136] ["set a new operator"] [task=testx] [unit="binlog replication"] ["new operator"="uuid: 5028d478-c40f-406b-8a20-65b23700b29c, pos: (mysql-bin|000001.000012, 53304437), op: SKIP, args: "]
[2020/07/13 18:16:16.633 +08:00] [INFO] [server.go:181] [request=OperateSubTask] [payload="op:Resume name:\"testx\" "]
[2020/07/13 18:16:16.633 +08:00] [INFO] [worker.go:320] ["operate subtask"] [component="worker controller"] [operation=Resume] [task=testx]
[2020/07/13 18:16:16.633 +08:00] [INFO] [server.go:229] [request=QueryTaskOperation] [payload="name:\"testx\" logID:15 "]
[2020/07/13 18:16:17.407 +08:00] [WARN] [task_checker.go:358] ["backoff skip auto resume task"] [component="task checker"] [task=testx] [latestResumeTime=2020/07/13 18:14:02.407 +08:00] [duration=5m0s]
[2020/07/13 18:16:17.407 +08:00] [INFO] [worker.go:802] ["start to execute operation"] [component="worker controller"] [oplog="{\"id\":15,\"task\":{\"op\":3,\"name\":\"testx\"},\"ts\":1594635376633535417}"]
[2020/07/13 18:16:17.407 +08:00] [INFO] [worker.go:881] ["resume sub task"] [component="worker controller"] [task=testx]
[2020/07/13 18:16:17.407 +08:00] [INFO] [subtask.go:660] ["wait condition between two units"] [subtask=testx] ["previous unit"=Load] [unit=Sync]
[2020/07/13 18:16:17.409 +08:00] [INFO] [subtask.go:690] ["relay binlog pos catchup loader end binlog pos"] [subtask=testx]
[2020/07/13 18:16:17.409 +08:00] [INFO] [subtask.go:467] ["resume with unit"] [subtask=testx] [unit=Sync]
[2020/07/13 18:16:17.409 +08:00] [INFO] [worker.go:893] ["end to execute operation"] [component="worker controller"] ["oplog ID"=15] []
[2020/07/13 18:16:17.438 +08:00] [INFO] [relay.go:568] ["flush meta finished"] [component="relay log"] [meta="master-uuid = 0652de9c-ddd6-11e9-8f03-00163e00241b.000001, relay-binlog = (mysql-bin.000012, 56003922), relay-binlog-gtid = "]
[2020/07/13 18:16:17.449 +08:00] [INFO] [syncer.go:1093] ["replicate binlog from checkpoint"] [task=testx] [unit="binlog replication"] [checkpoint="(mysql-bin|000001.000012, 53304003)"]
[2020/07/13 18:16:17.450 +08:00] [INFO] [reader.go:402] ["update relay UUIDs"] [task=testx] [unit="binlog replication"] [component="binlog reader"] ["old uuids"="[]"] [uuids="[0652de9c-ddd6-11e9-8f03-00163e00241b.000001]"]
[2020/07/13 18:16:17.450 +08:00] [INFO] [reader.go:95] ["start to check relay log file"] [task=testx] [unit="binlog replication"] [component="binlog reader"] [path=relay_log/0652de9c-ddd6-11e9-8f03-00163e00241b.000001/mysql-bin.000012] [position="(mysql-bin.000012, 53304003)"]
[2020/07/13 18:16:17.450 +08:00] [INFO] [mode.go:99] ["change count"] [task=testx] [unit="binlog replication"] ["previous count"=0] ["new count"=0]
[2020/07/13 18:16:17.450 +08:00] [INFO] [mode.go:99] ["change count"] [task=testx] [unit="binlog replication"] ["previous count"=0] ["new count"=1]
[2020/07/13 18:16:17.450 +08:00] [INFO] [reader.go:134] ["start reading"] [task=testx] [unit="binlog replication"] [component="binlog reader"] [position="(mysql-bin|000001.000012, 53304003)"]
[2020/07/13 18:16:17.450 +08:00] [INFO] [reader.go:197] ["start to parse relay log files in sub directory"] [task=testx] [unit="binlog replication"] [component="binlog reader"] [directory=relay_log/0652de9c-ddd6-11e9-8f03-00163e00241b.000001] [position="(mysql-bin.000012, 53304003)"]
[2020/07/13 18:16:17.450 +08:00] [INFO] [reader.go:320] ["rotate binlog"] [task=testx] [unit="binlog replication"] [component="binlog reader"] [position="(mysql-bin|000001.000012, 53304003)"]
[2020/07/13 18:16:17.450 +08:00] [INFO] [reader.go:346] ["start parse relay log file"] [task=testx] [unit="binlog replication"] [component="binlog reader"] [file=relay_log/0652de9c-ddd6-11e9-8f03-00163e00241b.000001/mysql-bin.000012] [offset=53304003]
[2020/07/13 18:16:17.450 +08:00] [INFO] [syncer.go:1417] [task=testx] [unit="binlog replication"] [event=rotate] [position="(mysql-bin|000001.000012, 53304003)"]
[2020/07/13 18:16:17.450 +08:00] [ERROR] [common.go:35] ["parse statement"] [sql="ALTER TABLE `fsk_erp_trade`.`et_device_order_round` \r\
MODIFY COLUMN `id` varchar(50) CHARACTER SET utf32 NOT NULL COMMENT '主键' FIRST,\r\
MODIFY COLUMN `round_detail` varchar(1000) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL COMMENT '查房详情' AFTER `round_type`"] [error="[parser:1115]Unknown character set: 'utf32'"]
[2020/07/13 18:16:17.450 +08:00] [ERROR] [ddl.go:71] ["encountered incompatible DDL in TiDB:\
\tplease confirm your DDL statement is correct and needed.\
\tfor TiDB compatible DDL, please see the docs:\
\t  English version: https://pingcap.com/docs/dev/reference/mysql-compatibility/#ddl\
\t  Chinese version: https://pingcap.com/docs-cn/dev/reference/mysql-compatibility/#ddl\
\tif the DDL is not needed, you can use a filter rule with \"*\" schema-pattern to ignore it.\
\t "] [task=testx] [unit="binlog replication"] [sql="ALTER TABLE `fsk_erp_trade`.`et_device_order_round` \r\
MODIFY COLUMN `id` varchar(50) CHARACTER SET utf32 NOT NULL COMMENT '主键' FIRST,\r\
MODIFY COLUMN `round_detail` varchar(1000) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL COMMENT '查房详情' AFTER `round_type`"]
[2020/07/13 18:16:17.451 +08:00] [ERROR] [syncer.go:1585] ["fail to parse statement"] [task=testx] [unit="binlog replication"] [event=query] [statement="ALTER TABLE `fsk_erp_trade`.`et_device_order_round` \r\
MODIFY COLUMN `id` varchar(50) CHARACTER SET utf32 NOT NULL COMMENT '主键' FIRST,\r\
MODIFY COLUMN `round_detail` varchar(1000) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL COMMENT '查房详情' AFTER `round_type`"] [schema=fsk_erp_trade] ["last position"="(mysql-bin|000001.000012, 53304003)"] [position="(mysql-bin|000001.000012, 53304437)"] ["gtid set"=NULL] [error="[code=11006:class=functional:scope=internal:level=high] encountered incompatible DDL in TiDB:\
\tplease confirm your DDL statement is correct and needed.\
\tfor TiDB compatible DDL, please see the docs:\
\t  English version: https://pingcap.com/docs/dev/reference/mysql-compatibility/#ddl\
\t  Chinese version: https://pingcap.com/docs-cn/dev/reference/mysql-compatibility/#ddl\
\tif the DDL is not needed, you can use a filter rule with \"*\" schema-pattern to ignore it.\
\t : parse statement: [parser:1115]Unknown character set: 'utf32'%!(EXTRA string=ALTER TABLE `fsk_erp_trade`.`et_device_order_round` \r\
MODIFY COLUMN `id` varchar(50) CHARACTER SET utf32 NOT NULL COMMENT '主键' FIRST,\r\
MODIFY COLUMN `round_detail` varchar(1000) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL COMMENT '查房详情' AFTER `round_type`)"]
[2020/07/13 18:16:17.451 +08:00] [INFO] [syncer.go:887] ["flushed checkpoint"] [task=testx] [unit="binlog replication"] [checkpoint="(mysql-bin|000001.000012, 53304003)(flushed (mysql-bin|000001.000012, 53304003))"]
[2020/07/13 18:16:17.451 +08:00] [INFO] [relay.go:113] ["current earliest active relay log"] [task=testx] [unit="binlog replication"] ["active relay log"=0652de9c-ddd6-11e9-8f03-00163e00241b.000001/mysql-bin.000012]
[2020/07/13 18:16:17.451 +08:00] [INFO] [mode.go:99] ["change count"] [task=testx] [unit="binlog replication"] ["previous count"=1] ["new count"=0]
[2020/07/13 18:16:17.451 +08:00] [INFO] [syncer.go:2032] ["print status routine exits"] [task=testx] [unit="binlog replication"] [error="context canceled"]
[2020/07/13 18:16:17.451 +08:00] [INFO] [reader.go:408] ["binlog reader closing"] [task=testx] [unit="binlog replication"] [component="binlog reader"]
[2020/07/13 18:16:17.451 +08:00] [WARN] [reader.go:137] ["parse relay finished"] [task=testx] [unit="binlog replication"] [component="binlog reader"] [error="parse relay log file mysql-bin.000012 from offset 53304003 in dir relay_log/0652de9c-ddd6-11e9-8f03-00163e00241b.000001: context canceled"]
[2020/07/13 18:16:17.451 +08:00] [INFO] [reader.go:413] ["binlog reader closed"] [task=testx] [unit="binlog replication"] [component="binlog reader"]
[2020/07/13 18:16:17.451 +08:00] [INFO] [checkpoint.go:464] ["rollback checkpoint"] [task=testx] [unit="binlog replication"] [component="remote checkpoint"] [checkpoint="(mysql-bin|000001.000012, 35492894)(flushed (mysql-bin|000001.000012, 35492894))"] [schema=fsk-systemctl] [table=sys_user]
[2020/07/13 18:16:17.451 +08:00] [INFO] [checkpoint.go:464] ["rollback checkpoint"] [task=testx] [unit="binlog replication"] [component="remote checkpoint"] [checkpoint="(mysql-bin|000001.000012, 39896667)(flushed (mysql-bin|000001.000012, 39896667))"] [schema=fsk-systemctl] [table=sys_code]
[2020/07/13 18:16:17.451 +08:00] [INFO] [checkpoint.go:464] ["rollback checkpoint"] [task=testx] [unit="binlog replication"] [component="remote checkpoint"] [checkpoint="(mysql-bin|000001.000012, 53302149)(flushed (mysql-bin|000001.000012, 53302149))"] [schema=fsk-systemctl] [table=sys_user_access]
[2020/07/13 18:16:17.451 +08:00] [INFO] [checkpoint.go:464] ["rollback checkpoint"] [task=testx] [unit="binlog replication"] [component="remote checkpoint"] [checkpoint="(mysql-bin|000001.000012, 35500041)(flushed (mysql-bin|000001.000012, 35500041))"] [schema=fsk-systemctl] [table=sys_user_data_authority]
[2020/07/13 18:16:17.451 +08:00] [INFO] [checkpoint.go:464] ["rollback checkpoint"] [task=testx] [unit="binlog replication"] [component="remote checkpoint"] [checkpoint="(mysql-bin|000001.000012, 35519864)(flushed (mysql-bin|000001.000012, 35519864))"] [schema=fsk_erp_trade] [table=et_assess_title_order]
[2020/07/13 18:16:17.451 +08:00] [INFO] [checkpoint.go:464] ["rollback checkpoint"] [task=testx] [unit="binlog replication"] [component="remote checkpoint"] [checkpoint="(mysql-bin|000001.000012, 51088501)(flushed (mysql-bin|000001.000012, 51088501))"] [schema=fsk_erp_trade] [table=et_device_order_data]
[2020/07/13 18:16:17.451 +08:00] [INFO] [checkpoint.go:464] ["rollback checkpoint"] [task=testx] [unit="binlog replication"] [component="remote checkpoint"] [checkpoint="(mysql-bin|000001.000012, 50230627)(flushed (mysql-bin|000001.000012, 50230627))"] [schema=fsk_erp_trade] [table=et_device_order_item]
[2020/07/13 18:16:17.451 +08:00] [INFO] [checkpoint.go:464] ["rollback checkpoint"] [task=testx] [unit="binlog replication"] [component="remote checkpoint"] [checkpoint="(mysql-bin|000001.000012, 35501499)(flushed (mysql-bin|000001.000012, 35501499))"] [schema=fsk-ehr] [table=ehr_staff_position]
[2020/07/13 18:16:17.451 +08:00] [INFO] [checkpoint.go:464] ["rollback checkpoint"] [task=testx] [unit="binlog replication"] [component="remote checkpoint"] [checkpoint="(mysql-bin|000001.000012, 35494366)(flushed (mysql-bin|000001.000012, 35494366))"] [schema=fsk-ehr] [table=ehr_staff]
[2020/07/13 18:16:17.453 +08:00] [INFO] [subtask.go:266] ["unit process returned"] [subtask=testx] [unit=Sync] [stage=Paused] [status="{\"totalEvents\":30785,\"totalTps\":11,\"recentTps\":1,\"masterBinlog\":\"(mysql-bin.000012, 56003922)\",\"masterBinlogGtid\":\"08e024dd-22db-11ea-837a-00163e04dda5:1-3918708\",\"syncerBinlog\":\"(mysql-bin|000001.000012, 53304003)\"}"]
[2020/07/13 18:16:17.453 +08:00] [ERROR] [subtask.go:285] ["unit process error"] [subtask=testx] [unit=Sync] ["error information"="{\"msg\":\"current pos (mysql-bin|000001.000012, 53304437): encountered incompatible DDL in TiDB:\\
\\tplease confirm your DDL statement is correct and needed.\\
\\tfor TiDB compatible DDL, please see the docs:\\
\\t  English version: https://pingcap.com/docs/dev/reference/mysql-compatibility/#ddl\\
\\t  Chinese version: https://pingcap.com/docs-cn/dev/reference/mysql-compatibility/#ddl\\
\\tif the DDL is not needed, you can use a filter rule with \\\"*\\\" schema-pattern to ignore it.\\
\\t : parse statement: [parser:1115]Unknown character set: 'utf32'%!!(MISSING)(EXTRA string=ALTER TABLE `fsk_erp_trade`.`et_device_order_round` \\r\\
MODIFY COLUMN `id` varchar(50) CHARACTER SET utf32 NOT NULL COMMENT '主键' FIRST,\\r\\
MODIFY COLUMN `round_detail` varchar(1000) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL COMMENT '查房详情' AFTER `round_type`)\",\"error\":{\"ErrCode\":11006,\"ErrClass\":2,\"ErrScope\":3,\"ErrLevel\":3,\"Message\":\"current pos (mysql-bin|000001.000012, 53304437): encountered incompatible DDL in TiDB:\\
\\tplease confirm your DDL statement is correct and needed.\\
\\tfor TiDB compatible DDL, please see the docs:\\
\\t  English version: https://pingcap.com/docs/dev/reference/mysql-compatibility/#ddl\\
\\t  Chinese version: https://pingcap.com/docs-cn/dev/reference/mysql-compatibility/#ddl\\
\\tif the DDL is not needed, you can use a filter rule with \\\"*\\\" schema-pattern to ignore it.\\
\\t : parse statement: [parser:1115]Unknown character set: 'utf32'%!!(MISSING)(EXTRA string=ALTER TABLE `fsk_erp_trade`.`et_device_order_round` \\r\\
MODIFY COLUMN `id` varchar(50) CHARACTER SET utf32 NOT NULL COMMENT '主键' FIRST,\\r\\
MODIFY COLUMN `round_detail` varchar(1000) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL COMMENT '查房详情' AFTER `round_type`)\",\"RawCause\":\"[parser:1115]Unknown character set: 'utf32'\"}}"]
[2020/07/13 18:16:17.634 +08:00] [INFO] [server.go:229] [request=QueryTaskOperation] [payload="name:\"testx\" logID:15 "]
[2020/07/13 18:16:22.407 +08:00] [WARN] [task_checker.go:358] ["backoff skip auto resume task"] [component="task checker"] [task=testx] [latestResumeTime=2020/07/13 18:14:02.407 +08:00] [duration=5m0s]
[2020/07/13 18:16:27.407 +08:00] [WARN] [task_checker.go:358] ["backoff skip auto resume task"] [component="task checker"] [task=testx] [latestResumeTime=2020/07/13 18:14:02.407 +08:00] [duration=5m0s]

来了老弟 · 2020 年7 月 13 日 11:03

是的，请问 mydupmers 部分使用 -x，black-write-list 使用 tbl-1 这种配置是否可行呢？

否则只能修改数据库名，- 变为 _