我们使用DM的SQL表达式过滤DML归档数据出现了异常，过滤规则在某些日期上不生效

TiDBer_2nbBAGRs · 2023 年3 月 10 日 01:48

【 TiDB 使用环境】生产环境
【 TiDB 版本】6.6.0
【复现路径】
我们设置了DM 的根据sql表达式过滤dml，设置规则为，上游mysql删除14天前的数据，下游tidb不做操作。

配置规则为 delete-value-expr: “timestampdiff(day, ORDER_DATE, NOW())>14”
规则中ORDER_DATE类型为 date

配置文件如下：

【遇到的问题：问题现象及影响】
一开始归档的时候，还是正常的，因为是第一次用这个功能，我一直是在手工删除 mysql 中表 cztycdb.pdc1_dwv_out_order 的数据，来观察tidb。在我删数据删到时间为 2023-01-28开始，我发现明明是14天之前的数据，上游mysql删掉，下游tidb也会删掉。
今天2023-03-10.我手工删除上游mysql 表 cztycdb.pdc1_dwv_out_order 2023-02-04的数据，发现tidb 2023-02-04的数据也没了

TiDBer_2nbBAGRs · 2023 年3 月 10 日 15:05

官方大佬有吗，能帮我瞧瞧吗

okenJiang · 2023 年3 月 10 日 16:15

麻烦提供一下上游的版本，以及对应的日志

TiDBer_2nbBAGRs · 2023 年3 月 11 日 07:06

上游 mysql 5.7
dm 版本 v6.5.0

tidb 集群 v6.50

TiDBer_2nbBAGRs · 2023 年3 月 11 日 07:23

是要 dm worker的日志吗

okenJiang · 2023 年3 月 11 日 07:35

Yes

TiDBer_2nbBAGRs · 2023 年3 月 11 日 08:00

data dir 没log文件
deploy dir 有个log

cd /data1/tidb/dm/deploy/dm-worker-8262/log/ && ls
文件如下
是昨天在 mysql 执行了归档 delete sql

那么文件应该就是

但是这个log文件里只有这种内容

[2023/03/10 08:37:39.345 +08:00] [INFO] [ddl.go:375] [“prepare to handle ddls”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427179382), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786300, endLocation: position: (mysql-binlog.001628, 427179492), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786301, lastLocation: position: (mysql-binlog.001628, 427179492), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786301, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:39.345 +08:00] [INFO] [syncer.go:3061] [“flush all jobs”] [task=mysql-pro-82] [unit=“binlog replication”] [“global checkpoint”=“location(position: (mysql-binlog.001628, 427179317), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786300) (flushed location(position: (mysql-binlog.001628, 427146368), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786268))”] [“flush job seq”=3237]
[2023/03/10 08:37:39.346 +08:00] [INFO] [checkpoint_flush_worker.go:67] [“about to sync flush checkpoint snapshot”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689480]
[2023/03/10 08:37:39.399 +08:00] [INFO] [checkpoint_flush_worker.go:108] [“sync flush checkpoint snapshot successfully”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689480] [pos=“position: (mysql-binlog.001628, 427179317), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786300”]
[2023/03/10 08:37:39.400 +08:00] [INFO] [syncer.go:1284] [“after sync flushed checkpoint, gc all causality keys”] [task=mysql-pro-82] [unit=“binlog replication”]
[2023/03/10 08:37:39.400 +08:00] [INFO] [ddl.go:403] [“start to handle ddls in normal mode”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [mode=normal] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427179382), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786300, endLocation: position: (mysql-binlog.001628, 427179492), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786301, lastLocation: position: (mysql-binlog.001628, 427179492), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786301, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:39.407 +08:00] [INFO] [checkpoint_flush_worker.go:67] [“about to sync flush checkpoint snapshot”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689481]
[2023/03/10 08:37:39.416 +08:00] [INFO] [checkpoint_flush_worker.go:108] [“sync flush checkpoint snapshot successfully”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689481] [pos=“position: (mysql-binlog.001628, 427179492), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786301”]
[2023/03/10 08:37:39.416 +08:00] [INFO] [syncer.go:1284] [“after sync flushed checkpoint, gc all causality keys”] [task=mysql-pro-82] [unit=“binlog replication”]
[2023/03/10 08:37:39.416 +08:00] [INFO] [ddl.go:444] [“finish to handle ddls in normal mode”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [mode=normal] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427179382), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786300, endLocation: position: (mysql-binlog.001628, 427179492), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786301, lastLocation: position: (mysql-binlog.001628, 427179492), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786301, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:40.539 +08:00] [INFO] [ddl.go:934] [“parse ddl”] [task=mysql-pro-82] [unit=“binlog replication”] [event=query] [“query event context”=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427228093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786342, endLocation: position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343, lastLocation: position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:40.540 +08:00] [INFO] [ddl.go:294] [“ready to split ddl”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427228093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786342, endLocation: position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343, lastLocation: position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:40.540 +08:00] [INFO] [ddl.go:314] [“resolve sql”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [appliedDDLs=“[“CREATE DATABASE IF NOT EXISTS test”]”] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427228093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786342, endLocation: position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343, lastLocation: position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:40.540 +08:00] [INFO] [ddl.go:375] [“prepare to handle ddls”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427228093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786342, endLocation: position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343, lastLocation: position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:40.540 +08:00] [INFO] [syncer.go:3061] [“flush all jobs”] [task=mysql-pro-82] [unit=“binlog replication”] [“global checkpoint”=“location(position: (mysql-binlog.001628, 427228028), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786342) (flushed location(position: (mysql-binlog.001628, 427179492), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786301))”] [“flush job seq”=3238]
[2023/03/10 08:37:40.541 +08:00] [INFO] [checkpoint_flush_worker.go:67] [“about to sync flush checkpoint snapshot”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689482]
[2023/03/10 08:37:40.589 +08:00] [INFO] [checkpoint_flush_worker.go:108] [“sync flush checkpoint snapshot successfully”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689482] [pos=“position: (mysql-binlog.001628, 427228028), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786342”]
[2023/03/10 08:37:40.589 +08:00] [INFO] [syncer.go:1284] [“after sync flushed checkpoint, gc all causality keys”] [task=mysql-pro-82] [unit=“binlog replication”]
[2023/03/10 08:37:40.589 +08:00] [INFO] [ddl.go:403] [“start to handle ddls in normal mode”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [mode=normal] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427228093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786342, endLocation: position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343, lastLocation: position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:40.596 +08:00] [INFO] [checkpoint_flush_worker.go:67] [“about to sync flush checkpoint snapshot”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689483]
[2023/03/10 08:37:40.605 +08:00] [INFO] [checkpoint_flush_worker.go:108] [“sync flush checkpoint snapshot successfully”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689483] [pos=“position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343”]
[2023/03/10 08:37:40.605 +08:00] [INFO] [syncer.go:1284] [“after sync flushed checkpoint, gc all causality keys”] [task=mysql-pro-82] [unit=“binlog replication”]
[2023/03/10 08:37:40.605 +08:00] [INFO] [ddl.go:444] [“finish to handle ddls in normal mode”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [mode=normal] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427228093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786342, endLocation: position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343, lastLocation: position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:41.734 +08:00] [INFO] [ddl.go:934] [“parse ddl”] [task=mysql-pro-82] [unit=“binlog replication”] [event=query] [“query event context”=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427264740), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786376, endLocation: position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377, lastLocation: position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:41.735 +08:00] [INFO] [ddl.go:294] [“ready to split ddl”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427264740), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786376, endLocation: position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377, lastLocation: position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:41.735 +08:00] [INFO] [ddl.go:314] [“resolve sql”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [appliedDDLs=“[“CREATE DATABASE IF NOT EXISTS test”]”] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427264740), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786376, endLocation: position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377, lastLocation: position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:41.735 +08:00] [INFO] [ddl.go:375] [“prepare to handle ddls”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427264740), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786376, endLocation: position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377, lastLocation: position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:41.735 +08:00] [INFO] [syncer.go:3061] [“flush all jobs”] [task=mysql-pro-82] [unit=“binlog replication”] [“global checkpoint”=“location(position: (mysql-binlog.001628, 427264675), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786376) (flushed location(position: (mysql-binlog.001628, 427228203), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786343))”] [“flush job seq”=3239]
[2023/03/10 08:37:41.736 +08:00] [INFO] [checkpoint_flush_worker.go:67] [“about to sync flush checkpoint snapshot”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689484]
[2023/03/10 08:37:41.791 +08:00] [INFO] [checkpoint_flush_worker.go:108] [“sync flush checkpoint snapshot successfully”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689484] [pos=“position: (mysql-binlog.001628, 427264675), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786376”]
[2023/03/10 08:37:41.791 +08:00] [INFO] [syncer.go:1284] [“after sync flushed checkpoint, gc all causality keys”] [task=mysql-pro-82] [unit=“binlog replication”]
[2023/03/10 08:37:41.791 +08:00] [INFO] [ddl.go:403] [“start to handle ddls in normal mode”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [mode=normal] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427264740), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786376, endLocation: position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377, lastLocation: position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:41.799 +08:00] [INFO] [checkpoint_flush_worker.go:67] [“about to sync flush checkpoint snapshot”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689485]
[2023/03/10 08:37:41.809 +08:00] [INFO] [checkpoint_flush_worker.go:108] [“sync flush checkpoint snapshot successfully”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689485] [pos=“position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377”]
[2023/03/10 08:37:41.809 +08:00] [INFO] [syncer.go:1284] [“after sync flushed checkpoint, gc all causality keys”] [task=mysql-pro-82] [unit=“binlog replication”]
[2023/03/10 08:37:41.809 +08:00] [INFO] [ddl.go:444] [“finish to handle ddls in normal mode”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [mode=normal] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427264740), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786376, endLocation: position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377, lastLocation: position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:42.915 +08:00] [INFO] [ddl.go:934] [“parse ddl”] [task=mysql-pro-82] [unit=“binlog replication”] [event=query] [“query event context”=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427294893), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786405, endLocation: position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406, lastLocation: position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:42.915 +08:00] [INFO] [ddl.go:294] [“ready to split ddl”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427294893), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786405, endLocation: position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406, lastLocation: position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:42.915 +08:00] [INFO] [ddl.go:314] [“resolve sql”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [appliedDDLs=“[“CREATE DATABASE IF NOT EXISTS test”]”] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427294893), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786405, endLocation: position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406, lastLocation: position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:42.915 +08:00] [INFO] [ddl.go:375] [“prepare to handle ddls”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427294893), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786405, endLocation: position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406, lastLocation: position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:42.915 +08:00] [INFO] [syncer.go:3061] [“flush all jobs”] [task=mysql-pro-82] [unit=“binlog replication”] [“global checkpoint”=“location(position: (mysql-binlog.001628, 427294828), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786405) (flushed location(position: (mysql-binlog.001628, 427264850), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786377))”] [“flush job seq”=3240]
[2023/03/10 08:37:42.916 +08:00] [INFO] [checkpoint_flush_worker.go:67] [“about to sync flush checkpoint snapshot”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689486]
[2023/03/10 08:37:42.960 +08:00] [INFO] [checkpoint_flush_worker.go:108] [“sync flush checkpoint snapshot successfully”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689486] [pos=“position: (mysql-binlog.001628, 427294828), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786405”]
[2023/03/10 08:37:42.960 +08:00] [INFO] [syncer.go:1284] [“after sync flushed checkpoint, gc all causality keys”] [task=mysql-pro-82] [unit=“binlog replication”]
[2023/03/10 08:37:42.960 +08:00] [INFO] [ddl.go:403] [“start to handle ddls in normal mode”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [mode=normal] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427294893), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786405, endLocation: position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406, lastLocation: position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:42.966 +08:00] [INFO] [checkpoint_flush_worker.go:67] [“about to sync flush checkpoint snapshot”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689487]
[2023/03/10 08:37:42.975 +08:00] [INFO] [checkpoint_flush_worker.go:108] [“sync flush checkpoint snapshot successfully”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689487] [pos=“position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406”]
[2023/03/10 08:37:42.975 +08:00] [INFO] [syncer.go:1284] [“after sync flushed checkpoint, gc all causality keys”] [task=mysql-pro-82] [unit=“binlog replication”]
[2023/03/10 08:37:42.975 +08:00] [INFO] [ddl.go:444] [“finish to handle ddls in normal mode”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [mode=normal] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427294893), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786405, endLocation: position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406, lastLocation: position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:44.091 +08:00] [INFO] [ddl.go:934] [“parse ddl”] [task=mysql-pro-82] [unit=“binlog replication”] [event=query] [“query event context”=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427337637), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786440, endLocation: position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441, lastLocation: position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:44.091 +08:00] [INFO] [ddl.go:294] [“ready to split ddl”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427337637), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786440, endLocation: position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441, lastLocation: position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:44.091 +08:00] [INFO] [ddl.go:314] [“resolve sql”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [appliedDDLs=“[“CREATE DATABASE IF NOT EXISTS test”]”] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427337637), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786440, endLocation: position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441, lastLocation: position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:44.091 +08:00] [INFO] [ddl.go:375] [“prepare to handle ddls”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427337637), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786440, endLocation: position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441, lastLocation: position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:44.091 +08:00] [INFO] [syncer.go:3061] [“flush all jobs”] [task=mysql-pro-82] [unit=“binlog replication”] [“global checkpoint”=“location(position: (mysql-binlog.001628, 427337572), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786440) (flushed location(position: (mysql-binlog.001628, 427295003), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786406))”] [“flush job seq”=3241]
[2023/03/10 08:37:44.092 +08:00] [INFO] [checkpoint_flush_worker.go:67] [“about to sync flush checkpoint snapshot”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689488]
[2023/03/10 08:37:44.135 +08:00] [INFO] [checkpoint_flush_worker.go:108] [“sync flush checkpoint snapshot successfully”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689488] [pos=“position: (mysql-binlog.001628, 427337572), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786440”]
[2023/03/10 08:37:44.135 +08:00] [INFO] [syncer.go:1284] [“after sync flushed checkpoint, gc all causality keys”] [task=mysql-pro-82] [unit=“binlog replication”]
[2023/03/10 08:37:44.135 +08:00] [INFO] [ddl.go:403] [“start to handle ddls in normal mode”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [mode=normal] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427337637), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786440, endLocation: position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441, lastLocation: position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:44.141 +08:00] [INFO] [checkpoint_flush_worker.go:67] [“about to sync flush checkpoint snapshot”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689489]
[2023/03/10 08:37:44.149 +08:00] [INFO] [checkpoint_flush_worker.go:108] [“sync flush checkpoint snapshot successfully”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689489] [pos=“position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441”]
[2023/03/10 08:37:44.149 +08:00] [INFO] [syncer.go:1284] [“after sync flushed checkpoint, gc all causality keys”] [task=mysql-pro-82] [unit=“binlog replication”]
[2023/03/10 08:37:44.149 +08:00] [INFO] [ddl.go:444] [“finish to handle ddls in normal mode”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [mode=normal] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427337637), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786440, endLocation: position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441, lastLocation: position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:45.306 +08:00] [INFO] [ddl.go:934] [“parse ddl”] [task=mysql-pro-82] [unit=“binlog replication”] [event=query] [“query event context”=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427391983), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786487, endLocation: position: (mysql-binlog.001628, 427392093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786488, lastLocation: position: (mysql-binlog.001628, 427392093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786488, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:45.306 +08:00] [INFO] [ddl.go:294] [“ready to split ddl”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427391983), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786487, endLocation: position: (mysql-binlog.001628, 427392093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786488, lastLocation: position: (mysql-binlog.001628, 427392093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786488, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:45.306 +08:00] [INFO] [ddl.go:314] [“resolve sql”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [appliedDDLs=“[“CREATE DATABASE IF NOT EXISTS test”]”] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427391983), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786487, endLocation: position: (mysql-binlog.001628, 427392093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786488, lastLocation: position: (mysql-binlog.001628, 427392093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786488, re-sync: , needHandleDDLs: , trackInfos: }”]
[2023/03/10 08:37:45.306 +08:00] [INFO] [ddl.go:375] [“prepare to handle ddls”] [task=mysql-pro-82] [unit=“binlog replication”] [component=ddl] [event=query] [queryEventContext=“{schema: test, originSQL: CREATE DATABASE IF NOT EXISTS test, startLocation: position: (mysql-binlog.001628, 427391983), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786487, endLocation: position: (mysql-binlog.001628, 427392093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786488, lastLocation: position: (mysql-binlog.001628, 427392093), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786488, re-sync: , needHandleDDLs: CREATE DATABASE IF NOT EXISTS test, trackInfos: {originDDL: CREATE DATABASE IF NOT EXISTS test, routedDDL: CREATE DATABASE IF NOT EXISTS test, sourceTables: test, targetTables: test}}”]
[2023/03/10 08:37:45.306 +08:00] [INFO] [syncer.go:3061] [“flush all jobs”] [task=mysql-pro-82] [unit=“binlog replication”] [“global checkpoint”=“location(position: (mysql-binlog.001628, 427391918), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786487) (flushed location(position: (mysql-binlog.001628, 427337747), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786441))”] [“flush job seq”=3242]
[2023/03/10 08:37:45.307 +08:00] [INFO] [checkpoint_flush_worker.go:67] [“about to sync flush checkpoint snapshot”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689490]
[2023/03/10 08:37:45.354 +08:00] [INFO] [checkpoint_flush_worker.go:108] [“sync flush checkpoint snapshot successfully”] [task=mysql-pro-82] [unit=“binlog replication”] [snapshot_id=1689490] [pos=“position: (mysql-binlog.001628, 427391918), gtid-set: adac6965-f5df-11eb-860f-286ed488c748:1-112786487”]
[2023/03/10 08:37:45.354 +08:00] [INFO] [syncer.go:1284] [“after sync flushed checkpoint, gc all causality keys”] [task=mysql-pro-82] [unit=“binlog replication”]

这个文件夹里的log文件貌似是不齐的

还有其他地方有log吗

1

okenJiang · 2023 年3 月 12 日 12:44

你这日志里面连 cztycdb.pdc1_dwv_out_order 这库和表都没有明显不对呀。。。你上其他文件里面看看，看看 worker.log 里有没有