DM 从上游分库迁移至下游tidb中存在少量重复数据

麻烦是朋友 · 2022 年11 月 10 日 02:28

【 TiDB 版本】
DM 版本：
Cluster type: dm
Cluster name: dm-calc
Cluster version: v6.1.0
Deploy user: tidb
SSH type: builtin

tiup dm -v
Starting component dm: /home/tidb/.tiup/components/dm/v1.10.2/tiup-dm -v
tiup version 1.10.2 tiup
Go Version: go1.18.3
Git Ref: v1.10.2
GitHash: 2de5b500c9fae6d418fa200ca150b8d5264d6b19
【环境描述】
上游：是分库分表的的环境。
下游：是tidb 环境，下游表结构比上游多一列ID ,id bigint(20) unsigned NOT NULL /*T![auto_rand] AUTO_RANDOM(5)，防止出现数据冲突情况。
DM同步状态正常：
“taskName”: “sharding_busi_detail”,
“taskStatus”: “Running”,
“sources”: [
“slave-mysql-x-xx-11851”,
“slave-mysql-x-xx-11852”,
“slave-mysql-x-xx-11853”,
“slave-mysql-x-xx-11854”,
“slave-mysql-x-xx-11855”,
“slave-mysql-x-xx-11856”,
“slave-mysql-x-xx-11857”
]
【遇到的问题】
拿着下游重复数据到上游查询只有一条记录，但是在TiDB环境中会有重复数据存在。
下游TiDB重复数据：（分组条件是上游表的主键）183条

重复数据366条记录

上游MySQL重复数据查询:
各个分库里查询后也只有一条数据。

是否能够提供一下排查思路。次文档已看过《专栏 - 加载中 | TiDB 社区

lance6716 · 2022 年11 月 10 日 02:57

麻烦贴一下上下游表结构

麻烦是朋友 · 2022 年11 月 10 日 03:11

上游MySQL表结构：
CREATE TABLE t_busi_detail (
did varchar(100) NOT NULL COMMENT ‘交易明细id’,
ewb_no varchar(100) NOT NULL COMMENT ‘运单号’,
serial_no varchar(100) DEFAULT NULL COMMENT ‘流水号’,
account_id varchar(100) NOT NULL COMMENT ‘帐号id’,
deposit_site_id bigint(20) NOT NULL COMMENT ‘开户行id’,
deposit_site_name varchar(100) NOT NULL COMMENT ‘开户行名’,
site_id bigint(20) NOT NULL COMMENT ‘开户网点id’,
site_name varchar(100) NOT NULL COMMENT ‘开户网点名’,
distribution_site_id bigint(20) NOT NULL COMMENT ‘开户户名所属分拨id’,
distribution_site_name varchar(100) NOT NULL COMMENT ‘开户户名所属分拨名’,
account_type int(11) NOT NULL COMMENT ‘账户类型’,
data_source tinyint(4) NOT NULL COMMENT ‘数据来源（用户：1，系统：0）’,
busi_id varchar(100) NOT NULL COMMENT ‘流水号（存储发生业务的数据id，计费业务则存储账单id）’,
charge_calculate_id varchar(100) NOT NULL COMMENT ‘账单计费的业务id，非计费数据存储0’,
amount decimal(10,2) NOT NULL,
fines_amount decimal(10,2) DEFAULT NULL COMMENT ‘赔付金额’,
transaction_type tinyint(4) NOT NULL COMMENT ‘交易类型（0：收入，1：支出）’,
balance decimal(20,2) NOT NULL COMMENT ‘交易后的余额’,
charge_itme_id bigint(8) NOT NULL COMMENT ‘费用项目id’,
parent_charge_item_id bigint(8) DEFAULT NULL COMMENT ‘所属费用项目id’,
send_time datetime DEFAULT NULL COMMENT ‘寄件时间’,
send_site_id bigint(8) DEFAULT NULL COMMENT ‘寄件网点id’,
send_site_name varchar(300) DEFAULT NULL COMMENT ‘寄件网点名’,
send_site_id_1 bigint(8) DEFAULT NULL COMMENT ‘寄件网点所属一级网点id’,
send_site_name_1 varchar(300) DEFAULT NULL COMMENT ‘寄件网点所属一级网点名’,
dispatch_site_id bigint(8) DEFAULT NULL COMMENT ‘派件网点id’,
dispatch_site_name varchar(300) DEFAULT NULL COMMENT ‘派件网点名’,
sign_site_id bigint(8) DEFAULT NULL COMMENT ‘签收网点id’,
sign_site_name varchar(300) DEFAULT NULL COMMENT ‘签收网点名’,
sign_site_id_1 bigint(8) DEFAULT NULL COMMENT ‘签收一级网点id’,
sign_site_id_name varchar(300) DEFAULT NULL COMMENT ‘签收一级网点名称’,
sign_time datetime DEFAULT NULL COMMENT ‘签收时间’,
fill_storehouse tinyint(1) DEFAULT NULL COMMENT ‘填仓’,
weight decimal(8,2) DEFAULT NULL COMMENT ‘运单实际重量’,
calc_weight decimal(8,2) DEFAULT NULL COMMENT ‘运单结算重量’,
piece bigint(5) DEFAULT NULL COMMENT ‘件数’,
goods_type_id bigint(8) DEFAULT NULL COMMENT ‘产品类型（字典表services_type）’,
send_customer_id varchar(100) DEFAULT NULL COMMENT ‘寄件客户id’,
send_customer_name varchar(300) DEFAULT NULL COMMENT ‘寄件客户名称’,
hedge_flag tinyint(4) DEFAULT NULL COMMENT ‘对冲（1：正常，0：对冲）’,
modify_time datetime DEFAULT NULL COMMENT ‘修改时间’,
created_by bigint(8) NOT NULL COMMENT ‘创建人id’,
created_name varchar(300) DEFAULT NULL COMMENT ‘创建人名称’,
created_time datetime NOT NULL COMMENT ‘创建时间’,
valid_flag tinyint(4) NOT NULL COMMENT ‘关联费用（1：与开户户名有关，0：只是经过开户行）’,
remark varchar(1000) DEFAULT NULL COMMENT ‘备注’,
sys_remark varchar(1000) DEFAULT NULL COMMENT ‘系统备注’,
ts datetime DEFAULT NULL COMMENT ‘时间戳’,
var_standby_1 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_2 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_3 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_4 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_5 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_6 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_7 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_8 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_9 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_10 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
goods_category int(10) DEFAULT NULL COMMENT ‘易碎品标识，1：易碎品 0：否’,
var_standby_12 int(10) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_13 int(10) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_14 int(10) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_15 int(10) DEFAULT NULL COMMENT ‘备用字段’,
wd_balance decimal(20,2) NOT NULL DEFAULT ‘0.00’ COMMENT ‘当次提现余额’,
PRIMARY KEY (did,created_time),
KEY idx_t_busi_detail_1 (busi_id,charge_itme_id,site_id) USING BTREE,
KEY idx_t_busi_detail_2 (site_id,created_time,charge_itme_id) USING BTREE,
KEY idx_t_busi_detail_3 (ewb_no) USING BTREE,
KEY I_t_busi_detail_04 (deposit_site_id,created_time),
KEY idx_t_busi_detail_4 (serial_no) USING BTREE,
KEY idx_t_busi_detail_5 (deposit_site_id,account_type,charge_itme_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT=‘交易明细表’
/*!50500 PARTITION BY RANGE COLUMNS(created_time)
(PARTITION P20220401 VALUES LESS THAN (‘2022-04-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20220402 VALUES LESS THAN (‘2022-04-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20220403 VALUES LESS THAN (‘2022-05-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20220501 VALUES LESS THAN (‘2022-05-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20220502 VALUES LESS THAN (‘2022-05-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20220503 VALUES LESS THAN (‘2022-06-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20220601 VALUES LESS THAN (‘2022-06-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20220602 VALUES LESS THAN (‘2022-06-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20220603 VALUES LESS THAN (‘2022-07-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20220701 VALUES LESS THAN (‘2022-07-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20220702 VALUES LESS THAN (‘2022-07-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20220703 VALUES LESS THAN (‘2022-08-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20220801 VALUES LESS THAN (‘2022-08-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20220802 VALUES LESS THAN (‘2022-08-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20220803 VALUES LESS THAN (‘2022-09-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20220901 VALUES LESS THAN (‘2022-09-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20220902 VALUES LESS THAN (‘2022-09-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20220903 VALUES LESS THAN (‘2022-10-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20221001 VALUES LESS THAN (‘2022-10-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20221002 VALUES LESS THAN (‘2022-10-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20221003 VALUES LESS THAN (‘2022-11-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20221101 VALUES LESS THAN (‘2022-11-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20221102 VALUES LESS THAN (‘2022-11-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20221103 VALUES LESS THAN (‘2022-12-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20221201 VALUES LESS THAN (‘2022-12-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20221202 VALUES LESS THAN (‘2022-12-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20221203 VALUES LESS THAN (‘2023-01-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20230101 VALUES LESS THAN (‘2023-01-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20230102 VALUES LESS THAN (‘2023-01-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20230103 VALUES LESS THAN (‘2023-02-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20230201 VALUES LESS THAN (‘2023-02-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20230202 VALUES LESS THAN (‘2023-02-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20230203 VALUES LESS THAN (‘2023-03-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20230301 VALUES LESS THAN (‘2023-03-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20230302 VALUES LESS THAN (‘2023-03-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20230303 VALUES LESS THAN (‘2023-04-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20230401 VALUES LESS THAN (‘2023-04-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20230402 VALUES LESS THAN (‘2023-04-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20230403 VALUES LESS THAN (‘2023-05-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20230501 VALUES LESS THAN (‘2023-05-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20230502 VALUES LESS THAN (‘2023-05-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20230503 VALUES LESS THAN (‘2023-06-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20230601 VALUES LESS THAN (‘2023-06-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20230602 VALUES LESS THAN (‘2023-06-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20230603 VALUES LESS THAN (‘2023-07-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20230701 VALUES LESS THAN (‘2023-07-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20230702 VALUES LESS THAN (‘2023-07-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20230703 VALUES LESS THAN (‘2023-08-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20230801 VALUES LESS THAN (‘2023-08-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20230802 VALUES LESS THAN (‘2023-08-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20230803 VALUES LESS THAN (‘2023-09-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20230901 VALUES LESS THAN (‘2023-09-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20230902 VALUES LESS THAN (‘2023-09-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20230903 VALUES LESS THAN (‘2023-10-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20231001 VALUES LESS THAN (‘2023-10-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20231002 VALUES LESS THAN (‘2023-10-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20231003 VALUES LESS THAN (‘2023-11-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20231101 VALUES LESS THAN (‘2023-11-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20231102 VALUES LESS THAN (‘2023-11-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20231103 VALUES LESS THAN (‘2023-12-01 00:00:00’) ENGINE = InnoDB,
PARTITION P20231201 VALUES LESS THAN (‘2023-12-10 00:00:00’) ENGINE = InnoDB,
PARTITION P20231202 VALUES LESS THAN (‘2023-12-20 00:00:00’) ENGINE = InnoDB,
PARTITION P20231203 VALUES LESS THAN (‘2024-01-01 00:00:00’) ENGINE = InnoDB,
PARTITION P_MAX VALUES LESS THAN (MAXVALUE) ENGINE = InnoDB) */

下游TiDB表结构：
CREATE TABLE t_busi_detail (
id bigint(20) unsigned NOT NULL /*T![auto_rand] AUTO_RANDOM(5) */,
did varchar(100) NOT NULL COMMENT ‘交易明细id’,
ewb_no varchar(100) NOT NULL COMMENT ‘运单号’,
serial_no varchar(100) DEFAULT NULL COMMENT ‘流水号’,
account_id varchar(100) NOT NULL COMMENT ‘帐号id’,
deposit_site_id bigint(20) NOT NULL COMMENT ‘开户行id’,
deposit_site_name varchar(100) NOT NULL COMMENT ‘开户行名’,
site_id bigint(20) NOT NULL COMMENT ‘开户网点id’,
site_name varchar(100) NOT NULL COMMENT ‘开户网点名’,
distribution_site_id bigint(20) NOT NULL COMMENT ‘开户户名所属分拨id’,
distribution_site_name varchar(100) NOT NULL COMMENT ‘开户户名所属分拨名’,
account_type int(11) NOT NULL COMMENT ‘账户类型’,
data_source tinyint(4) NOT NULL COMMENT ‘数据来源（用户：1，系统：0）’,
busi_id varchar(100) NOT NULL COMMENT ‘流水号（存储发生业务的数据id，计费业务则存储账单id）’,
charge_calculate_id varchar(100) NOT NULL COMMENT ‘账单计费的业务id，非计费数据存储0’,
amount decimal(10,2) NOT NULL,
fines_amount decimal(10,2) DEFAULT NULL COMMENT ‘赔付金额’,
transaction_type tinyint(4) NOT NULL COMMENT ‘交易类型（0：收入，1：支出）’,
balance decimal(20,2) NOT NULL COMMENT ‘交易后的余额’,
charge_itme_id bigint(8) NOT NULL COMMENT ‘费用项目id’,
parent_charge_item_id bigint(8) DEFAULT NULL COMMENT ‘所属费用项目id’,
send_time datetime DEFAULT NULL COMMENT ‘寄件时间’,
send_site_id bigint(8) DEFAULT NULL COMMENT ‘寄件网点id’,
send_site_name varchar(300) DEFAULT NULL COMMENT ‘寄件网点名’,
send_site_id_1 bigint(8) DEFAULT NULL COMMENT ‘寄件网点所属一级网点id’,
send_site_name_1 varchar(300) DEFAULT NULL COMMENT ‘寄件网点所属一级网点名’,
dispatch_site_id bigint(8) DEFAULT NULL COMMENT ‘派件网点id’,
dispatch_site_name varchar(300) DEFAULT NULL COMMENT ‘派件网点名’,
sign_site_id bigint(8) DEFAULT NULL COMMENT ‘签收网点id’,
sign_site_name varchar(300) DEFAULT NULL COMMENT ‘签收网点名’,
sign_site_id_1 bigint(8) DEFAULT NULL COMMENT ‘签收一级网点id’,
sign_site_id_name varchar(300) DEFAULT NULL COMMENT ‘签收一级网点名称’,
sign_time datetime DEFAULT NULL COMMENT ‘签收时间’,
fill_storehouse tinyint(1) DEFAULT NULL COMMENT ‘填仓’,
weight decimal(8,2) DEFAULT NULL COMMENT ‘运单实际重量’,
calc_weight decimal(8,2) DEFAULT NULL COMMENT ‘运单结算重量’,
piece bigint(5) DEFAULT NULL COMMENT ‘件数’,
goods_type_id bigint(8) DEFAULT NULL COMMENT ‘产品类型（字典表services_type）’,
send_customer_id varchar(100) DEFAULT NULL COMMENT ‘寄件客户id’,
send_customer_name varchar(300) DEFAULT NULL COMMENT ‘寄件客户名称’,
hedge_flag tinyint(4) DEFAULT NULL COMMENT ‘对冲（1：正常，0：对冲）’,
modify_time datetime DEFAULT NULL COMMENT ‘修改时间’,
created_by bigint(8) NOT NULL COMMENT ‘创建人id’,
created_name varchar(300) DEFAULT NULL COMMENT ‘创建人名称’,
created_time datetime NOT NULL COMMENT ‘创建时间’,
valid_flag tinyint(4) NOT NULL COMMENT ‘关联费用（1：与开户户名有关，0：只是经过开户行）’,
remark varchar(1000) DEFAULT NULL COMMENT ‘备注’,
sys_remark varchar(1000) DEFAULT NULL COMMENT ‘系统备注’,
ts datetime DEFAULT NULL COMMENT ‘时间戳’,
var_standby_1 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_2 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_3 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_4 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_5 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_6 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_7 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_8 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_9 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_10 varchar(50) DEFAULT NULL COMMENT ‘备用字段’,
goods_category int(10) DEFAULT NULL COMMENT ‘易碎品标识，1：易碎品 0：否’,
var_standby_12 int(10) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_13 int(10) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_14 int(10) DEFAULT NULL COMMENT ‘备用字段’,
var_standby_15 int(10) DEFAULT NULL COMMENT ‘备用字段’,
wd_balance decimal(20,2) NOT NULL DEFAULT ‘0.00’ COMMENT ‘当次提现余额’,
PRIMARY KEY (id) /*T![clustered_index] CLUSTERED */,
KEY idx_t_busi_detail_1 (did,created_time),
KEY idx_t_busi_detail_2 (busi_id,charge_itme_id,site_id),
KEY idx_t_busi_detail_3 (site_id,created_time,charge_itme_id),
KEY idx_t_busi_detail_4 (ewb_no),
KEY idx_t_busi_detail_5 (deposit_site_id,created_time),
KEY idx_t_busi_detail_6 (serial_no),
KEY idx_t_busi_detail_7 (deposit_site_id,account_type,charge_itme_id),
KEY idx_t_busi_detail_8 (created_time,modify_time),
KEY idx_t_busi_detail_9 (modify_time)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin /*T![auto_rand_base] AUTO_RANDOM_BASE=1673240265 */ COMMENT=‘交易明细表’

Min_Chen · 2022 年11 月 16 日 03:06

你好，麻烦提供任务配置 yaml，另外同步期间有没有做过什么操作？比如 DDL 或发生过异常。如果有请提供对应时间点和 dm worker 日志。

麻烦是朋友 · 2022 年11 月 16 日 07:49

配置文件：
cat sharding_busi_detail.yaml
name: “sharding_busi_detail”
task-mode: “all”

target-database:
host: “172.17.x.xx”
port: xxxx
user: “root”
password: “n7NLD6TpEuGe4kEnVhgxy0Dr67IMIXA=”

routes:
route-rule-0:
schema-pattern: “calcdb0*”
target-schema: “calcdb”

route-rule-1:
schema-pattern: “calcdb0*”
table-pattern: “t_busi_detail”
target-schema: “calcdb”
target-table: “t_busi_detail”

filters:
filter-rule-0:
schema-pattern: “calcdb0*”
events: [“all dml”]
action: Do

filter-rule-1:
schema-pattern: “calcdb0*”
table-pattern: “t_busi_detail”
events: [“truncate table”, “drop table”]
sql-pattern: [“ALTER\s+TABLE[\s\S]*ADD\s+PARTITION”, “ALTER\s+TABLE[\s\S]*DROP\s+PARTITION”]
action: Ignore

mysql-instances:

source-id: “slave-mysql-x-xx-11851”
block-allow-list: “bw-rule-1”
route-rules: [“route-rule-0”, “route-rule-1”]
filter-rules: [“filter-rule-0”, “filter-rule-1”]

source-id: “slave-mysql-x-xx-11852”
block-allow-list: “bw-rule-1”
route-rules: [“route-rule-0”, “route-rule-1”]
filter-rules: [“filter-rule-0”, “filter-rule-1”]

source-id: “slave-mysql-x-xx-11853”
block-allow-list: “bw-rule-1”
route-rules: [“route-rule-0”, “route-rule-1”]
filter-rules: [“filter-rule-0”, “filter-rule-1”]

source-id: “slave-mysql-x-xx-11854”
block-allow-list: “bw-rule-1”
route-rules: [“route-rule-0”, “route-rule-1”]
filter-rules: [“filter-rule-0”, “filter-rule-1”]

source-id: “slave-mysql-x-xx-11855”
block-allow-list: “bw-rule-1”
route-rules: [“route-rule-0”, “route-rule-1”]
filter-rules: [“filter-rule-0”, “filter-rule-1”]

source-id: “slave-mysql-x-xx-11856”
block-allow-list: “bw-rule-1”
route-rules: [“route-rule-0”, “route-rule-1”]
filter-rules: [“filter-rule-0”, “filter-rule-1”]

source-id: “slave-mysql-x-xx-11857”
block-allow-list: “bw-rule-1”
route-rules: [“route-rule-0”, “route-rule-1”]
filter-rules: [“filter-rule-0”, “filter-rule-1”]

block-allow-list:
bw-rule-1:
do-dbs: [“calcdb0*”]
do-tables:
- db-name: “calcdb0*”
tbl-name: “t_busi_detail”

初始化完数据会停下来，应为下游多了一列，已上游表结构为准，操作如下：
tiup dmctl --master-addr 172.17.x.xxx:8261 binlog-schema update -s slave-mysql-x-xx-11851 sharding_busi_detail calcdb01 t_sub_account_busi_detail --from-source
tiup dmctl --master-addr 172.17.x.xxx:8261 binlog-schema update -s slave-mysql-x-xx-11852 sharding_busi_detail calcdb02 t_sub_account_busi_detail --from-source
tiup dmctl --master-addr 172.17.x.xxx:8261 binlog-schema update -s slave-mysql-x-xx-11853 sharding_busi_detail calcdb03 t_sub_account_busi_detail --from-source
tiup dmctl --master-addr 172.17.x.xxx:8261 binlog-schema update -s slave-mysql-x-xx-11854 sharding_busi_detail calcdb04 t_sub_account_busi_detail --from-source
tiup dmctl --master-addr 172.17.x.xxx:8261 binlog-schema update -s slave-mysql-x-xx-11855 sharding_busi_detail calcdb05 t_sub_account_busi_detail --from-source
tiup dmctl --master-addr 172.17.x.xxx:8261 binlog-schema update -s slave-mysql-x-xx-11856 sharding_busi_detail calcdb06 t_sub_account_busi_detail --from-source
tiup dmctl --master-addr 172.17.x.xxx:8261 binlog-schema update -s slave-mysql-x-xx-11857 sharding_busi_detail calcdb07 t_sub_account_busi_detail --from-source

tiup dmctl --master-addr 172.17.x.xxx:8261 resume-task sharding_busi_detail

tiup dmctl --master-addr 172.17.x.xxx:8261 check-task sharding_busi_detail.yaml

麻烦是朋友 · 2023 年12 月 27 日 05:40

最终在下游创建联合主键，单独处理源头少量重复。

system · 2024 年2 月 25 日 05:41

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。

DM 从上游分库迁移至下游tidb中存在少量重复数据

mysql-instances:

source-id: “slave-mysql-x-xx-11851” block-allow-list: “bw-rule-1” route-rules: [“route-rule-0”, “route-rule-1”] filter-rules: [“filter-rule-0”, “filter-rule-1”]

source-id: “slave-mysql-x-xx-11852” block-allow-list: “bw-rule-1” route-rules: [“route-rule-0”, “route-rule-1”] filter-rules: [“filter-rule-0”, “filter-rule-1”]

source-id: “slave-mysql-x-xx-11853” block-allow-list: “bw-rule-1” route-rules: [“route-rule-0”, “route-rule-1”] filter-rules: [“filter-rule-0”, “filter-rule-1”]

source-id: “slave-mysql-x-xx-11854” block-allow-list: “bw-rule-1” route-rules: [“route-rule-0”, “route-rule-1”] filter-rules: [“filter-rule-0”, “filter-rule-1”]

source-id: “slave-mysql-x-xx-11855” block-allow-list: “bw-rule-1” route-rules: [“route-rule-0”, “route-rule-1”] filter-rules: [“filter-rule-0”, “filter-rule-1”]

source-id: “slave-mysql-x-xx-11856” block-allow-list: “bw-rule-1” route-rules: [“route-rule-0”, “route-rule-1”] filter-rules: [“filter-rule-0”, “filter-rule-1”]