dm同步忽略处理

【概述】 场景 + 问题概述
需求,使用dm同步上游数据到tidb,现在dm同步任务需要排除两个表同步
现配置如下:

block-allow-list:                     # 如果 DM 版本 <= v2.0.0-beta.2 则使用 black-white-list。
  h3_trade:
    do-dbs: ["~^h3_trade.*"]     # 迁移哪些库
    do-tables:                       # 迁移哪些表,测试时只迁移t_test_开头的表
    - db-name: "~.*"
      tbl-name: "t_business_ware_statistics_month,t_company_ware_statistics_day,t_coupon_separate,t_coupon_usage,t_hang_order,t_hang_order_detail,t_integral_exchange_record,t_medical_order_rel,t_member_coupon,t_oms_refund_order,t_oms_refund_order_detail,t_oms_retail_order,t_oms_retail_order_detail,t_order_detail,t_order_info,t_order_outbound,t_order_outbound_detail,t_order_pay_detail,t_order_transport_idempotent,t_prescription,t_prescription_detail,t_prescription_sale_detail,t_promotion_relation,t_user_customer_info,t_warehouse_separate_check_record"
    ignore-tables:
    - db-name: "~.*"
      tbl-name: "~^t_out_order_transport_*"

导致所有表均被过滤掉,
忽略表配置应如何处理?

使用这种方式直接报错:

block-allow-list:                     # 如果 DM 版本 <= v2.0.0-beta.2 则使用 black-white-list。
  h3_trade:
    do-dbs: ["~^h3_trade.*"]     # 迁移哪些库
    do-tables:                       # 迁移哪些表,测试时只迁移t_test_开头的表
    - db-name: "~.*"
      tbl-name: ["t_business_ware_statistics_month","t_company_ware_statistics_day","t_coupon_separate","t_coupon_usage","t_hang_order","t_hang_order_detail","t_integral_exchange_record","t_medical_order_rel","t_member_coupon","t_oms_refund_order","t_oms_refund_order_detail","t_oms_retail_order","t_oms_retail_order_detail","t_order_detail","t_order_info","t_order_outbound","t_order_outbound_detail","t_order_pay_detail","t_order_transport_idempotent","t_prescription","t_prescription_detail","t_prescription_sale_detail","t_promotion_relation","t_user_customer_info","t_warehouse_separate_check_record"]

报错:

    "result": false,
    "msg": "[code=20003:class=dm-master:scope=internal:level=medium], Message: decode task config failed, RawCause: yaml: unmarshal errors:\
  line 68: cannot unmarshal !!seq into string, Workaround: Please check the configuration file has correct YAML format.",
    "sources": [
    ]

可以了:

block-allow-list:                     # 如果 DM 版本 <= v2.0.0-beta.2 则使用 black-white-list。
  h3_trade:
    do-dbs: ["~^h3_trade.*"]     # 迁移哪些库
    ignore-tables:                       # 迁移哪些表,测试时只迁移t_test_开头的表
    - db-name: "~.*"
      tbl-name: "~^t_out_order_transport_.*"

但是如果配置多个表同步怎么设置?

多张表的话,需要通过写成正则表达方式,来进行处理。可以参考一下之前团队小伙伴写的一些 DM 的分享。https://docs.pingcap.com/zh/tidb-data-migration/stable/task-configuration-file#dm-任务配置文件介绍

2 个赞

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。