【TiDB 版本】V7.5.6
【遇到的问题：问题现象及影响】
将旧集群数据通过BR、lightning、DTS导入到新集群，然后通过dataworks继续回流增量数据，回流期间一直异常，违反唯一约束。相同的数据在mysql上重复执行了5次，均没有异常。
怀疑insert into on duplicate key update方式，MySQL与TiDB的实现逻辑不一样或者主键自增存在异常。
dataworks回流报错的语句，手动执行又可以执行成功。
在TiDB7.5.6版本执行存在异常，但在TiDB 4.0.11版本执行又可以正常执行。
TiDB 7.5.6链接信息
2025-09-01 21:02:12.351 [job-1849816697] INFO DBUtil - DatabaseType: MySql login url: jdbc:mysql://merchant-dsr-tidb.shizhuang-inc.com:4013/dw_merchant_data_themis?useSSL=false&allowLoadLocalInfile=false&allowLocalInfile=false&allowUrlInLocalInfile=false&autoDeserialize=false&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true with connection properties: {“password”:“***”,“user”:“dw_merchant_data_themis”} 2025-09-01 21:02:12.573 [job-1849816697] INFO DBUtil - choose jdbc driverName: MySQL Connector Java driverVersion: mysql-connector-java-5.1.49 ( Revision: ad86f36e100e104cd926c6b81c8cab9565750116 ) 2025-09-01 21:02:12.573 [job-1849816697] INFO DBUtil - databaseProductVersion: MySQL, databaseProductVersion: 8.0.11-TiDB-v7.5.6 2025-09-01 21:02:12.574 [job-1849816697] INFO DBUtil - user configured useSSL, url: jdbc:mysql://merchant-dsr-tidb.shizhuang-inc.com:4013/dw_merchant_data_themis?useSSL=false&allowLoadLocalInfile=false&allowLocalInfile=false&allowUrlInLocalInfile=false&autoDeserialize=false&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true, props null, skip check. 2025-09-01 21:02:12.574 [job-1849816697] INFO DBUtil - databaseProductVersion: 8.0.11-TiDB-v7.5.6, databaseMajorVersion: 8
TiDB 4.0.11链接信息
2025-09-01 21:55:50.248 [job-1849852898] INFO DBUtil - DatabaseType: MySql login url: jdbc:mysql://10.221.201.195:4000/merchant_data?useSSL=false&allowLoadLocalInfile=false&allowLocalInfile=false&allowUrlInLocalInfile=false&autoDeserialize=false&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true with connection properties: {“password”:“***”,“user”:“merchant_test”} 2025-09-01 21:55:50.491 [job-1849852898] INFO DBUtil - choose jdbc driverName: MySQL Connector Java driverVersion: mysql-connector-java-5.1.49 ( Revision: ad86f36e100e104cd926c6b81c8cab9565750116 ) 2025-09-01 21:55:50.491 [job-1849852898] INFO DBUtil - databaseProductVersion: MySQL, databaseProductVersion: 5.7.25-TiDB-v4.0.11
dataworks回流模式：
表结构：
CREATE TABLE
merchant_assessment_month_result (id bigint(20) NOT NULL AUTO_INCREMENT COMMENT ‘主键’,user_id bigint(20) NOT NULL,merchant_id bigint(20) NOT NULL ,merchant_name varchar(255) NOT NULL ,cycle_date int(8) NOT NULL ,overall_score bigint(20) ,overall_score_rank varchar(32),pay_order_cnt bigint(20),create_time datetime ,update_time datetime ,empower_gmv bigint(20) ,unempower_gmv bigint(20),unempower_collection_time bigint(20) ,unempower_pay_cnt bigint(20),empower_status int(4),merchant_type int(4) ,data_version varchar(8),PRIMARY KEY (
id),UNIQUE KEY
uk_merchant_id_cycle_date (merchant_id,cycle_date),KEY
idx_create_time (create_time),KEY
idx_update_time (update_time),KEY
idx_cycle_date_merchant_id (cycle_date,merchant_id),KEY
idx_ empower_status (empower_status)) ENGINE=InnoDB;
回流语句：
INSERT INTO %s (user_id,merchant_id,merchant_name,cycle_date,overall_score,overall_score_rank,pay_order_cnt,empower_gmv,unempower_gmv,unempower_pay_cnt,unempower_collection_time,empower_status,merchant_type,data_version) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?) ON DUPLICATE KEY UPDATE user_id=VALUES(user_id),merchant_id=VALUES(merchant_id),merchant_name=VALUES(merchant_name),cycle_date=VALUES(cycle_date),overall_score=VALUES(overall_score),overall_score_rank=VALUES(overall_score_rank),pay_order_cnt=VALUES(pay_order_cnt),empower_gmv=VALUES(empower_gmv),unempower_gmv=VALUES(unempower_gmv),unempower_pay_cnt=VALUES(unempower_pay_cnt),unempower_collection_time=VALUES(unempower_collection_time),empower_status=VALUES(empower_status),merchant_type=VALUES(merchant_type),data_version=VALUES(data_version)
回流任务报错：
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry ‘5688533-202508’ for key ‘merchant_assessment_month_result.uk_merchant_id_cycle_date’;
Code:[Framework-14], Description:[The dirty data transmitted by DataX exceeds user expectations. This error often occurs when a lot dirty data exists in the source data. Please carefully check the dirty data log information reported by DataX, or you can tune up the dirty data threshold value. ]. - Check for the number of dirty data entries has not passed. The limit is [0] entries, but [190] entries have been captured.
如果是稳定复现的话，可能是遇到bug了。
感谢分享解决方案