CDC同步速率过慢,日志频繁报写入冲突

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】v6.5.0
请教各位大神,我们cdc同步到下游tidb集群时同步速率很慢,每秒只有几千,远远达不到官方说的几万。查看监控是在下游写入锁冲突耗时较多,日志也是报了锁冲突的问题,下游只有一个cdc的写入源,不应该有冲突呀,所以比较奇怪,请各位帮忙看看。截图和报错如下。


[2024/07/25 21:22:39.116 +08:00] [WARN] [mysql.go:599] [“execute DMLs with error, retry later”] [error=“[CDC:ErrMySQLTxnError]MySQL txn error: Error 8022: Error: KV error safe to retry Error(Txn(Error(Mvcc(Error(TxnLockNotFound { start_ts: TimeStamp(451389358718059268), commit_ts: TimeStamp(451389360447947045), key: [116, 128, 0, 0, 0, 0, 0, 1, 18, 95, 105, 128, 0, 0, 0, 0, 0, 0, 2, 4, 0, 0, 147, 107, 65, 76, 199, 209, 1, 0, 0, 0, 0, 0, 0, 0, 0, 247, 3, 128, 0, 0, 0, 128, 85, 135, 102] }))))) {tableID=274, indexID=2, indexValues={162088866334673, , 2153088870, }} [try again later]”] [errorVerbose=“[CDC:ErrMySQLTxnError]MySQL txn error: Error 8022: Error: KV error safe to retry Error(Txn(Error(Mvcc(Error(TxnLockNotFound { start_ts: TimeStamp(451389358718059268), commit_ts: TimeStamp(451389360447947045), key: [116, 128, 0, 0, 0, 0, 0, 1, 18, 95, 105, 128, 0, 0, 0, 0, 0, 0, 2, 4, 0, 0, 147, 107, 65, 76, 199, 209, 1, 0, 0, 0, 0, 0, 0, 0, 0, 247, 3, 128, 0, 0, 0, 128, 85, 135, 102] }))))) {tableID=274, indexID=2, indexValues={162088866334673, , 2153088870, }} [try again later]\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1.3\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:575\ngithub.com/pingcap/tiflow/cdc/sinkv2/metrics.(*Statistics).RecordBatchExecution\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/metrics/statistics.go:74\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:531\ngithub.com/pingcap/tiflow/pkg/retry.run\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:508\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).Flush\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:158\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn.(*worker).doFlush\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/worker.go:209\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn.(*worker).runBackgroundLoop.func1\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/worker.go:163\nruntime.goexit\n\truntime/asm_amd64.s:1594”] [duration=6.644718322s] [query=COMMIT] [count=2] [startTs=“[451389089507183412,451389089664467762]”] [changefeed=default.bj-sjzt-to-sjbb-01]

看看 LAG 的指标


这个肯定不正常了,延迟已经是分钟级了(都 30 分钟了,半小时了…)

你先检查下,是数据问题,还是资源的问题导致的

调大并发数试试?