DELETE的时候出现slow prewrite request

【 TiDB 使用环境】生产环境
【 TiDB 版本】6.2.0 DMR
【复现路径】无法复现
【遇到的问题】:通过cloudcanal进行同步数据,从MYSQL同步到TIDB,但是在执行DELETE 的时候出现较大的延迟,是基于复合主键进行删除的
【资源配置】
【附件:日志】
tidb_10.170.0.26_4000.zip (4.0 KB)
tidb_10.170.0.27_4000.zip (3.9 KB)
tidb_10.170.0.28_4000.zip (4.9 KB)
tikv_10.170.0.29_20160.zip (6.6 KB)
tikv_10.170.0.30_20160.zip (5.4 KB)
tikv_10.170.0.31_20160.zip (6.1 KB)

cloudcanal任务日志如下:

com.clougence.cloudcanal.base.service.task.exception.runtime.ApplierHandlerException: parallel handle increment event failed
	at com.clougence.cloudcanal.tidb.worker.writer.Incre.TiDBIncreBatchApplyNoTxHandler.handle(TiDBIncreBatchApplyNoTxHandler.java:179)
	at com.clougence.cloudcanal.tidb.worker.writer.Incre.TiDBIncreBatchApplyNoTxHandler.handle(TiDBIncreBatchApplyNoTxHandler.java:55)
	at com.clougence.cloudcanal.task.applier.incre.IncrementDisruptorApplyHandler.onEvent(IncrementDisruptorApplyHandler.java:25)
	at com.clougence.cloudcanal.task.applier.incre.IncrementDisruptorApplyHandler.onEvent(IncrementDisruptorApplyHandler.java:12)
	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: com.clougence.cloudcanal.base.service.task.exception.runtime.ApplierHandlerException: apply batch data error.msg:SocketTimeoutException: Read timed out
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at com.clougence.cloudcanal.tidb.worker.writer.Incre.TiDBIncreBatchApplyNoTxHandler.handle(TiDBIncreBatchApplyNoTxHandler.java:164)
	... 6 more
Caused by: com.clougence.cloudcanal.base.service.task.exception.runtime.ApplierHandlerException: apply batch data error.msg:SocketTimeoutException: Read timed out
	at com.clougence.cloudcanal.base.service.task.executor.RdbOneByOneExecutor.exec(RdbOneByOneExecutor.java:82)
	at com.clougence.cloudcanal.base.service.task.applier.RdbIncreMultiSqlApply$DmlWorker.run(RdbIncreMultiSqlApply.java:121)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
Caused by: com.clougence.cloudcanal.base.service.task.exception.runtime.ApplierHandlerException: {"db":null,"schema":"jy8","table":"tbl_game_record","eventType":"DELETE","pks":[["114929357","1669966573221"]]} error,msg:SocketTimeoutException: Read timed out
	at com.clougence.cloudcanal.base.service.task.exception.IncreExceptionHandler.logAndSkipExIfNecessary(IncreExceptionHandler.java:36)
	at com.clougence.cloudcanal.base.service.task.executor.RdbOneByOneExecutor.deleteOneRow(RdbOneByOneExecutor.java:200)
	at com.clougence.cloudcanal.base.service.task.executor.RdbOneByOneExecutor.delete(RdbOneByOneExecutor.java:132)
	at com.clougence.cloudcanal.base.service.task.executor.RdbOneByOneExecutor.exec(RdbOneByOneExecutor.java:72)
	... 6 more
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

The last packet successfully received from the server was 60,062 milliseconds ago. The last packet sent successfully to the server was 60,064 milliseconds ago.
	at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
	at com.mysql.cj.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:370)
	at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:497)
	at com.clougence.cloudcanal.base.service.task.executor.RdbOneByOneExecutor.deleteOneRow(RdbOneByOneExecutor.java:192)
	... 8 more
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure

The last packet successfully received from the server was 60,062 milliseconds ago. The last packet sent successfully to the server was 60,064 milliseconds ago.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
	at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
	at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:546)
	at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:710)
	at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:649)
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:951)
	at com.mysql.cj.NativeSession.execSQL(NativeSession.java:1075)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:930)
	... 11 more
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at com.mysql.cj.protocol.ReadAheadInputStream.fill(ReadAheadInputStream.java:107)
	at com.mysql.cj.protocol.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:150)
	at com.mysql.cj.protocol.ReadAheadInputStream.read(ReadAheadInputStream.java:180)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:64)
	at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:63)
	at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:45)
	at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:52)
	at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:41)
	at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:54)
	at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:44)
	at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:540)
	... 16 more

TIDB上看delete的sql慢吗?

在dashboard看到这个DELETE执行要很久 ,最小 24秒 ,长的1.4min。但是这是基于主键删除数据的呀,这就有问题了;

delete from `dbn` . `tbl_game_record` where `id` = ? and `bet_time` = ? limit ?

id 和 bet_time 是联合主键

删除时间点集群的整体负载怎么样?



dsshboard看下这个慢sql的执行时间消耗是再哪些部分

[“run statement failed”] [conn=6195053131177213977] [schemaVersion=350] [error=“[tikv:9004]Resolve lock timeout”]

你的日志里面有很多锁冲突导致超时的。

这个我也看到了,但是按理说 ,主键DELETE应该不会有锁冲突。

是不是多个任务同时执行的?导致锁冲突?可以尝试减少并发,说不定会更快。

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。