使用fink-tidb-cdc-connector，tidb到tidb，数据同步一半就不同步了

TiDBer_FYLY5ka1 · 2022 年12 月 8 日 07:45

我在做一个从tidb同步到tidb的任务，总共数据28w，使用的是fink-tidb-cdc-connector，任务开始后，正常同步了20W数据，但是后面就不同步了，日志只是一直输出以下语句：

2022-12-08 14:56:31,904 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906988733300739, regionId: 3495290
2022-12-08 14:56:31,905 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906988733300739, regionId: 3495290
2022-12-08 14:56:32,868 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906988982337537, regionId: 3495286
2022-12-08 14:56:32,869 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906988982337537, regionId: 3495286
2022-12-08 14:56:32,869 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906988982337537, regionId: 3495286
2022-12-08 14:56:32,905 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906988995444739, regionId: 3495290
2022-12-08 14:56:32,905 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906988995444739, regionId: 3495290
2022-12-08 14:56:32,905 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906988995444739, regionId: 3495290
2022-12-08 14:56:33,857 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906989244481537, regionId: 3495286
2022-12-08 14:56:33,857 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906989244481537, regionId: 3495286
2022-12-08 14:56:33,858 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906989244481537, regionId: 3495286
2022-12-08 14:56:33,906 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906989257588738, regionId: 3495290
2022-12-08 14:56:33,906 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906989257588738, regionId: 3495290
2022-12-08 14:56:33,908 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906989257588738, regionId: 3495290
2022-12-08 14:56:34,870 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906989506625537, regionId: 3495286
2022-12-08 14:56:34,870 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906989506625537, regionId: 3495286
2022-12-08 14:56:34,870 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906989506625537, regionId: 3495286
2022-12-08 14:56:34,909 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906989519732737, regionId: 3495290
2022-12-08 14:56:34,909 INFO org.tikv.cdc.CDCClient [] - handle resolvedTs: 437906989519732737, regionId: 3495290

Billmay表妹 · 2022 年12 月 8 日 08:05

【 TiDB 版本】
【Flink 版本】
【Flink CDC版本】
贴一下

TiDBer_FYLY5ka1 · 2022 年12 月 8 日 08:07

tidb v5.3.0
flink-sql-connector-tidb-cdc 2.2.1

TiDBer_FYLY5ka1 · 2022 年12 月 8 日 08:08

flink 1.14.3

Billmay表妹 · 2022 年12 月 8 日 08:08

参考这个贴子，可以先升级到 2.3.0 试一下~

Billmay表妹 · 2022 年12 月 8 日 08:11

这报错可以去 Flink CDC 社区问一下，看 TiDB 这块未解的 bug 有好几个；
针对 ticdc 这边的 log 有异常吗？可以贴下

TiDBer_FYLY5ka1 · 2022 年12 月 8 日 08:20

log 没有异常在，他只是存量数据不能同步完全，增量数据是正常的，我尝试使用 2.3.0 版本，重新同步全量，现在正常了。

TiDBer_FYLY5ka1 · 2022 年12 月 8 日 08:21

感谢

Billmay表妹 · 2022 年12 月 8 日 08:29

好叻，那可能是低版本的 BUG 引起的，有其他问题再反馈~

TiDBer_FYLY5ka1 · 2022 年12 月 8 日 09:30

我尝试同步更大的表（tidb到tidb，千万级），发生以下警告：
2022-12-08 17:12:32,556 INFO com.ververica.cdc.connectors.tidb.TiKVRichParallelSourceFunction [] - read snapshot events
2022-12-08 17:12:32,860 WARN org.tikv.common.region.StoreHealthyChecker [] - store [127.0.0.1:3930] is not reachable
2022-12-08 17:20:55,822 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - The heartbeat of JobManager with id ae84acffcbb8019922f9dfbfa758b959 timed out.

随后便任务就停止了：
FlinkException: Disconnect from JobManager responsible for 37783358c880142452cfdc23a76b97c4.

at org.apache.flink.runtime.taskexecutor.TaskExecutor.disconnectJobManagerConnection(TaskExecutor.java:1654)

at org.apache.flink.runtime.taskexecutor.TaskExecutor.disconnectAndTryReconnectToJobManager(TaskExecutor.java:1218)

at org.apache.flink.runtime.taskexecutor.TaskExecutor.access$3900(TaskExecutor.java:183)

at org.apache.flink.runtime.taskexecutor.TaskExecutor$JobManagerHeartbeatListener.lambda$handleJobManagerConnectionLoss$0(TaskExecutor.java:2387)

at java.util.Optional.ifPresent(Optional.java:159)

at org.apache.flink.runtime.taskexecutor.TaskExecutor$JobManagerHeartbeatListener.handleJobManagerConnectionLoss(TaskExecutor.java:2385)

at org.apache.flink.runtime.taskexecutor.TaskExecutor$JobManagerHeartbeatListener.notifyHeartbeatTimeout(TaskExecutor.java:2368)

at org.apache.flink.runtime.heartbeat.HeartbeatMonitorImpl.run(HeartbeatMonitorImpl.java:155)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRunAsync$4(AkkaRpcActor.java:455)

at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)

at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:455)

at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:213)

at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)

at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)

at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)

at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)

at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)

at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)

at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)

at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)

at akka.actor.Actor.aroundReceive(Actor.scala:537)

at akka.actor.Actor.aroundReceive$(Actor.scala:535)

at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)

at akka.actor.ActorCell.invoke(ActorCell.scala:548)

at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)

at akka.dispatch.Mailbox.run(Mailbox.scala:231)

at akka.dispatch.Mailbox.exec(Mailbox.scala:243)

at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)

at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)

at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)

at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

Caused by: java.util.concurrent.TimeoutException: The heartbeat of JobManager with id ae84acffcbb8019922f9dfbfa758b959 timed out.

system · 2023 年2 月 6 日 09:30

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。