ticdc-5.4.0 无法正常同步

[信仰在空中飘扬]

使用场景:
tidb 集群20+ 节点，每天大量的插入数据，使用ticdc 同步tidb 的变更数据到下游kafka 中
由5.3.0 版本ticdc 依然存在延迟问题，于是升级tidb 到5.4.0 版本

使用后出现的问题：
ticdc 无法缓冲已经延迟的数据，出现panic 错误

[2022/02/18 12:51:16.319 +08:00] [INFO] [region_range_lock.go:246] ["tryLockRange stale"] [lockID=3] [regionID=4806118] [startKey=7480000000000001ff6a5f72e000000034ff06ac190000000000fa] [endKey=7480000000000001ff6a5f72e000000034ff16a0730000000000fa] [allOverlapping="[\"regionID: 4806118, ver: 4875, start: 7480000000000001ff6a5f72e000000034ff06ac190000000000fa, end: 7480000000000001ff6a5f72e000000034ff16a0730000000000fa\"]"]
[2022/02/18 12:51:16.319 +08:00] [INFO] [client.go:616] ["request expired"] [regionID=4806118] [span="[7480000000000001ff6a5f72e000000034ff06ac190000000000fa, 7480000000000001ff6a5f72e000000034ff16a0730000000000fa)"] [retrySpans="[]"]
[2022/02/18 12:51:17.473 +08:00] [PANIC] [canal.go:167] ["unsigned value not in type uint64"] [column="{\"name\":\"customer_sid\",\"type\":8,\"flag\":129,\"value\":\"0\"}"] [stack="github.com/pingcap/tiflow/cdc/sink/codec.getJavaSQLType\
\tgithub.com/pingcap/tiflow/cdc/sink/codec/canal.go:167\
github.com/pingcap/tiflow/cdc/sink/codec.(*canalEntryBuilder).buildColumn\
\tgithub.com/pingcap/tiflow/cdc/sink/codec/canal.go:280\
github.com/pingcap/tiflow/cdc/sink/codec.(*canalEntryBuilder).buildRowData\
\tgithub.com/pingcap/tiflow/cdc/sink/codec/canal.go:317\
github.com/pingcap/tiflow/cdc/sink/codec.(*CanalFlatEventBatchEncoder).newFlatMessageForDML\
\tgithub.com/pingcap/tiflow/cdc/sink/codec/canal_flat.go:178\
github.com/pingcap/tiflow/cdc/sink/codec.(*CanalFlatEventBatchEncoder).AppendRowChangedEvent\
\tgithub.com/pingcap/tiflow/cdc/sink/codec/canal_flat.go:321\
github.com/pingcap/tiflow/cdc/sink.(*mqSink).runWorker\
\tgithub.com/pingcap/tiflow/cdc/sink/mq.go:338\
github.com/pingcap/tiflow/cdc/sink.(*mqSink).run.func1\
\tgithub.com/pingcap/tiflow/cdc/sink/mq

此时ticdc 已经无法正常向下游同步数据。ticdc 监控大致如下

[zhenjiaogao]

这边先看下，有更新再跟帖回复 ~

[TiDBer_L3dm6WTj]

应该是新发布的版本吧，有空看看

[TiDBer_4oMELAVl]

panic错误应该是比较严重的错误了，5.4.0这个版本还没用过

[TiDBer_ZfRrIo1G]

遇到同样的问题…

[2022/03/01 18:27:11.489 +08:00] [PANIC] [canal.go:167] [“unsigned value not in type uint64”] [column="{“name”:“mch_id”,“type”:8,“flag”:193,“value”:null}"]
[stack=“github.com/pingcap/tiflow/cdc/sink/codec.getJavaSQLType\ \tgithub.com/pingcap/tiflow/cdc/sink/codec/canal.go:167\ github.com/pingcap/tiflow/cdc/sink/codec.(*canalEntryBuilder).buildColumn\ \tgithub.com/pingcap/tiflow/cdc/sink/codec/canal.go:280\ github.com/pingcap/tiflow/cdc/sink/codec.(*canalEntryBuilder).buildRowData\ \tgithub.com/pingcap/tiflow/cdc/sink/codec/canal.go:306\ github.com/pingcap/tiflow/cdc/sink/codec.(*CanalFlatEventBatchEncoder).newFlatMessageForDML\ \tgithub.com/pingcap/tiflow/cdc/sink/codec/canal_flat.go:178\ github.com/pingcap/tiflow/cdc/sink/codec.(*CanalFlatEventBatchEncoder).AppendRowChangedEvent\ \tgithub.com/pingcap/tiflow/cdc/sink/codec/canal_flat.go:321\ github.com/pingcap/tiflow/cdc/sink.(*mqSink).runWorker\ \tgithub.com/pingcap/tiflow/cdc/sink/mq.go:338\ github.com/pingcap/tiflow/cdc/sink.(*mqSink).run.func1\ \tgithub.com/pingcap/tiflow/cdc/sink/mq.go:267\ golang.org/x/sync/errgroup.(*Group).Go.func1\ \tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57”]

[maxshuang]

@信仰在空中飘扬
https://github.com/pingcap/tiflow/issues/4635
这个 issue 对应的 pr 已经修复了你的问题，是一样的，可以使用新的版本先规避下。
另外： 这个表的 customer_sid 字段是创建表就有的？还是后面 add column 新增？如果是新增的话看是不是能提供下对应的 DDL。
感谢

[maxshuang]

@TiDBer_ZfRrIo1G 你的表结构可以提供一下排查吗？
具体复现的 DDL/DML 序列看是否能一起提供下

关注： https://github.com/pingcap/tiflow/issues/4736

[LingJin (3AceShowHand)]

mch_id 这一列，允许存在 null 值，并且未插入，可以通过给该列赋一个默认值来绕过这个问题。

该问题已经有 PR，下个版本修复。