ticdc 反复重启

🪐 TiDB 技术问题
1

今天遇到一个问题发现所有 ticdc 的 resolve tso 都不往前推进了，tidb 的日志也不往前推进，请问有人要过吗，版本是 v5.4.1

报错如下

[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:452] ["[etcd worker] ============State Commit============="] [committed=false]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:456] ["[etcd worker] ==========Failed Etcd Txn Cmps=========="]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/workload/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-ticdc001\" mod_revision:0 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/status/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-ticdc001\" mod_revision:103984787 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/status/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-ticdc002\" mod_revision:103984787 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/workload/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-ticdc002\" mod_revision:0 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/status/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-ticdc007\" mod_revision:103984752 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/workload/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-ticdc007\" mod_revision:0 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/status/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-alltab\" mod_revision:103984787 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:MOD key:\"/tidb/cdc/task/workload/5fe2dac7-bea2-45cd-8768-184e721b92b3/mediav-base-alltab\" mod_revision:0 "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:459] ["[etcd worker] compare"] [cmp="target:VALUE key:\"/tidb/cdc/meta/ticdc-delete-etcd-key-count\" value:\"2395\" "]
[2022/09/21 11:53:31.837 +08:00] [INFO] [etcd_worker.go:462] ["[etcd worker] ============End Failed Etcd Txn Cmps============="]
[2022/09/21 11:53:32.583 +08:00] [INFO] [etcd_worker.go:444] ["[etcd worker] ==========Update State to ETCD=========="]
2

这些日志都是 info，也不是错误呢

3

额，我检查了下没有 WARN 日志，我尝试吧 changefeed remove 掉发现了一个报错

[2022/09/21 14:12:39.806 +08:00] [WARN] [changefeed.go:349] [“changefeed is removed, but state is not complete”] [state="{“ID”:“mediav-base-alltab”,“Info”:null,“Status”:null,“TaskPositions”:{},“TaskStatuses”:{},“Workloads”:{}}"]

4

删除任务变了一些协议开始报这个问题了

[2022/09/21 14:37:48.698 +08:00] [PANIC] [canal.go:167] [“unsigned value not in type uint64”] [column="{“name”:“xxx”,“type”:8,“flag”:225,“value”:null}"] [stack=“github.com/pingcap/tiflow/cdc/sink/codec.getJavaSQLType\ \tgithub.com/pingcap/tiflow/cdc/sink/codec/canal.go:167\ github.com/pingcap/tiflow/cdc/sink/codec.(*canalEntryBuilder).buildColumn\ \tgithub.com/pingcap/tiflow/cdc/sink/codec/canal.go:280\ github.com/pingcap/tiflow/cdc/sink/codec.(*canalEntryBuilder).buildRowData\ \tgithub.com/pingcap/tiflow/cdc/sink/codec/canal.go:306\ github.com/pingcap/tiflow/cdc/sink/codec.(*CanalFlatEventBatchEncoder).newFlatMessageForDML\ \tgithub.com/pingcap/tiflow/cdc/sink/codec/canal_flat.go:178\ github.com/pingcap/tiflow/cdc/sink/codec.(*CanalFlatEventBatchEncoder).AppendRowChangedEvent\ \tgithub.com/pingcap/tiflow/cdc/sink/codec/canal_flat.go:321\ github.com/pingcap/tiflow/cdc/sink.(*mqSink).runWorker\ \tgithub.com/pingcap/tiflow/cdc/sink/mq.go:338\ github.com/pingcap/tiflow/cdc/sink.(*mqSink).run.func1\ \tgithub.com/pingcap/tiflow/cdc/sink/mq.go:267\ golang.org/x/sync/errgroup.(*Group).Go.func1\ \tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57”]

5

-force，强制删除试一试

6

升级了tidb版本问题解决

7

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。