kafka: Failed to produce message to topic user-binlog: kafka: error encoding packet: invalid request size (5980520)

【 TiDB 使用环境】生产环境 /测试/ Poc
测试环境

【 TiDB 版本】
5.1.4

【复现路径】做过哪些操作出现的问题
【遇到的问题:问题现象及影响】
【资源配置】
【附件:截图/日志/监控】
tidb drainer 向 kafka 同步数据,drainer日志报错如下

[2023/03/17 17:05:46.675 +08:00] [FATAL] [kafka.go:223] [“fail to produce message to kafka, please check the state of kafka server”] [error=“kafka: Failed to produce message to topic user-binlog: kafka: error encoding packet: invalid request size (5980520)”] [stack=“github.com/pingcap/tidb-binlog/drainer/sync.(*KafkaSyncer).run.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/sync/kafka.go:223”]

调整如下drainer参数后依然报错
kafka-max-message-size = 1000
kafka-max-messages = 1

报错如下

[2023/03/17 17:16:37.051 +08:00] [FATAL] [kafka.go:223] [“fail to produce message to kafka, please check the state of kafka server”] [error=“kafka: Failed to produce message to topic user-binlog: kafka: error encoding packet: invalid request size (28106)”] [stack=“github.com/pingcap/tidb-binlog/drainer/sync.(*KafkaSyncer).run.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/sync/kafka.go:223”]

继续调大
invalid request size (28106)

使用 默认值时,报错如下,实际上kafka是正常的

[2023/03/17 18:35:56.416 +08:00] [ERROR] [syncer.go:488] [“Failed to close syncer”] [error=“fail to push msg to kafka after 30s, check if kafka is up and working”] [errorVerbose=“fail to push msg to kafka after 30s, check if kafka is up and working\ngithub.com/pingcap/tidb-binlog/drainer/sync.(*KafkaSyncer).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/sync/kafka.go:236\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371”]

[2023/03/17 18:36:06.417 +08:00] [ERROR] [util.go:62] [“Recovered from panic”] [err=“"Waiting too long for Syncer.run to quit."”] [“real stack”=“github.com/pingcap/tidb-binlog/drainer.(*taskGroup).start.func1.1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/util.go:64\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:965\ngithub.com/pingcap/tidb-binlog/drainer.(*Syncer).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:494\ngithub.com/pingcap/tidb-binlog/drainer.(*Syncer).Start\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:133\ngithub.com/pingcap/tidb-binlog/drainer.(*Server).Start.func4\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/server.go:290\ngithub.com/pingcap/tidb-binlog/drainer.(*taskGroup).start.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/util.go:72”] [name=syncer]

如果将 kafka-max-message-size 改为 50m,则报错如下

[2023/03/18 18:22:09.974 +08:00] [FATAL] [kafka.go:223] [“fail to produce message to kafka, please check the state of kafka server”] [error=“kafka: Failed to produce message to topic user-binlog: kafka server: Unexpected (unknown?) server error.”] [stack=“github.com/pingcap/tidb-binlog/drainer/sync.(*KafkaSyncer).run.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb-binlog/drainer/sync/kafka.go:223”]

5.X 不推荐使用 binlog 组件,推荐使用 cdc

kafka 消息限制,参考
https://docs.pingcap.com/zh/tidb/stable/ticdc-faq#为什么-ticdc-到-kafka-的同步任务延时越来越大

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。