创建ticdc同步任务时，指定的kafka版本有什么作用或者有没有必要指定kafka版本

liyuntang · 2025 年3 月 11 日 11:55

【 TiDB 使用环境】测试环境 kubernetes环境 opertator为1.5.4版本
【 TiDB 版本】企业版7.1.0
【遇到的问题：问题现象及影响】
经测试创建到kafka的ticdc通过任务时，kafka版本无需指定或者指定一个错误的版本数据也能正常同步，所以官方文档里说的“该值需要与下游 Kafka 的实际版本保持一致。”是否准确，或者说在创建同步任务时有没有必要指定kafka版本

【复现路径】
1、正常的ticdc任务(完全按照官方要求创建的任务)

2、运行insert.sh程序模拟tidb 数据写入

3、创建新任务test，将kafka的版本指定为3.5.1(kafka的真正版本号为2.8.1)，观察是否创建成功

任务创建成功了，我们运行kafka消费程序观察数据同步是否正常

数据同步正常，可以正常消费

4、重新创建这个任务(任务名test)，此次不指定版本号，观察是否创建成功

5、在没有指定kafka版本的情况下依然可以创建成功。我们运行kafka消费程序观察数据同步是否正常

数据同步正常，可以正常消费

那么，问题来了，kafka-version参数在不传、错传的情况下ticdc任务依然可以创建成功，且不影响数据消费，那么“该值需要与下游 Kafka 的实际版本保持一致。”是否准确，或者说在创建同步任务时有没有必要指定kafka版本，在不传、错传的情况下对数据同步、数据消费有什么影响

有猫万事足 · 2025 年3 月 11 日 15:01

ticdc访问kafka使用的是下面这个开源项目

需要指定版本也是这个项目的要求。
你把版本号这个静态变量放进去搜一下，就会发现里面用这个版本号做了很多代码的分支。

按理说不同的版本应该有不同的行为和访问协议，sarama才需要这么做。但这个又和你实际的实验结论感觉上不一致。
继续找几个pr进去继续翻。

github.com/IBM/sarama

Consider switching default Kafka version to at least 2.1

已打开 01:17AM - 29 Dec 22 UTC

ijuma

stale/exempt

The default Kafka version was set to `1.0` via #1787. This was a huge improvemen…t over the previous default and the hope was that proper api versions support would be supported some time later. 2.5 years have passed and the default remains [1.0](https://github.com/stanislavkozlovski/sarama/blob/5ab6809aecc5a50ee523d27983762d0abdcc0105/utils.go#L195) if I understand correctly (please correct me if this is wrong). I am working on a KIP to remove older protocol versions in Apache Kafka 4.0 and the proposed baseline is Apache Kafka 2.1: https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 The KIP is still under discussion (i.e. it hasn't been approved yet), but I wanted to: 1. Give an early heads up to this project since it would impact your users with the current default 2. Get your input on whether bumping the default Kafka version in the near term is something you would consider Thanks!

sarama在这个issue中准备把默认的kafka版本指定在v2.1.

同时给出了kafka那边的讨论链接：

https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0

直接看动机这一段

Motivation

The Apache Kafka project has retained support for all protocol API versions since Apache Kafka 0.8.0 (released 9+ years ago on December 2013). As time goes on, the cost of maintaining support for all these versions goes up (both in code complexity and the testing matrix) and the value goes down (as users switch to clients that rely on newer protocol API versions). Apache Kafka 4.0 presents a good opportunity to re-evaluate this policy since it includes other modernization initiatives like dropping support for zk mode and dropping support for message formats v0 and v1 (KIP-724). No date has been set for Apache Kafka 4.0, but it’s safe to say it won’t happen before the end of 2023.

We propose setting the new baseline for protocol API versions to Apache Kafka 2.1 (released 4+ years ago on Nov 2018). More concretely, for each protocol API, we will only keep the most recent version supported by Apache Kafka 2.1 as well as any version introduced after Apache Kafka 2.1. We deviate from the proposed baseline in a number of cases based on existing clients support. In order to determine client support for the various protocol API versions, we take the latest client release at the end of 2021 - this gives it a better chance of being widely used.

Apache Kafka 2.1 has been chosen as the baseline because it was released many years ago and it includes support for KIP-320 (required for proper fencing support when consuming) and KIP-110 (zstd is a popular compression mechanism).

Since some protocol APIs are used both for clients and inter-broker replication (e.g. Fetch, ListOffsets, etc.), this proposal also implies a minimum inter.broker.protocol.version of 2.1. That said, the removal of zk mode (also planned for Apache Kafka 4.0) will have even stricter requirements when it comes to this.

We believe there is merit in having a general protocol API versioning policy that can be applied mechanically on each new major version - we intend to submit a separate KIP for that. It would only apply from Apache Kafka 5.0 to give the ecosystem plenty of time to adapt.

里面的理由和细节还是比较丰富的，api的基线打算被定在2.1版本，并且api不打算向下兼容了。api的版本影响的内容也非常多，甚至会影响到kafka和zookeeper之间的互动方式。

所以我觉得结论是：

kafka-version这个参数是必须的，文档没有写错。不同的kafka版本会有截然不同的api访问方式。甚至和其他外围的组件例如zookeeper之类的互动也会有截然不同的模式。

在你不传的时候，ticdc默认指定的kafka版本是2.0.

github.com/pingcap/tiflow

pkg/sink/kafka/sarama.go

b67943012


      
          
          	"github.com/IBM/sarama"
          	"github.com/pingcap/errors"
          	"github.com/pingcap/log"
          	cerror "github.com/pingcap/tiflow/pkg/errors"
          	"github.com/pingcap/tiflow/pkg/security"
          	"go.uber.org/zap"
          )
          
          var (
          	defaultKafkaVersion = sarama.V2_0_0_0
          	maxKafkaVersion     = sarama.V2_8_0_0
          )
          
          // NewSaramaConfig return the default config and set the according version and metrics
          func NewSaramaConfig(ctx context.Context, o *Options) (*sarama.Config, error) {
          	config := sarama.NewConfig()
          	config.ClientID = o.ClientID
          	var err error
          	// Admin client would refresh metadata periodically,
          	// if metadata cannot be refreshed easily, this would indicate the network condition between the

你选择的版本号正好在你的kafka实际能支持的api协议的版本范围内。上面这个KIP-896是2025.3.6接受的。从这个版本往后最小的api版本应该是2.1.你的kafka的版本是2.8.1应该是正好能支持2.0和3.5.1这2个版本的api。
具体的差别，你可以自己翻翻sarama的实现。

liyuntang · 2025 年3 月 12 日 01:56

明白了，谢谢

system · 2025 年3 月 19 日 01:57

此话题已在最后回复的 7 天后被自动关闭。不再允许新回复。