TiCDC同步认证的kafka问题

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】v6.5.1
【复现路径】做过哪些操作出现的问题
【遇到的问题:问题现象及影响】
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【附件:截图/日志/监控】

使用TiCDC同步认证的kafka,报kafka认证失败的问题,

tiup ctl:v6.5.1 cdc changefeed create --pd=http://xxxxx:2379 --sink-uri=“kafka://xxxxx:9092/topic_name?kafka-version=2.7.1&sasl-user=tidb_cdc&sasl-password=xxxxxx&sasl-mechanism=PLAIN&max-message-bytes=1048576” --changefeed-id=“kafka-task” --sort-engine=“unified” --config=./changefeed.toml

特别说明:
1、kafka的密码不包含任何特殊字符,只有数字和字母的组合
2、kafka认证方式为SASL/PLAIN

报错:new sarama producer: Cluster authorization failed.

日志如下:
[2024/01/04 12:21:44.572 +08:00] [INFO] [base_client.go:378] [“[pd] switch leader”] [new-leader=http://XXXX:2379] [old-leader=]
[2024/01/04 12:21:44.572 +08:00] [INFO] [base_client.go:105] [“[pd] init cluster id”] [cluster-id=7073366411188089559]
[2024/01/04 12:21:44.572 +08:00] [INFO] [client.go:702] [“[pd] tso dispatcher created”] [dc-location=global]
[2024/01/04 12:21:44.585 +08:00] [INFO] [sink.go:313] [“succeed to parse parameter from sink uri”] [protocol=default] [txnAtomicity=]
[2024/01/04 12:21:44.903 +08:00] [INFO] [sink.go:313] [“succeed to parse parameter from sink uri”] [protocol=default] [txnAtomicity=]
[2024/01/04 12:21:44.927 +08:00] [INFO] [client.go:783] [“[pd] stop fetching the pending tso requests due to context canceled”] [dc-location=global]
[2024/01/04 12:21:44.927 +08:00] [INFO] [client.go:720] [“[pd] exit tso dispatcher”] [dc-location=global]
[2024/01/04 12:21:44.927 +08:00] [INFO] [middleware.go:48] [/api/v2/changefeeds] [status=500] [method=POST] [path=/api/v2/changefeeds] [query=] [ip=xxxxx] [user-agent=Go-http-client/1.1] [client-version=v6.5.1] [error=“[CDC:ErrKafkaNewSaramaProducer]new sarama producer: Cluster authorization failed.”] [errorVerbose=“[CDC:ErrKafkaNewSaramaProducer]new sarama producer: Cluster authorization failed.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/mq.NewKafkaDMLSink\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/mq/kafka_dml_sink.go:72\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/factory.New\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/factory/factory.go:68\ngithub.com/pingcap/tiflow/cdc/sink.Validate\n\tgithub.com/pingcap/tiflow/cdc/sink/validator.go:63\ngithub.com/pingcap/tiflow/cdc/api/v2.APIV2HelpersImpl.verifyCreateChangefeedConfig\n\tgithub.com/pingcap/tiflow/cdc/api/v2/api_helpers.go:240\ngithub.com/pingcap/tiflow/cdc/api/v2.(*OpenAPIV2).createChangefeed\n\tgithub.com/pingcap/tiflow/cdc/api/v2/changefeed.go:81\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.ForwardToOwnerMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:95\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.ErrorHandleMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:64\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.LogMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:38\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/middleware.CheckServerReadyMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:103\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/gin-gonic/gin.CustomRecoveryWithWriter.func1\n\tgithub.com/gin-gonic/gin@v1.8.1/recovery.go:101\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\tgithub.com/gin-gonic/gin@v1.8.1/gin.go:616\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\tgithub.com/gin-gonic/gin@v1.8.1/gin.go:572\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2947\nnet/http.(*conn).serve\n\tnet/http/server.go:1991\nruntime.goexit\n\truntime/asm_amd64.s:1594”] [duration=371.528001ms]

kafka是什么。

kafka的版本对吗,确认下是用的2.7.1?

topic名字写的对吗

topic名称保证是对的

kafka版本确认是2.7.1

我们生产CDC同步用的2.3版本kafka,没有使用鉴权,也没遇到这问题 :rofl:

密码是否正确,是否授权,防火墙是否关闭等等都要检查一下

把plain小写试试?(猜的)

看下是否是kafka用户的权限问题,TiCDC通过ACL连接Kafka是有一定的权限要求的:

ACL 授权TiCDC 能够正常工作所需的最小权限集合如下:
(1)对 Topic 资源类型 的 Create 和 Write 权限。
(2)对 Cluster 资源类型的 DescribeConfigs 权限。

参考:
TiCDC 使用 Kafka 的认证与授权

可以看下:TiCDC 无法接入kafka.

1 个赞

也不是这个原因,我们的kafka认证方式的SASL/PLAIN,也不是权限的问题;我自己写代码,同样的topic,同样的账号密码,认证和写入数据完全是没有问题的

高级…