TiCDC 创建抽取任务报错Error: [CDC:ErrVersionIncompatible]version is incompatible

🪐 TiDB 技术问题 备份&数据迁移
,
1

【 TiDB 使用环境`】
测试环境

【 TiDB 版本】
$ tiup cluster display calctidb-cluster
Cluster name: calctidb-cluster
Cluster version: v6.0.0
Deploy user: tidb
SSH type: builtin

【 cdc 版本】
$ tiup cdc version
tiup is checking updates for component cdc …
Starting component cdc: /home/tidb/.tiup/components/cdc/v6.1.0/cdc version
** Release Version: v6.1.0**
Git Commit Hash: 9f5e3cea2fae0a3dce6cc5ca79e3bcd654b8aa99
Git Branch: heads/refs/tags/v6.1.0
UTC Build Time: 2022-06-07 11:54:04
Go Version: go version go1.18.2 linux/amd64
Failpoint Build: false

【 OS 版本】
CentOS Linux release 7.8.2003 (Core)

【 tiup 版本】
[tidb@ptidbcalcmanage01 ~]$ tiup --version
1.10.2 tiup
Go Version: go1.18.3
Git Ref: v1.10.2
GitHash: 2de5b500c9fae6d418fa200ca150b8d5264d6b19

【遇到的问题】
在TiDB 5.0.0 版本创建TiCDC同步任务，报如下错误。
Error: [CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha
在offline 模式升级TiDB 到6.0.0 版本创建TiCDC同步任务，还是报如下错误。
Error: [CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha

【复现路径】
1：报错时候对TiDB集群与tiup组件都进行升级操作。
2：发现还是报同样的错误新，随后将CDC节点缩容后在扩容。担心TiCDC在TiDB 5.0版本创建，保留了就的信息，从新安装了一次。

【问题现象及影响】
TiCDC节点日志信息
[2022/07/08 16:26:44.919 +08:00] [INFO] [server.go:355] ["/data03/cdc-8300 is set as data-dir (1206GB available), sort-dir=/data03/cdc-8300/tmp/sorter. It is recommended that the disk for data-dir at least have 500GB available space"]
[2022/07/08 16:26:44.922 +08:00] [WARN] [check.go:79] [“check TiKV version failed”] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\ \tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ github.com/pingcap/errors.(*Error).GenWithStackByArgs\ \tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ github.com/pingcap/tiflow/pkg/version.CheckStoreVersion\ \tgithub.com/pingcap/tiflow/pkg/version/check.go:178\ github.com/pingcap/tiflow/pkg/version.CheckClusterVersion\ \tgithub.com/pingcap/tiflow/pkg/version/check.go:74\ github.com/pingcap/tiflow/cdc.(*Server).Run\ \tgithub.com/pingcap/tiflow/cdc/server.go:184\ github.com/pingcap/tiflow/pkg/cmd/server.(*options).run\ \tgithub.com/pingcap/tiflow/pkg/cmd/server/server.go:149\ github.com/pingcap/tiflow/pkg/cmd/server.NewCmdServer.func1\ \tgithub.com/pingcap/tiflow/pkg/cmd/server/server.go:297\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.2.1/command.go:856\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.2.1/command.go:974\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.2.1/command.go:902\ github.com/pingcap/tiflow/pkg/cmd.Run\ \tgithub.com/pingcap/tiflow/pkg/cmd/cmd.go:50\ main.main\ \t./main.go:22\ runtime.main\ \truntime/proc.go:250\ runtime.goexit\ \truntime/asm_amd64.s:1571"]

【创建任务命令】
tiup cdc cli changefeed create --pd=http://xxx.xx.3.115:2379 --sink-uri=“kafka:///xxx.xx.8.24:15502,/xxx.xx.8.25:15502,172.18.8.26:15502/t_pay_detail?kafka-version=3.1.0&max-message-bytes=10485760&replication-factor=3” --changefeed-id=“t-pay-detail” --config /home/tidb/cdc-conf/t_pay_detail.toml

2

cdc 版本和 tidb 版本要保持一致，部署 ticdc 可以用 tiup cluster scale-out cdc.yaml 方式, 会自动部署同一个版本的 cdc.

3

这个操作也做过，结果还是一样

4

你的环境现在 ticdc & tidb 版本已经完全一致了是吗？ 是的话，如果可以，清理一下 metadata (会删除所有 changefeed): cdc cli unsafe reset

5

–CDC 版本
[tidb@ptidbcalcmanage01 ~]$ tiup cdc version
tiup is checking updates for component cdc …
A new version of cdc is available:
The latest version: v6.1.0
Local installed version: v6.0.0
Update current component: tiup update cdc
Update all components: tiup update --all

Starting component cdc: /home/tidb/.tiup/components/cdc/v6.0.0/cdc version
Release Version: v6.0.0
Git Commit Hash: 2f363f653a49fc557ebaab1afa7a47f900fbfc8f
Git Branch: heads/refs/tags/v6.0.0
UTC Build Time: 2022-03-31 10:05:34
Go Version: go version go1.18 linux/amd64
Failpoint Build: false
–TiDB 版本
tiup cluster list;
tiup is checking updates for component cluster …
Starting component cluster: /home/tidb/.tiup/components/cluster/v1.10.2/tiup-cluster list
Name User Version Path PrivateKey

calctidb-cluster tidb v6.0.0 /home/tidb/.tiup/storage/cluster/clusters/calctidb-cluster /home/tidb/.tiup/storage/cluster/clusters/calctidb-cluster/ssh/id_rsa
–创建任务还是报同样的错误。
[tidb@ptidbcalcmanage01 cdc-conf]$ tiup cdc cli changefeed create --pd=http://xxx.xx.3.113:2379 --sink-uri=“kafka://xxx.xx.8.24:15502,xxx.xx.8.25:15502,xxx.xx.8.26:15502/t_pay_detail?kafka-version=3.1.0&max-message-bytes=10485760&replication-factor=3” --changefeed-id=“t-pay-detail” --config /home/tidb/cdc-conf/t_pay_detail.toml
tiup is checking updates for component cdc …
A new version of cdc is available:
The latest version: v6.1.0
Local installed version: v6.0.0
Update current component: tiup update cdc
Update all components: tiup update --all

Starting component cdc: /home/tidb/.tiup/components/cdc/v6.0.0/cdc cli changefeed create --pd=http://xxx.xx.3.113:2379 --sink-uri=kafka://xxx.xx.8.24:15502,xxx.xx.8.25:15502,xxx.xx.8.26:15502/t_pay_detail?kafka-version=3.1.0&max-message-bytes=10485760&replication-factor=3 --changefeed-id=t-pay-detail --config /home/tidb/cdc-conf/t_pay_detail.toml
Error: [CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha
Usage:
cdc cli changefeed create [flags]

Flags:
-c, --changefeed-id string Replication task (changefeed) ID
–config string Path of the configuration file
–cyclic-filter-replica-ids uints (Experimental) Cyclic replication filter replica ID of changefeed (default [])
–cyclic-replica-id uint (Experimental) Cyclic replication replica ID of changefeed
–cyclic-sync-ddl (Experimental) Cyclic replication sync DDL of changefeed (default true)
–disable-gc-check Disable GC safe point check
-h, --help help for create
–no-confirm Don’t ask user whether to ignore ineligible table
–opts key=value Extra options, in the key=value format
–sink-uri string sink uri
–sort-engine string sort engine used for data sort (default “unified”)
–start-ts uint Start ts of changefeed
–sync-interval duration (Experimental) Set the interval for syncpoint in replication(default 10min) (default 10m0s)
–sync-point (Experimental) Set and Record syncpoint in replication(default off)
–target-ts uint Target ts of changefeed
–tz string timezone used when checking sink uri (changefeed timezone is determined by cdc server) (default “SYSTEM”)

Global Flags:
–ca string CA certificate path for TLS connection
–cert string Certificate path for TLS connection
-i, --interact Run cdc cli with readline
–key string Private key path for TLS connection
–log-level string log level (etc: debug|info|warn|error) (default “warn”)
–pd string PD address, use ‘,’ to separate multiple PDs (default “http://127.0.0.1:2379”)

[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha

未添加–pd参数报错
[tidb@ptidbcalcmanage01 ~]$ tiup cdc cli unsafe reset
tiup is checking updates for component cdc …
A new version of cdc is available:
The latest version: v6.1.0
Local installed version: v6.0.0
Update current component: tiup update cdc
Update all components: tiup update --all

Starting component cdc: /home/tidb/.tiup/components/cdc/v6.0.0/cdc cli unsafe reset
Confirm that you know what this command will do and use it at your own risk [Y/N]
Y

Error: fail to open PD client, please check pd address “http://127.0.0.1:2379”: context deadline exceeded
Usage:
cdc cli unsafe reset [flags]

Flags:
-h, --help help for reset

Global Flags:
–ca string CA certificate path for TLS connection
–cert string Certificate path for TLS connection
-i, --interact Run cdc cli with readline
–key string Private key path for TLS connection
–log-level string log level (etc: debug|info|warn|error) (default “warn”)
–no-confirm Don’t ask user whether to confirm executing meta command
–pd string PD address, use ‘,’ to separate multiple PDs (default “http://127.0.0.1:2379”)

fail to open PD client, please check pd address “http://127.0.0.1:2379”: context deadline exceeded

添加–pd参数报错:CDC:ErrVersionIncompatible
[tidb@ptidbcalcmanage01 ~]$ tiup cdc cli unsafe reset --pd=http://xxx.xx.3.115:2379
tiup is checking updates for component cdc …
A new version of cdc is available:
The latest version: v6.1.0
Local installed version: v6.0.0
Update current component: tiup update cdc
Update all components: tiup update --all

Starting component cdc: /home/tidb/.tiup/components/cdc/v6.0.0/cdc cli unsafe reset --pd=http://xxx.xx.3.115:2379
Confirm that you know what this command will do and use it at your own risk [Y/N]
Y
Error: [CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha
Usage:
cdc cli unsafe reset [flags]

Flags:
-h, --help help for reset

Global Flags:
–ca string CA certificate path for TLS connection
–cert string Certificate path for TLS connection
-i, --interact Run cdc cli with readline
–key string Private key path for TLS connection
–log-level string log level (etc: debug|info|warn|error) (default “warn”)
–no-confirm Don’t ask user whether to confirm executing meta command
–pd string PD address, use ‘,’ to separate multiple PDs (default “http://127.0.0.1:2379”)

[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha

6

缩容后清理也是同样问题。
tiup cluster scale-in calctidb-cluster --node xxx.xx.3.116:8300
tiup cluster scale-in calctidb-cluster --node xxx.xx.3.117:8300

[tidb@ptidbcalcmanage01 ~]$ tiup cdc cli unsafe reset --pd=http://xxx.xx.3.115:2379
tiup is checking updates for component cdc …
A new version of cdc is available:
The latest version: v6.1.0
Local installed version: v6.0.0
Update current component: tiup update cdc
Update all components: tiup update --all

Starting component cdc: /home/tidb/.tiup/components/cdc/v6.0.0/cdc cli unsafe reset --pd=http://xxx.xx.3.115:2379
Confirm that you know what this command will do and use it at your own risk [Y/N]
Y
Error: [CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha
Usage:
cdc cli unsafe reset [flags]

Flags:
-h, --help help for reset

Global Flags:
–ca string CA certificate path for TLS connection
–cert string Certificate path for TLS connection
-i, --interact Run cdc cli with readline
–key string Private key path for TLS connection
–log-level string log level (etc: debug|info|warn|error) (default “warn”)
–no-confirm Don’t ask user whether to confirm executing meta command
–pd string PD address, use ‘,’ to separate multiple PDs (default “http://127.0.0.1:2379”)

[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.0 is not supported, the minimal compatible version is 5.1.0-alpha

7

通过TIDB官方给的信息确认遇到BUG，回复：我内部复现了你们这个问题，原因是跟你们集群从 v5.0.0 升级到 v6.0.0 版本有关。v6.0.0 的 pd 存在 BUG ，解决方案是将集群升级到 v6.1.0 版本，我这边验证升级之后可以解决
确认问题：
在pd中存在5.0.0的版本信息：
curl http://xxx.xx.3.115:2379/pd/api/v1/stores?state=0&state=1&state=2
{
“count”: 36,
“stores”: [
{
“store”: {
“id”: 10,
“address”: “xxx.xx.3.76:20172”,
“state”: 2,
“labels”: [
{
“key”: “host”,
“value”: “xxx.xx.3.76” --------------->以前主机IP
}
],
"version": “5.0.0”,
** “status_address”: “xxx.xx.3.76:20182”,**
“git_hash”: “7706b9634bd901c9fe8dbe6a556025abbfd0793d”,
“start_timestamp”: 1625815970,
“deploy_path”: “/tidb/tikv2/bin”,
“last_heartbeat”: 1631172405972679111,
“node_state”: 3,
“state_name”: “Tombstone”
},
升级集群版本后，查询还有此信息，使用如下命令清理，在6.0.0版本清理不了。
tiup ctl:v6.0.0 pd -u xxx.xx.3.115:2379 store remove-tombstone
再次验证后，没有5.0.0信息，再次创建cdc任务可以正常创建。
希望此贴对大家又有帮助。

1 个赞
8

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。