TiDB 从 5.0.1 升级到 5.2.0 时 CDC 报错导致集群升级失败

🪐 TiDB 技术问题
1

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：
【 TiDB 使用环境】测试环境
【概述】tidb版本从5.0.1升级到5.2.0时，cdc报错，导致集群升级失败
【背景】tidb集群升级
【现象】cdc提示不能从5.0.1升级，集群升级失败
【业务影响】
【TiDB 版本】5.0.1
【附件】

  1. TiUP Cluster Display 信息

  2. TiUP Cluster Edit Config 信息

  3. TiDB- Overview 监控

  • 对应模块日志（包含问题前后1小时日志）
    报错日志如下：
    $ tiup cluster upgrade test-cluster v5.2.0 --wait-timeout 300
    Starting component cluster: /home/tidb/.tiup/components/cluster/v1.5.5/tiup-cluster upgrade test-cluster v5.2.0 --wait-timeout 300
    This operation will upgrade tidb v5.0.1 cluster test-cluster to v5.2.0.
    Do you want to continue? [y/N]:(default=N) y
    Upgrading cluster…
  • [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/ssh/id_rsa.pub
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x90
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x88
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x88
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x89
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x90
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x88
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x80
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x90
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x89
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x89
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x89
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x90
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x90
  • [Parallel] - UserSSH: user=tidb, host=xxx.xxx.xx.x90
  • [ Serial ] - Download: component=alertmanager, version=, os=linux, arch=amd64
  • [ Serial ] - Download: component=cdc, version=v5.2.0, os=linux, arch=amd64
  • [ Serial ] - Download: component=tiflash, version=v5.2.0, os=linux, arch=amd64
  • [ Serial ] - Download: component=tikv, version=v5.2.0, os=linux, arch=amd64
  • [ Serial ] - Download: component=prometheus, version=v5.2.0, os=linux, arch=amd64
  • [ Serial ] - Download: component=pd, version=v5.2.0, os=linux, arch=amd64
  • [ Serial ] - Download: component=tidb, version=v5.2.0, os=linux, arch=amd64
  • [ Serial ] - Download: component=grafana, version=v5.2.0, os=linux, arch=amd64
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x90, directories=’/data/tidb/deploy/data.alertmanager’
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x89, directories=’/data/tidb/deploy/data’
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x89, directories=’’
  • [ Serial ] - BackupComponent: component=tidb, currentVersion=v5.0.1, remote=xxx.xxx.xx.x89:/data/tidb/deploy
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x90, directories=’/data/tidb/deploy/prometheus2.0.0.data.metrics’
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x89, directories=’/data/tidb/deploy/cdc-8300/data’
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x90, directories=’’
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x88, directories=’/data/tidb/deploy/data’
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x90, directories=’’
  • [ Serial ] - BackupComponent: component=tidb, currentVersion=v5.0.1, remote=xxx.xxx.xx.x90:/data/tidb/deploy
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x89, directories=’/data/tidb/deploy/data.pd’
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x88, directories=’/data/tidb/deploy/data.pd’
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x88, directories=’’
  • [ Serial ] - BackupComponent: component=tidb, currentVersion=v5.0.1, remote=xxx.xxx.xx.x88:/data/tidb/deploy
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x80, directories=’/data/tidb/data/tiflash-9000’
  • [ Serial ] - CopyComponent: component=grafana, version=v5.2.0, remote=xxx.xxx.xx.x90:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x90, directories=’/data/tidb/deploy/data’
  • [ Serial ] - Mkdir: host=xxx.xxx.xx.x90, directories=’/data/tidb/deploy/data.pd’
  • [ Serial ] - CopyComponent: component=tidb, version=v5.2.0, remote=xxx.xxx.xx.x88:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=tidb, version=v5.2.0, remote=xxx.xxx.xx.x89:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=tidb, version=v5.2.0, remote=xxx.xxx.xx.x90:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - BackupComponent: component=tikv, currentVersion=v5.0.1, remote=xxx.xxx.xx.x88:/data/tidb/deploy
  • [ Serial ] - BackupComponent: component=pd, currentVersion=v5.0.1, remote=xxx.xxx.xx.x88:/data/tidb/deploy
  • [ Serial ] - BackupComponent: component=tikv, currentVersion=v5.0.1, remote=xxx.xxx.xx.x89:/data/tidb/deploy
  • [ Serial ] - CopyComponent: component=prometheus, version=v5.2.0, remote=xxx.xxx.xx.x90:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - BackupComponent: component=tiflash, currentVersion=v5.0.1, remote=xxx.xxx.xx.x80:/data/tidb/deploy/tiflash-9000
  • [ Serial ] - BackupComponent: component=pd, currentVersion=v5.0.1, remote=xxx.xxx.xx.x90:/data/tidb/deploy
  • [ Serial ] - BackupComponent: component=pd, currentVersion=v5.0.1, remote=xxx.xxx.xx.x89:/data/tidb/deploy
  • [ Serial ] - BackupComponent: component=tikv, currentVersion=v5.0.1, remote=xxx.xxx.xx.x90:/data/tidb/deploy
  • [ Serial ] - CopyComponent: component=tikv, version=v5.2.0, remote=xxx.xxx.xx.x89:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=pd, version=v5.2.0, remote=xxx.xxx.xx.x90:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=tikv, version=v5.2.0, remote=xxx.xxx.xx.x88:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=pd, version=v5.2.0, remote=xxx.xxx.xx.x88:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - BackupComponent: component=cdc, currentVersion=v5.0.1, remote=xxx.xxx.xx.x89:/data/tidb/deploy/cdc-8300
  • [ Serial ] - CopyComponent: component=tiflash, version=v5.2.0, remote=xxx.xxx.xx.x80:/data/tidb/deploy/tiflash-9000 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=pd, version=v5.2.0, remote=xxx.xxx.xx.x89:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=alertmanager, version=, remote=xxx.xxx.xx.x90:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=cdc, version=v5.2.0, remote=xxx.xxx.xx.x89:/data/tidb/deploy/cdc-8300 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=tikv, version=v5.2.0, remote=xxx.xxx.xx.x90:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x89, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/pd-2379.service, deploy_dir=/data/tidb/deploy, data_dir=[/data/tidb/deploy/data.pd], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - BackupComponent: component=prometheus, currentVersion=v5.0.1, remote=xxx.xxx.xx.x90:/data/tidb/deploy
  • [ Serial ] - BackupComponent: component=alertmanager, currentVersion=v5.0.1, remote=xxx.xxx.xx.x90:/data/tidb/deploy
  • [ Serial ] - CopyComponent: component=prometheus, version=v5.2.0, remote=xxx.xxx.xx.x90:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=alertmanager, version=, remote=xxx.xxx.xx.x90:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x88, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/pd-2379.service, deploy_dir=/data/tidb/deploy, data_dir=[/data/tidb/deploy/data.pd], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x89, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/cdc-8300.service, deploy_dir=/data/tidb/deploy/cdc-8300, data_dir=[/data/tidb/deploy/cdc-8300/data], log_dir=/data/tidb/deploy/cdc-8300/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x90, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/alertmanager-9093.service, deploy_dir=/data/tidb/deploy, data_dir=[/data/tidb/deploy/data.alertmanager], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x90, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/prometheus-9090.service, deploy_dir=/data/tidb/deploy, data_dir=[/data/tidb/deploy/prometheus2.0.0.data.metrics], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x88, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/tikv-20160.service, deploy_dir=/data/tidb/deploy, data_dir=[/data/tidb/deploy/data], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x89, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/tikv-20160.service, deploy_dir=/data/tidb/deploy, data_dir=[/data/tidb/deploy/data], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - BackupComponent: component=grafana, currentVersion=v5.0.1, remote=xxx.xxx.xx.x90:/data/tidb/deploy
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x90, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/tidb-4000.service, deploy_dir=/data/tidb/deploy, data_dir=[], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - CopyComponent: component=grafana, version=v5.2.0, remote=xxx.xxx.xx.x90:/data/tidb/deploy os=linux, arch=amd64
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x89, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/tidb-4000.service, deploy_dir=/data/tidb/deploy, data_dir=[], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x90, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/pd-2379.service, deploy_dir=/data/tidb/deploy, data_dir=[/data/tidb/deploy/data.pd], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x88, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/tidb-4000.service, deploy_dir=/data/tidb/deploy, data_dir=[], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x90, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/grafana-3000.service, deploy_dir=/data/tidb/deploy, data_dir=[], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x90, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/tikv-20160.service, deploy_dir=/data/tidb/deploy, data_dir=[/data/tidb/deploy/data], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xx.x80, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/tiflash-9000.service, deploy_dir=/data/tidb/deploy/tiflash-9000, data_dir=[/data/tidb/data/tiflash-9000], log_dir=/data/tidb/deploy/tiflash-9000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
  • [ Serial ] - UpgradeCluster
    Upgrading component tiflash
    Restarting instance xxx.xxx.xx.x80:9000
    Restart instance xxx.xxx.xx.x80:9000 success
    Upgrading component cdc
    Restarting instance xxx.xxx.xx.x89:8300

Error: failed to restart: xxx.xxx.xx.x89 cdc-8300.service, please check the instance’s log(/data/tidb/deploy/cdc-8300/log) for more detail.: timed out waiting for port 8300 to be started after 5m0s

Verbose debug logs has been written to /home/tidb/.tiup/logs/tiup-cluster-debug-2021-08-31-11-26-00.log.
Error: run /home/tidb/.tiup/components/cluster/v1.5.5/tiup-cluster (wd:/home/tidb/.tiup/data/ShdELY2) failed: exit status 1

cdc日志：
[2021/08/31 11:23:01.424 +08:00] [INFO] [helper.go:49] [“init log”] [file=/data/tidb/deploy/cdc-8300/log/cdc.log] [level=info]
[2021/08/31 11:23:01.425 +08:00] [INFO] [version.go:47] [“Welcome to Change Data Capture (CDC)”] [release-version=v5.2.0] [git-hash=81c22b1c1b2041e2806160d8c7e1105a70815ff5] [git-branch=heads/refs/tags/v5.2.0] [utc-build-time=“2021-08-26 08:37:01”] [go-version=“go version go1.16.4 linux/amd64”] [failpoint-build=false]
[2021/08/31 11:23:01.425 +08:00] [INFO] [server.go:67] [“creating CDC server”] [pd-addrs="[http://xxx.xxx.xx.x88:2379,http://xxx.xxx.xx.x89:2379,http://xxx.xxx.xx.x90:2379]"] [config="{“addr”:“0.0.0.0:8300”,“advertise-addr”:“xxx.xxx.xx.x89:8300”,“log-file”:"/data/tidb/deploy/cdc-8300/log/cdc.log",“log-level”:“info”,“log”:{“file”:{“max-size”:300,“max-days”:0,“max-backups”:0}},“data-dir”:"/data/tidb/deploy/cdc-8300/data",“gc-ttl”:86400,“tz”:“System”,“capture-session-ttl”:10,“owner-flush-interval”:200000000,“processor-flush-interval”:100000000,“sorter”:{“num-concurrent-worker”:4,“chunk-size-limit”:134217728,“max-memory-percentage”:30,“max-memory-consumption”:17179869184,“num-workerpool-goroutine”:16,“sort-dir”:"/tmp/sorter"},“security”:{“ca-path”:"",“cert-path”:"",“key-path”:"",“cert-allowed-cn”:null},“per-table-memory-quota”:20971520,“kv-client”:{“worker-concurrent”:8,“worker-pool-size”:0,“region-scan-limit”:40}}"]
[2021/08/31 11:23:01.425 +08:00] [INFO] [client.go:214] ["[pd] create pd client with endpoints"] [pd-address="[http://xxx.xxx.xx.x88:2379,http://xxx.xxx.xx.x89:2379,http://xxx.xxx.xx.x90:2379]"]
[2021/08/31 11:23:01.429 +08:00] [INFO] [base_client.go:346] ["[pd] switch leader"] [new-leader=http://xxx.xxx.xx.x88:2379] [old-leader=]
[2021/08/31 11:23:01.429 +08:00] [INFO] [base_client.go:126] ["[pd] init cluster id"] [cluster-id=6793561162943471910]
[2021/08/31 11:23:01.429 +08:00] [INFO] [client.go:238] ["[pd] create tso dispatcher"] [dc-location=global]
[2021/08/31 11:23:01.431 +08:00] [INFO] [server.go:273] ["/data/tidb/deploy/cdc-8300/data is set as data-dir (128GB available), sort-dir=/data/tidb/deploy/cdc-8300/data/tmp/sorter. It is recommended that the disk for data-dir at least have 500GB available space"]
[2021/08/31 11:23:01.431 +08:00] [WARN] [check.go:83] [“check TiKV version failed”] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.1 is not supported, the minimal compatible version is 5.1.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.1 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\ \tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ github.com/pingcap/errors.(*Error).GenWithStackByArgs\ \tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ github.com/pingcap/ticdc/pkg/version.CheckStoreVersion\ \tgithub.com/pingcap/ticdc/pkg/version/check.go:166\ github.com/pingcap/ticdc/pkg/version.CheckClusterVersion\ \tgithub.com/pingcap/ticdc/pkg/version/check.go:78\ github.com/pingcap/ticdc/cdc.(*Server).Run\ \tgithub.com/pingcap/ticdc/cdc/server.go:145\ github.com/pingcap/ticdc/pkg/cmd/server.(*options).run\ \tgithub.com/pingcap/ticdc/pkg/cmd/server/server.go:127\ github.com/pingcap/ticdc/pkg/cmd/server.NewCmdServer.func1\ \tgithub.com/pingcap/ticdc/pkg/cmd/server/server.go:273\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:842\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.0.0/command.go:950\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:887\ github.com/pingcap/ticdc/cmd.Execute\ \tgithub.com/pingcap/ticdc/cmd/root.go:38\ main.main\ \tcommand-line-arguments/main.go:22\ runtime.main\ \truntime/proc.go:225\ runtime.goexit\ \truntime/asm_amd64.s:1371"]
[2021/08/31 11:23:01.434 +08:00] [WARN] [check.go:83] [“check TiKV version failed”] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.1 is not supported, the minimal compatible version is 5.1.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.1 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\ \tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ github.com/pingcap/errors.(*Error).GenWithStackByArgs\ \tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ github.com/pingcap/ticdc/pkg/version.CheckStoreVersion\ \tgithub.com/pingcap/ticdc/pkg/version/check.go:166\ github.com/pingcap/ticdc/pkg/version.CheckClusterVersion\ \tgithub.com/pingcap/ticdc/pkg/version/check.go:78\ github.com/pingcap/ticdc/cdc.(*Server).Run\ \tgithub.com/pingcap/ticdc/cdc/server.go:145\ github.com/pingcap/ticdc/pkg/cmd/server.(*options).run\ \tgithub.com/pingcap/ticdc/pkg/cmd/server/server.go:127\ github.com/pingcap/ticdc/pkg/cmd/server.NewCmdServer.func1\ \tgithub.com/pingcap/ticdc/pkg/cmd/server/server.go:273\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:842\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.0.0/command.go:950\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:887\ github.com/pingcap/ticdc/cmd.Execute\ \tgithub.com/pingcap/ticdc/cmd/root.go:38\ main.main\ \tcommand-line-arguments/main.go:22\ runtime.main\ \truntime/proc.go:225\ runtime.goexit\ \truntime/asm_amd64.s:1371"]
[2021/08/31 11:23:01.436 +08:00] [WARN] [check.go:83] [“check TiKV version failed”] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.1 is not supported, the minimal compatible version is 5.1.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 5.0.1 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\ \tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ github.com/pingcap/errors.(*Error).GenWithStackByArgs\ \tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ github.com/pingcap/ticdc/pkg/version.CheckStoreVersion\ \tgithub.com/pingcap/ticdc/pkg/version/check.go:166\ github.com/pingcap/ticdc/pkg/version.CheckClusterVersion\ \tgithub.com/pingcap/ticdc/pkg/version/check.go:78\ github.com/pingcap/ticdc/cdc.(*Server).Run\ \tgithub.com/pingcap/ticdc/cdc/server.go:145\ github.com/pingcap/ticdc/pkg/cmd/server.(*options).run\ \tgithub.com/pingcap/ticdc/pkg/cmd/server/server.go:127\ github.com/pingcap/ticdc/pkg/cmd/server.NewCmdServer.func1\ \tgithub.com/pingcap/ticdc/pkg/cmd/server/server.go:273\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:842\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.0.0/command.go:950\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:887\ github.com/pingcap/ticdc/cmd.Execute\ \tgithub.com/pingcap/ticdc/cmd/root.go:38\ main.main\ \tcommand-line-arguments/main.go:22\ runtime.main\ \truntime/proc.go:225\ runtime.goexit\ \truntime/asm_amd64.s:1371"]
[2021/08/31 11:23:01.438 +08:00] [ERROR] [server.go:129] [“run server”] [error="[CDC:ErrVersionIncompatible]version is incompatible: PD 5.0.1 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\ \tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ github.com/pingcap/errors.(*Error).GenWithStackByArgs\ \tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ github.com/pingcap/ticdc/pkg/version.CheckClusterVersion\ \tgithub.com/pingcap/ticdc/pkg/version/check.go:131\ github.com/pingcap/ticdc/cdc.(*Server).Run\ \tgithub.com/pingcap/ticdc/cdc/server.go:145\ github.com/pingcap/ticdc/pkg/cmd/server.(*options).run\ \tgithub.com/pingcap/ticdc/pkg/cmd/server/server.go:127\ github.com/pingcap/ticdc/pkg/cmd/server.NewCmdServer.func1\ \tgithub.com/pingcap/ticdc/pkg/cmd/server/server.go:273\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:842\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.0.0/command.go:950\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:887\ github.com/pingcap/ticdc/cmd.Execute\ \tgithub.com/pingcap/ticdc/cmd/root.go:38\ main.main\ \tcommand-line-arguments/main.go:22\ runtime.main\ \truntime/proc.go:225\ runtime.goexit\ \truntime/asm_amd64.s:1371"]
[2021/08/31 11:23:01.438 +08:00] [INFO] [client.go:510] ["[pd] exit tso dispatcher"] [dc-location=global]

1 个赞
2

建议一下，如果不支持从某个版本A直接升级到版本B，是不是在升级前给出提示并终止升级操作比较好呢？像现在升级到一半的时候，提示不支持从5.0.1升级到5.2.0，不知道需要做哪些回退操作才能正常升级到5.2.0了。

3

HI, 非常抱歉，我们尽快分析一下遇到的问题。理论上是可以支持升级的，CDC 也是可以的。

从反馈日志看，目前升级 TiCDC 的版本是 v5.1.0 ，想确认一下，目前 TiDB 集群拓扑，可以发一下。通过 tiup display 操作检查一下。

1 个赞
4

$ tiup cluster display test-cluster
Starting component cluster: /home/tidb/.tiup/components/cluster/v1.5.5/tiup-cluster display test-cluster
Cluster type: tidb
Cluster name: test-cluster
Cluster version: v5.0.1
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://xxx.xxx.xx.x89:2379/dashboard
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

xxx.xxx.xx.x90:9093 alertmanager xxx.xxx.xx.x90 9093/9094 linux/x86_64 Up /data/tidb/deploy/data.alertmanager /data/tidb/deploy
xxx.xxx.xx.x89:8300 cdc xxx.xxx.xx.x89 8300 linux/x86_64 Down data /data/tidb/deploy/cdc-8300
xxx.xxx.xx.x90:3000 grafana xxx.xxx.xx.x90 3000 linux/x86_64 Up - /data/tidb/deploy
xxx.xxx.xx.x88:2379 pd xxx.xxx.xx.x88 2379/2380 linux/x86_64 Up|L /data/tidb/deploy/data.pd /data/tidb/deploy
xxx.xxx.xx.x89:2379 pd xxx.xxx.xx.x89 2379/2380 linux/x86_64 Up|UI /data/tidb/deploy/data.pd /data/tidb/deploy
xxx.xxx.xx.x90:2379 pd xxx.xxx.xx.x90 2379/2380 linux/x86_64 Up /data/tidb/deploy/data.pd /data/tidb/deploy
xxx.xxx.xx.x90:9090 prometheus xxx.xxx.xx.x90 9090 linux/x86_64 Up /data/tidb/deploy/prometheus2.0.0.data.metrics /data/tidb/deploy
xxx.xxx.xx.x88:4000 tidb xxx.xxx.xx.x88 4000/10080 linux/x86_64 Up - /data/tidb/deploy
xxx.xxx.xx.x89:4000 tidb xxx.xxx.xx.x89 4000/10080 linux/x86_64 Up - /data/tidb/deploy
xxx.xxx.xx.x90:4000 tidb xxx.xxx.xx.x90 4000/10080 linux/x86_64 Up - /data/tidb/deploy
xxx.xxx.xx.x80:9000 tiflash xxx.xxx.xx.x80 9000/8123/3930/20170/20292/8234 linux/x86_64 Up /data/tidb/data/tiflash-9000 /data/tidb/deploy/tiflash-9000
xxx.xxx.xx.x88:20160 tikv xxx.xxx.xx.x88 20160/20180 linux/x86_64 Up /data/tidb/deploy/data /data/tidb/deploy
xxx.xxx.xx.x89:20160 tikv xxx.xxx.xx.x89 20160/20180 linux/x86_64 Up /data/tidb/deploy/data /data/tidb/deploy
xxx.xxx.xx.x90:20160 tikv xxx.xxx.xx.x90 20160/20180 linux/x86_64 Up /data/tidb/deploy/data /data/tidb/deploy
Total nodes: 14

这个是执行了升级操作命令之后的display情况，ticdc升级前的版本是5.0.1，从报错日志看，升级完后，ticdc重启的时候，需要校验tikv的版本是否在5.1.0-alpha以上，否则中断退出，但这时tikv还没有重启，所以此刻的版本是5.0.1

5

非常抱歉，我们会尽快修复这个问题

6

Hi，可以关注一下这个 PR 进展，后续修复会通过 patch 方式。目前如果升级 TiCDC 失败，一个先不做升级。忽略 TiCDC 的升级。

7

你好，我想了解一下：
1、如果我现在选择保持现状不变，等后续patch补丁可以继续成功升级到5.2吗？（目前集群除了ticdc不正常其他都是正常的，对于测试环境短期内ticdc不正常可以容忍）
2、我现在选择忽略ticdc的升级，把其他组件升级到5.2，需要如何操作？后续ticdc可以单独升级吗？

8

速度挺快的，tiup在1.5.6版本已经修复了这个问题，tidb版本已经顺利升级到5.2.0，感谢！

1 个赞
9

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。