tidb从v6.1.2升级到6.5.0 tidb节点提示端口4000失败["unexpected error"] [error="TiKV version must greater than or equal to 6.2.0-alpha"]

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】V6.1.2
【复现路径】 tiup cluster upgrade tidb-test v6.5.0
之后有尝试离线升级,提示成功了,但是tidb节点再也起不来了
【遇到的问题:问题现象及影响】tikv pd可以正常升级,tidb在升级时候提示 Error: failed to restart: 10.125.76.252 tidb-4000.service, please check the instance’s log(/data/tidb-deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s
【资源配置】
【附件:截图/日志/监控】

2023-02-09T19:10:16.738+0800	DEBUG	TaskFinish	{"task": "StartCluster", "error": "failed to start tidb: failed to start: 10.125.76.252 tidb-4000.service, please check the instance's log(/data/tidb-deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s", "errorVerbose": "timed out waiting for port 4000 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(*WaitFor).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ngithub.com/pingcap/tiup/pkg/cluster/spec.PortStarted\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:116\ngithub.com/pingcap/tiup/pkg/cluster/spec.(*BaseInstance).Ready\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:148\ngithub.com/pingcap/tiup/pkg/cluster/operation.startInstance\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:373\ngithub.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:502\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1581\nfailed to start: 10.125.76.252 tidb-4000.service, please check the instance's log(/data/tidb-deploy/tidb-4000/log) for more detail.\nfailed to start tidb"}

4000端口未被占用

缩容也报错

[2023/02/09 20:05:33.521 +08:00] [FATAL] [terror.go:300] ["unexpected error"] [error="TiKV version must greater than or equal to 6.2.0-alpha"] [stack="github.com/pingcap/tidb/parser/terror.MustNil\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:300\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:312\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:214\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] [stack="github.com/pingcap/tidb/parser/terror.MustNil\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:300\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:312\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:214\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]

tikv的错误日志

看下tikv版本、
升级文档 :
https://docs.pingcap.com/zh/tidb/stable/upgrade-tidb-using-tiup

参考下这个处理”在启动TIDB的时候 4000端口无法起来

非端口备用,非防火墙问题

asktug 搜索下相关案例 尝试下自己找下答案

没有类似案例

看tidb报错日志是tikv没有升级成功啊。

感觉是的 推荐看看 tikv 版本先

看看tikv吧

建议描述一下过程
1、升级评估如何?
2、升级的前置检查tiup cluster check --cluster结果如何?

tiup cluster display tidb-test 看下集群状态。脱敏


–force参数 强制 性的干。 看到你这是测试环境

tikv版本已经是6.5.0,tikv报错信息

image
这个日志 贴出来核心的看看

scale-out 增加一个tidb 。能成功不 ?

一样的报错端口4000的问题,哪怕是一个没有装过tidb 4000不可能被占用的机器都不行

2023-02-10T13:41:05.548+0800 DEBUG retry error {error: operation timed out after 2m0s}
2023-02-10T13:41:05.549+0800 DEBUG TaskFinish {task: Start new instances, error: failed to start tidb: failed to start: 10.125.72.101 tidb-4000.service, please check the instance’s log(/data/tidb-deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s, errorVerbose: timed out waiting for port 4000 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(*WaitFor).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ngithub.com/pingcap/tiup/pkg/cluster/spec.PortStarted\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:116\ngithub.com/pingcap/tiup/pkg/cluster/spec.(*BaseInstance).Ready\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:148\ngithub.com/pingcap/tiup/pkg/cluster/operation.startInstance\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:373\ngithub.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:502\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1581\nfailed to start: 10.125.72.101 tidb-4000.service, please check the instance’s log(/data/tidb-deploy/tidb-4000/log) for more detail.\nfailed to start tidb}
2023-02-10T13:41:05.549+0800 INFO Execute command finished {code: 1, error: failed to start tidb: failed to start: 10.125.72.101 tidb-4000.service, please check the instance’s log(/data/tidb-deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s, errorVerbose: timed out waiting for port 4000 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(*WaitFor).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ngithub.com/pingcap/tiup/pkg/cluster/spec.PortStarted\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:116\ngithub.com/pingcap/tiup/pkg/cluster/spec.(*BaseInstance).Ready\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:148\ngithub.com/pingcap/tiup/pkg/cluster/operation.startInstance\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:373\ngithub.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:502\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1581\nfailed to start: 10.125.72.101 tidb-4000.service, please check the instance’s log(/data/tidb-deploy/tidb-4000/log) for more detail.\nfailed to start tidb}

版本已经升级成6.5.0了