tidb6.3 升级报错

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】
【遇到的问题】
原先部署的tidb 为1pd 1kv 1tidb 我先升级为1pd 3kv 1tidb 然后升级tidb6.2 到6.3 结果就报错了
【复现路径】做过哪些操作出现的问题
【问题现象及影响】
2022-10-13T16:53:40.879+0800 ERROR CheckPoint {“instance”: “10.8.35.62:20160”, “error”: “failed to evict store leader 10.8.35.62: metric tikv_raftstore_region_count{type=“leader”} not found”, “errorVerbose”: “metric tikv_raftstore_region_count{type=“leader”} not found\ngithub.com/pingcap/tiup/pkg/cluster/spec.genLeaderCounter.func1\ \tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:456\ github.com/pingcap/tiup/pkg/cluster/api.(*PDClient).EvictStoreLeader\ \tgithub.com/pingcap/tiup/pkg/cluster/api/pdapi.go:474\ github.com/pingcap/tiup/pkg/cluster/spec.(*TiKVInstance).PreRestart\ \tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:360\ github.com/pingcap/tiup/pkg/cluster/operation.upgradeInstance\ \tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:215\ github.com/pingcap/tiup/pkg/cluster/operation.Upgrade\ \tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:169\ github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade.func1\ \tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:210\ github.com/pingcap/tiup/pkg/cluster/task.(*Func).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/task/func.go:34\ github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:86\ github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade\ \tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:219\ github.com/pingcap/tiup/components/cluster/command.newUpgradeCmd.func1\ \tgithub.com/pingcap/tiup/components/cluster/command/upgrade.go:41\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.5.0/command.go:872\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.5.0/command.go:990\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.5.0/command.go:918\ github.com/pingcap/tiup/components/cluster/command.newReplayCmd.func1\ \tgithub.com/pingcap/tiup/components/cluster/command/replay.go:59\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.5.0/command.go:872\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.5.0/command.go:990\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.5.0/command.go:918\ github.com/pingcap/tiup/components/cluster/command.Execute\ \tgithub.com/pingcap/tiup/components/cluster/command/root.go:297\ main.main\ \tgithub.com/pingcap/tiup/components/cluster/main.go:23\ runtime.main\ \truntime/proc.go:250\ runtime.goexit\ \truntime/asm_amd64.s:1594\ failed to evict store leader 10.8.35.62”, “hash”: “1a4714d7146fa85240a1ff4ef7451df719e0b4f0”, “func”: “github.com/pingcap/tiup/pkg/cluster/operation.upgradeInstance”, “hit”: false}
2022-10-13T16:53:40.880+0800 DEBUG setting replication config: leader-schedule-limit=4
2022-10-13T16:53:40.880+0800 ERROR failed counting leader {“host”: “10.8.35.62:20160”, “status addr”: “http://10.8.35.62:20180/metrics”, “error”: “GET request for URL “http://10.8.35.62:20180/metrics” returned HTTP status 503 Service Unavailable”}
2022-10-13T16:53:40.881+0800 DEBUG setting replication config: region-schedule-limit=2048
2022-10-13T16:53:40.882+0800 DEBUG TaskFinish {“task”: “UpgradeCluster”, “error”: “failed to evict store leader 10.8.35.62: metric tikv_raftstore_region_count{type=“leader”} not found”, “errorVerbose”: “metric tikv_raftstore_region_count{type=“leader”} not found\ngithub.com/pingcap/tiup/pkg/cluster/spec.genLeaderCounter.func1\ \tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:456\ github.com/pingcap/tiup/pkg/cluster/api.(*PDClient).EvictStoreLeader\ \tgithub.com/pingcap/tiup/pkg/cluster/api/pdapi.go:474\ github.com/pingcap/tiup/pkg/cluster/spec.(*TiKVInstance).PreRestart\ \tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:360\ github.com/pingcap/tiup/pkg/cluster/operation.upgradeInstance\ \tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:215\ github.com/pingcap/tiup/pkg/cluster/operation.Upgrade\ \tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:169\ github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade.func1\ \tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:210\ github.com/pingcap/tiup/pkg/cluster/task.(*Func).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/task/func.go:34\ github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:86\ github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade\ \tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:219\ github.com/pingcap/tiup/components/cluster/command.newUpgradeCmd.func1\ \tgithub.com/pingcap/tiup/components/cluster/command/upgrade.go:41\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.5.0/command.go:872\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.5.0/command.go:990\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.5.0/command.go:918\ github.com/pingcap/tiup/components/cluster/command.newReplayCmd.func1\ \tgithub.com/pingcap/tiup/components/cluster/command/replay.go:59\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.5.0/command.go:872\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.5.0/command.go:990\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.5.0/command.go:918\ github.com/pingcap/tiup/components/cluster/command.Execute\ \tgithub.com/pingcap/tiup/components/cluster/command/root.go:297\ main.main\ \tgithub.com/pingcap/tiup/components/cluster/main.go:23\ runtime.main\ \truntime/proc.go:250\ runtime.goexit\ \truntime/asm_amd64.s:1594\ failed to evict store leader 10.8.35.62”}
2022-10-13T16:53:40.882+0800 INFO Execute command finished {“code”: 1, “error”: “failed to evict store leader 10.8.35.62: metric tikv_raftstore_region_count{type=“leader”} not found”, “errorVerbose”: “metric tikv_raftstore_region_count{type=“leader”} not found\ngithub.com/pingcap/tiup/pkg/cluster/spec.genLeaderCounter.func1\ \tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:456\ github.com/pingcap/tiup/pkg/cluster/api.(*PDClient).EvictStoreLeader\ \tgithub.com/pingcap/tiup/pkg/cluster/api/pdapi.go:474\ github.com/pingcap/tiup/pkg/cluster/spec.(*TiKVInstance).PreRestart\ \tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:360\ github.com/pingcap/tiup/pkg/cluster/operation.upgradeInstance\ \tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:215\ github.com/pingcap/tiup/pkg/cluster/operation.Upgrade\ \tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:169\ github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade.func1\ \tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:210\ github.com/pingcap/tiup/pkg/cluster/task.(*Func).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/task/func.go:34\ github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:86\ github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade\ \tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:219\ github.com/pingcap/tiup/components/cluster/command.newUpgradeCmd.func1\ \tgithub.com/pingcap/tiup/components/cluster/command/upgrade.go:41\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.5.0/command.go:872\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.5.0/command.go:990\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.5.0/command.go:918\ github.com/pingcap/tiup/components/cluster/command.newReplayCmd.func1\ \tgithub.com/pingcap/tiup/components/cluster/command/replay.go:59\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.5.0/command.go:872\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.5.0/command.go:990\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.5.0/command.go:918\ github.com/pingcap/tiup/components/cluster/command.Execute\ \tgithub.com/pingcap/tiup/components/cluster/command/root.go:297\ main.main\ \tgithub.com/pingcap/tiup/components/cluster/main.go:23\ runtime.main\ \truntime/proc.go:250\ runtime.goexit\ \truntime/asm_amd64.s:1594\ failed to evict store leader 10.8.35.62”}
【附件】

请提供各个组件的 version 信息,如 cdc/tikv,可通过执行 cdc version/tikv-server --version 获取。

“instance”: “10.8.35.62:20160”,
failed to evict store leader 10.8.35.62: metric tikv_raftstore_region_count{type=“leader”} not found

数据的状态不太对劲,日志描述得很清楚了

应该是有tikv已经down了,可以offline升级

都不是这个原因 是升级前单tikv副本改成了3 得同步完成才能升级