【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】
【遇到的问题】
原先部署的tidb 为1pd 1kv 1tidb 我先升级为1pd 3kv 1tidb 然后升级tidb6.2 到6.3 结果就报错了
【复现路径】做过哪些操作出现的问题
【问题现象及影响】
2022-10-13T16:53:40.879+0800 ERROR CheckPoint {“instance”: “10.8.35.62:20160”, “error”: “failed to evict store leader 10.8.35.62: metric tikv_raftstore_region_count{type="leader"} not found”, “errorVerbose”: “metric tikv_raftstore_region_count{type="leader"} not found
github.com/pingcap/tiup/pkg/cluster/spec.genLeaderCounter.func1
\tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:456
github.com/pingcap/tiup/pkg/cluster/api.(*PDClient).EvictStoreLeader
\tgithub.com/pingcap/tiup/pkg/cluster/api/pdapi.go:474
github.com/pingcap/tiup/pkg/cluster/spec.(*TiKVInstance).PreRestart
\tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:360
github.com/pingcap/tiup/pkg/cluster/operation.upgradeInstance
\tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:215
github.com/pingcap/tiup/pkg/cluster/operation.Upgrade
\tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:169
github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade.func1
\tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:210
github.com/pingcap/tiup/pkg/cluster/task.(*Func).Execute
\tgithub.com/pingcap/tiup/pkg/cluster/task/func.go:34
github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute
\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:86
github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade
\tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:219
github.com/pingcap/tiup/components/cluster/command.newUpgradeCmd.func1
\tgithub.com/pingcap/tiup/components/cluster/command/upgrade.go:41
github.com/spf13/cobra.(*Command).execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
\tgithub.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:918
github.com/pingcap/tiup/components/cluster/command.newReplayCmd.func1
\tgithub.com/pingcap/tiup/components/cluster/command/replay.go:59
github.com/spf13/cobra.(*Command).execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
\tgithub.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:918
github.com/pingcap/tiup/components/cluster/command.Execute
\tgithub.com/pingcap/tiup/components/cluster/command/root.go:297
main.main
\tgithub.com/pingcap/tiup/components/cluster/main.go:23
runtime.main
\truntime/proc.go:250
runtime.goexit
\truntime/asm_amd64.s:1594
failed to evict store leader 10.8.35.62”, “hash”: “1a4714d7146fa85240a1ff4ef7451df719e0b4f0”, “func”: “github.com/pingcap/tiup/pkg/cluster/operation.upgradeInstance”, “hit”: false}
2022-10-13T16:53:40.880+0800 DEBUG setting replication config: leader-schedule-limit=4
2022-10-13T16:53:40.880+0800 ERROR failed counting leader {“host”: “10.8.35.62:20160”, “status addr”: “http://10.8.35.62:20180/metrics”, “error”: “GET request for URL "http://10.8.35.62:20180/metrics\” returned HTTP status 503 Service Unavailable"}
2022-10-13T16:53:40.881+0800 DEBUG setting replication config: region-schedule-limit=2048
2022-10-13T16:53:40.882+0800 DEBUG TaskFinish {“task”: “UpgradeCluster”, “error”: “failed to evict store leader 10.8.35.62: metric tikv_raftstore_region_count{type="leader"} not found”, “errorVerbose”: “metric tikv_raftstore_region_count{type="leader"} not found
github.com/pingcap/tiup/pkg/cluster/spec.genLeaderCounter.func1
\tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:456
github.com/pingcap/tiup/pkg/cluster/api.(*PDClient).EvictStoreLeader
\tgithub.com/pingcap/tiup/pkg/cluster/api/pdapi.go:474
github.com/pingcap/tiup/pkg/cluster/spec.(*TiKVInstance).PreRestart
\tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:360
github.com/pingcap/tiup/pkg/cluster/operation.upgradeInstance
\tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:215
github.com/pingcap/tiup/pkg/cluster/operation.Upgrade
\tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:169
github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade.func1
\tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:210
github.com/pingcap/tiup/pkg/cluster/task.(*Func).Execute
\tgithub.com/pingcap/tiup/pkg/cluster/task/func.go:34
github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute
\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:86
github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade
\tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:219
github.com/pingcap/tiup/components/cluster/command.newUpgradeCmd.func1
\tgithub.com/pingcap/tiup/components/cluster/command/upgrade.go:41
github.com/spf13/cobra.(*Command).execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
\tgithub.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:918
github.com/pingcap/tiup/components/cluster/command.newReplayCmd.func1
\tgithub.com/pingcap/tiup/components/cluster/command/replay.go:59
github.com/spf13/cobra.(*Command).execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
\tgithub.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:918
github.com/pingcap/tiup/components/cluster/command.Execute
\tgithub.com/pingcap/tiup/components/cluster/command/root.go:297
main.main
\tgithub.com/pingcap/tiup/components/cluster/main.go:23
runtime.main
\truntime/proc.go:250
runtime.goexit
\truntime/asm_amd64.s:1594
failed to evict store leader 10.8.35.62”}
2022-10-13T16:53:40.882+0800 INFO Execute command finished {“code”: 1, “error”: “failed to evict store leader 10.8.35.62: metric tikv_raftstore_region_count{type="leader"} not found”, “errorVerbose”: “metric tikv_raftstore_region_count{type="leader"} not found
github.com/pingcap/tiup/pkg/cluster/spec.genLeaderCounter.func1
\tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:456
github.com/pingcap/tiup/pkg/cluster/api.(*PDClient).EvictStoreLeader
\tgithub.com/pingcap/tiup/pkg/cluster/api/pdapi.go:474
github.com/pingcap/tiup/pkg/cluster/spec.(*TiKVInstance).PreRestart
\tgithub.com/pingcap/tiup/pkg/cluster/spec/tikv.go:360
github.com/pingcap/tiup/pkg/cluster/operation.upgradeInstance
\tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:215
github.com/pingcap/tiup/pkg/cluster/operation.Upgrade
\tgithub.com/pingcap/tiup/pkg/cluster/operation/upgrade.go:169
github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade.func1
\tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:210
github.com/pingcap/tiup/pkg/cluster/task.(*Func).Execute
\tgithub.com/pingcap/tiup/pkg/cluster/task/func.go:34
github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute
\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:86
github.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Upgrade
\tgithub.com/pingcap/tiup/pkg/cluster/manager/upgrade.go:219
github.com/pingcap/tiup/components/cluster/command.newUpgradeCmd.func1
\tgithub.com/pingcap/tiup/components/cluster/command/upgrade.go:41
github.com/spf13/cobra.(*Command).execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
\tgithub.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:918
github.com/pingcap/tiup/components/cluster/command.newReplayCmd.func1
\tgithub.com/pingcap/tiup/components/cluster/command/replay.go:59
github.com/spf13/cobra.(*Command).execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
\tgithub.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
\tgithub.com/spf13/cobra@v1.5.0/command.go:918
github.com/pingcap/tiup/components/cluster/command.Execute
\tgithub.com/pingcap/tiup/components/cluster/command/root.go:297
main.main
\tgithub.com/pingcap/tiup/components/cluster/main.go:23
runtime.main
\truntime/proc.go:250
runtime.goexit
\truntime/asm_amd64.s:1594
failed to evict store leader 10.8.35.62”}
【附件】
请提供各个组件的 version 信息，如 cdc/tikv，可通过执行 cdc version/tikv-server --version 获取。