tidb upgrade to v6.6.0 遇到tikv Still waitting for 1 store leaders to transfer

[lmdb (Time is money , my friend！— Goblin)]

【 TiDB 使用环境】测试/ Poc
【 TiDB 版本】v.6.6.0
【复现路径】 upgrade tidb cluster from v 6.4.0 to v 6.6.0
【遇到的问题：问题现象及影响】
在upgrade tikv 的时候提示 Still waitting for 1 store leaders to transfer异常，最后 tikv 被重启 ，
【资源配置】
【附件：截图/日志/监控】

####操作记录

[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$ tiup cluster list
tiup is checking updates for component cluster …
Starting component cluster: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster list
Name User Version Path PrivateKey

---

tidb-cluster tidb v6.4.0 /home/tidb/.tiup/storage/cluster/clusters/tidb-cluster /home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
[tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$ tiup cluster upgrade tidb-cluster v6.6.0
tiup is checking updates for component cluster …
Starting component cluster: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster upgrade tidb-cluster v6.6.0
Before the upgrade, it is recommended to read the upgrade guide at https://docs.pingcap.com/tidb/stable/upgrade-tidb-using-tiup and finish the preparation steps.
This operation will upgrade tidb v6.4.0 cluster tidb-cluster to v6.6.0.
Do you want to continue? [y/N]:(default=N) y
Upgrading cluster…

• [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa.pub
• [Parallel] - UserSSH: user=tidb, host=192.168.2.21
• [Parallel] - UserSSH: user=tidb, host=192.168.2.22
• [Parallel] - UserSSH: user=tidb, host=192.168.2.22
• [Parallel] - UserSSH: user=tidb, host=192.168.2.151
• [Parallel] - UserSSH: user=tidb, host=192.168.2.21
• [Parallel] - UserSSH: user=tidb, host=192.168.2.22
• [Parallel] - UserSSH: user=tidb, host=192.168.2.21
• [Parallel] - UserSSH: user=tidb, host=192.168.2.151
• [Parallel] - UserSSH: user=tidb, host=192.168.2.151
• [Parallel] - UserSSH: user=tidb, host=192.168.2.151
• [Parallel] - UserSSH: user=tidb, host=192.168.2.22
• [Parallel] - UserSSH: user=tidb, host=192.168.2.151
• [Parallel] - UserSSH: user=tidb, host=192.168.2.151
• [Parallel] - UserSSH: user=tidb, host=192.168.2.21
• [ Serial ] - Download: component=prometheus, version=v6.6.0, os=linux, arch=amd64
• [ Serial ] - Download: component=pd, version=v6.6.0, os=linux, arch=amd64
• [ Serial ] - Download: component=tiflash, version=v6.6.0, os=linux, arch=amd64
• [ Serial ] - Download: component=tikv, version=v6.6.0, os=linux, arch=amd64
• [ Serial ] - Download: component=tidb, version=v6.6.0, os=linux, arch=amd64
• [ Serial ] - Download: component=grafana, version=v6.6.0, os=linux, arch=amd64
• [ Serial ] - Download: component=alertmanager, version=, os=linux, arch=amd64
• [ Serial ] - Mkdir: host=192.168.2.151, directories=‘/data/tidb-data/pd-2379’
• [ Serial ] - Mkdir: host=192.168.2.21, directories=‘/data/tidb-data/tiflash-9000’
• [ Serial ] - Mkdir: host=192.168.2.22, directories=‘/data/tidb-data/tiflash-9000’
• [ Serial ] - Mkdir: host=192.168.2.22, directories=‘/data/tidb-data/pd-2379’
• [ Serial ] - Mkdir: host=192.168.2.21, directories=‘/data/tidb-data/pd-2379’
• [ Serial ] - BackupComponent: component=tiflash, currentVersion=v6.4.0, remote=192.168.2.21:/data/tidb-deploy/tiflash-9000
• [ Serial ] - BackupComponent: component=pd, currentVersion=v6.4.0, remote=192.168.2.21:/data/tidb-deploy/pd-2379
• [ Serial ] - BackupComponent: component=pd, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/pd-2379
• [ Serial ] - BackupComponent: component=pd, currentVersion=v6.4.0, remote=192.168.2.22:/data/tidb-deploy/pd-2379
• [ Serial ] - BackupComponent: component=tiflash, currentVersion=v6.4.0, remote=192.168.2.22:/data/tidb-deploy/tiflash-9000
• [ Serial ] - CopyComponent: component=pd, version=v6.6.0, remote=192.168.2.22:/data/tidb-deploy/pd-2379 os=linux, arch=amd64
• [ Serial ] - CopyComponent: component=tiflash, version=v6.6.0, remote=192.168.2.22:/data/tidb-deploy/tiflash-9000 os=linux, arch=amd64
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.22, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/pd-2379.service, deploy_dir=/data/tidb-deploy/pd-2379, data_dir=[/data/tidb-data/pd-2379], log_dir=/data/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - Mkdir: host=192.168.2.151, directories=‘/data/tidb-data/tikv-20160’
• [ Serial ] - CopyComponent: component=pd, version=v6.6.0, remote=192.168.2.151:/data/tidb-deploy/pd-2379 os=linux, arch=amd64
• [ Serial ] - CopyComponent: component=pd, version=v6.6.0, remote=192.168.2.21:/data/tidb-deploy/pd-2379 os=linux, arch=amd64
• [ Serial ] - BackupComponent: component=tikv, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/tikv-20160
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.22, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tiflash-9000.service, deploy_dir=/data/tidb-deploy/tiflash-9000, data_dir=[/data/tidb-data/tiflash-9000], log_dir=/data/tidb-deploy/tiflash-9000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - Mkdir: host=192.168.2.21, directories=‘/data/tidb-data/tikv-20160’
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/pd-2379.service, deploy_dir=/data/tidb-deploy/pd-2379, data_dir=[/data/tidb-data/pd-2379], log_dir=/data/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - BackupComponent: component=tikv, currentVersion=v6.4.0, remote=192.168.2.21:/data/tidb-deploy/tikv-20160
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.21, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/pd-2379.service, deploy_dir=/data/tidb-deploy/pd-2379, data_dir=[/data/tidb-data/pd-2379], log_dir=/data/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - Mkdir: host=192.168.2.22, directories=‘/data/tidb-data/tikv-20160’
• [ Serial ] - BackupComponent: component=tikv, currentVersion=v6.4.0, remote=192.168.2.22:/data/tidb-deploy/tikv-20160
• [ Serial ] - Mkdir: host=192.168.2.151, directories=‘’
• [ Serial ] - BackupComponent: component=tidb, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/tidb-4000
• [ Serial ] - CopyComponent: component=tikv, version=v6.6.0, remote=192.168.2.22:/data/tidb-deploy/tikv-20160 os=linux, arch=amd64
• [ Serial ] - CopyComponent: component=tidb, version=v6.6.0, remote=192.168.2.151:/data/tidb-deploy/tidb-4000 os=linux, arch=amd64
• [ Serial ] - CopyComponent: component=tiflash, version=v6.6.0, remote=192.168.2.21:/data/tidb-deploy/tiflash-9000 os=linux, arch=amd64
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.22, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tikv-20160.service, deploy_dir=/data/tidb-deploy/tikv-20160, data_dir=[/data/tidb-data/tikv-20160], log_dir=/data/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - Mkdir: host=192.168.2.21, directories=‘’
• [ Serial ] - BackupComponent: component=tidb, currentVersion=v6.4.0, remote=192.168.2.21:/data/tidb-deploy/tidb-4000
• [ Serial ] - CopyComponent: component=tikv, version=v6.6.0, remote=192.168.2.151:/data/tidb-deploy/tikv-20160 os=linux, arch=amd64
• [ Serial ] - CopyComponent: component=tikv, version=v6.6.0, remote=192.168.2.21:/data/tidb-deploy/tikv-20160 os=linux, arch=amd64
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tidb-4000.service, deploy_dir=/data/tidb-deploy/tidb-4000, data_dir=[], log_dir=/data/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - Mkdir: host=192.168.2.22, directories=‘’
• [ Serial ] - BackupComponent: component=tidb, currentVersion=v6.4.0, remote=192.168.2.22:/data/tidb-deploy/tidb-4000
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tikv-20160.service, deploy_dir=/data/tidb-deploy/tikv-20160, data_dir=[/data/tidb-data/tikv-20160], log_dir=/data/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - Mkdir: host=192.168.2.151, directories=‘/data/tidb-data/prometheus-9090’
• [ Serial ] - BackupComponent: component=prometheus, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/prometheus-9090
• [ Serial ] - CopyComponent: component=tidb, version=v6.6.0, remote=192.168.2.21:/data/tidb-deploy/tidb-4000 os=linux, arch=amd64
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.21, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tiflash-9000.service, deploy_dir=/data/tidb-deploy/tiflash-9000, data_dir=[/data/tidb-data/tiflash-9000], log_dir=/data/tidb-deploy/tiflash-9000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.21, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tidb-4000.service, deploy_dir=/data/tidb-deploy/tidb-4000, data_dir=[], log_dir=/data/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - Mkdir: host=192.168.2.151, directories=‘’
• [ Serial ] - BackupComponent: component=grafana, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/grafana-3000
• [ Serial ] - Mkdir: host=192.168.2.151, directories=‘/data/tidb-data/alertmanager-9093’
• [ Serial ] - BackupComponent: component=alertmanager, currentVersion=v6.4.0, remote=192.168.2.151:/data/tidb-deploy/alertmanager-9093
• [ Serial ] - CopyComponent: component=tidb, version=v6.6.0, remote=192.168.2.22:/data/tidb-deploy/tidb-4000 os=linux, arch=amd64
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.22, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tidb-4000.service, deploy_dir=/data/tidb-deploy/tidb-4000, data_dir=[], log_dir=/data/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - CopyComponent: component=alertmanager, version=, remote=192.168.2.151:/data/tidb-deploy/alertmanager-9093 os=linux, arch=amd64
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/alertmanager-9093.service, deploy_dir=/data/tidb-deploy/alertmanager-9093, data_dir=[/data/tidb-data/alertmanager-9093], log_dir=/data/tidb-deploy/alertmanager-9093/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - CopyComponent: component=prometheus, version=v6.6.0, remote=192.168.2.151:/data/tidb-deploy/prometheus-9090 os=linux, arch=amd64
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/prometheus-9090.service, deploy_dir=/data/tidb-deploy/prometheus-9090, data_dir=[/data/tidb-data/prometheus-9090], log_dir=/data/tidb-deploy/prometheus-9090/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.21, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/tikv-20160.service, deploy_dir=/data/tidb-deploy/tikv-20160, data_dir=[/data/tidb-data/tikv-20160], log_dir=/data/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - CopyComponent: component=grafana, version=v6.6.0, remote=192.168.2.151:/data/tidb-deploy/grafana-3000 os=linux, arch=amd64
• [ Serial ] - InitConfig: cluster=tidb-cluster, user=tidb, host=192.168.2.151, path=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache/grafana-3000.service, deploy_dir=/data/tidb-deploy/grafana-3000, data_dir=[], log_dir=/data/tidb-deploy/grafana-3000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/config-cache
• [ Serial ] - UpgradeCluster
  Upgrading component tiflash
  Restarting instance 192.168.2.21:9000
  Restart instance 192.168.2.21:9000 success
  Restarting instance 192.168.2.22:9000
  Restart instance 192.168.2.22:9000 success
  Upgrading component pd
  Restarting instance 192.168.2.151:2379
  Restart instance 192.168.2.151:2379 success
  Restarting instance 192.168.2.22:2379
  Restart instance 192.168.2.22:2379 success
  Restarting instance 192.168.2.21:2379
  Restart instance 192.168.2.21:2379 success
  Upgrading component tikv
  Evicting 1 leaders from store 192.168.2.151:20160…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Restarting instance 192.168.2.151:20160
  Restart instance 192.168.2.151:20160 success
  Evicting 1 leaders from store 192.168.2.21:20160…
  Still waitting for 1 store leaders to transfer…
  Still waitting for 1 store leaders to transfer…
  Restarting instance 192.168.2.21:20160
  Restart instance 192.168.2.21:20160 success
  Evicting 2 leaders from store 192.168.2.22:20160…
  Still waitting for 2 store leaders to transfer…
  Still waitting for 2 store leaders to transfer…
  Still waitting for 2 store leaders to transfer…
  Restarting instance 192.168.2.22:20160
  Restart instance 192.168.2.22:20160 success
  Upgrading component tidb
  Restarting instance 192.168.2.151:4000
  Restart instance 192.168.2.151:4000 success
  Restarting instance 192.168.2.21:4000
  Restart instance 192.168.2.21:4000 success
  Restarting instance 192.168.2.22:4000
  Restart instance 192.168.2.22:4000 success
  Upgrading component prometheus
  Restarting instance 192.168.2.151:9090
  Restart instance 192.168.2.151:9090 success
  Upgrading component grafana
  Restarting instance 192.168.2.151:3000
  Restart instance 192.168.2.151:3000 success
  Upgrading component alertmanager
  Restarting instance 192.168.2.151:9093
  Restart instance 192.168.2.151:9093 success
  Stopping component node_exporter
  Stopping instance 192.168.2.151
  Stopping instance 192.168.2.21
  Stopping instance 192.168.2.22
  Stop 192.168.2.21 success
  Stop 192.168.2.151 success
  Stop 192.168.2.22 success
  Stopping component blackbox_exporter
  Stopping instance 192.168.2.151
  Stopping instance 192.168.2.21
  Stopping instance 192.168.2.22
  Stop 192.168.2.21 success
  Stop 192.168.2.151 success
  Stop 192.168.2.22 success
  Starting component node_exporter
  Starting instance 192.168.2.151
  Starting instance 192.168.2.21
  Starting instance 192.168.2.22
  Start 192.168.2.21 success
  Start 192.168.2.151 success
  Start 192.168.2.22 success
  Starting component blackbox_exporter
  Starting instance 192.168.2.151
  Starting instance 192.168.2.21
  Starting instance 192.168.2.22
  Start 192.168.2.151 success
  Start 192.168.2.22 success
  Start 192.168.2.21 success
  Upgraded cluster tidb-cluster successfully
  [tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
  [tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
  [tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$
  [tidb@tidb-mdb tidb-community-server-v6.6.0-linux-amd64]$ cd
  [tidb@tidb-mdb ~]$
  [tidb@tidb-mdb ~]$
  [tidb@tidb-mdb ~]$ tiup cluster list
  tiup is checking updates for component cluster …
  Starting component cluster: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster list
  Name User Version Path PrivateKey

---

tidb-cluster tidb v6.6.0 /home/tidb/.tiup/storage/cluster/clusters/tidb-cluster /home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa
[tidb@tidb-mdb ~]$

[lmdb (Time is money , my friend！— Goblin)]

采用的滚动upgrade 的方式， 同时也注意到有关tikv的 升级的说明， * 滚动升级会逐个升级所有的组件。升级 TiKV 期间，会逐个将 TiKV 上的所有 leader 切走再停止该 TiKV 实例。默认超时时间为 5 分钟（300 秒），超时后会直接停止该实例。

我感觉 这个tidb cluster 的tikv的leader 是不正常切走的

[tidb菜鸟一只 (小菜一颗)]

看一下对应节点的tikv日志

[xfworld (魔幻之翼)]

为啥不用 6.5.X 测呢，要从6.4.x 升级到 6.6.0… 要体验新特性？

[海石花47]

现在线上6.1，不敢直接升级，怕踩雷。。

[ffeenn (ThatBoy)]

这是升级完成了呀。没异常，属于正常。

[lmdb (Time is money , my friend！— Goblin)]

嗯， 在搞测试。 应该是重启tikv 也是正常情况。

[xfworld (魔幻之翼)]

马上要出 7.0.0 了～

[lmdb (Time is money , my friend！— Goblin)]

有点体验不过来呢， 支持

[BraveChen (我人傻了)]

这不是正常现象吗，滚动重启

[h5n1]

日志是正常造作的日志，等待leader迁移到其他节点， 迁移过程慢有可能是待迁到的region副本apply落后