关于 TIDB 扩容 与 缩容

🪐 TiDB 技术问题 部署&运维管理
,
1

为提高效率，提问时请提供以下信息，问题描述清晰可优先响应。

  • 【TiDB 版本】：
    $ tiup cluster display bx_tidb
    Starting component cluster: display bx_tidb
    TiDB Cluster: bx_tidb
    TiDB Version: v4.0.2
    ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

10.10.101.181:9093 alertmanager 10.10.101.181 9093/9094 linux/x86_64 Up /tidb_install/tidb-data/alertmanager-9093 /tidb_install/tidb-deploy/alertmanager-9093
10.10.101.181:3000 grafana 10.10.101.181 3000 linux/x86_64 Up - /tidb_install/tidb-deploy/grafana-3000
10.10.101.16:2379 pd 10.10.101.16 2379/2380 linux/x86_64 Up /tidb_install/tidb-data/pd-2379 /tidb_install/tidb-deploy/pd-2379
10.10.101.181:2379 pd 10.10.101.181 2379/2380 linux/x86_64 Up /tidb_install/tidb-data/pd-2379 /tidb_install/tidb-deploy/pd-2379
10.10.101.182:2379 pd 10.10.101.182 2379/2380 linux/x86_64 Up|L|UI /tidb_install/tidb-data/pd-2379 /tidb_install/tidb-deploy/pd-2379
10.10.101.183:2379 pd 10.10.101.183 2379/2380 linux/x86_64 Up /tidb_install/tidb-data/pd-2379 /tidb_install/tidb-deploy/pd-2379
10.10.101.181:9090 prometheus 10.10.101.181 9090 linux/x86_64 Up /tidb_install/tidb-data/prometheus-8249 /tidb_install/tidb-deploy/prometheus-8249
10.10.101.16:4000 tidb 10.10.101.16 4000/10080 linux/x86_64 Up - /tidb_install/tidb-deploy/tidb-4000
10.10.101.181:4000 tidb 10.10.101.181 4000/10080 linux/x86_64 Up - /tidb_install/tidb-deploy/tidb-4000
10.10.101.182:4000 tidb 10.10.101.182 4000/10080 linux/x86_64 Up - /tidb_install/tidb-deploy/tidb-4000
10.10.101.183:4000 tidb 10.10.101.183 4000/10080 linux/x86_64 Up - /tidb_install/tidb-deploy/tidb-4000
10.10.101.16:20160 tikv 10.10.101.16 20160/20180 linux/x86_64 Up /tidb_install/tidb-data/tikv-20160 /tidb_install/tidb-deploy/tikv-20160
10.10.101.181:20160 tikv 10.10.101.181 20160/20180 linux/x86_64 Up /tidb_install/tidb-data/tikv-20160 /tidb_install/tidb-deploy/tikv-20160
10.10.101.182:20160 tikv 10.10.101.182 20160/20180 linux/x86_64 Up /tidb_install/tidb-data/tikv-20160 /tidb_install/tidb-deploy/tikv-20160
10.10.101.183:20160 tikv 10.10.101.183 20160/20180 linux/x86_64 Up /tidb_install/tidb-data/tikv-20160 /tidb_install/tidb-deploy/tikv-20160

  • 【问题描述】：
    手工执行命令 ：tiup cluster scale-in bx_tidb --node 10.10.101.183:20160
    将 183 上的 TIKV 移除
    执行如下 ：
    $ tiup cluster scale-in bx_tidb --node 10.10.101.183:20160
    Starting component cluster: scale-in bx_tidb --node 10.10.101.183:20160
    This operation will delete the 10.10.101.183:20160 nodes in bx_tidb and all their data.
    Do you want to continue? [y/N]: y
    Scale-in nodes…
  • [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/ssh/id_rsa.pub
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.181
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.183
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.16
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.183
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.181
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.182
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.16
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.182
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.181
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.16
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.183
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.181
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.181
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.181
  • [Parallel] - UserSSH: user=tidb, host=10.10.101.182
  • [ Serial ] - ClusterOperate: operation=ScaleInOperation, options={Roles:[] Nodes:[10.10.101.183:20160] Force:false SSHTimeout:5 OptTimeout:120 APITimeout:300 IgnoreConfigCheck:false RetainDataRoles:[] RetainDataNodes:[]}
    Delete store 4 10.10.101.183:20160 from the cluster success
    The component tikv will be destroyed when display cluster info when it become tombstone, maybe exists in several minutes or hours
  • [ Serial ] - UpdateMeta: cluster=bx_tidb, deleted=''
  • [ Serial ] - UpdateTopology: cluster=bx_tidb
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.181, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/alertmanager-9093.service, deploy_dir=/tidb_install/tidb-deploy/alertmanager-9093, data_dir=[/tidb_install/tidb-data/alertmanager-9093], log_dir=/tidb_install/tidb-deploy/alertmanager-9093/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.182, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/pd-2379.service, deploy_dir=/tidb_install/tidb-deploy/pd-2379, data_dir=[/tidb_install/tidb-data/pd-2379], log_dir=/tidb_install/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.181, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/pd-2379.service, deploy_dir=/tidb_install/tidb-deploy/pd-2379, data_dir=[/tidb_install/tidb-data/pd-2379], log_dir=/tidb_install/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.182, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/tidb-4000.service, deploy_dir=/tidb_install/tidb-deploy/tidb-4000, data_dir=[], log_dir=/tidb_install/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.181, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/tikv-20160.service, deploy_dir=/tidb_install/tidb-deploy/tikv-20160, data_dir=[/tidb_install/tidb-data/tikv-20160], log_dir=/tidb_install/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.183, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/tidb-4000.service, deploy_dir=/tidb_install/tidb-deploy/tidb-4000, data_dir=[], log_dir=/tidb_install/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.181, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/prometheus-9090.service, deploy_dir=/tidb_install/tidb-deploy/prometheus-8249, data_dir=[/tidb_install/tidb-data/prometheus-8249], log_dir=/tidb_install/tidb-deploy/prometheus-8249/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.181, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/grafana-3000.service, deploy_dir=/tidb_install/tidb-deploy/grafana-3000, data_dir=[], log_dir=/tidb_install/tidb-deploy/grafana-3000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.182, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/tikv-20160.service, deploy_dir=/tidb_install/tidb-deploy/tikv-20160, data_dir=[/tidb_install/tidb-data/tikv-20160], log_dir=/tidb_install/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.16, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/tidb-4000.service, deploy_dir=/tidb_install/tidb-deploy/tidb-4000, data_dir=[], log_dir=/tidb_install/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.183, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/pd-2379.service, deploy_dir=/tidb_install/tidb-deploy/pd-2379, data_dir=[/tidb_install/tidb-data/pd-2379], log_dir=/tidb_install/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.16, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/tikv-20160.service, deploy_dir=/tidb_install/tidb-deploy/tikv-20160, data_dir=[/tidb_install/tidb-data/tikv-20160], log_dir=/tidb_install/tidb-deploy/tikv-20160/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.181, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/tidb-4000.service, deploy_dir=/tidb_install/tidb-deploy/tidb-4000, data_dir=[], log_dir=/tidb_install/tidb-deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
  • [ Serial ] - InitConfig: cluster=bx_tidb, user=tidb, host=10.10.101.16, path=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache/pd-2379.service, deploy_dir=/tidb_install/tidb-deploy/pd-2379, data_dir=[/tidb_install/tidb-data/pd-2379], log_dir=/tidb_install/tidb-deploy/pd-2379/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/bx_tidb/config-cache
    Scaled cluster bx_tidb in successfully

缩容后查看集群状态：

$ tiup cluster display bx_tidb
Starting component cluster: display bx_tidb
TiDB Cluster: bx_tidb
TiDB Version: v4.0.2
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

10.10.101.181:9093 alertmanager 10.10.101.181 9093/9094 linux/x86_64 Up /tidb_install/tidb-data/alertmanager-9093 /tidb_install/tidb-deploy/alertmanager-9093
10.10.101.181:3000 grafana 10.10.101.181 3000 linux/x86_64 Up - /tidb_install/tidb-deploy/grafana-3000
10.10.101.16:2379 pd 10.10.101.16 2379/2380 linux/x86_64 Up /tidb_install/tidb-data/pd-2379 /tidb_install/tidb-deploy/pd-2379
10.10.101.181:2379 pd 10.10.101.181 2379/2380 linux/x86_64 Up /tidb_install/tidb-data/pd-2379 /tidb_install/tidb-deploy/pd-2379
10.10.101.182:2379 pd 10.10.101.182 2379/2380 linux/x86_64 Up|L|UI /tidb_install/tidb-data/pd-2379 /tidb_install/tidb-deploy/pd-2379
10.10.101.183:2379 pd 10.10.101.183 2379/2380 linux/x86_64 Up /tidb_install/tidb-data/pd-2379 /tidb_install/tidb-deploy/pd-2379
10.10.101.181:9090 prometheus 10.10.101.181 9090 linux/x86_64 Up /tidb_install/tidb-data/prometheus-8249 /tidb_install/tidb-deploy/prometheus-8249
10.10.101.16:4000 tidb 10.10.101.16 4000/10080 linux/x86_64 Up - /tidb_install/tidb-deploy/tidb-4000
10.10.101.181:4000 tidb 10.10.101.181 4000/10080 linux/x86_64 Up - /tidb_install/tidb-deploy/tidb-4000
10.10.101.182:4000 tidb 10.10.101.182 4000/10080 linux/x86_64 Up - /tidb_install/tidb-deploy/tidb-4000
10.10.101.183:4000 tidb 10.10.101.183 4000/10080 linux/x86_64 Up - /tidb_install/tidb-deploy/tidb-4000
10.10.101.16:20160 tikv 10.10.101.16 20160/20180 linux/x86_64 Disconnected /tidb_install/tidb-data/tikv-20160 /tidb_install/tidb-deploy/tikv-20160
10.10.101.181:20160 tikv 10.10.101.181 20160/20180 linux/x86_64 Up /tidb_install/tidb-data/tikv-20160 /tidb_install/tidb-deploy/tikv-20160
10.10.101.182:20160 tikv 10.10.101.182 20160/20180 linux/x86_64 Disconnected /tidb_install/tidb-data/tikv-20160 /tidb_install/tidb-deploy/tikv-20160
10.10.101.183:20160 tikv 10.10.101.183 20160/20180 linux/x86_64 Pending Offline /tidb_install/tidb-data/tikv-20160 /tidb_install/tidb-deploy/tikv-20160

10.10.101.183:20160 tikv 可以看到状态为 Pending Offline；
我看到官方文档说要等待一段时间 状态修改为 Tombstone 才是下线成功；想问一下这个等待时间需要多久！！

%E5%9B%BE%E7%89%87
%E5%9B%BE%E7%89%87878×552 21.4 KB

2
  1. 您好，这个时间和您节点里的 region 数量，以及集群负载有关系。
  2. 如果想要加快调度，可以参考文档 章节 节点下线速度慢

https://docs.pingcap.com/zh/tidb/stable/pd-scheduling-best-practices#region-merge-速度慢

3

收到 ，多谢哈！！

4

:handshake:

5

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。