新增TIFLASH节点无法启动-no leader

🪐 TiDB 技术问题 TiFlash
1

【 TiDB 使用环境`】测试环境
【 TiDB 版本】5.4
【遇到的问题】
tidb + pd 一台服务器
tikv 三台服务器
部署完新增（扩容)一个tiflash节点，运行 tiup cluster scale-out tidb-test scale-out.yaml -p 的结果为：

[tidb@bogon ~]$ tiup cluster scale-out tidb-test scale-out.yaml -p
tiup is checking updates for component cluster …
Starting component cluster: /home/tidb/.tiup/components/cluster/v1.9.4/tiup-cluster /home/tidb/.tiup/components/cluster/v1.9.4/tiup-cluster scale-out tidb-test scale-out.yaml -p
Input SSH password:

  • Detect CPU Arch
    • Detecting node 192.168.5.162 … Done
      Please confirm your topology:
      Cluster type: tidb
      Cluster name: tidb-test
      Cluster version: v5.4.0
      Role Host Ports OS/Arch Directories

tiflash 192.168.5.162 9000/8123/3930/20170/20292/8234 linux/x86_64 /data/tidb-deploy/tiflash-9000,/data/tidb-data/tiflash-9000
Attention:
1. If the topology is not what you expected, check your yaml file.
2. Please confirm there is no port/directory conflicts in same host.
Do you want to continue? [y/N]: (default=N) y

  • [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa.pub
  • [Parallel] - UserSSH: user=tidb, host=192.168.5.162
  • [Parallel] - UserSSH: user=tidb, host=192.168.5.163
  • [Parallel] - UserSSH: user=tidb, host=192.168.5.162
  • [Parallel] - UserSSH: user=tidb, host=192.168.5.164
  • [Parallel] - UserSSH: user=tidb, host=192.168.5.162
  • [Parallel] - UserSSH: user=tidb, host=192.168.5.165
  • [Parallel] - UserSSH: user=tidb, host=192.168.5.162
  • [Parallel] - UserSSH: user=tidb, host=192.168.5.162
  • Download TiDB components
    • Download tiflash:v5.4.0 (linux/amd64) … Done
  • Initialize target host environments
  • Deploy TiDB instance
    • Deploy instance tiflash -> 192.168.5.162:9000 … Done
  • Copy certificate to remote host
  • Generate scale-out config
    • Generate scale-out config tiflash -> 192.168.5.162:9000 … Done
  • Init monitor config
  • Check status
    Enabling component tiflash
    Enabling instance 192.168.5.162:9000
    Enable instance 192.168.5.162:9000 success
    Enabling component node_exporter
    Enabling instance 192.168.5.162
    Enable 192.168.5.162 success
    Enabling component blackbox_exporter
    Enabling instance 192.168.5.162
    Enable 192.168.5.162 success
  • [ Serial ] - Save meta
  • [ Serial ] - Start new instances
    Starting component tiflash

Error: failed to start tiflash: error requesting http://192.168.5.162:2379/pd/api/v1/config/replicate, response: no leader
, code 503

Verbose debug logs has been written to /home/tidb/.tiup/logs/tiup-cluster-debug-2022-04-25-11-14-32.log.

2

tiup-cluster-debug-2022-04-25-11-14-32.log (45.2 KB)

3

启动 tiflash 的时候需要去 PD 中 get 些信息，可能是 PD 发生了 leader 切换等原因导致 请求失败。
重新尝试启动一直同样错吗？感觉等 PD Leader 选出来了就可以起来了。
requesting http://192.168.5.162:2379/pd/api/v1/config/replicate, response: no leader, code 503

如果，可精准复现，同时可以查下 PD 当前的 Leader 状态，并反馈下 TiFlash 日志（在 TiFlash Deploy 目录下）。