4.0.7升级5.0.2报ssh连接错误

🪐 TiDB 技术问题 部署&运维管理
1

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：

【TiDB 版本】：
目前为v4.0.7

【概述】：场景 + 问题概述
v4.0.7升级到v5.0.2时报错
[root@k8s-master02 ~]# tiup cluster upgrade tidb_xiqu_test v5.0.2
Starting component cluster: /root/.tiup/components/cluster/v1.5.1/tiup-cluster upgrade tidb_xiqu_test v5.0.2
This operation will upgrade tidb v4.0.7 cluster tidb_xiqu_test to v5.0.2.
Do you want to continue? [y/N]:(default=N) y
Upgrading cluster…

  • [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/ssh/id_rsa.pub
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [Parallel] - UserSSH: user=tidb2, host=10.103.236.178
  • [ Serial ] - Download: component=cdc, version=v5.0.2, os=linux, arch=amd64
  • [ Serial ] - Download: component=pd, version=v5.0.2, os=linux, arch=amd64
  • [ Serial ] - Download: component=tikv, version=v5.0.2, os=linux, arch=amd64
  • [ Serial ] - Download: component=prometheus, version=v5.0.2, os=linux, arch=amd64
  • [ Serial ] - Download: component=tiflash, version=v5.0.2, os=linux, arch=amd64
  • [ Serial ] - Download: component=grafana, version=v5.0.2, os=linux, arch=amd64
  • [ Serial ] - Download: component=pump, version=v5.0.2, os=linux, arch=amd64
  • [ Serial ] - Download: component=drainer, version=v5.0.2, os=linux, arch=amd64
  • [ Serial ] - Download: component=tidb, version=v5.0.2, os=linux, arch=amd64
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’’
  • [ Serial ] - BackupComponent: component=grafana, currentVersion=v4.0.7, remote=10.103.236.178:/data/tidb-deploy/grafana-3000
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’/data/tidb-data/pd-3379’
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’/data/tidb-data/pump-8250’
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’/data/tidb-data/tikv-20160’
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’/data/tidb-data/tikv-20161’
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’/data/tidb-data/tikv-20162’
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’’
  • [ Serial ] - BackupComponent: component=tidb, currentVersion=v4.0.7, remote=10.103.236.178:/data/tidb-deploy/tidb-4000
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’/data/tidb-data/drainer-8249’
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’/data/tidb-data/cdc-8300’
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’/data/tidb-data/tiflash-9000’
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’/data/tidb-data/prometheus-9090’
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’/data/tidb-data/drainer-8239’
  • [ Serial ] - Mkdir: host=10.103.236.178, directories=’/data/tidb-data/pump-8251’
  • [ Serial ] - CopyComponent: component=grafana, version=v5.0.2, remote=10.103.236.178:/data/tidb-deploy/grafana-3000 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=tidb, version=v5.0.2, remote=10.103.236.178:/data/tidb-deploy/tidb-4000 os=linux, arch=amd64
  • [ Serial ] - BackupComponent: component=drainer, currentVersion=v4.0.7, remote=10.103.236.178:/data/tidb-deploy/drainer-8239
  • [ Serial ] - BackupComponent: component=cdc, currentVersion=v4.0.7, remote=10.103.236.178:/data/tidb-deploy/cdc-8300
  • [ Serial ] - BackupComponent: component=tiflash, currentVersion=v4.0.7, remote=10.103.236.178:/data/tidb-deploy/tiflash-9000
  • [ Serial ] - BackupComponent: component=pump, currentVersion=v4.0.7, remote=10.103.236.178:/data/tidb-deploy/pump-8250
  • [ Serial ] - BackupComponent: component=pump, currentVersion=v4.0.7, remote=10.103.236.178:/data/tidb-deploy/pump-8251
  • [ Serial ] - BackupComponent: component=tikv, currentVersion=v4.0.7, remote=10.103.236.178:/data/tidb-deploy/tikv-20160
  • [ Serial ] - BackupComponent: component=pd, currentVersion=v4.0.7, remote=10.103.236.178:/data/tidb-deploy/pd-3379
  • [ Serial ] - BackupComponent: component=prometheus, currentVersion=v4.0.7, remote=10.103.236.178:/data/tidb-deploy/prometheus-9090
  • [ Serial ] - BackupComponent: component=drainer, currentVersion=v4.0.7, remote=10.103.236.178:/data/tidb-deploy/drainer-8249
  • [ Serial ] - BackupComponent: component=tikv, currentVersion=v4.0.7, remote=10.103.236.178:/data/tidb-deploy/tikv-20162
  • [ Serial ] - CopyComponent: component=cdc, version=v5.0.2, remote=10.103.236.178:/data/tidb-deploy/cdc-8300 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=tiflash, version=v5.0.2, remote=10.103.236.178:/data/tidb-deploy/tiflash-9000 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=drainer, version=v5.0.2, remote=10.103.236.178:/data/tidb-deploy/drainer-8239 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=pump, version=v5.0.2, remote=10.103.236.178:/data/tidb-deploy/pump-8251 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=pump, version=v5.0.2, remote=10.103.236.178:/data/tidb-deploy/pump-8250 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=tikv, version=v5.0.2, remote=10.103.236.178:/data/tidb-deploy/tikv-20160 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=pd, version=v5.0.2, remote=10.103.236.178:/data/tidb-deploy/pd-3379 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=prometheus, version=v5.0.2, remote=10.103.236.178:/data/tidb-deploy/prometheus-9090 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=drainer, version=v5.0.2, remote=10.103.236.178:/data/tidb-deploy/drainer-8249 os=linux, arch=amd64
  • [ Serial ] - CopyComponent: component=tikv, version=v5.0.2, remote=10.103.236.178:/data/tidb-deploy/tikv-20162 os=linux, arch=amd64
  • [ Serial ] - InitConfig: cluster=tidb_xiqu_test, user=tidb2, host=10.103.236.178, path=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache/pump-8251.service, deploy_dir=/data/tidb-deploy/pump-8251, data_dir=[/data/tidb-data/pump-8251], log_dir=/data/tidb-deploy/pump-8251/log, cache_dir=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache
  • [ Serial ] - InitConfig: cluster=tidb_xiqu_test, user=tidb2, host=10.103.236.178, path=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache/pump-8250.service, deploy_dir=/data/tidb-deploy/pump-8250, data_dir=[/data/tidb-data/pump-8250], log_dir=/data/tidb-deploy/pump-8250/log, cache_dir=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache
  • [ Serial ] - InitConfig: cluster=tidb_xiqu_test, user=tidb2, host=10.103.236.178, path=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache/drainer-8239.service, deploy_dir=/data/tidb-deploy/drainer-8239, data_dir=[/data/tidb-data/drainer-8239], log_dir=/data/tidb-deploy/drainer-8239/log, cache_dir=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache
  • [ Serial ] - InitConfig: cluster=tidb_xiqu_test, user=tidb2, host=10.103.236.178, path=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache/drainer-8249.service, deploy_dir=/data/tidb-deploy/drainer-8249, data_dir=[/data/tidb-data/drainer-8249], log_dir=/data/tidb-deploy/drainer-8249/log, cache_dir=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache
  • [ Serial ] - InitConfig: cluster=tidb_xiqu_test, user=tidb2, host=10.103.236.178, path=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache/tidb-4000.service, deploy_dir=/data/tidb-deploy/tidb-4000, data_dir=[], log_dir=/data/tidb-deploy/tidb-4000/log, cache_dir=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache
  • [ Serial ] - InitConfig: cluster=tidb_xiqu_test, user=tidb2, host=10.103.236.178, path=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache/pd-3379.service, deploy_dir=/data/tidb-deploy/pd-3379, data_dir=[/data/tidb-data/pd-3379], log_dir=/data/tidb-deploy/pd-3379/log, cache_dir=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache
  • [ Serial ] - InitConfig: cluster=tidb_xiqu_test, user=tidb2, host=10.103.236.178, path=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache/cdc-8300.service, deploy_dir=/data/tidb-deploy/cdc-8300, data_dir=[/data/tidb-data/cdc-8300], log_dir=/data/tidb-deploy/cdc-8300/log, cache_dir=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache
  • [ Serial ] - InitConfig: cluster=tidb_xiqu_test, user=tidb2, host=10.103.236.178, path=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache/prometheus-9090.service, deploy_dir=/data/tidb-deploy/prometheus-9090, data_dir=[/data/tidb-data/prometheus-9090], log_dir=/data/tidb-deploy/prometheus-9090/log, cache_dir=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache
  • [ Serial ] - InitConfig: cluster=tidb_xiqu_test, user=tidb2, host=10.103.236.178, path=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache/grafana-3000.service, deploy_dir=/data/tidb-deploy/grafana-3000, data_dir=[], log_dir=/data/tidb-deploy/grafana-3000/log, cache_dir=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache
  • [ Serial ] - InitConfig: cluster=tidb_xiqu_test, user=tidb2, host=10.103.236.178, path=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache/tikv-20162.service, deploy_dir=/data/tidb-deploy/tikv-20162, data_dir=[/data/tidb-data/tikv-20162], log_dir=/data/tidb-deploy/tikv-20162/log, cache_dir=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache
  • [ Serial ] - InitConfig: cluster=tidb_xiqu_test, user=tidb2, host=10.103.236.178, path=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache/tikv-20160.service, deploy_dir=/data/tidb-deploy/tikv-20160, data_dir=[/data/tidb-data/tikv-20160], log_dir=/data/tidb-deploy/tikv-20160/log, cache_dir=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache
  • [ Serial ] - InitConfig: cluster=tidb_xiqu_test, user=tidb2, host=10.103.236.178, path=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache/tiflash-9000.service, deploy_dir=/data/tidb-deploy/tiflash-9000, data_dir=[/data/tidb-data/tiflash-9000], log_dir=/data/tidb-deploy/tiflash-9000/log, cache_dir=/root/.tiup/storage/cluster/clusters/tidb_xiqu_test/config-cache

Error: executor.ssh.execute_failed: Failed to execute command over SSH for ‘tidb2@10.103.236.178:22’ {ssh_stderr: , ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin sudo -H bash -c “test -d /data/tidb-data || (mkdir -p /data/tidb-data && chown tidb2:$(id -g -n tidb2) /data/tidb-data)”}, cause: ssh: handshake failed: read tcp 10.103.236.178:35676->10.103.236.178:22: read: connection reset by peer

Verbose debug logs has been written to /root/.tiup/logs/tiup-cluster-debug-2021-06-22-10-39-10.log.
Error: run /root/.tiup/components/cluster/v1.5.1/tiup-cluster (wd:/root/.tiup/data/Sb1lAdU) failed: exit status 1

【附件】：

2
  1. 查了下，之前的帖子有类似问题
    tiup 安装过程中出现，在copy file这个过程中
  2. 麻烦执行下 tiup cluster display 展示下信息，多谢。
  3. 调整 ssh 的链接数再试试 ，记得重启sshd服务 service sshd restart
    https://docs.pingcap.com/zh/tidb/stable/tiup-faq#集群部署期间tiup-收到报错-ssh-handshake-failed-read-tcp-1010103438980---101010343600-read-connection-reset-by-peer
3

是ssh连接数导致的（我检查了ssh配置，我们这里是20）

谢谢！

4

:handshake:

5

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。