tiup 部署集群,扩容报错ERROR Run Command Timeout!

使用tiup扩容两台机器 配置如下
tikv_servers:

  • host: 192.168.1.159
    ssh_port: 22
    port: 20171
    status_port: 20181
    deploy_dir: /data/deploy1
    data_dir: /data/deploy1/data
    log_dir: /data/deploy1/log

  • host: 192.168.1.160
    ssh_port: 22
    port: 20171
    status_port: 20181
    deploy_dir: /data/deploy1
    data_dir: /data/deploy1/data
    log_dir: /data/deploy1/log

执行命令 tiup cluster scale-out tidb_cluster scale-out.yaml --ssh-timeout 60

tikv一共已有15个节点,其中存在一个实例两个tikv的情况,也存在单个tikv的情况

每次执行都会有某个tikv节点报超时错误,每次都是不同的tikv节点,报错如下
2020-06-17T00:38:49.370+0800 INFO Start tikv 192.168.1.153:20171 success
2020-06-17T00:40:28.702+0800 INFO SSHCommand {“host”: “192.168.1.156”, “port”: “22”, “cmd”: “PATH=$PATH:/usr/bin:/usr/sbin sudo -H -u root bash -c “systemctl daemon-reload && systemctlstart tikv-20172.service && systemctl enable tikv-20172.service””, “stdout”: “”, “stderr”: “Run Command Timeout!\ ”}
2020-06-17T00:40:28.702+0800 ERROR Run Command Timeout!

2020-06-17T00:40:28.803+0800 INFO SSHCommand {“host”: “192.168.1.154”, “port”: “22”, “cmd”: “PATH=$PATH:/usr/bin:/usr/sbin sudo -H -u root bash -c “systemctl daemon-reload && systemctlstart tikv-20172.service && systemctl enable tikv-20172.service””, “stdout”: “”, “stderr”: “Run Command Timeout!\ ”}
2020-06-17T00:40:28.803+0800 ERROR Run Command Timeout!

2020-06-17T00:40:28.804+0800 DEBUG TaskFinish {“task”: “ClusterOperate: operation=StartOperation, options={Roles:[] Nodes:[] Force:false SSHTimeout:0 OptTimeout:60 APITimeout:0}”, “error”: “failed to start: failed to start tikv: failed to start: tikv 192.168.1.156:20172: executor.ssh.execute_timedout: Execute command over SSH timedout for ‘tidb@192.168.1.156:22’ {ssh_stderr: Run Command Timeout!\ , ssh_stdout: , ssh_command: PATH=$PATH:/usr/bin:/usr/sbin sudo -H -u root bash -c “systemctl daemon-reload && systemctl start tikv-20172.service && systemctl enable tikv-20172.service”}”, “errorVerbose”: “executor.ssh.execute_timedout: Execute command over SSH timedout for ‘tidb@192.168.1.156:22’ {ssh_stderr: Run Command Timeout!\ , ssh_stdout: , ssh_command: PATH=$PATH:/usr/bin:/usr/sbin sudo -H -u root bash -c “systemctl daemon-reload && systemctl start tikv-20172.service && systemctl enable tikv-20172.service”}\ at github.com/pingcap-incubator/tiup-cluster/pkg/executor.(*SSHExecutor).Execute()\ \t/home/jenkins/agent/workspace/tiup-cluster-release/pkg/executor/ssh.go:165\ at github.com/pingcap-incubator/tiup-cluster/pkg/module.(*SystemdModule).Execute()\ \t/home/jenkins/agent/workspace/tiup-cluster-release/pkg/module/systemd.go:89\ at github.com/pingcap-incubator/tiup-cluster/pkg/operation.startInstance()\ \t/home/jenkins/agent/workspace/tiup-cluster-release/pkg/operation/action.go:451\ at github.com/pingcap-incubator/tiup-cluster/pkg/operation.StartComponent.func1()\ \t/home/jenkins/agent/workspace/tiup-cluster-release/pkg/operation/action.go:504\ at golang.org/x/sync/errgroup.(*Group).Go.func1()\ \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57\ at runtime.goexit()\ \t/usr/local/go/src/runtime/asm_amd64.s:1357\ failed to start: tikv 192.168.1.156:20172\ngithub.com/pingcap-incubator/tiup-cluster/pkg/operation.startInstance\ \t/home/jenkins/agent/workspace/tiup-cluster-release/pkg/operation/action.go:461\ github.com/pingcap-incubator/tiup-cluster/pkg/operation.StartComponent.func1\ \t/home/jenkins/agent/workspace/tiup-cluster-release/pkg/operation/action.go:504\ golang.org/x/sync/errgroup.(*Group).Go.func1\ \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57\ runtime.goexit\ \t/usr/local/go/src/runtime/asm_amd64.s:1357\ failed to start tikv\ failed to start”}
2020-06-17T00:40:28.804+0800 INFO Execute command finished {“code”: 1, “error”: “failed to start: failed to start tikv: failed to start: tikv 192.168.1.156:20172: executor.ssh.execute_timedout: Execute command over SSH timedout for ‘tidb@192.168.1.156:22’ {ssh_stderr: Run Command Timeout!\ , ssh_stdout: , ssh_command: PATH=$PATH:/usr/bin:/usr/sbin sudo -H -u root bash -c “systemctl daemon-reload && systemctl start tikv-20172.service && systemctl enable tikv-20172.service”}”, “errorVerbose”: “executor.ssh.execute_timedout: Execute command over SSH timedout for ‘tidb@192.168.1.156:22’ {ssh_stderr: Run Command Timeout!\ , ssh_stdout: , ssh_command: PATH=$PATH:/usr/bin:/usr/sbin sudo -H -u root bash -c “systemctl daemon-reload && systemctl start tikv-20172.service&& systemctl enable tikv-20172.service”}\ at github.com/pingcap-incubator/tiup-cluster/pkg/executor.(*SSHExecutor).Execute()\ \t/home/jenkins/agent/workspace/tiup-cluster-release/pkg/executor/ssh.go:165\ at github.com/pingcap-incubator/tiup-cluster/pkg/module.(*SystemdModule).Execute()\ \t/home/jenkins/agent/workspace/tiup-cluster-release/pkg/module/systemd.go:89\ at github.com/pingcap-incubator/tiup-cluster/pkg/operation.startInstance()\ \t/home/jenkins/agent/workspace/tiup-cluster-release/pkg/operation/action.go:451\ at github.com/pingcap-incubator/tiup-cluster/pkg/operation.StartComponent.func1()\ \t/home/jenkins/agent/workspace/tiup-cluster-release/pkg/operation/action.go:504\ at golang.org/x/sync/errgroup.(*Group).Go.func1()\ \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57\ at runtime.goexit()\ \t/usr/local/go/src/runtime/asm_amd64.s:1357\ failed to start: tikv 192.168.1.156:20172\ngithub.com/pingcap-incubator/tiup-cluster/pkg/operation.startInstance\ \t/home/jenkins/agent/workspace/tiup-cluster-release/pkg/operation/action.go:461\ github.com/pingcap-incubator/tiup-cluster/pkg/operation.StartComponent.func1\ \t/home/jenkins/agent/workspace/tiup-cluster-release/pkg/operation/action.go:504\ golang.org/x/sync/errgroup.(*Group).Go.func1\ \t/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57\ runtime.goexit\ \t/usr/local/go/src/runtime/asm_amd64.s:1357\ failed to start tikv\ failed to start”}

tiup version v1.0.0 tiup
Go Version: go1.13
Git Branch: master
GitHash: 5796405
Build Time: 2020-05-28 18:15:15 +0800

tidb版本 v4.0.0

你好,

可以将 --ssh-timeout 设置一个更长的时间。

可以使用 tiup update --self && tiup update --all 升级以下 tiup 及其组件。

–ssh-timeout 我设置过600s 但是实际上报错超时时间不受这个参数影响,并没有过去十分钟才报超时

tiup-cluster-debug-2020-06-16-23-51-43.log
tiup-cluster-debug-2020-06-16-23-56-20.log
tiup-cluster-debug-2020-06-17-00-06-59.log
tiup-cluster-debug-2020-06-17-00-11-26.log
tiup-cluster-debug-2020-06-17-00-16-41.log
tiup-cluster-debug-2020-06-17-00-22-29.log
tiup-cluster-debug-2020-06-17-00-26-42.log
tiup-cluster-debug-2020-06-17-00-32-43.log
tiup-cluster-debug-2020-06-17-00-40-28.log
tiup-cluster-debug-2020-06-17-00-48-42.log

我这边设置不同的超时时间,操作了10次,每次报错都是不同的tikv超时,麻烦能告知一下原理吗?为什么会超时,我测试环境没有这个问题

你好,

请按照提示先进行 tiup 的升级工作,然后再次 scale-out 看是否成功,否则将最新的 debug 日志上传一下,这边看下具体信息。

这个晚点操作看下

:ok_hand:

集群使用tiup操作,扩容缩容都会出现如下警告,然后显示error exit ,但是结果是成功的,请问这个是为什么呢?警告级别的错误为什么最后显示不是成功而是错误退出的结果?

2020-06-19T00:15:30.032+0800 DEBUG TaskFinish {“task”: “UpdateTopology: cluster=tidb-cluster”, “error”: “etcdserver: duplicate key given in txn request”}
2020-06-19T00:15:30.032+0800 INFO Execute command finished {“code”: 1, “error”: “etcdserver: duplicate key given in txn request”, “errorVerbose”: “etcdserver: duplicate key given in txn request\ngithub.com/pingcap/errors.AddStack\ \tgithub.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\ github.com/pingcap/errors.Trace\ \tgithub.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\ github.com/pingcap/tiup/components/cluster/command.scaleIn\ \tgithub.com/pingcap/tiup@/components/cluster/command/scale_in.go:151\ github.com/pingcap/tiup/components/cluster/command.newScaleInCmd.func1\ \tgithub.com/pingcap/tiup@/components/cluster/command/scale_in.go:56\ github.com/spf13/cobra.(*Command).execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:842\ github.com/spf13/cobra.(*Command).ExecuteC\ \tgithub.com/spf13/cobra@v1.0.0/command.go:950\ github.com/spf13/cobra.(*Command).Execute\ \tgithub.com/spf13/cobra@v1.0.0/command.go:887\ github.com/pingcap/tiup/components/cluster/command.Execute\ \tgithub.com/pingcap/tiup@/components/cluster/command/root.go:220\ main.main\ \tgithub.com/pingcap/tiup@/components/cluster/main.go:19\ runtime.main\ \truntime/proc.go:203\ runtime.goexit\ \truntime/asm_amd64.s:1357”}

在 tiup 低版本是个已知问题,请看下当前 tiup 版本,并上传下完整返回结果

tiup --version
tiup cluster --version

升级方式楼上有描述

tiup-cluster-debug-2020-06-19-00-09-22.log (419.9 KB) tiup-cluster-debug-2020-06-19-00-15-30.log (15.5 KB)

日志文件见附件

tiup是执行前更新到最新版本

tiup version v1.0.4 tiup
Go Version: go1.13
Git Branch: v1.0
GitHash: 59fb77b

用新版的tiup第一次执行还是会报run command timeout,
想起之前把tikv_gc_life_time时间修改了1h,所以先把这个参数恢复默认10m0s
再次执行扩容正常扩容了,报错和上面日志的缩容日志一样,不知道是否有关系

昨天发布的 1.0.5 哈,可以先不更新,不影响。

可以将 ssh-timeout 调整到 300,甚至更大

tiup cluster scale-out tidb-cluster scale-out_15960.yaml --ssh-timeout 6

应该没有必然联系,

请提供以下信息,便于排查,当前是否扩容成功

  1. scale-out_15960.yaml 文件内容
  2. pd-ctl store/member 信息,看下当前节点状态,反馈下 display 信息,确认下与 pd-ctl 是否对应

这个就是扩容的yaml信息

display信息

store&member信息 (17.0 KB)

store和member信息

你好,

从 pd-ctl 和 display 信息来看,当前集群状态时正常的,tiup 扩容这个报错可以先忽略。

对于该报错我们会查下是什么原因,如果有信息需要反馈我们会继续跟帖

好的,感谢

:wink: