我用tiup升级时（由v4.0.0-rc.2升级到v4.0.0）tiflash 报 operation timed out after 1m0s

🪐 TiDB 技术问题
1

我环境是：单机使用tiup部署的v4.0.0-rc.2
现在是升级到v4.0.0时报错，如下：
Restarting component tidb
Restarting instance 10.103.236.178
Restart 10.103.236.178 success
Restarting component tiflash
Restarting instance 10.103.236.178
retry error: operation timed out after 1m0s
10.103.236.178 failed to restart: timed out waiting for port 9000 to be started after 1m0s

Error: failed to upgrade: failed to restart tiflash: 10.103.236.178 failed to restart: timed out waiting for port 9000 to be started after 1m0s: timed out waiting for port 9000 to be started after 1m0s

Verbose debug logs has been written to /root/logs/tiup-cluster-debug-2020-05-29-15-00-48.log.
Error: run /root/.tiup/components/cluster/v0.6.2/cluster (wd:/root/.tiup/data/S0MG7DU) failed: exit status 1

tiflash.zip (645.3 KB)

2

我们看下日志会尽快答复，感谢反馈

3

请问是否有在 TiDB 中对这个表 daas_ops.assets 执行过把某一列从 int modify 为 tinyint 的操作吗？
可以尝试把该列重新 modify column 设置为 int，然后再重启 tiflash 试试

4

通过tiup cluster display tidb_xiqu_test命令查看是v4.0.0-rc.2 [root@k8s-master02 ~]# tiup cluster display tidb_xiqu_test Starting component cluster: /root/.tiup/components/cluster/v0.6.2/cluster display tidb_xiqu_test TiDB Cluster: tidb_xiqu_test TiDB Version: v4.0.0-rc.2

但是通过sql来查看却是v4.0.0 mysql [daas_ops]> select tidb_version(); ±-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | tidb_version() | ±-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Release Version: v4.0.0 Edition: Community Git Commit Hash: 689a6b6439ae7835947fcaccf329a3fc303986cb Git Branch: heads/refs/tags/v4.0.0 UTC Build Time: 2020-05-28 01:37:40 GoVersion: go1.13 Race Enabled: false TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306 Check Table Before Drop: false | ±-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.01 sec)

mysql [daas_ops]> select * from information_schema.cluster_info; ±--------±---------------------±---------------------±-------------------±-----------------------------------------±--------------------------±-----------------+ | TYPE | INSTANCE | STATUS_ADDRESS | VERSION | GIT_HASH | START_TIME | UPTIME | ±--------±---------------------±---------------------±-------------------±-----------------------------------------±--------------------------±-----------------+ | tidb | 10.xx.xx.xx:4000 | 10.xx.xx.xx:10080 | 5.7.25-TiDB-v4.0.0 | 689a6b6439ae7835947fcaccf329a3fc303986cb | 2020-05-29T16:00:50+08:00 | 46m29.296732506s | | pd | 10.xx.xx.xx:3379 | 10.xx.xx.xx:3379 | 4.0.0 | 56d4c3d2237f5bf6fb11a794731ed1d95c8020c2 | 2020-05-29T15:59:29+08:00 | 47m50.296744696s | | tikv | 10.xx.xx.xx:20162 | 10.xx.xx.xx:20182 | 4.0.0 | 198a2cea01734ce8f46d55a29708f123f9133944 | 2020-05-29T16:00:49+08:00 | 46m30.296750106s | | tikv | 10.xx.xx.xx:20160 | 10.xx.xx.xx:20180 | 4.0.0 | 198a2cea01734ce8f46d55a29708f123f9133944 | 2020-05-29T16:00:40+08:00 | 46m39.296753879s | | tikv | 10.xx.xx.xx:20161 | 10.xx.xx.xx:20181 | 4.0.0 | 198a2cea01734ce8f46d55a29708f123f9133944 | 2020-05-29T16:00:45+08:00 | 46m34.296757006s | | tiflash | 10.xx.xx.xx:3930 | 10.xx.xx.xx:20292 | v4.0.0 | c51c2c5c18860aaef3b5853f24f8e9cefea167eb | 2020-05-29T16:04:07+08:00 | 43m12.296760476s | ±--------±---------------------±---------------------±-------------------±-----------------------------------------±--------------------------±-----------------+ 6 rows in set (0.02 sec)

通过命令tiflash来查看，发现已经有新版本的tiflash [root@k8s-master02 tiflash-9000]# ll total 1211128 drwxrwxr-x 3 tidb2 tidb2 28 May 29 16:52 bin drwxr-xr-x 3 tidb2 tidb2 28 May 25 12:27 bin.old.v4.0.0-rc.2 drwxr-xr-x 2 tidb2 tidb2 100 May 29 14:06 conf -rw------- 1 tidb2 tidb2 2323374080 May 29 14:14 core.31595 drwxr-xr-x 2 tidb2 tidb2 4096 May 29 16:47 log drwxr-xr-x 2 tidb2 tidb2 35 May 25 12:27 scripts [root@k8s-master02 tiflash-9000]# [root@k8s-master02 tiflash-9000]# bin/tiflash/tiflash --version bin/tiflash/tiflash: error while loading shared libraries: libtiflash_proxy.so: cannot open shared object file: No such file or directory [root@k8s-master02 tiflash-9000]# [root@k8s-master02 tiflash-9000]# LD_LIBRARY_PATH=bin/tiflash/ bin/tiflash/tiflash --version TiFlash 4.0.0 Release Version: v4.0.0 Git Commit Hash: c51c2c5c18860aaef3b5853f24f8e9cefea167eb Git Branch: heads/refs/tags/v4.0.0 UTC Build Time: 2020-05-28 02:57:47

Raft Proxy Git Commit Hash: a1192bd198f42cd3921129b0ecceffa9a386588e Git Commit Branch: HEAD UTC Build Time: 2020-05-28 02:26:54 Rust Version: rustc 1.42.0-nightly (0de96d37f 2019-12-19) [root@k8s-master02 tiflash-9000]# [root@k8s-master02 tiflash-9000]# LD_LIBRARY_PATH=bin.old.v4.0.0-rc.2/tiflash/ bin.old.v4.0.0-rc.2/tiflash/tiflash --version TiFlash 4.0.0 Release Version: v4.0.0-rc.2 Git Commit Hash: 09bd9e6b9a271b1fcd25c676083104a97f18739a Git Branch: heads/refs/tags/v4.0.0-rc.2 UTC Build Time: 2020-05-11 10:46:33

Raft Proxy Git Commit Hash: 146b2b417dd15b9ad6ae877fca3e0469b3a91829 Git Commit Branch: HEAD UTC Build Time: 2020-05-11 10:11:09 Rust Version: rustc 1.42.0-nightly (0de96d37f 2019-12-19)

5

再次执行升级后，产生的相关日志tiflash_2.zip (2.6 MB)

6

收到，帮忙确认下在升级过程中是否有 drop table 操作。

7

开发人员到现在还没给我反馈，后面我跟他们确认后，再来回复一次。抱歉

8

好的。