执行升级后报如下错误
tiup cluster upgrade tidb-test v4.0.0
Restarting component tiflash
Restarting instance 172.18.181.6
retry error: operation timed out after 1m0s
172.18.181.6 failed to restart: timed out waiting for port 9000 to be started after 1m0s
Error: failed to upgrade: failed to restart tiflash: 192.168.181.6 failed to restart: timed out waiting for port 9000 to be started after 1m0s: timed out waiting for port 9000 to be started after 1m0s
Verbose debug logs has been written to /home/tidb/logs/tiup-cluster-debug-2020-06-29-20-26-05.log.
Error: run `/home/tidb/.tiup/components/cluster/v1.0.7/tiup-cluster` (wd:/home/tidb/.tiup/data/S3IqRWN) failed: exit status 1
tiflash所在机器查看日志报错如下:
[tidb@dev7 tidb-deploy]$ cd tiflash-9000/
[tidb@dev7 tiflash-9000]$ ll
总用量 4
drwxr-xr-x 3 tidb tidb 21 6月 29 20:24 bin
drwxr-xr-x 3 tidb tidb 21 6月 29 20:24 bin.old.v4.0.0-rc
drwxr-xr-x 2 tidb tidb 87 6月 29 21:31 conf
drwxr-xr-x 2 tidb tidb 4096 6月 29 20:24 log
drwxr-xr-x 2 tidb tidb 28 6月 23 03:56 scripts
[tidb@dev7 tiflash-9000]$ pwd
/tidb-deploy/tiflash-9000
[tidb@dev7 tiflash-9000]$ ls /log
tiflash_cluster_manager.log tiflash.log.11 tiflash.log.2 tiflash.log.8
tiflash_error.log tiflash.log.12 tiflash.log.3 tiflash.log.9
tiflash.log tiflash.log.13 tiflash.log.4 tiflash_tikv.log
tiflash.log.0 tiflash.log.14 tiflash.log.5 tiflash_tikv.log.2020-06-29-12:24:56
tiflash.log.1 tiflash.log.15 tiflash.log.6
tiflash.log.10 tiflash.log.16 tiflash.log.7
[tidb@dev7 tiflash-9000]$ tail -1000f tiflash_error.log
tail: 无法打开"tiflash_error.log" 读取数据: 没有那个文件或目录
tail: 没有剩余文件
[tidb@dev7 tiflash-9000]$ tail -1000f log/tiflash_error.log
2020.06.23 04:03:38.491313 [ 7 ] <Warning> pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 77 store_id: 5)
2020.06.23 04:04:11.867121 [ 17 ] <Warning> pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 7 store_id: 1)
2020.06.23 04:04:40.480288 [ 17 ] <Warning> pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 77 store_id: 5)
2020.06.23 04:06:57.043714 [ 17 ] <Warning> pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 7 store_id: 1)
2020.06.23 04:17:02.641782 [ 5 ] <Warning> pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 77 store_id: 5)
2020.06.23 04:17:18.717236 [ 5 ] <Warning> pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 7 store_id: 1)
2020.06.23 04:17:36.378698 [ 5 ] <Warning> pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 77 store_id: 5)
2020.06.23 04:19:51.718644 [ 5 ] <Warning> pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 7 store_id: 1)
2020.06.23 04:23:20.400907 [ 5 ] <Warning> pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 77 store_id: 5)
2020.06.23 04:35:59.248611 [ 11 ] <Warning> pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 7 store_id: 1)
2020.06.23 04:41:09.146777 [ 17 ] <Warning> pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 46 store_id: 4)
2020.06.23 06:31:10.635586 [ 5 ] <Warning> pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 7 store_id: 1)
2020.06.23 08:12:14.881873 [ 20 ] <Warning> pingcap.tikv: region {6,5,2} find error: EpochNotMatch current epoch of region 6 is conf_ver: 5 version: 3, but you sent conf_ver: 5 version: 2
2020.06.24 14:27:05.928733 [ 9 ] <Error> pingcap.tikv: Get Failed4: Deadline Exceeded
2020.06.24 14:27:11.022742 [ 9 ] <Error> pingcap.tikv: Get Failed4: Deadline Exceeded
2020.06.24 14:27:14.024087 [ 9 ] <Error> pingcap.tikv: Get Failed14: Connection reset by peer
2020.06.24 14:27:14.332150 [ 9 ] <Error> pingcap.tikv: Get Failed14: failed to connect to all addresses
2020.06.24 14:27:15.332264 [ 9 ] <Error> pingcap.tikv: Get Failed14: failed to connect to all addresses
2020.06.24 14:27:16.756154 [ 9 ] <Error> pingcap.tikv: Get Failed14: failed to connect to all addresses
2020.06.24 14:27:19.813190 [ 9 ] <Error> pingcap.tikv: Get Failed14: failed to connect to all addresses
2020.06.24 14:27:24.022110 [ 9 ] <Error> pingcap.tikv: Get Failed14: failed to connect to all addresses
2020.06.24 14:27:25.869636 [ 9 ] <Error> pingcap.tikv: Get Failed14: failed to connect to all addresses
2020.06.24 14:27:27.580239 [ 9 ] <Error> pingcap.tikv: Get Failed14: failed to connect to all addresses
2020.06.24 14:27:29.662999 [ 9 ] <Error> pingcap.tikv: Get Failed14: failed to connect to all addresses
2020.06.24 14:27:31.983739 [ 9 ] <Error> pingcap.tikv: Get Failed14: failed to connect to all addresses
2020.06.24 14:27:33.821525 [ 9 ] <Error> pingcap.tikv: Get Failed14: failed to connect to all addresses
2020.06.24 14:27:36.055393 [ 9 ] <Error> pingcap.tikv: Get Failed14: failed to connect to all addresses
2020.06.24 14:27:37.711725 [ 9 ] <Error> pingcap.tikv: Get Failed14: failed to connect to all addresses
2020.06.24 14:27:40.205996 [ 9 ] <Warning> SchemaSyncService: Schema sync failed by Exception: Get Failed14: failed to connect to all addresses
2020.06.24 14:29:45.775965 [ 4 ] <Warning> pingcap.tikv: region {99561,5,3} find error: peer is not leader for region 99561, leader may Some(id: 99564 store_id: 5)