生产环境 tiup 升级v4.0.2 到v4.0.4 tiflash 等待超时 timed out waiting for port 9000 to be started after 2m0s

🪐 TiDB 技术问题 TiFlash
2

  1. debug 日志：
    2020-08-20T01:35:19.560+0800 DEBUG TaskFinish {“task”: “UpgradeCluster”, “error”: “\t192.168.254.12 failed to restart: timed out waiting for port 9000 to be started after 2m0s: timed out waiting for port 9000 to be started after 2m0s”, “errorVerbose”: "timed out waiting for port 9000 to be started after

  2. tiflahs 日志：
    2020.08.20 01:33:26.561529 [ 1 ] DeltaMergeStore[db_45.t_3062]: Release DeltaMerge Store start [db_45.t_3062]
    2020.08.20 01:33:26.561537 [ 1 ] DeltaMergeStore[db_45.t_3062]: Release DeltaMerge Store end [db_45.t_3062]
    2020.08.20 01:33:28.130962 [ 1 ] Application: Let tiflash proxy shutdown
    2020.08.20 01:33:28.130994 [ 1 ] Application: Wait for tiflash proxy thread to join

  3. tiflash_error日志的时间是问题发生前的
    2020.08.20 00:31:38.310447 [ 23 ] pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 47 store_id: 4)
    2020.08.20 00:31:59.289970 [ 13 ] pingcap.tikv: region {6,5,2} find error: peer is not leader for region 6, leader may Some(id: 7 store_id: 1)

  4. 稍等我们再分析下