启动集群报这个错Error: failed to start tikv: failed to start: 172.24.0.119 tikv-20160.service, please check the instance’s log(/data/tidb-deploy/tikv-20160/log) for more detail.: timed out waiting for port 20160 to be started after 2m0s,集群启动不起来
但是tikv日志里面没有有用报错信息
tikv.log (38.9 KB)
tikv启动时间长
根据日志，建议检查一下磁盘和内存情况。
另外，报错和截图好像不太一致，172.24.0.119这个tikv是启动的，但是报错显示没有启动成功
磁盘和内存情况是正常的,单个tikv在挂载2000G的节点上部署的,就会出现tikv启动很慢,tiup启动失败后,这tikv自己又成功启动起来了
你可以检查一下，失败之后到自动启动成功的这段时间内，tikv的日志，看看是不是一直在输出什么东西，而且没有报错
没有启动起来的tikv没有报error,从前面我提供的日志可以看到,但是其他的节点报了 [2021/11/26 11:19:44.162 +08:00] [ERROR] [raft_client.rs:407] [“connection aborted”] [addr=172.24.0.119:20160] [receiver_err=“Some(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: “failed to connect to all addresses”, details: [] }))”] [sink_error=“Some(RpcFinished(Some(RpcStatus { code: 14-UNAVAILABLE, message: “failed to connect to all addresses”, details: [] })))”] [store_id=5]
初步怀疑是启动tikv的时候，tikv加载 block cache时间太长，所以tiup提示超时了。其实tikv还在加载，等加载完了，就成功启动了。加载时间长可能就是磁盘I/O的问题
嗯嗯 ,那我怎么确定呢,因为tikv在1000G内,是不出现这个问题的,我怎么确定是什么原因呢? 是磁盘I/O的问题还是tikv启动的问题,所以我新开了一个贴https://asktug.com/t/topic/273316
启动很慢的tikv的日志中:[2021/11/26 17:00:48.838 +08:00] [INFO] [server.rs:300] [“using config”] [config="{“log-level”:“info”,“log-file”:"/data/tidb-deploy/tikv-20160/log/tikv.log",“log-format”:“text”,“slow-log-file”:"",“slow-log-threshold”:“1s”,“log-rotation-timespan”:“1d”,“log-rotation-size”:“300MiB”,“panic-when-unexpected-key-or-data”:false,“enable-io-snoop”:true,“abort-on-panic”:false,“memory-usage-limit”:“34359738KiB”,“memory-usage-high-water”:0.9,“readpool”:{“unified”:{“min-thread-count”:8,“max-thread-count”:16,“stack-size”:“10MiB”,“max-tasks-per-worker”:2000},“storage”:{“use-unified-pool”:false,“high-concurrency”:8,“normal-concurrency”:12,“low-concurrency”:8,“max-tasks-per-worker-high”:2000,“max-tasks-per-worker-normal”:2000,“max-tasks-per-worker-low”:2000,“stack-size”:“10MiB”},“coprocessor”:{“use-unified-pool”:true,“high-concurrency”:14,“normal-concurrency”:14,“low-concurrency”:14,“max-tasks-per-worker-high”:2000,“max-tasks-per-worker-normal”:2000,“max-tasks-per-worker-low”:2000,“stack-size”:“10MiB”}},“server”:{“addr”:“0.0.0.0:20160”,“advertise-addr”:“172.24.0.119:20160”,“status-addr”:“0.0.0.0:20180”,“advertise-status-addr”:“172.24.0.119:20180”,“status-thread-pool-size”:1,“max-grpc-send-msg-len”:10485760,“raft-client-grpc-send-msg-buffer”:524288,“raft-client-queue-size”:8192,“raft-msg-max-batch-size”:128,“grpc-compression-type”:“none”,“grpc-concurrency”:8,“grpc-concurrent-stream”:1024,“grpc-raft-conn-num”:1,“grpc-memory-pool-quota”:9223372036854775807,“grpc-stream-initial-window-size”:“2MiB”,“grpc-keepalive-time”:“10s”,“grpc-keepalive-timeout”:“3s”,“concurrent-send-snap-limit”:32,“concurrent-recv-snap-limit”:32,“end-point-recursion-limit”:1000,“end-point-stream-channel-size”:8,“end-point-batch-row-limit”:64,“end-point-stream-batch-row-limit”:128,“end-point-enable-batch-if-possible”:true,“end-point-request-max-handle-duration”:“1m”,“end-point-max-concurrency”:18,“snap-max-write-bytes-per-sec”:“100MiB”,“snap-max-total-size”:“0KiB”,“stats-concurrency”:1,“heavy-load-threshold”:300,“heavy-load-wait-duration”:“1ms”,“enable-request-batch”:false,“background-thread-count”:3,“end-point-slow-log-threshold”:“1s”,“forward-max-connections-per-address”:4,“labels”:{}},“storage”:{“data-dir”:"/data/tidb-data/tikv-20160",“gc-ratio-threshold”:1.1,“max-key-size”:4096,“scheduler-concurrency”:524288,“scheduler-worker-pool-size”:8,“scheduler-pending-write-threshold”:“100MiB”,“reserve-space”:“5GiB”,“enable-async-apply-prewrite”:false,“enable-ttl”:false,“ttl-check-poll-interval”:“12h”,“block-cache”:{“shared”:true,“capacity”:“21151MiB”,“num-shard-bits”:6,“strict-capacity-limit”:false,“high-pri-pool-ratio”:0.8,“memory-allocator”:“nodump”},“io-rate-limit”:{“max-bytes-per-sec”:“0KiB”,“mode”:“write-only”,“strict”:false,“foreground-read-priority”:“high”,“foreground-write-priority”:“high”,“flush-priority”:“high”,“level-zero-compaction-priority”:“medium”,“compaction-priority”:“low”,“replication-priority”:“high”,“load-balance-priority”:“high”,“gc-priority”:“high”,“import-priority”:“medium”,“export-priority”:“medium”,“other-priority”:“high”}},“pd”:{“endpoints”:[“172.24.0.118:2379”,“172.24.0.116:2379”,“172.24.0.114:2379”],“retry-interval”:“300ms”,“retry-max-count”:9223372036854775807,“retry-log-every”:10,“update-interval”:“10m”,“enable-forwarding”:false},“metric”:{“job”:“tikv”},“raftstore”:{“prevote”:true,“raftdb-path”:"/data/tidb-data/tikv-20160/raft",“capacity”:“0KiB”,“raft-base-tick-interval”:“1s”,“raft-heartbeat-ticks”:2,“raft-election-timeout-ticks”:10,“raft-min-election-timeout-ticks”:10,“raft-max-election-timeout-ticks”:20,“raft-max-size-per-msg”:“1MiB”,“raft-max-inflight-msgs”:256,“raft-entry-max-size”:“8MiB”,“raft-log-gc-tick-interval”:“10s”,“raft-log-gc-threshold”:50,“raft-log-gc-count-limit”:73728,“raft-log-gc-size-limit”:“72MiB”,“raft-log-reserve-max-ticks”:6,“raft-engine-purge-interval”:“10s”,“raft-entry-cache-life-time”:“30s”,“raft-reject-transfer-leader-duration”:“3s”,“split-region-check-tick-interval”:“10s”,“region-split-check-diff”:“6MiB”,“region-compact-check-interval”:“5m”,“region-compact-check-step”:100,“region-compact-min-tombstones”:10000,“region-compact-tombstones-percent”:30,“pd-heartbeat-tick-interval”:“1m”,“pd-store-heartbeat-tick-interval”:“10s”,“snap-mgr-gc-tick-interval”:“1m”,“snap-gc-timeout”:“4h”,“lock-cf-compact-interval”:“10m”,“lock-cf-compact-bytes-threshold”:“256MiB”,“notify-capacity”:40960,“messages-per-tick”:4096,“max-peer-down-duration”:“5m”,“max-leader-missing-duration”:“2h”,“abnormal-leader-missing-duration”:“10m”,“peer-stale-state-check-interval”:“5m”,“leader-transfer-max-log-lag”:128,“snap-apply-batch-size”:“10MiB”,“consistency-check-interval”:“0s”,“report-region-flow-interval”:“1m”,“raft-store-max-leader-lease”:“9s”,“right-derive-when-split”:true,“allow-remove-leader”:false,“merge-max-log-gap”:10,“merge-check-tick-interval”:“2s”,“use-delete-range”:false,“cleanup-import-sst-interval”:“10m”,“local-read-batch-size”:1024,“apply-max-batch-size”:256,“apply-pool-size”:5,“apply-reschedule-duration”:“5s”,“apply-low-priority-pool-size”:1,“store-max-batch-size”:256,“store-pool-size”:5,“store-reschedule-duration”:“5s”,“store-low-priority-pool-size”:0,“future-poll-size”:1,“hibernate-regions”:true,“dev-assert”:false,“apply-yield-duration”:“500ms”,“perf-level”:1},“coprocessor”:{“split-region-on-table”:false,“batch-split-limit”:10,“region-max-size”:“144MiB”,“region-split-size”:“96MiB”,“region-max-keys”:1440000,“region-split-keys”:960000,“consistency-check-method”:“mvcc”,“perf-level”:2},“coprocessor-v2”:{“coprocessor-plugin-directory”:null},“rocksdb”:{“info-log-level”:“info”,“wal-recovery-mode”:2,“wal-dir”:"",“wal-ttl-seconds”:0,“wal-size-limit”:“0KiB”,“max-total-wal-size”:“4GiB”,“max-background-jobs”:12,“max-background-flushes”:4,“max-manifest-file-size”:“128MiB”,“create-if-missing”:true,“max-open-files”:40960,“enable-statistics”:true,“stats-dump-period”:“10m”,“compaction-readahead-size”:“0KiB”,“info-log-max-size”:“1GiB”,“info-log-roll-time”:“0s”,“info-log-keep-log-file-num”:10,“info-log-dir”:"",“rate-bytes-per-sec”:“10GiB”,“rate-limiter-refill-period”:“100ms”,“rate-limiter-mode”:2,“rate-limiter-auto-tuned”:true,“bytes-per-sync”:“1MiB”,“wal-bytes-per-sync”:“512KiB”,“max-sub-compactions”:3,“writable-file-max-buffer-size”:“1MiB”,“use-direct-io-for-flush-and-compaction”:false,“enable-pipelined-write”:true,“enable-multi-batch-write”:true,“enable-unordered-write”:false,“defaultcf”:{“block-size”:“64KiB”,“block-cache-size”:“12GiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:true,“optimize-filters-for-hits”:true,“whole-key-filtering”:true,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“zstd”,“zstd”,“zstd”,“zstd”,“zstd”],“write-buffer-size”:“128MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“512MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:4,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:3,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“192GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:false,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“enable-compaction-guard”:true,“compaction-guard-min-output-file-size”:“8MiB”,“compaction-guard-max-output-file-size”:“128MiB”,“bottommost-level-compression”:“zstd”,“bottommost-zstd-compression-dict-size”:0,“bottommost-zstd-compression-sample-size”:0,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“normal”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“writecf”:{“block-size”:“64KiB”,“block-cache-size”:“6GiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:true,“optimize-filters-for-hits”:false,“whole-key-filtering”:false,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“lz4”,“lz4”,“lz4”,“zstd”,“zstd”],“write-buffer-size”:“128MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“512MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:4,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:3,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“192GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:false,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“enable-compaction-guard”:true,“compaction-guard-min-output-file-size”:“8MiB”,“compaction-guard-max-output-file-size”:“128MiB”,“bottommost-level-compression”:“zstd”,“bottommost-zstd-compression-dict-size”:0,“bottommost-zstd-compression-sample-size”:0,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“read-only”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“lockcf”:{“block-size”:“16KiB”,“block-cache-size”:“671MiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:true,“optimize-filters-for-hits”:false,“whole-key-filtering”:true,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“no”,“no”,“no”,“no”,“no”],“write-buffer-size”:“32MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“128MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:1,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:0,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“192GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:false,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“enable-compaction-guard”:false,“compaction-guard-min-output-file-size”:“8MiB”,“compaction-guard-max-output-file-size”:“128MiB”,“bottommost-level-compression”:“disable”,“bottommost-zstd-compression-dict-size”:0,“bottommost-zstd-compression-sample-size”:0,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“read-only”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“raftcf”:{“block-size”:“16KiB”,“block-cache-size”:“128MiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:true,“optimize-filters-for-hits”:true,“whole-key-filtering”:true,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“no”,“no”,“no”,“no”,“no”],“write-buffer-size”:“128MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“128MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:1,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:0,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“192GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:false,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“enable-compaction-guard”:false,“compaction-guard-min-output-file-size”:“8MiB”,“compaction-guard-max-output-file-size”:“128MiB”,“bottommost-level-compression”:“disable”,“bottommost-zstd-compression-dict-size”:0,“bottommost-zstd-compression-sample-size”:0,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“read-only”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“titan”:{“enabled”:false,“dirname”:"",“disable-gc”:false,“max-background-gc”:4,“purge-obsolete-files-period”:“10s”}},“raftdb”:{“wal-recovery-mode”:2,“wal-dir”:"",“wal-ttl-seconds”:0,“wal-size-limit”:“0KiB”,“max-total-wal-size”:“4GiB”,“max-background-jobs”:12,“max-background-flushes”:1,“max-manifest-file-size”:“20MiB”,“create-if-missing”:true,“max-open-files”:40960,“enable-statistics”:true,“stats-dump-period”:“10m”,“compaction-readahead-size”:“0KiB”,“info-log-max-size”:“1GiB”,“info-log-roll-time”:“0s”,“info-log-keep-log-file-num”:10,“info-log-dir”:"",“info-log-level”:“info”,“max-sub-compactions”:2,“writable-file-max-buffer-size”:“1MiB”,“use-direct-io-for-flush-and-compaction”:false,“enable-pipelined-write”:true,“enable-unordered-write”:false,“allow-concurrent-memtable-write”:true,“bytes-per-sync”:“1MiB”,“wal-bytes-per-sync”:“512KiB”,“defaultcf”:{“block-size”:“64KiB”,“block-cache-size”:“2GiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:false,“optimize-filters-for-hits”:true,“whole-key-filtering”:true,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“zstd”,“zstd”,“zstd”,“zstd”,“zstd”],“write-buffer-size”:“128MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“512MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:4,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:0,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“192GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:false,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“enable-compaction-guard”:false,“compaction-guard-min-output-file-size”:“8MiB”,“compaction-guard-max-output-file-size”:“128MiB”,“bottommost-level-compression”:“disable”,“bottommost-zstd-compression-dict-size”:0,“bottommost-zstd-compression-sample-size”:0,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“normal”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“titan”:{“enabled”:false,“dirname”:"",“disable-gc”:false,“max-background-gc”:4,“purge-obsolete-files-period”:“10s”}},“raft-engine”:{“enable”:false,“dir”:"/data/tidb-data/tikv-20160/raft-engine",“recovery-mode”:“tolerate-corrupted-tail-records”,“bytes-per-sync”:“256KiB”,“target-file-size”:“128MiB”,“purge-threshold”:“10GiB”,“cache-limit”:“1GiB”,“batch-compression-threshold”:“8KiB”},“security”:{“ca-path”:"",“cert-path”:"",“key-path”:"",“cert-allowed-cn”:[],“redact-info-log”:null,“encryption”:{“data-encryption-method”:“plaintext”,“data-key-rotation-period”:“7d”,“enable-file-dictionary-log”:true,“file-dictionary-rewrite-threshold”:1000000,“master-key”:{“type”:“plaintext”},“previous-master-key”:{“type”:“plaintext”}}},“import”:{“num-threads”:8,“stream-channel-window”:128,“import-mode-timeout”:“10m”},“backup”:{“num-threads”:13,“batch-size”:8,“sst-max-size”:“144MiB”},“pessimistic-txn”:{“wait-for-lock-timeout”:“1s”,“wake-up-delay-duration”:“20ms”,“pipelined”:true},“gc”:{“ratio-threshold”:1.1,“batch-keys”:512,“max-write-bytes-per-sec”:“0KiB”,“enable-compaction-filter”:true,“compaction-filter-skip-version-check”:false},“split”:{“qps-threshold”:3000,“split-balance-score”:0.25,“split-contained-score”:0.5,“detect-times”:10,“sample-num”:20,“sample-threshold”:100,“byte-threshold”:31457280},“cdc”:{“min-ts-interval”:“1s”,“hibernate-regions-compatible”:true,“incremental-scan-threads”:4,“incremental-scan-concurrency”:6,“incremental-scan-speed-limit”:“128MiB”,“sink-memory-quota”:“512MiB”,“old-value-cache-memory-quota”:“512MiB”},“resolved-ts”:{“enable”:true,“advance-ts-interval”:“1s”,“scan-lock-pool-size”:2},“resource-metering”:{“enabled”:false,“agent-address”:"",“report-agent-interval”:“1m”,“max-resource-groups”:2000,“precision”:“1s”}}"]
[2021/11/26 17:04:00.283 +08:00] [INFO] [mod.rs:118] [“encryption: none of key dictionary and file dictionary are found.”]
在tikv的启动过程,这两个输出日志间隔就是4分钟以上,这4分钟tikv启动服务在干什么呢?为什么和磁盘大小有关系呢?磁盘越大.tikv启动时间就越长
tikv启动的时候会去日志中加载之前在内存当中的缓存文件，相当于是读文件加载到内存，对磁盘要求比较大，所以你可以检查一下这个时间段的磁盘I/O，如果使用率非常大，可能就是这个原因了
我也遇到这样的问题，应该怎么解决呢
增加启动超时时间 tiup cluster start ${TIDB_CLUSTER_NAME} --wait-timeout 600；
出现这样的问题，直接手动启动服务即可，通过tiup查看状态就恢复了，再次reload 就正常了，如果是滚动升级 遇到这样还需要注意是否每个版本都升级成功了，不成功需要重新滚起来
，TIUP还是有待改进
是的 重启也是一种方式解决的方式
太难了，历时近2天，最后手动一个个拉起来
tiup cluster start ${TIDB_CLUSTER_NAME} --wait-timeout 600； 加超时时间呀,有试过吗?
你说的没有试过，这是超时10分钟吗？配置文件里有配置超时，tikv-client.max-batch-wait-time: 2000000
运维过程中涉及到很多操作：指定 systemctl 启动/停止服务，等待端口上线/下线等，每个操作可能会消耗数秒。 --wait-timeout 用于设置每个步骤的最长等待时间（单位为秒），超时后报错退出
你说的tikv-client.max-batch-wait-time意思是:
max-batch-wait-time 纳秒批量将此期间的数据包封装成一个大包发送给 TiKV 节点，仅在 tikv-client.max-batch-size 值大于 0 时有效，不推荐修改该值。此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。