【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】
【复现路径】做过哪些操作出现的问题
【遇到的问题：问题现象及影响】
encryption: none of key dictionary and file dictionary are found."
【资源配置】*进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面服务器四的节点起不来
日志里指定check的log文件里有什么可用的信息么？
500G磁盘？另外两个都是50TB，这个相差确实有点大。
日志里没有error级别的日志么？
就是87站点的tikv.log文件。如果没有敏感信息的话，可以发出来大家一起看看。
哦，忽然想到还有一种可能，就是这台服务器资源不足，导致启动慢，超过2分钟，所以报错超时了，但实际还在启动，可以过10分钟display一下，看看集群状态是不是都启动了。我们之前遇到过这种情况，也是日志里一点信息没有，然后一直重启。最后还是因为有人过来讨论问题，讨论用了十几分钟，讨论完一看，虽然报错了，但是集群状态显示都启动了。
最后一行提示welcome to tikv，这句意思应该就是成功启动了。display集群看看，是不是状态是已启动？
没有成功，这个最后一行是开始启动的意思最上边也是这句
循环出现的话，那估计就是资源不足导致重启了。可以看看服务器的资源是不是打满了。包括cpu、内存、硬盘空间。
单独看资源很充足，一点都没有，不知道是不是资源不平衡影响的
日志文本能贴出来吗，图片看不清楚
[2025/01/22 16:08:30.527 +08:00] [INFO] [lib.rs:91] [“Welcome to TiKV”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [lib.rs:96] [“Release Version: 8.5.1”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [lib.rs:96] [“Edition: Community”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [lib.rs:96] [“Git Commit Hash: 25897b73e1f0e93f616c764dbe65ed7db02edd07”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [lib.rs:96] [“Git Commit Branch: HEAD”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [lib.rs:96] [“UTC Build Time: 2025-01-14 05:02:29”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [lib.rs:96] [“Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [lib.rs:96] [“Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [lib.rs:96] [“Profile: dist_release”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [fips.rs:40] [“OpenSSL FIPS mode is disabled”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [mod.rs:125] [“cgroup quota: memory=None, cpu=None, cores={}”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [mod.rs:132] [“memory limit in bytes: 50332991488, cpu cores quota: 16”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [WARN] [lib.rs:526] [“environment variable TZ is missing, using /etc/localtime”] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [config.rs:914] [“kernel parameters”] [value=32768] [param=net.core.somaxconn] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [config.rs:914] [“kernel parameters”] [value=0] [param=net.ipv4.tcp_syncookies] [thread_id=1]
[2025/01/22 16:08:30.527 +08:00] [INFO] [config.rs:914] [“kernel parameters”] [value=0] [param=vm.swappiness] [thread_id=1]
[2025/01/22 16:08:30.530 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=10.212.200.86:2379] [thread_id=1]
[2025/01/22 16:08:30.530 +08:00] [INFO] [] [“TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter”] [thread_id=1]
[2025/01/22 16:08:30.533 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://10.212.200.86:2379] [thread_id=1]
[2025/01/22 16:08:30.535 +08:00] [INFO] [util.rs:639] [“connecting to PD endpoint”] [endpoints=http://10.212.200.86:2379] [thread_id=1]
[2025/01/22 16:08:30.537 +08:00] [INFO] [util.rs:809] [“connected to PD member”] [endpoints=http://10.212.200.86:2379] [thread_id=1]
[2025/01/22 16:08:30.537 +08:00] [INFO] [util.rs:631] [“all PD endpoints are consistent”] [endpoints=“["10.212.200.86:2379"]”] [thread_id=1]
[2025/01/22 16:08:30.537 +08:00] [INFO] [common.rs:327] [“connect to PD cluster”] [cluster_id=7462611306883308244] [thread_id=1]
[2025/01/22 16:08:30.542 +08:00] [INFO] [config.rs:403] [“using default coprocessor quota”] [quota=ReadableSize(6291623936)] [thread_id=1]
[2025/01/22 16:08:30.542 +08:00] [INFO] [mod.rs:2599] [“readpool.storage.use-unified-pool is not set, set to true by default”] [thread_id=1]
[2025/01/22 16:08:30.542 +08:00] [INFO] [mod.rs:2622] [“readpool.coprocessor.use-unified-pool is not set, set to true by default”] [thread_id=1]
[2025/01/22 16:08:30.542 +08:00] [WARN] [mod.rs:2028] [“raft-engine.batch-compression-threshold 8KiB should be adpative to the size of async-io. Set it to 4KiB instead.”] [thread_id=1]
[2025/01/22 16:08:30.543 +08:00] [INFO] [config.rs:403] [“using default coprocessor quota”] [quota=ReadableSize(6291623936)] [thread_id=1]
[2025/01/22 16:08:30.544 +08:00] [INFO] [common.rs:447] [“beginning system configuration check”] [thread_id=1]
[2025/01/22 16:08:30.544 +08:00] [INFO] [config.rs:1101] [“data dir”] [mount_fs=“FsInfo { tp: "xfs", opts: "rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota", mnt_dir: "/home", fsname: "/dev/mapper/rl-home" }”] [data_path=/home/tidb-data/tikv-20160] [thread_id=1]
[2025/01/22 16:08:30.544 +08:00] [WARN] [config.rs:1104] [“not on SSD device”] [data_path=/home/tidb-data/tikv-20160] [thread_id=1]
[2025/01/22 16:08:30.544 +08:00] [INFO] [config.rs:1101] [“data dir”] [mount_fs=“FsInfo { tp: "xfs", opts: "rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota", mnt_dir: "/home", fsname: "/dev/mapper/rl-home" }”] [data_path=/home/tidb-data/tikv-20160/raft] [thread_id=1]
[2025/01/22 16:08:30.544 +08:00] [WARN] [config.rs:1104] [“not on SSD device”] [data_path=/home/tidb-data/tikv-20160/raft] [thread_id=1]
[2025/01/22 16:08:30.544 +08:00] [INFO] [common.rs:123] [“using config”] [config=“{"slow-log-file":"","slow-log-threshold":"1s","panic-when-unexpected-key-or-data":false,"abort-on-panic":false,"memory-usage-limit":"37749743615B","memory-usage-high-water":0.9,"log":{"level":"info","format":"text","enable-timestamp":true,"file":{"filename":"/home/tidb-deploy/tikv-20160/log/tikv.log","max-size":300,"max-days":0,"max-backups":0}},"memory":{"enable-heap-profiling":true,"profiling-sample-per-bytes":"512KiB","enable-thread-exclusive-arena":true},"quota":{"foreground-cpu-time":0,"foreground-write-bandwidth":"0KiB","foreground-read-bandwidth":"0KiB","max-delay-duration":"500ms","background-cpu-time":0,"background-write-bandwidth":"0KiB","background-read-bandwidth":"0KiB","enable-auto-tune":false},"readpool":{"unified":{"min-thread-count":1,"max-thread-count":12,"stack-size":"10MiB","max-tasks-per-worker":2000,"auto-adjust-pool-size":false},"storage":{"use-unified-pool":true,"high-concurrency":8,"normal-concurrency":8,"low-concurrency":8,"max-tasks-per-worker-high":2000,"max-tasks-per-worker-normal":2000,"max-tasks-per-worker-low":2000,"stack-size":"10MiB"},"coprocessor":{"use-unified-pool":true,"high-concurrency":12,"normal-concurrency":12,"low-concurrency":12,"max-tasks-per-worker-high":2000,"max-tasks-per-worker-normal":2000,"max-tasks-per-worker-low":2000,"stack-size":"10MiB"}},"server":{"addr":"0.0.0.0:20160","advertise-addr":"10.212.200.87:20160","status-addr":"0.0.0.0:20180","advertise-status-addr":"10.212.200.87:20180","status-thread-pool-size":1,"max-grpc-send-msg-len":10485760,"raft-client-grpc-send-msg-buffer":524288,"raft-client-queue-size":16384,"raft-msg-max-batch-size":256,"grpc-compression-type":"none","grpc-gzip-compression-level":2,"grpc-min-message-size-to-compress":4096,"grpc-concurrency":5,"grpc-concurrent-stream":1024,"grpc-raft-conn-num":1,"grpc-memory-pool-quota":"9223372036854775807B","grpc-stream-initial-window-size":"2MiB","grpc-keepalive-time":"10s","grpc-keepalive-timeout":"3s","concurrent-send-snap-limit":32,"concurrent-recv-snap-limit":32,"end-point-recursion-limit":1000,"end-point-stream-channel-size":8,"end-point-batch-row-limit":64,"end-point-stream-batch-row-limit":128,"end-point-enable-batch-if-possible":true,"end-point-request-max-handle-duration":"1m","end-point-max-concurrency":16,"end-point-perf-level":0,"end-point-memory-quota":"6144164KiB","snap-io-max-bytes-per-sec":"100MiB","snap-max-total-size":"0KiB","snap-min-ingest-size":"2MiB","stats-concurrency":1,"heavy-load-threshold":75,"heavy-load-wait-duration":null,"enable-request-batch":true,"background-thread-count":2,"end-point-slow-log-threshold":"1s","forward-max-connections-per-address":4,"reject-messages-on-memory-ratio":0.2,"simplify-metrics":false,"health-feedback-interval":"1s","labels":{}},"storage":{"data-dir":"/home/tidb-data/tikv-20160","engine":"raft-kv","gc-ratio-threshold":1.1,"max-key-size":8192,"scheduler-concurrency":524288,"scheduler-worker-pool-size":8,"scheduler-pending-write-threshold":"100MiB","reserve-space":"5GiB","reserve-raft-space":"1GiB","enable-async-apply-prewrite":false,"api-version":1,"enable-ttl":false,"background-error-recovery-window":"1h","ttl-check-poll-interval":"12h","txn-status-cache-capacity":5120000,"memory-quota":"256MiB","flow-control":{"enable":true,"soft-pending-compaction-bytes-limit":"192GiB","hard-pending-compaction-bytes-limit":"1TiB","memtables-threshold":5,"l0-files-threshold":20},"block-cache":{"shared":null,"capacity":"22649846169B","num-shard-bits":6,"strict-capacity-limit":false,"high-pri-pool-ratio":0.8,"low-pri-pool-ratio":0.2,"memory-allocator":"nodump"},"io-rate-limit":{"max-bytes-per-sec":"0KiB","mode":"write-only","strict":false,"foreground-read-priority":"high","foreground-write-priority":"high","flush-priority":"high","level-zero-compaction-priority":"medium","compaction-priority":"low","replication-priority":"high","load-balance-priority":"high","gc-priority":"high","import-priority":"medium","export-priority":"medium","other-priority":"high"}},"pd":{"endpoints":["10.212.200.86:2379"],"retry-interval":"300ms","retry-max-count":9223372036854775807,"retry-log-every":10,"update-interval":"10m","enable-forwarding":false},"metric":{"job":"tikv"},"raftstore":{"prevote":true,"raftdb-path":"/home/tidb-data/tikv-20160/raft","capacity":"0KiB","raft-base-tick-interval":"1s","raft-heartbeat-ticks":2,"raft-election-timeout-ticks":10,"raft-min-election-timeout-ticks":10,"raft-max-election-timeout-ticks":20,"raft-max-size-per-msg":"1MiB","raft-max-inflight-msgs":256,"raft-entry-max-size":"8MiB","raft-log-compact-sync-interval":"2s","raft-log-gc-tick-interval":"3s","request-voter-replicated-index-interval":"5m","raft-log-gc-threshold":50,"raft-log-gc-count-limit":196608,"raft-log-gc-size-limit":"192MiB","max-apply-unpersisted-log-limit":1024,"follower-read-max-log-gap":100,"raft-log-reserve-max-ticks":6,"raft-engine-purge-interval":"10s","max-manual-flush-rate":3.0,"raft-entry-cache-life-time":"30s","split-region-check-tick-interval":"10s","region-split-check-diff":"16MiB","region-compact-check-interval":"5m","region-compact-check-step":100,"region-compact-min-tombstones":10000,"region-compact-tombstones-percent":30,"region-compact-min-redundant-rows":50000,"region-compact-redundant-rows-percent":20,"pd-heartbeat-tick-interval":"1m","pd-store-heartbeat-tick-interval":"10s","pd-report-min-resolved-ts-interval":"1s","snap-mgr-gc-tick-interval":"1m","snap-gc-timeout":"4h","snap-wait-split-duration":"34s","lock-cf-compact-interval":"10m","lock-cf-compact-bytes-threshold":"256MiB","periodic-full-compact-start-times":,"periodic-full-compact-start-max-cpu":0.1,"notify-capacity":40960,"messages-per-tick":4096,"max-peer-down-duration":"10m","max-leader-missing-duration":"2h","abnormal-leader-missing-duration":"10m","peer-stale-state-check-interval":"5m","gc-peer-check-interval":"1m","leader-transfer-max-log-lag":128,"snap-apply-batch-size":"10MiB","snap-apply-copy-symlink":false,"region-worker-tick-interval":"1s","clean-stale-ranges-tick":10,"consistency-check-interval":"0s","report-region-flow-interval":"1m","raft-store-max-leader-lease":"9s","check-leader-lease-interval":"2s250ms","renew-leader-lease-advance-duration":"2s250ms","right-derive-when-split":true,"merge-max-log-gap":10,"merge-check-tick-interval":"2s","use-delete-range":false,"snap-generator-pool-size":2,"cleanup-import-sst-interval":"10m","local-read-batch-size":1024,"apply-max-batch-size":256,"apply-pool-size":2,"apply-reschedule-duration":"5s","apply-low-priority-pool-size":1,"store-max-batch-size":256,"store-pool-size":2,"store-reschedule-duration":"5s","store-low-priority-pool-size":0,"store-io-pool-size":1,"store-io-notify-capacity":40960,"future-poll-size":1,"hibernate-regions":true,"dev-assert":false,"apply-yield-duration":"500ms","apply-yield-write-size":"32KiB","perf-level":0,"evict-cache-on-memory-ratio":0.1,"cmd-batch":true,"cmd-batch-concurrent-ready-max-count":1,"raft-write-size-limit":"1MiB","raft-write-batch-size-hint":"8KiB","raft-write-wait-duration":"20us","waterfall-metrics":true,"io-reschedule-concurrent-max-count":4,"io-reschedule-hotpot-duration":"5s","inspect-interval":"100ms","inspect-kvdb-interval":"2s","inspect-cpu-util-thd":0.4,"slow-trend-unsensitive-cause":10.0,"slow-trend-unsensitive-result":0.5,"slow-trend-network-io-factor":0.0,"reactive-memory-lock-tick-interval":"2s","reactive-memory-lock-timeout-tick":5,"report-region-buckets-tick-interval":"10s","check-long-uncommitted-interval":"10s","long-uncommitted-base-threshold":"20s","max-entry-cache-warmup-duration":"1s","max-snapshot-file-raw-size":"100MiB","unreachable-backoff":"10s","enable-v2-compatible-learner":false,"min-pending-apply-region-count":10,"skip-manual-compaction-in-clean-up-worker":false},"coprocessor":{"split-region-on-table":false,"batch-split-limit":10,"region-max-size":"384MiB","region-split-size":"256MiB","region-max-keys":3840000,"region-split-keys":2560000,"consistency-check-method":"mvcc","enable-region-bucket":null,"region-bucket-size":"50MiB","region-size-threshold-for-approximate":"750MiB","prefer-approximate-bucket":true,"region-bucket-merge-size-ratio":0.33},"coprocessor-v2":{"coprocessor-plugin-directory":null},"rocksdb":{"wal-recovery-mode":2,"wal-dir":"","wal-ttl-seconds":0,"wal-size-limit":"0KiB","max-total-wal-size":"4GiB","max-background-jobs":9,"max-background-flushes":3,"max-manifest-file-size":"128MiB","create-if-missing":true,"max-open-files":40960,"stats-dump-period":"10m","compaction-readahead-size":"0KiB","info-log-dir":"","rate-bytes-per-sec":"10GiB","rate-limiter-refill-period":"100ms","rate-limiter-mode":2,"rate-limiter-auto-tuned":true,"bytes-per-sync":"1MiB","wal-bytes-per-sync":"512KiB","max-sub-compactions":3,"writable-file-max-buffer-size":"1MiB","use-direct-io-for-flush-and-compaction":false,"enable-pipelined-write":false,"enable-multi-batch-write":null,"enable-unordered-write":false,"allow-concurrent-memtable-write":true,"write-buffer-limit":null,"track-and-verify-wals-in-manifest":true,"defaultcf":{"block-size":"32KiB","block-cache-size":null,"disable-block-cache":false,"cache-index-and-filter-blocks":true,"pin-l0-filter-and-index-blocks":true,"use-bloom-filter":true,"optimize-filters-for-hits":true,"optimize-filters-for-memory":false,"whole-key-filtering":true,"bloom-filter-bits-per-key":10,"block-based-bloom-filter":false,"ribbon-filter-above-level":null,"read-amp-bytes-per-bit":0,"compression-per-level":["no","no","lz4","lz4","lz4","zstd","zstd"],"write-buffer-size":"128MiB","write-buffer-limit":null,"max-write-buffer-number":5,"min-write-buffer-number-to-merge":1,"max-bytes-for-level-base":"512MiB","target-file-size-base":null,"level0-file-num-compaction-trigger":4,"level0-slowdown-writes-trigger":20,"level0-stop-writes-trigger":20,"max-compaction-bytes":"2GiB","compaction-pri":3,"dynamic-level-bytes":true,"num-levels":7,"max-bytes-for-level-multiplier":10,"compaction-style":0,"disable-auto-compactions":false,"disable-write-stall":true,"soft-pending-compaction-bytes-limit":"192GiB","hard-pending-compaction-bytes-limit":"1TiB","force-consistency-checks":false,"prop-size-index-distance":4194304,"prop-keys-index-distance":40960,"enable-doubly-skiplist":true,"enable-compaction-guard":true,"compaction-guard-min-output-file-size":"8MiB","compaction-guard-max-output-file-size":"128MiB","bottommost-level-compression":"zstd","bottommost-zstd-compression-dict-size":0,"bottommost-zstd-compression-sample-size":0,"prepopulate-block-cache":"disabled","format-version":null,"checksum":"crc32c","max-compactions":null,"ttl":null,"periodic-compaction-seconds":null,"titan":{"min-blob-size":"32KiB","blob-file-compression":"zstd","zstd-dict-size":"0KiB","shared-blob-cache":true,"blob-cache-size":"0KiB","min-gc-batch-size":"16MiB","max-gc-batch-size":"64MiB","discardable-ratio":0.5,"merge-small-file-threshold":"8MiB","blob-run-mode":"normal","level-merge":false,"range-merge":true,"max-sorted-runs":20}},"writecf":{"block-size":"32KiB","block-cache-size":null,"disable-block-cache":false,"cache-index-and-filter-blocks":true,"pin-l0-filter-and-index-blocks":true,"use-bloom-filter":true,"optimize-filters-for-hits":false,"optimize-filters-for-memory":false,"whole-key-filtering":false,"bloom-filter-bits-per-key":10,"block-based-bloom-filter":false,"ribbon-filter-above-level":null,"read-amp-bytes-per-bit":0,"compression-per-level":["no","no","lz4","lz4","lz4","zstd","zstd"],"write-buffer-size":"128MiB","write-buffer-limit":null,"max-write-buffer-number":5,"min-write-buffer-number-to-merge":1,"max-bytes-for-level-base":"512MiB","target-file-size-base":null,"level0-file-num-compaction-trigger":4,"level0-slowdown-writes-trigger":20,"level0-stop-writes-trigger":20,"max-compaction-bytes":"2GiB","compaction-pri":3,"dynamic-level-bytes":true,"num-levels":7,"max-bytes-for-level-multiplier":10,"compaction-style":0,"disable-auto-compactions":false,"disable-write-stall":true,"soft-pending-compaction-bytes-limit":"192GiB","hard-pending-compaction-bytes-limit":"1TiB","force-consistency-checks":false,"prop-size-index-distance":4194304,"prop-keys-index-distance":40960,"enable-doubly-skiplist":true,"enable-compaction-guard":true,"compaction-guard-min-output-file-size":"8MiB","compaction-guard-max-output-file-size":"128MiB","bottommost-level-compression":"zstd","bottommost-zstd-compression-dict-size":0,"bottommost-zstd-compression-sample-size":0,"prepopulate-block-cache":"disabled","format-version":null,"checksum":"crc32c","max-compactions":null,"ttl":null,"periodic-compaction-seconds":null,"titan":{"min-blob-size":null,"blob-file-compression":"zstd","zstd-dict-size":"0KiB","shared-blob-cache":true,"blob-cache-size":"0KiB","min-gc-batch-size":"16MiB","max-gc-batch-size":"64MiB","discardable-ratio":0.5,"merge-small-file-threshold":"8MiB","blob-run-mode":"read-only","level-merge":false,"range-merge":true,"max-sorted-runs":20}},"lockcf":{"block-size":"16KiB","block-cache-size":null,"disable-block-cache":false,"cache-index-and-filter-blocks":true,"pin-l0-filter-and-index-blocks":true,"use-bloom-filter":true,"optimize-filters-for-hits":false,"optimize-filters-for-memory":false,"whole-key-filtering":true,"bloom-filter-bits-per-key":10,"block-based-bloom-filter":false,"ribbon-filter-above-level":null,"read-amp-bytes-per-bit":0,"compression-per-level":["no","no","no","no","no","no","no"],"write-buffer-size":"32MiB","write-buffer-limit":null,"max-write-buffer-number":5,"min-write-buffer-number-to-merge":1,"max-bytes-for-level-base":"128MiB","target-file-size-base":null,"level0-file-num-compaction-trigger":1,"level0-slowdown-writes-trigger":20,"level0-stop-writes-trigger":20,"max-compaction-bytes":"2GiB","compaction-pri":0,"dynamic-level-bytes":true,"num-levels":7,"max-bytes-for-level-multiplier":10,"compaction-style":0,"disable-auto-compactions":false,"disable-write-stall":true,"soft-pending-compaction-bytes-limit":"192GiB","hard-pending-compaction-bytes-limit":"1TiB","force-consistency-checks":false,"prop-size-index-distance":4194304,"prop-keys-index-distance":40960,"enable-doubly-skiplist":true,"enable-compaction-guard":null,"compaction-guard-min-output-file-size":"8MiB","compaction-guard-max-output-file-size":"128MiB","bottommost-level-compression":"disable","bottommost-zstd-compression-dict-size":0,"bottommost-zstd-compression-sample-size":0,"prepopulate-block-cache":"disabled","format-version":null,"checksum":"crc32c","max-compactions":null,"ttl":null,"periodic-compaction-seconds":null,"titan":{"min-blob-size":null,"blob-file-compression":"zstd","zstd-dict-size":"0KiB","shared-blob-cache":true,"blob-cache-size":"0KiB","min-gc-batch-size":"16MiB","max-gc-batch-size":"64MiB","discardable-ratio":0.5,"merge-small-file-threshold":"8MiB","blob-run-mode":"read-only","level-merge":false,"range-merge":true,"max-sorted-runs":20}},"raftcf":{"block-size":"16KiB","block-cache-size":null,"disable-block-cache":false,"cache-index-and-filter-blocks":true,"pin-l0-filter-and-index-blocks":true,"use-bloom-filter":true,"optimize-filters-for-hits":true,"optimize-filters-for-memory":false,"whole-key-filtering":true,"bloom-filter-bits-per-key":10,"block-based-bloom-filter":false,"ribbon-filter-above-level":null,"read-amp-bytes-per-bit":0,"compression-per-level":["no","no","no","no","no","no","no"],"write-buffer-size":"128MiB","write-buffer-limit":null,"max-write-buffer-number":5,"min-write-buffer-number-to-merge":1,"max-bytes-for-level-base":"128MiB","target-file-size-base":null,"level0-file-num-compaction-trigger":1,"level0-slowdown-writes-trigger":20,"level0-stop-writes-trigger":20,"max-compaction-bytes":"2GiB","compaction-pri":0,"dynamic-level-bytes":true,"num-levels":7,"max-bytes-for-level-multiplier":10,"compaction-style":0,"disable-auto-compactions":false,"disable-write-stall":true,"soft-pending-compaction-bytes-limit":"192GiB","hard-pending-compaction-bytes-limit":"1TiB","force-consistency-checks":false,"prop-size-index-distance":4194304,"prop-keys-index-distance":40960,"enable-doubly-skiplist":true,"enable-compaction-guard":null,"compaction-guard-min-output-file-size":"8MiB","compaction-guard-max-output-file-size":"128MiB","bottommost-level-compression":"disable","bottommost-zstd-compression-dict-size":0,"bottommost-zstd-compression-sample-size":0,"prepopulate-block-cache":"disabled","format-version":null,"checksum":"crc32c","max-compactions":null,"ttl":null,"periodic-compaction-seconds":null,"titan":{"min-blob-size":null,"blob-file-compression":"zstd","zstd-dict-size":"0KiB","shared-blob-cache":true,"blob-cache-size":"0KiB","min-gc-batch-size":"16MiB","max-gc-batch-size":"64MiB","discardable-ratio":0.5,"merge-small-file-threshold":"8MiB","blob-run-mode":"read-only","level-merge":false,"range-merge":true,"max-sorted-runs":20}},"titan":{"enabled":false,"dirname":"","disable-gc":false,"max-background-gc":1,"purge-obsolete-files-period":"10s"}},"raftdb":{"wal-recovery-mode":2,"wal-dir":"","wal-ttl-seconds":0,"wal-size-limit":"0KiB","max-total-wal-size":"4GiB","max-background-jobs":4,"max-background-flushes":1,"max-manifest-file-size":"20MiB","create-if-missing":true,"max-open-files":40960,"stats-dump-period":"10m","compaction-readahead-size":"0KiB","info-log-dir":"","max-sub-compactions":2,"writable-file-max-buffer-size":"1MiB","use-direct-io-for-flush-and-compaction":false,"enable-pipelined-write":true,"enable-unordered-write":false,"allow-concurrent-memtable-write":true,"bytes-per-sync":"1MiB","wal-bytes-per-sync":"512KiB","defaultcf":{"block-size":"64KiB","block-cache-size":null,"disable-block-cache":false,"cache-index-and-filter-blocks":true,"pin-l0-filter-and-index-blocks":true,"use-bloom-filter":false,"optimize-filters-for-hits":true,"optimize-filters-for-memory":false,"whole-key-filtering":true,"bloom-filter-bits-per-key":10,"block-based-bloom-filter":false,"ribbon-filter-above-level":null,"read-amp-bytes-per-bit":0,"compression-per-level":["no","no","lz4","lz4","lz4","zstd","zstd"],"write-buffer-size":"128MiB","write-buffer-limit":null,"max-write-buffer-number":5,"min-write-buffer-number-to-merge":1,"max-bytes-for-level-base":"512MiB","target-file-size-base":null,"level0-file-num-compaction-trigger":4,"level0-slowdown-writes-trigger":20,"level0-stop-writes-trigger":20,"max-compaction-bytes":"2GiB","compaction-pri":0,"dynamic-level-bytes":true,"num-levels":7,"max-bytes-for-level-multiplier":10,"compaction-style":0,"disable-auto-compactions":false,"disable-write-stall":false,"soft-pending-compaction-bytes-limit":"192GiB","hard-pending-compaction-bytes-limit":"1TiB","force-consistency-checks":false,"prop-size-index-distance":4194304,"prop-keys-index-distance":40960,"enable-doubly-skiplist":true,"enable-compaction-guard":null,"compaction-guard-min-output-file-size":"8MiB","compaction-guard-max-output-file-size":"128MiB","bottommost-level-compression":"disable","bottommost-zstd-compression-dict-size":0,"bottommost-zstd-compression-sample-size":0,"prepopulate-block-cache":"disabled","format-version":2,"checksum":"crc32c","max-compactions":null,"ttl":null,"periodic-compaction-seconds":null,"titan":{"min-blob-size":null,"blob-file-compression":"zstd","zstd-dict-size":"0KiB","shared-blob-cache":true,"blob-cache-size":"0KiB","min-gc-batch-size":"16MiB","max-gc-batch-size":"64MiB","discardable-ratio":0.5,"merge-small-file-threshold":"8MiB","blob-run-mode":"normal","level-merge":false,"range-merge":true,"max-sorted-runs":20}},"titan":{"enabled":false,"dirname":"","disable-gc":false,"max-background-gc":1,"purge-obsolete-files-period":"10s"}},"raft-engine":{"enable":true,"dir":"/home/tidb-data/tikv-20160/raft-engine","spill-dir":null,"recovery-mode":"tolerate-corrupted-tail-records","recovery-read-block-size":"16KiB","recovery-threads":4,"batch-compression-threshold":"4KiB","compression-level":null,"bytes-per-sync":null,"format-version":2,"target-file-size":"128MiB","purge-threshold":"10GiB","purge-rewrite-threshold":"1GiB","purge-rewrite-garbage-ratio":0.6,"memory-limit":"7549948723B","enable-log-recycle":true,"prefill-for-recycle":false,"prefill-limit":null},"security":{"ca-path":"","cert-path":"","key-path":"","cert-allowed-cn":,"redact-info-log":false,"encryption":{"data-encryption-method":"plaintext","data-key-rotation-period":"7d","enable-file-dictionary-log":true,"file-dictionary-rewrite-threshold":1000000,"master-key":{"type":"plaintext"},"previous-master-key":{"type":"plaintext"}}},"import":{"num-threads":8,"stream-channel-window":128,"import-mode-timeout":"10m","memory-use-ratio":0.3},"backup":{"num-threads":8,"batch-size":8,"sst-max-size":"384MiB","enable-auto-tune":true,"auto-tune-remain-threads":3,"auto-tune-refresh-interval":"1m","io-thread-size":2,"s3-multi-part-size":"5MiB","hadoop":{"home":"","linux-user":""}},"log-backup":{"min-ts-interval":"10s","max-flush-interval":"3m","num-threads":8,"enable":true,"temp-path":"/home/tidb-data/tikv-20160/log-backup-temp","file-size-limit":"256MiB","initial-scan-pending-memory-quota":"512MiB","initial-scan-rate-limit":"60MiB","initial-scan-concurrency":6},"pessimistic-txn":{"wait-for-lock-timeout":"1s","wake-up-delay-duration":"20ms","pipelined":true,"in-memory":true,"in-memory-peer-size-limit":"512KiB","in-memory-instance-size-limit":"100MiB"},"gc":{"ratio-threshold":1.1,"batch-keys":512,"max-write-bytes-per-sec":"0KiB","enable-compaction-filter":true,"compaction-filter-skip-version-check":false,"num-threads":1},"split":{"qps-threshold":3000,"split-balance-score":0.25,"split-contained-score":0.5,"detect-times":10,"sample-num":20,"sample-threshold":100,"byte-threshold":31457280,"grpc-thread-cpu-overload-threshold-ratio":0.5,"unified-read-pool-thread-cpu-overload-threshold-ratio":0.8,"region-cpu-overload-threshold-ratio":0.25},"cdc":{"min-ts-interval":"1s","hibernate-regions-compatible":true,"incremental-scan-threads":4,"incremental-scan-concurrency":6,"incremental-scan-concurrency-limit":10000,"incremental-scan-speed-limit":"128MiB","incremental-fetch-speed-limit":"512MiB","incremental-scan-ts-filter-ratio":0.2,"tso-worker-threads":1,"sink-memory-quota":"512MiB","old-value-cache-memory-quota":"512MiB"},"resolved-ts":{"enable":true,"advance-ts-interval":"20s","scan-lock-pool-size":2,"memory-quota":"256MiB","incremental-scan-concurrency":6},"resource-metering":{"receiver-address":"","report-receiver-interval":"1m","max-resource-groups":100,"precision":"1s"},"causal-ts":{"renew-interval":"100ms","renew-batch-min-size":100,"renew-batch-max-size":8192,"alloc-ahead-buffer":"3s"},"resource-control":{"enabled":true,"priority-ctl-strategy":"moderate"},"in-memory-engine":{"enable":false,"capacity":null,"evict-threshold":null,"stop-load-threshold":null,"gc-run-interval":"3m","load-evict-interval":"5m","mvcc-amplification-threshold":10,"cross-check-interval":"0s"}}”] [thread_id=1]
[2025/01/22 16:08:30.545 +08:00] [INFO] [resource_group.rs:151] [“add resource group”] [ru=2147483647] [name=default] [thread_id=1]
[2025/01/22 16:08:30.546 +08:00] [WARN] [worker.rs:392] [“adjust duration too small, skip adjustment.”] [dur=1ms] [thread_id=17]
[2025/01/22 16:08:30.547 +08:00] [INFO] [resource_group.rs:151] [“add resource group”] [ru=2147483647] [name=default] [thread_id=16]
[2025/01/22 16:08:30.547 +08:00] [INFO] [service.rs:70] [“pd meta client creating watch stream.”] [rev=5484] [path=resource_group/settings] [thread_id=16]
[2025/01/22 16:08:30.547 +08:00] [INFO] [service.rs:193] [“load controller config”] [config=“RequestUnitConfig { read_base_cost: 0.125, read_cost_per_byte: 1.52587890625e-5, write_base_cost: 1.0, write_cost_per_byte: 0.0009765625, read_cpu_ms_cost: 0.3333333333333333 }”] [thread_id=17]
[2025/01/22 16:08:30.576 +08:00] [INFO] [mod.rs:130] [“encryption: none of key dictionary and file dictionary are found.”] [thread_id=1]
[2025/01/22 16:08:30.576 +08:00] [INFO] [mod.rs:549] [“encryption is disabled.”] [thread_id=1]
[2025/01/22 16:08:30.577 +08:00] [INFO] [engine.rs:93] [“Recovering raft logs takes 1.100608ms”] [thread_id=1]
权限？防火墙，单个节点出问题这个概率不小？另外这个机器跟别的机器有什么不同？国产操作系统？遇到过7.1安装麒麟+飞腾的虚机，安装后启动不了，也不报错，后来安装了developtool才好。
系统是一样Rocky linux 9配置上有差别，其他两个TIkV是386内存38T存储20cpu，这个起不来的是36内存6T存储16cpu
root权限，防火墙关了
换了个服务器好了
换了一台配置高的还是和之前一样配置的？
内存配置是之前的两倍，存储小了一半
配置不一样就不好说是不是其他问题了。可能是内存问题，也可能是其他未发现的问题。
