基于tiup方式，某个tikv节点无法启动

🪐 TiDB 技术问题 部署&运维管理
1

为提高效率，提问时请提供以下信息，问题描述清晰可优先响应。

  • 【TiDB 版本】： v4.0.0

  • 【问题描述】：
    tikv节点214无法启动，处于down的状态
    对应的log日志如下：

    [2020/09/08 02:01:20.713 +00:00] [INFO] [mod.rs:335] [“starting working thread”] [worker=addr-resolver]
    [2020/09/08 02:01:20.714 +00:00] [INFO] [mod.rs:335] [“starting working thread”] [worker=region-collector-worker]
    [2020/09/08 02:01:23.024 +00:00] [FATAL] [lib.rs:481] [“create /tmp/1001_TIKV_LOCK_FILES failed”] [backtrace=“stack backtrace:\ 0: tikv_util::set_panic_hook::{{closure}}\ at components/tikv_util/src/lib.rs:480\ 1: std::panicking::rust_panic_with_hook\ at src/libstd/panicking.rs:475\ 2: rust_begin_unwind\ at src/libstd/panicking.rs:375\ 3: std::panicking::begin_panic_fmt\ at src/libstd/panicking.rs:326\ 4: cmd::server::TiKVServer::check_conflict_addr::{{closure}}\ at cmd/src/server.rs:263\ 5: core::result::Result<T,E>::unwrap_or_else\ at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/result.rs:841\ cmd::server::TiKVServer::check_conflict_addr\ at cmd/src/server.rs:262\ cmd::server::run_tikv\ at cmd/src/server.rs:94\ 6: tikv_server::main\ at cmd/src/bin/tikv-server.rs:159\ 7: std::rt::lang_start::{{closure}}\ at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/rt.rs:67\ 8: main\ 9: __libc_start_main\ 10: \ ”] [location=cmd/src/server.rs:263] [thread_name=main]
    [2020/09/08 02:01:38.268 +00:00] [INFO] [lib.rs:93] [“Welcome to TiKV”]
    [2020/09/08 02:01:38.268 +00:00] [INFO] [lib.rs:95] []
    [2020/09/08 02:01:38.268 +00:00] [INFO] [lib.rs:95] [“Release Version: 4.0.0”]
    [2020/09/08 02:01:38.268 +00:00] [INFO] [lib.rs:95] [“Edition: Community”]
    [2020/09/08 02:01:38.268 +00:00] [INFO] [lib.rs:95] [“Git Commit Hash: 198a2cea01734ce8f46d55a29708f123f9133944”]
    [2020/09/08 02:01:38.268 +00:00] [INFO] [lib.rs:95] [“Git Commit Branch: heads/refs/tags/v4.0.0”]
    [2020/09/08 02:01:38.268 +00:00] [INFO] [lib.rs:95] [“UTC Build Time: 2020-05-27 12:09:22”]
    [2020/09/08 02:01:38.268 +00:00] [INFO] [lib.rs:95] [“Rust Version: rustc 1.42.0-nightly (0de96d37f 2019-12-19)”]
    [2020/09/08 02:01:38.268 +00:00] [INFO] [lib.rs:95] [“Enable Features: jemalloc portable sse protobuf-codec”]
    [2020/09/08 02:01:38.268 +00:00] [INFO] [lib.rs:95] [“Profile: dist_release”]
    [2020/09/08 02:01:38.281 +00:00] [INFO] [mod.rs:48] [“memory limit in bytes: 12589617152, cpu cores quota: 16”]
    [2020/09/08 02:01:38.281 +00:00] [WARN] [lib.rs:528] [“environment variable TZ is missing, using /etc/localtime“]
    [2020/09/08 02:01:38.281 +00:00] [INFO] [config.rs:529] [“kernel parameters”] [value=32768] [param=net.core.somaxconn]
    [2020/09/08 02:01:38.281 +00:00] [INFO] [config.rs:529] [“kernel parameters”] [value=0] [param=net.ipv4.tcp_syncookies]
    [2020/09/08 02:01:38.281 +00:00] [INFO] [config.rs:529] [“kernel parameters”] [value=0] [param=vm.swappiness]
    [2020/09/08 02:01:38.282 +00:00] [INFO] [util.rs:398] [“connecting to PD endpoint”] [endpoints=10.12.5.113:2379]
    [2020/09/08 02:01:38.283 +00:00] [INFO] [] [“New connected subchannel at 0x7f8643e39180 for subchannel 0x7f864e219a00”]
    [2020/09/08 02:01:38.284 +00:00] [INFO] [util.rs:398] [“connecting to PD endpoint”] [endpoints=10.12.5.114:2379]
    [2020/09/08 02:01:38.284 +00:00] [INFO] [] [“New connected subchannel at 0x7f8643e39240 for subchannel 0x7f864e219a00”]
    [2020/09/08 02:01:38.285 +00:00] [INFO] [util.rs:398] [“connecting to PD endpoint”] [endpoints=10.12.5.115:2379]
    [2020/09/08 02:01:38.286 +00:00] [INFO] [] [“New connected subchannel at 0x7f8643e39300 for subchannel 0x7f864e219a00”]
    [2020/09/08 02:01:38.287 +00:00] [INFO] [util.rs:398] [“connecting to PD endpoint”] [endpoints=http://10.12.5.115:2379]
    [2020/09/08 02:01:38.287 +00:00] [INFO] [] [“New connected subchannel at 0x7f8643e393c0 for subchannel 0x7f864e219a00”]
    [2020/09/08 02:01:38.288 +00:00] [INFO] [util.rs:398] [“connecting to PD endpoint”] [endpoints=http://10.12.5.113:2379]
    [2020/09/08 02:01:38.288 +00:00] [INFO] [] [“New connected subchannel at 0x7f8643e39480 for subchannel 0x7f864e219a00”]
    [2020/09/08 02:01:38.289 +00:00] [INFO] [util.rs:457] [“connected to PD leader”] [endpoints=http://10.12.5.113:2379]
    [2020/09/08 02:01:38.289 +00:00] [INFO] [util.rs:386] [“all PD endpoints are consistent”] [endpoints=”[“10.12.5.113:2379”, “10.12.5.114:2379”, “10.12.5.115:2379”]”]
    [2020/09/08 02:01:38.289 +00:00] [INFO] [server.rs:242] [“connect to PD cluster”] [cluster_id=6807312755917103041]
    [2020/09/08 02:01:38.454 +00:00] [INFO] [server.rs:834] [“beginning system configuration check”]
    [2020/09/08 02:01:38.454 +00:00] [INFO] [config.rs:709] [“data dir”] [mount_fs=“FsInfo { tp: “ext4”, opts: “rw,noatime,nodelalloc,data=ordered”, mnt_dir: “/”, fsname: “/dev/sda2” }”] [data_path=/home/tidb/deploy/data]
    [2020/09/08 02:01:38.454 +00:00] [WARN] [config.rs:712] [“not on SSD device”] [data_path=/home/tidb/deploy/data]
    [2020/09/08 02:01:38.454 +00:00] [INFO] [config.rs:709] [“data dir”] [mount_fs=“FsInfo { tp: “ext4”, opts: “rw,noatime,nodelalloc,data=ordered”, mnt_dir: “/”, fsname: “/dev/sda2” }”] [data_path=/home/tidb/deploy/data/raft]
    [2020/09/08 02:01:38.454 +00:00] [WARN] [config.rs:712] [“not on SSD device”] [data_path=/home/tidb/deploy/data/raft]
    [2020/09/08 02:01:38.455 +00:00] [INFO] [server.rs:212] [“using config”] [config="{“cfg-path”:“conf/tikv.toml”,“log-level”:“info”,“log-file”:"/home/tidb/deploy/log/tikv.log",“slow-log-file”:"",“slow-log-threshold”:“1s”,“log-rotation-timespan”:“1d”,“log-rotation-size”:“300MiB”,“panic-when-unexpected-key-or-data”:false,“readpool”:{“unified”:{“min-thread-count”:1,“max-thread-count”:12,“stack-size”:“10MiB”,“max-tasks-per-worker”:2000},“storage”:{“use-unified-pool”:null,“high-concurrency”:12,“normal-concurrency”:12,“low-concurrency”:12,“max-tasks-per-worker-high”:2000,“max-tasks-per-worker-normal”:2000,“max-tasks-per-worker-low”:2000,“stack-size”:“10MiB”},“coprocessor”:{“use-unified-pool”:null,“high-concurrency”:12,“normal-concurrency”:12,“low-concurrency”:12,“max-tasks-per-worker-high”:2000,“max-tasks-per-worker-normal”:2000,“max-tasks-per-worker-low”:2000,“stack-size”:“10MiB”}},“server”:{“addr”:“0.0.0.0:20160”,“advertise-addr”:“10.12.5.214:20160”,“status-addr”:“10.12.5.214:20180”,“status-thread-pool-size”:1,“max-grpc-send-msg-len”:10485760,“grpc-compression-type”:“none”,“grpc-concurrency”:4,“grpc-concurrent-stream”:1024,“grpc-raft-conn-num”:1,“grpc-memory-pool-quota”:9223372036854775807,“grpc-stream-initial-window-size”:“2MiB”,“grpc-keepalive-time”:“10s”,“grpc-keepalive-timeout”:“3s”,“concurrent-send-snap-limit”:32,“concurrent-recv-snap-limit”:32,“end-point-recursion-limit”:1000,“end-point-stream-channel-size”:8,“end-point-batch-row-limit”:64,“end-point-stream-batch-row-limit”:128,“end-point-enable-batch-if-possible”:true,“end-point-request-max-handle-duration”:“1m”,“end-point-max-concurrency”:16,“snap-max-write-bytes-per-sec”:“100MiB”,“snap-max-total-size”:“0KiB”,“stats-concurrency”:1,“heavy-load-threshold”:300,“heavy-load-wait-duration”:“1ms”,“enable-request-batch”:true,“request-batch-enable-cross-command”:false,“request-batch-wait-duration”:“1ms”,“labels”:{}},“storage”:{“data-dir”:"/home/tidb/deploy/data",“gc-ratio-threshold”:1.1,“max-key-size”:4096,“scheduler-concurrency”:524288,“scheduler-worker-pool-size”:8,“scheduler-pending-write-threshold”:“100MiB”,“reserve-space”:“2GiB”,“block-cache”:{“shared”:true,“capacity”:“8GiB”,“num-shard-bits”:6,“strict-capacity-limit”:false,“high-pri-pool-ratio”:0.8,“memory-allocator”:“nodump”}},“pd”:{“endpoints”:[“10.12.5.113:2379”,“10.12.5.114:2379”,“10.12.5.115:2379”],“retry-interval”:“300ms”,“retry-max-count”:9223372036854775807,“retry-log-every”:10},“metric”:{“interval”:“15s”,“address”:"",“job”:“tikv”},“raftstore”:{“sync-log”:false,“prevote”:true,“raftdb-path”:"/home/tidb/deploy/data/raft",“capacity”:1209462790553,“raft-base-tick-interval”:“1s”,“raft-heartbeat-ticks”:2,“raft-election-timeout-ticks”:10,“raft-min-election-timeout-ticks”:10,“raft-max-election-timeout-ticks”:20,“raft-max-size-per-msg”:“1MiB”,“raft-max-inflight-msgs”:256,“raft-entry-max-size”:“8MiB”,“raft-log-gc-tick-interval”:“10s”,“raft-log-gc-threshold”:50,“raft-log-gc-count-limit”:73728,“raft-log-gc-size-limit”:“72MiB”,“raft-entry-cache-life-time”:“30s”,“raft-reject-transfer-leader-duration”:“3s”,“split-region-check-tick-interval”:“10s”,“region-split-check-diff”:“6MiB”,“region-compact-check-interval”:“5m”,“clean-stale-peer-delay”:“11m”,“region-compact-check-step”:100,“region-compact-min-tombstones”:10000,“region-compact-tombstones-percent”:30,“pd-heartbeat-tick-interval”:“1m”,“pd-store-heartbeat-tick-interval”:“10s”,“snap-mgr-gc-tick-interval”:“1m”,“snap-gc-timeout”:“4h”,“lock-cf-compact-interval”:“10m”,“lock-cf-compact-bytes-threshold”:“256MiB”,“notify-capacity”:40960,“messages-per-tick”:4096,“max-peer-down-duration”:“5m”,“max-leader-missing-duration”:“2h”,“abnormal-leader-missing-duration”:“10m”,“peer-stale-state-check-interval”:“5m”,“leader-transfer-max-log-lag”:10,“snap-apply-batch-size”:“10MiB”,“consistency-check-interval”:“0s”,“report-region-flow-interval”:“1m”,“raft-store-max-leader-lease”:“9s”,“right-derive-when-split”:true,“allow-remove-leader”:false,“merge-max-log-gap”:10,“merge-check-tick-interval”:“10s”,“use-delete-range”:false,“cleanup-import-sst-interval”:“10m”,“local-read-batch-size”:1024,“apply-max-batch-size”:256,“apply-pool-size”:2,“apply-reschedule-duration”:“5s”,“store-max-batch-size”:256,“store-pool-size”:2,“store-reschedule-duration”:“5s”,“future-poll-size”:1,“hibernate-regions”:true,“early-apply”:true,“apply-yield-duration”:“500ms”},“coprocessor”:{“split-region-on-table”:false,“batch-split-limit”:10,“region-max-size”:“144MiB”,“region-split-size”:“96MiB”,“region-max-keys”:1440000,“region-split-keys”:960000},“rocksdb”:{“wal-recovery-mode”:2,“wal-dir”:"",“wal-ttl-seconds”:0,“wal-size-limit”:“0KiB”,“max-total-wal-size”:“4GiB”,“max-background-jobs”:8,“max-manifest-file-size”:“128MiB”,“create-if-missing”:true,“max-open-files”:40960,“enable-statistics”:true,“stats-dump-period”:“10m”,“compaction-readahead-size”:“0KiB”,“info-log-max-size”:“1GiB”,“info-log-roll-time”:“0s”,“info-log-keep-log-file-num”:10,“info-log-dir”:"",“rate-bytes-per-sec”:“0KiB”,“rate-limiter-mode”:2,“auto-tuned”:false,“bytes-per-sync”:“1MiB”,“wal-bytes-per-sync”:“512KiB”,“max-sub-compactions”:3,“writable-file-max-buffer-size”:“1MiB”,“use-direct-io-for-flush-and-compaction”:false,“enable-pipelined-write”:true,“enable-multi-batch-write”:true,“enable-unordered-write”:false,“defaultcf”:{“block-size”:“64KiB”,“block-cache-size”:“3001MiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:true,“optimize-filters-for-hits”:true,“whole-key-filtering”:true,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“lz4”,“lz4”,“lz4”,“zstd”,“zstd”],“write-buffer-size”:“128MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“512MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:4,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:3,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“64GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:true,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“normal”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“writecf”:{“block-size”:“64KiB”,“block-cache-size”:“1800MiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:true,“optimize-filters-for-hits”:false,“whole-key-filtering”:false,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“lz4”,“lz4”,“lz4”,“zstd”,“zstd”],“write-buffer-size”:“128MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“512MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:4,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:3,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“64GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:true,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“read-only”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“lockcf”:{“block-size”:“16KiB”,“block-cache-size”:“256MiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:true,“optimize-filters-for-hits”:false,“whole-key-filtering”:true,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“no”,“no”,“no”,“no”,“no”],“write-buffer-size”:“32MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“128MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:1,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:0,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“64GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:true,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“read-only”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“raftcf”:{“block-size”:“16KiB”,“block-cache-size”:“128MiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:true,“optimize-filters-for-hits”:true,“whole-key-filtering”:true,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“no”,“no”,“no”,“no”,“no”],“write-buffer-size”:“128MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“128MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:1,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:0,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“64GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:true,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“read-only”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“titan”:{“enabled”:false,“dirname”:"",“disable-gc”:false,“max-background-gc”:4,“purge-obsolete-files-period”:“10s”}},“raftdb”:{“wal-recovery-mode”:2,“wal-dir”:"",“wal-ttl-seconds”:0,“wal-size-limit”:“0KiB”,“max-total-wal-size”:“4GiB”,“max-background-jobs”:4,“max-manifest-file-size”:“20MiB”,“create-if-missing”:true,“max-open-files”:40960,“enable-statistics”:true,“stats-dump-period”:“10m”,“compaction-readahead-size”:“0KiB”,“info-log-max-size”:“1GiB”,“info-log-roll-time”:“0s”,“info-log-keep-log-file-num”:10,“info-log-dir”:"",“max-sub-compactions”:2,“writable-file-max-buffer-size”:“1MiB”,“use-direct-io-for-flush-and-compaction”:false,“enable-pipelined-write”:true,“enable-unordered-write”:false,“allow-concurrent-memtable-write”:true,“bytes-per-sync”:“1MiB”,“wal-bytes-per-sync”:“512KiB”,“defaultcf”:{“block-size”:“64KiB”,“block-cache-size”:“256MiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:false,“optimize-filters-for-hits”:true,“whole-key-filtering”:true,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“lz4”,“lz4”,“lz4”,“zstd”,“zstd”],“write-buffer-size”:“128MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“512MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:4,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:0,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“64GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:true,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“normal”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“titan”:{“enabled”:false,“dirname”:"",“disable-gc”:false,“max-background-gc”:4,“purge-obsolete-files-period”:“10s”}},“security”:{“ca-path”:"",“cert-path”:"",“key-path”:"",“cert-allowed-cn”:[],“encryption”:{“data-encryption-method”:“plaintext”,“data-key-rotation-period”:“7d”,“master-key”:{“type”:“plaintext”},“previous-master-key”:{“type”:“plaintext”}}},“import”:{“num-threads”:8,“stream-channel-window”:128},“pessimistic-txn”:{“enabled”:true,“wait-for-lock-timeout”:“1s”,“wake-up-delay-duration”:“20ms”,“pipelined”:false},“gc”:{“ratio-threshold”:1.1,“batch-keys”:512,“max-write-bytes-per-sec”:“0KiB”},“split”:{“qps-threshold”:3000,“split-balance-score”:0.25,“split-contained-score”:0.5,“detect-times”:10,“sample-num”:20,“sample-threshold”:100}}"]
    [2020/09/08 02:01:38.455 +00:00] [INFO] [mod.rs:335] [“starting working thread”] [worker=addr-resolver]
    [2020/09/08 02:01:38.455 +00:00] [INFO] [mod.rs:335] [“starting working thread”] [worker=region-collector-worker]
    [2020/09/08 02:01:40.766 +00:00] [FATAL] [lib.rs:481] [“create /tmp/1001_TIKV_LOCK_FILES failed”] [backtrace=“stack backtrace:\ 0: tikv_util::set_panic_hook::{{closure}}\ at components/tikv_util/src/lib.rs:480\ 1: std::panicking::rust_panic_with_hook\ at src/libstd/panicking.rs:475\ 2: rust_begin_unwind\ at src/libstd/panicking.rs:375\ 3: std::panicking::begin_panic_fmt\ at src/libstd/panicking.rs:326\ 4: cmd::server::TiKVServer::check_conflict_addr::{{closure}}\ at cmd/src/server.rs:263\ 5: core::result::Result<T,E>::unwrap_or_else\ at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/result.rs:841\ cmd::server::TiKVServer::check_conflict_addr\ at cmd/src/server.rs:262\ cmd::server::run_tikv\ at cmd/src/server.rs:94\ 6: tikv_server::main\ at cmd/src/bin/tikv-server.rs:159\ 7: std::rt::lang_start::{{closure}}\ at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/rt.rs:67\ 8: main\ 9: __libc_start_main\ 10: \ ”] [location=cmd/src/server.rs:263] [thread_name=main]

若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出的打印结果，请务必全选并复制粘贴上传。

2

tikv出问题时的log日志如下：
tikv_tmp.log (442.1 KB)

3

目前根据报错，创建/tmp文件夹供该节点tidb用户使用。此时tikv的log信息如下：

[2020/09/08 02:25:25.199 +00:00] [INFO] [raft.rs:902] ["17716890 received message from 17716890"] [term=17137] [msg=MsgRequestPreVote] [from=17716890] [id=17716890] [raft_id=17716890] [region_id=117628]
[2020/09/08 02:25:25.199 +00:00] [INFO] [raft.rs:923] ["[logterm: 17136, index: 28591] sent request to 21894985"] [msg=MsgRequestPreVote] [term=17137] [id=21894985] [log_index=28591] [log_term=17136] [raft_id=17716890] [region_id=117628]
[2020/09/08 02:25:25.624 +00:00] [INFO] [apply.rs:1142] ["execute admin command"] [command="cmd_type: ChangePeer change_peer { peer { id: 21917839 store_id: 335855 } }"] [index=507] [term=468] [peer_id=21879463] [region_id=9314839]
[2020/09/08 02:25:25.624 +00:00] [INFO] [apply.rs:1467] ["exec ConfChange"] [epoch="conf_ver: 1338 version: 7481"] [type=AddNode] [peer_id=21879463] [region_id=9314839]
[2020/09/08 02:25:25.624 +00:00] [INFO] [apply.rs:1522] ["add peer successfully"] [region="id: 9314839 start_key: 7480000000000003FFBB00000000000000F8 end_key: 7480000000000003FFBE00000000000000F8 region_epoch { conf_ver: 1338 version: 7481 } peers { id: 19882997 store_id: 6506924 } peers { id: 21879463 store_id: 17388737 } peers { id: 21917839 store_id: 335855 is_learner: true }"] [peer="id: 21917839 store_id: 335855"] [peer_id=21879463] [region_id=9314839]
[2020/09/08 02:25:25.665 +00:00] [INFO] [apply.rs:1142] ["execute admin command"] [command="cmd_type: ChangePeer change_peer { peer { id: 21917841 store_id: 6506924 } }"] [index=21897] [term=21564] [peer_id=17565306] [region_id=9317067]
[2020/09/08 02:25:25.665 +00:00] [INFO] [apply.rs:1467] ["exec ConfChange"] [epoch="conf_ver: 1341 version: 7505"] [type=AddNode] [peer_id=17565306] [region_id=9317067]
[2020/09/08 02:25:25.665 +00:00] [INFO] [apply.rs:1522] ["add peer successfully"] [region="id: 9317067 start_key: 7480000000000004FF0300000000000000F8 end_key: 7480000000000004FF0600000000000000F8 region_epoch { conf_ver: 1341 version: 7505 } peers { id: 17565306 store_id: 17388737 } peers { id: 21900416 store_id: 21741644 } peers { id: 21917841 store_id: 6506924 is_learner: true }"] [peer="id: 21917841 store_id: 6506924"] [peer_id=17565306] [region_id=9317067]
[2020/09/08 02:25:25.807 +00:00] [INFO] [peer.rs:746] ["failed to schedule peer tick"] [err="sending on a disconnected channel"] [tick=PD_HEARTBEAT] [peer_id=21908183] [region_id=319408]
[2020/09/08 02:25:25.814 +00:00] [INFO] [peer.rs:746] ["failed to schedule peer tick"] [err="sending on a disconnected channel"] [tick=PD_HEARTBEAT] [peer_id=21878388] [region_id=533498]
[2020/09/08 02:25:25.861 +00:00] [INFO] [peer.rs:746] ["failed to schedule peer tick"] [err="sending on a disconnected channel"] [tick=PD_HEARTBEAT] [peer_id=21913027] [region_id=16389746]
[2020/09/08 02:25:25.871 +00:00] [INFO] [peer.rs:467] ["deleting applied snap file"] [snap_file=232517_1194_5805] [peer_id=21929739] [region_id=232517]
[2020/09/08 02:25:25.882 +00:00] [INFO] [peer.rs:467] ["deleting applied snap file"] [snap_file=274624_4386_12541] [peer_id=21914769] [region_id=274624]
[2020/09/08 02:25:25.908 +00:00] [INFO] [peer.rs:467] ["deleting applied snap file"] [snap_file=2240384_6286_6239] [peer_id=21914750] [region_id=2240384]
[2020/09/08 02:25:26.564 +00:00] [INFO] [peer.rs:467] ["deleting applied snap file"] [snap_file=6746480_5410_7661] [peer_id=21929658] [region_id=6746480]
[2020/09/08 02:25:26.585 +00:00] [INFO] [peer.rs:467] ["deleting applied snap file"] [snap_file=6807172_597_1334] [peer_id=21929571] [region_id=6807172]
[2020/09/08 02:25:26.604 +00:00] [INFO] [peer.rs:467] ["deleting applied snap file"] [snap_file=7796869_589_2597] [peer_id=21914743] [region_id=7796869]

不知道这样操作是否正确

4
  1. tikv-server 进程的启动用户有 /tmp 目录的读写权限吗？
  2. 手动创建 /tmp 目录是指之前没有 /tmp 目录？
  3. 手动创建 /tmp 目录后，目前 tikv-server 提供服务正常吗？可以通过 pd-ctl 执行 store 查看节点状态，看下这个 tikv 节点上有没有 region count 和 leader count，如果有，说明正常提供服务