tikv 无法启动,terminate called after throwing an instance of 'std::bad_alloc'

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:
【 TiDB 使用环境】tiup 部署的单机集群,v4.0.7
【概述】场景+问题概述

tikv 报错日志:
: Invalid conf pair: prof:true
: Invalid conf pair: prof_active:false
terminate called after throwing an instance of ‘std::bad_alloc’
what(): std::bad_alloc

服务器内存是充足的。

【集群配置】:

global:
user: “tidb”
ssh_port: 22

monitored:
node_exporter_port: 9100
blackbox_exporter_port: 9115

deploy_dir: “/tidb/deploy/yms-develop/monitored”
data_dir: “/tidb/data/yms-develop/monitored”
log_dir: “/tidb/deploy/yms-develop/monitored/log”

server_configs:
tidb:
log.slow-threshold: 300
binlog.enable: false
binlog.ignore-error: false
performance.txn-total-size-limit: 10737418240

tikv:
readpool.storage.use-unified-pool: true
readpool.coprocessor.use-unified-pool: true
readpool.unified.max-thread-count: 10
storage.block-cache.shared: true
storage.block-cache.capacity: “16GB”

pd:
schedule.leader-schedule-limit: 4
schedule.region-schedule-limit: 2048
schedule.replica-schedule-limit: 64
replication.enable-placement-rules: true
replication.location-labels:
- host

pd_servers:

  • host: dev-k8s1
    name: “pd”
    client_port: 2300
    peer_port: 12300
    deploy_dir: “/tidb/deploy/yms-develop/pd”
    data_dir: “/tidb/data/yms-develop/pd”
    log_dir: “/tidb/deploy/yms-develop/pd/log”
    tidb_servers:
  • host: dev-k8s1
    port: 4000
    status_port: 10080
    deploy_dir: “/tidb/deploy/yms-develop/tidb”
    log_dir: “/tidb/deploy/yms-develop/tidb/log”
    tikv_servers:
  • host: dev-k8s1
    port: 20160
    status_port: 20180
    deploy_dir: “/tidb/deploy/yms-develop/tikv”
    data_dir: “/tidb/data/yms-develop/tikv”
    log_dir: “/tidb/deploy/yms-develop/tikv/log”
    config:
    server.labels: { host: “dev-k8s1” }

1)部署前有没有用tiup check和apply下?
2)目前看报错是jemalloc的问题,你的主机OS版本和官方要求的匹配吗?

运行了,我环境已经运行了2个月了,昨天重启服务器出现了这种清理。
OS是centos 7.6 3.10.0-957.el7.x86_64

看看硬件有没有报错,特别是内存这块的内核日志

内存没问题,在服务器日志里看到了这个:
EXT4-fs error (device rbd0): ext4_lookup:1441: inode #786445: comm tikv-server: deleted inode referenced: 786560

这个是tikv的挂载device。

文件系统问题,看看是否可以用fsck修复,安全期间可以,先把这个主机上的tikv缩容掉。

文件系统没有问题;
这是tikv启动的参数,麻烦帮忙看下有没有问题。

[2022/01/12 12:34:27.117 +08:00] [INFO] [config.rs:752] [“data dir”] [mount_fs=“FsInfo { tp: “ext4”, opts: “rw,noatime,nodelalloc,stripe=1024,data=ordered”, mnt_dir: “/tidb/data/yms-develop”, fsname: “/dev/rbd0” }”] [data_path=/tidb/data/yms-develop/tikv]
[2022/01/12 12:34:27.119 +08:00] [INFO] [config.rs:752] [“data dir”] [mount_fs=“FsInfo { tp: “ext4”, opts: “rw,noatime,nodelalloc,stripe=1024,data=ordered”, mnt_dir: “/tidb/data/yms-develop”, fsname: “/dev/rbd0” }”] [data_path=/tidb/data/yms-develop/tikv/raft]
[2022/01/12 12:34:27.119 +08:00] [INFO] [server.rs:212] [“using config”] [config="{“log-level”:“info”,“log-file”:"/tidb/deploy/yms-develop/tikv/log/tikv.log",“log-format”:“text”,“slow-log-file”:"",“slow-log-threshold”:“1s”,“log-rotation-timespan”:“1d”,“log-rotation-size”:“300MiB”,“panic-when-unexpected-key-or-data”:false,“readpool”:{“unified”:{“min-thread-count”:1,“max-thread-count”:10,“stack-size”:“10MiB”,“max-tasks-per-worker”:2000},“storage”:{“use-unified-pool”:true,“high-concurrency”:8,“normal-concurrency”:8,“low-concurrency”:8,“max-tasks-per-worker-high”:2000,“max-tasks-per-worker-normal”:2000,“max-tasks-per-worker-low”:2000,“stack-size”:“10MiB”},“coprocessor”:{“use-unified-pool”:true,“high-concurrency”:24,“normal-concurrency”:24,“low-concurrency”:24,“max-tasks-per-worker-high”:2000,“max-tasks-per-worker-normal”:2000,“max-tasks-per-worker-low”:2000,“stack-size”:“10MiB”}},“server”:{“addr”:“0.0.0.0:20160”,“advertise-addr”:“dev-k8s1:20160”,“status-addr”:“0.0.0.0:20180”,“advertise-status-addr”:“dev-k8s1:20180”,“status-thread-pool-size”:1,“max-grpc-send-msg-len”:10485760,“grpc-compression-type”:“none”,“grpc-concurrency”:4,“grpc-concurrent-stream”:1024,“grpc-raft-conn-num”:1,“grpc-memory-pool-quota”:9223372036854775807,“grpc-stream-initial-window-size”:“2MiB”,“grpc-keepalive-time”:“10s”,“grpc-keepalive-timeout”:“3s”,“concurrent-send-snap-limit”:32,“concurrent-recv-snap-limit”:32,“end-point-recursion-limit”:1000,“end-point-stream-channel-size”:8,“end-point-batch-row-limit”:64,“end-point-stream-batch-row-limit”:128,“end-point-enable-batch-if-possible”:true,“end-point-request-max-handle-duration”:“1m”,“end-point-max-concurrency”:30,“snap-max-write-bytes-per-sec”:“100MiB”,“snap-max-total-size”:“0KiB”,“stats-concurrency”:1,“heavy-load-threshold”:300,“heavy-load-wait-duration”:“1ms”,“enable-request-batch”:true,“request-batch-enable-cross-command”:false,“request-batch-wait-duration”:“1ms”,“labels”:{“host”:“dev-k8s1”}},“storage”:{“data-dir”:"/tidb/data/yms-develop/tikv",“gc-ratio-threshold”:1.1,“max-key-size”:4096,“scheduler-concurrency”:524288,“scheduler-worker-pool-size”:8,“scheduler-pending-write-threshold”:“100MiB”,“reserve-space”:“2GiB”,“block-cache”:{“shared”:true,“capacity”:“16GiB”,“num-shard-bits”:6,“strict-capacity-limit”:false,“high-pri-pool-ratio”:0.8,“memory-allocator”:“nodump”}},“pd”:{“endpoints”:[“dev-k8s1:2300”],“retry-interval”:“300ms”,“retry-max-count”:9223372036854775807,“retry-log-every”:10,“update-interval”:“10m”},“metric”:{“interval”:“15s”,“address”:"",“job”:“tikv”},“raftstore”:{“sync-log”:true,“prevote”:true,“raftdb-path”:"/tidb/data/yms-develop/tikv/raft",“capacity”:“0KiB”,“raft-base-tick-interval”:“1s”,“raft-heartbeat-ticks”:2,“raft-election-timeout-ticks”:10,“raft-min-election-timeout-ticks”:10,“raft-max-election-timeout-ticks”:20,“raft-max-size-per-msg”:“1MiB”,“raft-max-inflight-msgs”:256,“raft-entry-max-size”:“8MiB”,“raft-log-gc-tick-interval”:“10s”,“raft-log-gc-threshold”:50,“raft-log-gc-count-limit”:73728,“raft-log-gc-size-limit”:“72MiB”,“raft-entry-cache-life-time”:“30s”,“raft-reject-transfer-leader-duration”:“3s”,“split-region-check-tick-interval”:“10s”,“region-split-check-diff”:“6MiB”,“region-compact-check-interval”:“5m”,“clean-stale-peer-delay”:“11m”,“region-compact-check-step”:100,“region-compact-min-tombstones”:10000,“region-compact-tombstones-percent”:30,“pd-heartbeat-tick-interval”:“1m”,“pd-store-heartbeat-tick-interval”:“10s”,“snap-mgr-gc-tick-interval”:“1m”,“snap-gc-timeout”:“4h”,“lock-cf-compact-interval”:“10m”,“lock-cf-compact-bytes-threshold”:“256MiB”,“notify-capacity”:40960,“messages-per-tick”:4096,“max-peer-down-duration”:“5m”,“max-leader-missing-duration”:“2h”,“abnormal-leader-missing-duration”:“10m”,“peer-stale-state-check-interval”:“5m”,“leader-transfer-max-log-lag”:10,“snap-apply-batch-size”:“10MiB”,“consistency-check-interval”:“0s”,“report-region-flow-interval”:“1m”,“raft-store-max-leader-lease”:“9s”,“right-derive-when-split”:true,“allow-remove-leader”:false,“merge-max-log-gap”:10,“merge-check-tick-interval”:“10s”,“use-delete-range”:false,“cleanup-import-sst-interval”:“10m”,“local-read-batch-size”:1024,“apply-max-batch-size”:256,“apply-pool-size”:2,“apply-reschedule-duration”:“5s”,“store-max-batch-size”:256,“store-pool-size”:2,“store-reschedule-duration”:“5s”,“future-poll-size”:1,“hibernate-regions”:false,“hibernate-timeout”:“10m”,“early-apply”:true,“dev-assert”:false,“apply-yield-duration”:“500ms”,“perf-level”:1},“coprocessor”:{“split-region-on-table”:false,“batch-split-limit”:10,“region-max-size”:“144MiB”,“region-split-size”:“96MiB”,“region-max-keys”:1440000,“region-split-keys”:960000},“rocksdb”:{“wal-recovery-mode”:2,“wal-dir”:"",“wal-ttl-seconds”:0,“wal-size-limit”:“0KiB”,“max-total-wal-size”:“4GiB”,“max-background-jobs”:8,“max-manifest-file-size”:“128MiB”,“create-if-missing”:true,“max-open-files”:40960,“enable-statistics”:true,“stats-dump-period”:“10m”,“compaction-readahead-size”:“0KiB”,“info-log-max-size”:“1GiB”,“info-log-roll-time”:“0s”,“info-log-keep-log-file-num”:10,“info-log-dir”:"",“rate-bytes-per-sec”:“0KiB”,“rate-limiter-mode”:2,“auto-tuned”:false,“bytes-per-sync”:“1MiB”,“wal-bytes-per-sync”:“512KiB”,“max-sub-compactions”:3,“writable-file-max-buffer-size”:“1MiB”,“use-direct-io-for-flush-and-compaction”:false,“enable-pipelined-write”:true,“enable-multi-batch-write”:true,“enable-unordered-write”:false,“defaultcf”:{“block-size”:“64KiB”,“block-cache-size”:“36861MiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:true,“optimize-filters-for-hits”:true,“whole-key-filtering”:true,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“lz4”,“lz4”,“lz4”,“zstd”,“zstd”],“write-buffer-size”:“128MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“512MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:4,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:3,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“64GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:true,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“normal”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“writecf”:{“block-size”:“64KiB”,“block-cache-size”:“22116MiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:true,“optimize-filters-for-hits”:false,“whole-key-filtering”:false,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“lz4”,“lz4”,“lz4”,“zstd”,“zstd”],“write-buffer-size”:“128MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“512MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:4,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:3,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“64GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:true,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“read-only”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“lockcf”:{“block-size”:“16KiB”,“block-cache-size”:“1GiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:true,“optimize-filters-for-hits”:false,“whole-key-filtering”:true,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“no”,“no”,“no”,“no”,“no”],“write-buffer-size”:“32MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“128MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:1,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:0,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“64GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:true,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“read-only”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“raftcf”:{“block-size”:“16KiB”,“block-cache-size”:“128MiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:true,“optimize-filters-for-hits”:true,“whole-key-filtering”:true,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“no”,“no”,“no”,“no”,“no”],“write-buffer-size”:“128MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“128MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:1,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:0,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“64GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:true,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“read-only”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“titan”:{“enabled”:false,“dirname”:"",“disable-gc”:false,“max-background-gc”:4,“purge-obsolete-files-period”:“10s”}},“raftdb”:{“wal-recovery-mode”:2,“wal-dir”:"",“wal-ttl-seconds”:0,“wal-size-limit”:“0KiB”,“max-total-wal-size”:“4GiB”,“max-background-jobs”:4,“max-manifest-file-size”:“20MiB”,“create-if-missing”:true,“max-open-files”:40960,“enable-statistics”:true,“stats-dump-period”:“10m”,“compaction-readahead-size”:“0KiB”,“info-log-max-size”:“1GiB”,“info-log-roll-time”:“0s”,“info-log-keep-log-file-num”:10,“info-log-dir”:"",“max-sub-compactions”:2,“writable-file-max-buffer-size”:“1MiB”,“use-direct-io-for-flush-and-compaction”:false,“enable-pipelined-write”:true,“enable-unordered-write”:false,“allow-concurrent-memtable-write”:true,“bytes-per-sync”:“1MiB”,“wal-bytes-per-sync”:“512KiB”,“defaultcf”:{“block-size”:“64KiB”,“block-cache-size”:“2GiB”,“disable-block-cache”:false,“cache-index-and-filter-blocks”:true,“pin-l0-filter-and-index-blocks”:true,“use-bloom-filter”:false,“optimize-filters-for-hits”:true,“whole-key-filtering”:true,“bloom-filter-bits-per-key”:10,“block-based-bloom-filter”:false,“read-amp-bytes-per-bit”:0,“compression-per-level”:[“no”,“no”,“lz4”,“lz4”,“lz4”,“zstd”,“zstd”],“write-buffer-size”:“128MiB”,“max-write-buffer-number”:5,“min-write-buffer-number-to-merge”:1,“max-bytes-for-level-base”:“512MiB”,“target-file-size-base”:“8MiB”,“level0-file-num-compaction-trigger”:4,“level0-slowdown-writes-trigger”:20,“level0-stop-writes-trigger”:36,“max-compaction-bytes”:“2GiB”,“compaction-pri”:0,“dynamic-level-bytes”:true,“num-levels”:7,“max-bytes-for-level-multiplier”:10,“compaction-style”:0,“disable-auto-compactions”:false,“soft-pending-compaction-bytes-limit”:“64GiB”,“hard-pending-compaction-bytes-limit”:“256GiB”,“force-consistency-checks”:true,“prop-size-index-distance”:4194304,“prop-keys-index-distance”:40960,“enable-doubly-skiplist”:true,“titan”:{“min-blob-size”:“1KiB”,“blob-file-compression”:“lz4”,“blob-cache-size”:“0KiB”,“min-gc-batch-size”:“16MiB”,“max-gc-batch-size”:“64MiB”,“discardable-ratio”:0.5,“sample-ratio”:0.1,“merge-small-file-threshold”:“8MiB”,“blob-run-mode”:“normal”,“level-merge”:false,“range-merge”:true,“max-sorted-runs”:20,“gc-merge-rewrite”:false}},“titan”:{“enabled”:false,“dirname”:"",“disable-gc”:false,“max-background-gc”:4,“purge-obsolete-files-period”:“10s”}},“security”:{“ca-path”:"",“cert-path”:"",“key-path”:"",“cert-allowed-cn”:[],“encryption”:{“data-encryption-method”:“plaintext”,“data-key-rotation-period”:“7d”,“master-key”:{“type”:“plaintext”},“previous-master-key”:{“type”:“plaintext”}}},“import”:{“num-threads”:8,“stream-channel-window”:128},“backup”:{“num-threads”:22},“pessimistic-txn”:{“enabled”:true,“wait-for-lock-timeout”:“1s”,“wake-up-delay-duration”:“20ms”,“pipelined”:false},“gc”:{“ratio-threshold”:1.1,“batch-keys”:512,“max-write-bytes-per-sec”:“0KiB”},“split”:{“qps-threshold”:3000,“split-balance-score”:0.25,“split-contained-score”:0.5,“detect-times”:10,“sample-num”:20,“sample-threshold”:100},“cdc”:{“min-ts-interval”:“1s”}}"]
[2022/01/12 12:34:27.119 +08:00] [INFO] [mod.rs:335] [“starting working thread”] [worker=addr-resolver]
[2022/01/12 12:34:27.120 +08:00] [INFO] [mod.rs:335] [“starting working thread”] [worker=region-collector-worker]
[2022/01/12 12:34:27.120 +08:00] [INFO] [mod.rs:85] [“encryption: none of key dictionary and file dictionary are found.”]
[2022/01/12 12:34:27.120 +08:00] [INFO] [mod.rs:374] [“encryption is disabled.”]

已经运行2个月,参数问题的肯能性很低很低,大概率是文件系统的问题,最近硬件有没有告警? 主机有没有自动重启? raid卡是否有问题?

硬件上,没有检测出问题。我们是基于虚拟机部署的,昨天重启了宿主机。

请问出现该问题是发生在宿主机重启之后吗?目前集群中 tikv 节点还在报错吗?