default rocksdb doesn't exist, but raftdb or raft engine exists

🪐 TiDB 技术问题
1

【 TiDB 使用环境】
【概述】成功过一次，删除集群重装之后就无法启动
【背景】安装部署
【现象】tikv无法启动
【业务影响】
【TiDB 版本】 v5.4.0
【附件】

  • 问题tikv的日志
[2022/03/17 08:49:37.585 +00:00] [INFO] [server.rs:347] ["connect to PD cluster"][cluster_id=7075982584926865257]
[2022/03/17 08:49:37.585 +00:00] [INFO] [config.rs:1982] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/03/17 08:49:37.585 +00:00] [INFO] [config.rs:2005] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/03/17 08:49:37.586 +00:00] [FATAL] [setup.rs:302] ["invalid configuration: default rocksdb doesn't exist, but raftdb or raft engine exists"]
2

1）删除之前是什么版本？很可能是之前的版本遗留和当前的版本不兼容；
2）登录tikv实例上，tikv-20160/{raftdb.info,rocksdb.info}的日志是否有报错？

3

是同一个版本的，都是这两天的事情，把问题节点排除掉，集群可以启动

问题节点
rocksdb.info是空的
raftdb.info日志

RocksDB version: 6.4.6
Git sha rocksdb_build_git_sha:@f236fe443d06337ca2538414c62f70ba17b8e409@
Compile date Jan 25 2022
DB SUMMARY
SST files in /tidb-data/tikv-20160/raft dir, Total Num: 0, files: 
Write Ahead Log file in /tidb-data/tikv-20160/raft: 
                        Options.error_if_exists: 0
                      Options.create_if_missing: 1
                        Options.paranoid_checks: 1
                                    Options.env: 0x7f60e08eaca0
                               Options.info_log: 0x7f60e08eacd0
               Options.max_file_opening_threads: 16
                             Options.statistics: 0x7f60e087e9b0
                              Options.use_fsync: 0
                      Options.max_log_file_size: 1073741824
                 Options.max_manifest_file_size: 20971520
                  Options.log_file_time_to_roll: 0
                      Options.keep_log_file_num: 10
                   Options.recycle_log_file_num: 0
                        Options.allow_fallocate: 1
                       Options.allow_mmap_reads: 0
                      Options.allow_mmap_writes: 0
                       Options.use_direct_reads: 0
                       Options.use_direct_io_for_flush_and_compaction: 0
         Options.create_missing_column_families: 0
                             Options.db_log_dir: 
                                Options.wal_dir: /tidb-data/tikv-20160/raft
               Options.table_cache_numshardbits: 6
                     Options.max_subcompactions: 2
                        Options.WAL_ttl_seconds: 0
                      Options.WAL_size_limit_MB: 0
            Options.manifest_preallocation_size: 4194304
                    Options.is_fd_close_on_exec: 1
                  Options.advise_random_on_open: 1
                   Options.db_write_buffer_size: 0
                   Options.write_buffer_manager: 0x7f60e08ead60
        Options.access_hint_on_compaction_start: 1
 Options.new_table_reader_for_compaction_inputs: 0
          Options.random_access_max_buffer_size: 1048576
                     Options.use_adaptive_mutex: 0
                           Options.rate_limiter: (nil)
    Options.sst_file_manager.rate_bytes_per_sec: 0
                      Options.wal_recovery_mode: 2
                 Options.enable_thread_tracking: 0
                 Options.enable_pipelined_write: 1
                 Options.unordered_write: 0
              Options.enable_multi_thread_write: 0
        Options.allow_concurrent_memtable_write: 1
     Options.enable_write_thread_adaptive_yield: 1
            Options.write_thread_max_yield_usec: 100
           Options.write_thread_slow_yield_usec: 3
                              Options.row_cache: None
                             Options.wal_filter: None
            Options.avoid_flush_during_recovery: 0
            Options.allow_ingest_behind: 0
            Options.preserve_deletes: 0
            Options.two_write_queues: 0
            Options.manual_wal_flush: 0
            Options.atomic_flush: 0
            Options.avoid_unnecessary_blocking_io: 0
                Options.persist_stats_to_disk: 0
                Options.log_readahead_size: 0
            Options.max_background_jobs: 4
            Options.base_background_compactions: -1
            Options.max_background_compactions: 3
            Options.avoid_flush_during_shutdown: 0
          Options.writable_file_max_buffer_size: 1048576
            Options.delayed_write_rate : 16777216
            Options.max_total_wal_size: 4294967296
            Options.delete_obsolete_files_period_micros: 21600000000
                  Options.stats_dump_period_sec: 600
                Options.stats_persist_period_sec: 600
                Options.stats_history_buffer_size: 1048576
                         Options.max_open_files: 40960
                         Options.bytes_per_sync: 1048576
                     Options.wal_bytes_per_sync: 524288
                  Options.strict_bytes_per_sync: 0
      Options.compaction_readahead_size: 0
                 Options.max_background_flushes: 1
Compression algorithms supported:
	kZSTDNotFinalCompression supported: 1
	kZSTD supported: 1
	kXpressCompression supported: 0
	kLZ4HCCompression supported: 1
	kLZ4Compression supported: 1
	kBZip2Compression supported: 1
	kZlibCompression supported: 1
	kSnappyCompression supported: 1
Fast CRC32 supported: Not supported on x86
[2022/03/17 09:57:33.072 +00:00][5][INFO] [db/db_impl/db_impl_open.cc:250] Creating manifest 1 
[2022/03/17 09:57:33.240 +00:00][5][INFO] [db/version_set.cc:4362] Recovering from manifest file: /tidb-data/tikv-20160/raft/MANIFEST-000001
[2022/03/17 09:57:33.240 +00:00][5][INFO] [db/column_family.cc:481] --------------- Options for column family [default]:
              Options.comparator: leveldb.BytewiseComparator
          Options.merge_operator: None
       Options.compaction_filter: None
       Options.compaction_filter_factory: None
 Options.sst_partitioner_factory: None
        Options.memtable_factory: DoublySkipListFactory
           Options.table_factory: BlockBasedTable
           table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f60e085fe78)
  cache_index_and_filter_blocks: 1
  cache_index_and_filter_blocks_with_high_priority: 1
  pin_l0_filter_and_index_blocks_in_cache: 1
  pin_top_level_index_and_filter: 1
  index_type: 0
  data_block_index_type: 0
  index_shortening: 1
  data_block_hash_table_util_ratio: 0.750000
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7f60e0836e90
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 2224029696
    num_shard_bits : 6
    strict_capacity_limit : 0
    memory_allocator : JemallocNodumpAllocator
    high_pri_pool_ratio: 0.800
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 65536
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  metadata_block_size: 4096
  partition_filters: 0
  use_delta_encoding: 1
  filter_policy: nullptr
  whole_key_filtering: 1
  verify_compression: 0
  read_amp_bytes_per_bit: 0
  format_version: 2
  enable_index_compression: 1
  block_align: 0
       Options.write_buffer_size: 134217728
 Options.max_write_buffer_number: 5
       Options.compression[0]: NoCompression
       Options.compression[1]: NoCompression
       Options.compression[2]: LZ4
       Options.compression[3]: LZ4
       Options.compression[4]: LZ4
       Options.compression[5]: ZSTD
       Options.compression[6]: ZSTD
                 Options.bottommost_compression: Disabled
      Options.prefix_extractor: nullptr
  Options.memtable_insert_with_hint_prefix_extractor: RaftPrefixSliceTransform
            Options.num_levels: 7
       Options.min_write_buffer_number_to_merge: 1
    Options.max_write_buffer_number_to_maintain: 0
           Options.bottommost_compression_opts.window_bits: -14
                 Options.bottommost_compression_opts.level: 32767
              Options.bottommost_compression_opts.strategy: 0
        Options.bottommost_compression_opts.max_dict_bytes: 0
        Options.bottommost_compression_opts.zstd_max_train_bytes: 0
                 Options.bottommost_compression_opts.enabled: true
           Options.compression_opts.window_bits: -14
                 Options.compression_opts.level: 32767
              Options.compression_opts.strategy: 0
        Options.compression_opts.max_dict_bytes: 0
        Options.compression_opts.zstd_max_train_bytes: 0
                 Options.compression_opts.enabled: false
     Options.level0_file_num_compaction_trigger: 4
         Options.level0_slowdown_writes_trigger: 10000
             Options.level0_stop_writes_trigger: 10000
                  Options.target_file_size_base: 8388608
            Options.target_file_size_multiplier: 1
               Options.max_bytes_for_level_base: 536870912
                     Options.snap_refresh_nanos: 0
Options.level_compaction_dynamic_level_bytes: 1
         Options.max_bytes_for_level_multiplier: 10.000000
Options.max_bytes_for_level_multiplier_addtl[0]: 1
Options.max_bytes_for_level_multiplier_addtl[1]: 1
Options.max_bytes_for_level_multiplier_addtl[2]: 1
Options.max_bytes_for_level_multiplier_addtl[3]: 1
Options.max_bytes_for_level_multiplier_addtl[4]: 1
Options.max_bytes_for_level_multiplier_addtl[5]: 1
Options.max_bytes_for_level_multiplier_addtl[6]: 1
      Options.max_sequential_skip_in_iterations: 8
                   Options.max_compaction_bytes: 2147483648
                       Options.arena_block_size: 16777216
  Options.soft_pending_compaction_bytes_limit: 0
  Options.hard_pending_compaction_bytes_limit: 0
      Options.rate_limit_delay_max_milliseconds: 100
               Options.disable_auto_compactions: 0
               Options.disable_write_stall: 0
                       Options.compaction_style: kCompactionStyleLevel
                         Options.compaction_pri: kByCompensatedSize
Options.compaction_options_universal.size_ratio: 1
Options.compaction_options_universal.min_merge_width: 2
Options.compaction_options_universal.max_merge_width: 4294967295
Options.compaction_options_universal.max_size_amplification_percent: 200
Options.compaction_options_universal.compression_size_percent: -1
Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
Options.compaction_options_fifo.max_table_files_size: 1073741824
Options.compaction_options_fifo.allow_compaction: 0
                  Options.table_properties_collectors: 
                  Options.inplace_update_support: 0
                Options.inplace_update_num_locks: 10000
              Options.memtable_prefix_bloom_size_ratio: 0.000000
              Options.memtable_whole_key_filtering: 0
  Options.memtable_huge_page_size: 0
                          Options.bloom_locality: 0
                   Options.max_successive_merges: 0
               Options.optimize_filters_for_hits: 1
               Options.paranoid_file_checks: 0
               Options.force_consistency_checks: 0
               Options.report_bg_io_stats: 0
                              Options.ttl: 0
         Options.periodic_compaction_seconds: 0
4

对应tikv节点的日志多发出来看看 ，特别是报错前后的。

5

https://docs.pingcap.com/zh/tidb/stable/tikv-configuration-file#raft-engine
看下是不是设置raft engine了

6

没有设置 默认配置

7 
[2022/03/18 01:47:13.878 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:47:13.879 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:47:13.879 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:47:13.879 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:47:13.879 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:47:13.879 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/03/18 01:47:13.879 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:47:13.879 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:47:13.879 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:47:13.879 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={3, 0, 2, 1}"]
[2022/03/18 01:47:13.879 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:47:13.879 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:47:13.879 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:47:13.879 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:47:13.879 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:47:13.879 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:47:13.880 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:47:13.881 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ffae802e150 for subchannel 0x7ffaed015cc0"]
[2022/03/18 01:47:13.882 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:47:13.884 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ffae782e150 for subchannel 0x7ffaed015e80"]
[2022/03/18 01:47:13.885 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:47:13.887 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ffae702e150 for subchannel 0x7ffaed015cc0"]
[2022/03/18 01:47:13.889 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:47:13.890 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ffae682e150 for subchannel 0x7ffaed015e80"]
[2022/03/18 01:47:13.892 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:47:13.894 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ffae602e150 for subchannel 0x7ffaed015cc0"]
[2022/03/18 01:47:13.896 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:47:13.896 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:47:13.897 +00:00] [INFO] [server.rs:347] ["connect to PD cluster"] [cluster_id=7076006086108832074]
[2022/03/18 01:47:13.897 +00:00] [INFO] [config.rs:1982] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:47:13.897 +00:00] [INFO] [config.rs:2005] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:47:13.898 +00:00] [FATAL] [setup.rs:302] ["invalid configuration: default rocksdb doesn't exist, but raftdb or raft engine exists"]
[2022/03/18 01:47:29.597 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:47:29.597 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:47:29.597 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:47:29.597 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:47:29.597 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:47:29.597 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/03/18 01:47:29.597 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:47:29.597 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:47:29.597 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:47:29.597 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={1, 3, 0, 2}"]
[2022/03/18 01:47:29.597 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:47:29.597 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:47:29.598 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:47:29.598 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:47:29.598 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:47:29.601 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:47:29.601 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:47:29.605 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f5628e2e150 for subchannel 0x7f562e615cc0"]
[2022/03/18 01:47:29.607 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:47:29.608 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f562842e150 for subchannel 0x7f562e615e80"]
[2022/03/18 01:47:29.610 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:47:29.612 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f562802e150 for subchannel 0x7f562e615cc0"]
[2022/03/18 01:47:29.615 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:47:29.617 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f5627c2e150 for subchannel 0x7f562e615e80"]
[2022/03/18 01:47:29.618 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:47:29.620 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f562782e150 for subchannel 0x7f562e615cc0"]
[2022/03/18 01:47:29.622 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:47:29.623 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:47:29.624 +00:00] [INFO] [server.rs:347] ["connect to PD cluster"] [cluster_id=7076006086108832074]
[2022/03/18 01:47:29.624 +00:00] [INFO] [config.rs:1982] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:47:29.624 +00:00] [INFO] [config.rs:2005] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:47:29.624 +00:00] [FATAL] [setup.rs:302] ["invalid configuration: default rocksdb doesn't exist, but raftdb or raft engine exists"]
[2022/03/18 01:47:45.372 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:47:45.372 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:47:45.372 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:47:45.372 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:47:45.372 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:47:45.372 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/03/18 01:47:45.372 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:47:45.372 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:47:45.372 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:47:45.372 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={1, 2, 3, 0}"]
[2022/03/18 01:47:45.372 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:47:45.372 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:47:45.372 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:47:45.372 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:47:45.372 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:47:45.374 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:47:45.375 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:47:45.376 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f958ba2e150 for subchannel 0x7f9590a15cc0"]
[2022/03/18 01:47:45.378 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:47:45.379 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f958a62e150 for subchannel 0x7f9590a15e80"]
[2022/03/18 01:47:45.380 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:47:45.382 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f958a42e150 for subchannel 0x7f9590a15cc0"]
[2022/03/18 01:47:45.385 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:47:45.386 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f958a02e150 for subchannel 0x7f9590a15e80"]
[2022/03/18 01:47:45.388 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:47:45.389 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f9589c2e150 for subchannel 0x7f9590a15cc0"]
[2022/03/18 01:47:45.391 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:47:45.391 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:47:45.392 +00:00] [INFO] [server.rs:347] ["connect to PD cluster"] [cluster_id=7076006086108832074]
[2022/03/18 01:47:45.392 +00:00] [INFO] [config.rs:1982] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:47:45.392 +00:00] [INFO] [config.rs:2005] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:47:45.393 +00:00] [FATAL] [setup.rs:302] ["invalid configuration: default rocksdb doesn't exist, but raftdb or raft engine exists"]
[2022/03/18 01:48:00.892 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:48:00.892 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:48:00.892 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:48:00.892 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:48:00.892 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:48:00.892 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/03/18 01:48:00.892 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:48:00.892 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:48:00.893 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:48:00.893 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={1, 2, 0, 3}"]
[2022/03/18 01:48:00.893 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:48:00.893 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:48:00.893 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:48:00.893 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:48:00.893 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:48:00.895 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:48:00.896 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:48:00.898 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f1706a2d150 for subchannel 0x7f170c215cc0"]
[2022/03/18 01:48:00.899 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:48:00.901 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f170602e150 for subchannel 0x7f170c215e80"]
[2022/03/18 01:48:00.903 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:48:00.904 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f1705c2e150 for subchannel 0x7f170c215cc0"]
[2022/03/18 01:48:00.907 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:48:00.908 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f170582e150 for subchannel 0x7f170c215e80"]
[2022/03/18 01:48:00.910 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:48:00.911 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f170542e150 for subchannel 0x7f170c215cc0"]
[2022/03/18 01:48:00.913 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:48:00.914 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:48:00.914 +00:00] [INFO] [server.rs:347] ["connect to PD cluster"] [cluster_id=7076006086108832074]
[2022/03/18 01:48:00.915 +00:00] [INFO] [config.rs:1982] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:48:00.915 +00:00] [INFO] [config.rs:2005] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:48:00.915 +00:00] [FATAL] [setup.rs:302] ["invalid configuration: default rocksdb doesn't exist, but raftdb or raft engine exists"]
[2022/03/18 01:48:16.535 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:48:16.536 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:48:16.536 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:48:16.536 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:48:16.536 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:48:16.536 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/03/18 01:48:16.536 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:48:16.536 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:48:16.536 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:48:16.536 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={0, 2, 1, 3}"]
[2022/03/18 01:48:16.536 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:48:16.536 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:48:16.536 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:48:16.536 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:48:16.536 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:48:16.539 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:48:16.540 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:48:16.541 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ff40da2e150 for subchannel 0x7ff412615cc0"]
[2022/03/18 01:48:16.543 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:48:16.544 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ff40d02d150 for subchannel 0x7ff412615e80"]
[2022/03/18 01:48:16.546 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:48:16.548 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ff40c82e150 for subchannel 0x7ff412615cc0"]
[2022/03/18 01:48:16.550 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:48:16.551 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ff40c42e150 for subchannel 0x7ff412615e80"]
[2022/03/18 01:48:16.553 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:48:16.555 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7ff40c02e150 for subchannel 0x7ff412615cc0"]
[2022/03/18 01:48:16.558 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:48:16.558 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:48:16.559 +00:00] [INFO] [server.rs:347] ["connect to PD cluster"] [cluster_id=7076006086108832074]
[2022/03/18 01:48:16.559 +00:00] [INFO] [config.rs:1982] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:48:16.559 +00:00] [INFO] [config.rs:2005] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:48:16.559 +00:00] [FATAL] [setup.rs:302] ["invalid configuration: default rocksdb doesn't exist, but raftdb or raft engine exists"]
[2022/03/18 01:48:32.310 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:48:32.310 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:48:32.310 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:48:32.310 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:48:32.310 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:48:32.310 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/03/18 01:48:32.310 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:48:32.310 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:48:32.310 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:48:32.310 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={1, 2, 0, 3}"]
[2022/03/18 01:48:32.310 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:48:32.310 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:48:32.310 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:48:32.310 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:48:32.310 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:48:32.314 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:48:32.315 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:48:32.317 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f216bc2e150 for subchannel 0x7f2170e15cc0"]
[2022/03/18 01:48:32.318 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:48:32.320 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f216b22e150 for subchannel 0x7f2170e15e80"]
[2022/03/18 01:48:32.322 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:48:32.323 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f216aa2e150 for subchannel 0x7f2170e15cc0"]
[2022/03/18 01:48:32.326 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:48:32.328 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f216a62e150 for subchannel 0x7f2170e15e80"]
[2022/03/18 01:48:32.329 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:48:32.331 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f216a22e150 for subchannel 0x7f2170e15cc0"]
[2022/03/18 01:48:32.334 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:48:32.334 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:48:32.335 +00:00] [INFO] [server.rs:347] ["connect to PD cluster"] [cluster_id=7076006086108832074]
[2022/03/18 01:48:32.335 +00:00] [INFO] [config.rs:1982] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:48:32.335 +00:00] [INFO] [config.rs:2005] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:48:32.336 +00:00] [FATAL] [setup.rs:302] ["invalid configuration: default rocksdb doesn't exist, but raftdb or raft engine exists"]
[2022/03/18 01:48:47.960 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:48:47.960 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:48:47.960 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:48:47.960 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:48:47.960 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:48:47.960 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/03/18 01:48:47.960 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:48:47.960 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:48:47.960 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:48:47.960 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={0, 1, 2, 3}"]
[2022/03/18 01:48:47.960 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:48:47.960 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:48:47.960 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:48:47.960 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:48:47.960 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:48:47.964 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:48:47.964 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:48:47.968 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f31a102d150 for subchannel 0x7f31a6215cc0"]
[2022/03/18 01:48:47.969 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:48:47.971 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f31a082e150 for subchannel 0x7f31a6215e80"]
[2022/03/18 01:48:47.973 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:48:47.975 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f31a002e150 for subchannel 0x7f31a6215cc0"]
[2022/03/18 01:48:47.978 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:48:47.979 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f319fc2e150 for subchannel 0x7f31a6215e80"]
[2022/03/18 01:48:47.981 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:48:47.984 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f319f82e150 for subchannel 0x7f31a6215cc0"]
[2022/03/18 01:48:47.986 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:48:47.987 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:48:47.988 +00:00] [INFO] [server.rs:347] ["connect to PD cluster"] [cluster_id=7076006086108832074]
[2022/03/18 01:48:47.988 +00:00] [INFO] [config.rs:1982] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:48:47.988 +00:00] [INFO] [config.rs:2005] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:48:47.988 +00:00] [FATAL] [setup.rs:302] ["invalid configuration: default rocksdb doesn't exist, but raftdb or raft engine exists"]
[2022/03/18 01:49:03.410 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:49:03.410 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:49:03.410 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:49:03.410 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:49:03.410 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:49:03.410 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
8 
[2022/03/18 01:49:03.410 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:49:03.410 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:49:03.410 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:49:03.410 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={0, 2, 1, 3}"]
[2022/03/18 01:49:03.411 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:49:03.411 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:49:03.411 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:49:03.411 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:49:03.411 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:49:03.414 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:49:03.415 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:49:03.416 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f53a2e2e150 for subchannel 0x7f53a8815cc0"]
[2022/03/18 01:49:03.418 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:49:03.419 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f53a282d150 for subchannel 0x7f53a8815e80"]
[2022/03/18 01:49:03.421 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:49:03.423 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f53a222e150 for subchannel 0x7f53a8815b00"]
[2022/03/18 01:49:03.425 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:49:03.427 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f53a1e2e150 for subchannel 0x7f53a8815cc0"]
[2022/03/18 01:49:03.429 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:49:03.430 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f53a1a2e150 for subchannel 0x7f53a8815b00"]
[2022/03/18 01:49:03.433 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:49:03.433 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:49:19.057 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:49:19.057 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:49:19.057 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:49:19.057 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:49:19.057 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:49:19.057 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/03/18 01:49:19.057 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:49:19.057 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:49:19.057 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:49:19.057 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={0, 1, 2, 3}"]
[2022/03/18 01:49:19.057 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:49:19.057 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:49:19.057 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:49:19.057 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:49:19.058 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:49:19.060 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:49:19.061 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:49:19.062 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f58dc82e150 for subchannel 0x7f58e2015cc0"]
[2022/03/18 01:49:19.064 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:49:19.065 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f58dc02e150 for subchannel 0x7f58e2015e80"]
[2022/03/18 01:49:19.067 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:49:19.069 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f58db82e150 for subchannel 0x7f58e2015b00"]
[2022/03/18 01:49:19.071 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:49:19.073 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f58db42e150 for subchannel 0x7f58e2015cc0"]
[2022/03/18 01:49:19.076 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:49:19.078 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f58db02e150 for subchannel 0x7f58e2015b00"]
[2022/03/18 01:49:19.080 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:49:19.081 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:49:19.082 +00:00] [INFO] [server.rs:347] ["connect to PD cluster"] [cluster_id=7076006086108832074]
[2022/03/18 01:49:19.082 +00:00] [INFO] [config.rs:1982] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:49:19.082 +00:00] [INFO] [config.rs:2005] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:49:19.082 +00:00] [FATAL] [setup.rs:302] ["invalid configuration: default rocksdb doesn't exist, but raftdb or raft engine exists"]
[2022/03/18 01:49:34.758 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:49:34.758 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:49:34.758 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:49:34.758 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:49:34.758 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:49:34.758 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/03/18 01:49:34.758 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:49:34.758 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:49:34.758 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:49:34.758 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={3, 1, 2, 0}"]
[2022/03/18 01:49:34.758 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:49:34.758 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:49:34.758 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:49:34.758 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:49:34.758 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:49:34.760 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:49:34.761 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:49:34.762 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f4f50e2e150 for subchannel 0x7f4f56415cc0"]
[2022/03/18 01:49:34.764 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:49:34.765 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f4f5062e150 for subchannel 0x7f4f56415e80"]
[2022/03/18 01:49:34.767 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:49:34.769 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f4f4fc2e150 for subchannel 0x7f4f56415b00"]
[2022/03/18 01:49:34.771 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:49:34.772 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f4f4f82e150 for subchannel 0x7f4f56415cc0"]
[2022/03/18 01:49:34.774 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:49:34.776 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f4f4f42e150 for subchannel 0x7f4f56415b00"]
[2022/03/18 01:49:34.778 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:49:34.779 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:49:34.780 +00:00] [INFO] [server.rs:347] ["connect to PD cluster"] [cluster_id=7076006086108832074]
[2022/03/18 01:49:34.780 +00:00] [INFO] [config.rs:1982] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:49:34.780 +00:00] [INFO] [config.rs:2005] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:49:34.780 +00:00] [FATAL] [setup.rs:302] ["invalid configuration: default rocksdb doesn't exist, but raftdb or raft engine exists"]
[2022/03/18 01:49:50.264 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:49:50.264 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:49:50.264 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:49:50.264 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:49:50.264 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:49:50.264 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/03/18 01:49:50.264 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:49:50.264 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:49:50.264 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:49:50.264 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={0, 1, 2, 3}"]
[2022/03/18 01:49:50.264 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:49:50.264 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:49:50.264 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:49:50.264 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:49:50.264 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:49:50.268 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:49:50.269 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:49:50.270 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f87d202e150 for subchannel 0x7f87d7615cc0"]
[2022/03/18 01:49:50.272 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:49:50.273 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f87d1c2e150 for subchannel 0x7f87d7615e80"]
[2022/03/18 01:49:50.274 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:49:50.276 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f87d182e150 for subchannel 0x7f87d7615cc0"]
[2022/03/18 01:49:50.279 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:49:50.280 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f87d142e150 for subchannel 0x7f87d7615e80"]
[2022/03/18 01:49:50.282 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:49:50.284 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f87d0c2e150 for subchannel 0x7f87d7615cc0"]
[2022/03/18 01:49:50.286 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:49:50.287 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:49:50.287 +00:00] [INFO] [server.rs:347] ["connect to PD cluster"] [cluster_id=7076006086108832074]
[2022/03/18 01:49:50.288 +00:00] [INFO] [config.rs:1982] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:49:50.288 +00:00] [INFO] [config.rs:2005] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:49:50.288 +00:00] [FATAL] [setup.rs:302] ["invalid configuration: default rocksdb doesn't exist, but raftdb or raft engine exists"]
[2022/03/18 01:50:05.587 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:50:05.588 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:50:05.588 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:50:05.588 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:50:05.588 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:50:05.588 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/03/18 01:50:05.588 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:50:05.588 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:50:05.588 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:50:05.588 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={1, 2, 0, 3}"]
[2022/03/18 01:50:05.588 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:50:05.588 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:50:05.588 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:50:05.588 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:50:05.588 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:50:05.591 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:50:05.592 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:50:05.594 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f0d5e62e150 for subchannel 0x7f0d63615cc0"]
[2022/03/18 01:50:05.596 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:50:05.597 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f0d5de2e150 for subchannel 0x7f0d63615e80"]
[2022/03/18 01:50:05.600 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:50:05.601 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f0d5d42e150 for subchannel 0x7f0d63615cc0"]
[2022/03/18 01:50:05.604 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:50:05.606 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f0d5d02e150 for subchannel 0x7f0d63615e80"]
[2022/03/18 01:50:05.608 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:50:05.609 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f0d5c82e150 for subchannel 0x7f0d63615cc0"]
[2022/03/18 01:50:05.612 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:50:05.613 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:50:20.895 +00:00] [INFO] [lib.rs:81] ["Welcome to TiKV"]
[2022/03/18 01:50:20.896 +00:00] [INFO] [lib.rs:86] ["Release Version:   5.4.0"]
[2022/03/18 01:50:20.896 +00:00] [INFO] [lib.rs:86] ["Edition:           Community"]
[2022/03/18 01:50:20.896 +00:00] [INFO] [lib.rs:86] ["Git Commit Hash:   b5262299604df88711d9ed4b84d43e9c507749a2"]
[2022/03/18 01:50:20.896 +00:00] [INFO] [lib.rs:86] ["Git Commit Branch: heads/refs/tags/v5.4.0"]
[2022/03/18 01:50:20.896 +00:00] [INFO] [lib.rs:86] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2022/03/18 01:50:20.896 +00:00] [INFO] [lib.rs:86] ["Rust Version:      rustc 1.56.0-nightly (2faabf579 2021-07-27)"]
[2022/03/18 01:50:20.896 +00:00] [INFO] [lib.rs:86] ["Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"]
[2022/03/18 01:50:20.896 +00:00] [INFO] [lib.rs:86] ["Profile:           dist_release"]
[2022/03/18 01:50:20.896 +00:00] [INFO] [mod.rs:73] ["cgroup quota: memory=9223372036854771712, cpu=None, cores={0, 2, 3, 1}"]
[2022/03/18 01:50:20.896 +00:00] [INFO] [mod.rs:80] ["memory limit in bytes: 6378902528, cpu cores quota: 4"]
[2022/03/18 01:50:20.896 +00:00] [WARN] [lib.rs:511] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2022/03/18 01:50:20.896 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=32768] [param=net.core.somaxconn]
[2022/03/18 01:50:20.896 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=net.ipv4.tcp_syncookies]
[2022/03/18 01:50:20.896 +00:00] [INFO] [config.rs:731] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2022/03/18 01:50:20.896 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.101:2379]
[2022/03/18 01:50:20.896 +00:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/03/18 01:50:20.898 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f8583c33120 for subchannel 0x7f8588e15cc0"]
[2022/03/18 01:50:20.900 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.102:2379]
[2022/03/18 01:50:20.901 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f858322d150 for subchannel 0x7f8588e15e80"]
[2022/03/18 01:50:20.903 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=192.168.10.103:2379]
[2022/03/18 01:50:20.908 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f8582c2e150 for subchannel 0x7f8588e15cc0"]
[2022/03/18 01:50:20.911 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.102:2379]
[2022/03/18 01:50:20.913 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f858242e150 for subchannel 0x7f8588e15e80"]
[2022/03/18 01:50:20.915 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:50:20.918 +00:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f858282e150 for subchannel 0x7f8588e15b00"]
[2022/03/18 01:50:20.921 +00:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://192.168.10.103:2379]
[2022/03/18 01:50:20.921 +00:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"192.168.10.101:2379\", \"192.168.10.102:2379\", \"192.168.10.103:2379\"]"]
[2022/03/18 01:50:20.923 +00:00] [INFO] [server.rs:347] ["connect to PD cluster"] [cluster_id=7076006086108832074]
[2022/03/18 01:50:20.923 +00:00] [INFO] [config.rs:1982] ["readpool.storage.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:50:20.923 +00:00] [INFO] [config.rs:2005] ["readpool.coprocessor.use-unified-pool is not set, set to true by default"]
[2022/03/18 01:50:20.923 +00:00] [FATAL] [setup.rs:302] ["invalid configuration: default rocksdb doesn't exist, but raftdb or raft engine exists"]
9

发了完整的日志，字数限制分成两部分了

10

default rocksdb 不存在，而raftdb存在，严重怀疑之前的安装清理后，没有清理干净。
进入tikv-20160目录后 查看下stat db/IDENTITY 和 stat raft/IDENTITY的时间。

12

stat raft/IDENTITY

  File: raft/IDENTITY
  Size: 37              Blocks: 8          IO Block: 4096   regular file
Device: 802h/2050d      Inode: 9568288     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/    tidb)   Gid: ( 1001/    tidb)
Access: 2022-03-18 02:03:02.849015017 +0000
Modify: 2022-03-18 02:03:02.849015017 +0000
Change: 2022-03-18 02:03:02.881014017 +0000
 Birth: -

stat db/IDENTITY
这个找不到文件

tikv里的文件

-rw-r--r-- 1 tidb tidb      18413 Mar 18 02:03 last_tikv.toml
-rw-r--r-- 1 tidb tidb          0 Mar 18 02:03 LOCK
drwxr-xr-x 2 tidb tidb       4096 Mar 18 02:03 raft
-rw-r--r-- 1 tidb tidb      10197 Mar 18 02:03 raftdb.info
-rw-r--r-- 1 tidb tidb          0 Mar 18 02:03 rocksdb.info
-rw-r--r-- 1 tidb tidb 4911799091 Mar 18 02:03 space_placeholder_file

可以启动的tikv的文件

drwxr-xr-x 2 tidb tidb       4096 Mar 18 02:03 db
drwxr-xr-x 4 tidb tidb       4096 Mar 18 02:03 import
-rw-r--r-- 1 tidb tidb      18413 Mar 18 02:03 last_tikv.toml
-rw-r--r-- 1 tidb tidb          0 Mar 18 02:03 LOCK
drwxr-xr-x 2 tidb tidb       4096 Mar 18 02:03 raft
-rw-r--r-- 1 tidb tidb      43216 Mar 18 02:13 raftdb.info
-rw-r--r-- 1 tidb tidb      71918 Mar 18 02:13 rocksdb.info
drwxr-xr-x 2 tidb tidb       4096 Mar 18 02:03 snap
-rw-r--r-- 1 tidb tidb 3091928432 Mar 18 02:03 space_placeholder_file
13

这个文件找不到，那就是跟上面比较吻合（default rocksdb doesn’t exist和rocksdb.info是空的），启动的时候无法创建default rocksdb实例所需要的文件。
1）部署的时候 ，参数模板是默认的吗？有没有修改过？
2）解决问题比较简单，通过缩容和扩容来处理；
3）看下tikv-20160目录下last_tikv.toml参数文件 对default rocksdb相关的参数。

14

1）你这个是物理机还是虚拟机？ 这异常tikv的主机和其他正常的主机有什么不同？
2) 看看tidb-deploy/tikv-20160/log/tikv.log 日志有什么异常？

15

应该是删除集群的时候没执行pure命令吧，导致tidb的pd里还有kv的信息

16

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。