tikv raw_put出现大量超时

[jwongz]

【TiKV 使用环境】生产环境
【TiKV 版本】v6.5.1
【操作系统】centos7.9
【部署方式】云上部署（什么云）/机器部署（什么机器配置、什么硬盘）
3台物理机、 512GB内存、每台物理机2块7.6TB nvme盘、每块nvme盘对应一个tikv服务

【集群数据量】
store size: 400GB
region size:4TB

【集群节点数】
3台物理机 6个tikv-server
【问题复现路径】做过哪些操作出现的问题
从其他集群通过raw_batch_put 导入数据
qps如图：

batch size:

【遇到的问题：问题现象及影响】
raw_put 超时：

"ErrorMessage":"wait recvLoop: context deadline exceeded","ErrorMessageVerbose":"context deadline exceeded\nwait recvLoop"

tikv server timeout\ngithub.com/tikv/client-go/v2/error.init\n\t/root/gopro/src/gateway/vendor/github.com/tikv/client-go/v2/error/error.go:53\nruntime.doInit\n\t/root/go/src/runtime/proc.go:6506\nruntime.doInit\n\t/root/go/src/runtime/proc.go:6483\nruntime.doInit\n\t/root/go/src/runtime/proc.go:6483\nruntime.doInit\n\t/root/go/src/runtime/proc.go:6483\nruntime.doInit\n\t/root/go/src/runtime/proc.go:6483\nruntime.doInit\n\t/root/go/src/runtime/proc.go:6483\nruntime.doInit\n\t/root/go/src/runtime/proc.go:6483\nruntime.main\n\t/root/go/src/runtime/proc.go:233\nruntime.goexit\n\t/root/go/src/runtime/asm_amd64.s:1598\ngithub.com/tikv/client-go/v2/internal/retry.(*Backoffer).BackoffWithCfgAndMaxSleep\n\t/root/gopro/src/gateway/vendor/github.com/tikv/client-go/v2/internal/retry/backoff.go:160\ngithub.com/tikv/client-go/v2/internal/retry.(*Backoffer).Backoff\n\t/root/gopro/src/gateway/vendor/github.com/tikv/client-go/v2/internal/retry/backoff.go:120\ngithub.com/tikv/client-go/v2/rawkv.(*Client).sendReq\n\t/root/gopro/src/gateway/vendor/github.com/tikv/client-go/v2/rawkv/rawkv.go:710\ngithub.com/tikv/client-go/v2/rawkv.(*Client).PutWithTTL\n\t/root/gopro/src/gateway/vendor/github.com/tikv/client-go/v2/rawkv/rawkv.go:345\ngithub.com/tikv/client-go/v2/rawkv.(*Client).Put\n\t/root/gopro/src/gateway/vendor/github.com/tikv/client-go/v2/rawkv/rawkv.go:398\ngateway/pkg/tikv.(*Client).Put\n\t/root/gopro/src/gateway/pkg/tikv/tikv.go:227\ngateway/pkg/tikv.

raw_get请求正常

【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【复制黏贴 ERROR 报错的日志】
【其他附件：截图/日志/监控】
平时有一点点网络丢包

凌晨04:16左右
pending peer陡增

hot-region-scheduler到达限制值

scheduler operator timeout

tikv-server内存暴涨，storage.block-cache.cap只配了128MB：

log replication reject暴涨

raft log lag暴涨：

raw_put， new与finish有GAP：

snapshot陡降：

磁盘吞吐，持续几天有些压力，但似乎还没到瓶颈：

【恢复手段】
重启107节点上的两个tikv-server后，恢复正常

【根因】
暂时未知，现象和https://github.com/tikv/tikv/issues/14740 很像，但没出现issue里面的leader跌0的现象，咨询了开发大佬 pingyu，他说6.5版本没有这个问题

[Billmay表妹 (表妹)]

v6.5 有点老了，有机会可以升到 v8.5

[Billmay表妹 (表妹)]

如果问题定位的话，首先版本较老，再者 rawkv 大家用得少，估计很难有环境帮你复现

[jwongz]

还在评估从v6.5.1升到v6.5.12还是升级到v8.5.3，生产环境不敢尝试太新的版本

[pingyu]

store size: 400GB
region size:4TB

确认一下这两个值是正确的吗？
如果是正确的，region size 4TB 的目的是什么

[jwongz]

grafana显示的，有什么办法获取准确值吗？

[jwongz]

rawkv不需要启动协程定期执行gc的吧？

[pingyu]

打开了 API v2 吗？请发一下 TiKV 的配置。

[jwongz]

应该没有开v2，故障时 storage.block-cache.cap是128MB，现在改为125GB

补一个mem trace 图

[quota]
foreground-cpu-time = 0
foreground-write-bandwidth = "0KiB"
foreground-read-bandwidth = "0KiB"
max-delay-duration = "500ms"
background-cpu-time = 0
background-write-bandwidth = "0KiB"
background-read-bandwidth = "0KiB"
enable-auto-tune = false
[readpool.unified]
min-thread-count = 1
max-thread-count = 25
stack-size = "10MiB"
max-tasks-per-worker = 2000
auto-adjust-pool-size = false

[readpool.storage]
use-unified-pool = false
high-concurrency = 8
normal-concurrency = 8
low-concurrency = 8
max-tasks-per-worker-high = 2000
max-tasks-per-worker-normal = 2000
max-tasks-per-worker-low = 2000
stack-size = "10MiB"

[readpool.coprocessor]
use-unified-pool = true
high-concurrency = 25
normal-concurrency = 25
low-concurrency = 25
max-tasks-per-worker-high = 2000
max-tasks-per-worker-normal = 2000
max-tasks-per-worker-low = 2000
stack-size = "10MiB"

[server]
addr = "0.0.0.0:20160"
advertise-addr = "x.111:20160"
status-addr = "0.0.0.0:20180"
advertise-status-addr = "x.111:20180"
status-thread-pool-size = 1
max-grpc-send-msg-len = 10485760
raft-client-grpc-send-msg-buffer = 524288
raft-client-queue-size = 8192
raft-msg-max-batch-size = 128
grpc-compression-type = "none"
grpc-gzip-compression-level = 2
grpc-min-message-size-to-compress = 4096
grpc-concurrency = 5
grpc-concurrent-stream = 1024
grpc-raft-conn-num = 1
grpc-memory-pool-quota = "9223372036854775807B"
grpc-stream-initial-window-size = "2MiB"
grpc-keepalive-time = "10s"
grpc-keepalive-timeout = "3s"
concurrent-send-snap-limit = 32
concurrent-recv-snap-limit = 32
end-point-recursion-limit = 1000
end-point-stream-channel-size = 8
end-point-batch-row-limit = 64
end-point-stream-batch-row-limit = 128
end-point-enable-batch-if-possible = true
end-point-request-max-handle-duration = "1m"
end-point-max-concurrency = 32
end-point-perf-level = 0
snap-max-write-bytes-per-sec = "100MiB"
snap-max-total-size = "0KiB"
stats-concurrency = 1
heavy-load-threshold = 75
enable-request-batch = true
background-thread-count = 3
end-point-slow-log-threshold = "1s"
forward-max-connections-per-address = 4
reject-messages-on-memory-ratio = 0.2
simplify-metrics = false

[storage]
data-dir = "/data0/tikv-data/tikv-20160"
gc-ratio-threshold = 1.1
max-key-size = 8192
scheduler-concurrency = 524288
scheduler-worker-pool-size = 8
scheduler-pending-write-threshold = "100MiB"
reserve-space = "5GiB"
reserve-raft-space = "1GiB"
enable-async-apply-prewrite = false
api-version = 1
enable-ttl = false
background-error-recovery-window = "1h"
ttl-check-poll-interval = "12h"

[storage.flow-control]
enable = true
soft-pending-compaction-bytes-limit = "192GiB"
hard-pending-compaction-bytes-limit = "1TiB"
memtables-threshold = 5
l0-files-threshold = 20

[storage.block-cache]
shared = true
capacity = "131960168B"
num-shard-bits = 6
strict-capacity-limit = false
high-pri-pool-ratio = 0.8
memory-allocator = "nodump"

[storage.io-rate-limit]
max-bytes-per-sec = "0KiB"
mode = "write-only"
strict = false
foreground-read-priority = "high"
foreground-write-priority = "high"
flush-priority = "high"
level-zero-compaction-priority = "medium"
compaction-priority = "low"
replication-priority = "high"
load-balance-priority = "high"
gc-priority = "high"
import-priority = "medium"
export-priority = "medium"
other-priority = "high"

[raftstore]
prevote = true
raftdb-path = "/data0/tikv-data/tikv-20160/raft"
capacity = "0KiB"
raft-base-tick-interval = "1s"
raft-heartbeat-ticks = 2
raft-election-timeout-ticks = 10
raft-min-election-timeout-ticks = 10
raft-max-election-timeout-ticks = 20
raft-max-size-per-msg = "1MiB"
raft-max-inflight-msgs = 256
raft-entry-max-size = "8MiB"
raft-log-compact-sync-interval = "2s"
raft-log-gc-tick-interval = "3s"
raft-log-gc-threshold = 50
raft-log-gc-count-limit = 73728
raft-log-gc-size-limit = "72MiB"
raft-log-reserve-max-ticks = 6
raft-engine-purge-interval = "10s"
raft-entry-cache-life-time = "30s"
split-region-check-tick-interval = "10s"
region-split-check-diff = "6MiB"
region-compact-check-interval = "5m"
region-compact-check-step = 100
region-compact-min-tombstones = 10000
region-compact-tombstones-percent = 30
pd-heartbeat-tick-interval = "1m"
pd-store-heartbeat-tick-interval = "10s"
snap-mgr-gc-tick-interval = "1m"
snap-gc-timeout = "4h"
lock-cf-compact-interval = "10m"
lock-cf-compact-bytes-threshold = "256MiB"
notify-capacity = 40960
messages-per-tick = 4096
max-peer-down-duration = "10m"
max-leader-missing-duration = "2h"
abnormal-leader-missing-duration = "10m"
peer-stale-state-check-interval = "5m"
leader-transfer-max-log-lag = 128
snap-apply-batch-size = "10MiB"
region-worker-tick-interval = "1s"
clean-stale-ranges-tick = 10
consistency-check-interval = "0s"
report-region-flow-interval = "1m"
raft-store-max-leader-lease = "9s"
check-leader-lease-interval = "2s250ms"
renew-leader-lease-advance-duration = "2s250ms"
right-derive-when-split = true
merge-max-log-gap = 10
merge-check-tick-interval = "2s"
use-delete-range = false
snap-generator-pool-size = 2
cleanup-import-sst-interval = "10m"
local-read-batch-size = 1024
apply-max-batch-size = 256
apply-pool-size = 2
apply-reschedule-duration = "5s"
apply-low-priority-pool-size = 1
store-max-batch-size = 256
store-pool-size = 2
store-reschedule-duration = "5s"
store-low-priority-pool-size = 0
store-io-pool-size = 0
store-io-notify-capacity = 40960
future-poll-size = 1
hibernate-regions = true
dev-assert = false
apply-yield-duration = "500ms"
apply-yield-write-size = "32KiB"
perf-level = 0
evict-cache-on-memory-ratio = 0.0
cmd-batch = true
cmd-batch-concurrent-ready-max-count = 1
raft-write-size-limit = "1MiB"
waterfall-metrics = true
io-reschedule-concurrent-max-count = 4
io-reschedule-hotpot-duration = "5s"
inspect-interval = "500ms"
report-min-resolved-ts-interval = "1s"
reactive-memory-lock-tick-interval = "2s"
reactive-memory-lock-timeout-tick = 5
report-region-buckets-tick-interval = "10s"
check-long-uncommitted-interval = "10s"
long-uncommitted-base-threshold = "20s"
max-entry-cache-warmup-duration = "1s"
max-snapshot-file-raw-size = "100MiB"
unreachable-backoff = "10s"
[coprocessor]
split-region-on-table = false
batch-split-limit = 10
region-max-size = "144MiB"
region-split-size = "96MiB"
region-max-keys = 1440000
region-split-keys = 960000
consistency-check-method = "mvcc"
enable-region-bucket = false
region-bucket-size = "96MiB"
region-size-threshold-for-approximate = "1440MiB"
prefer-approximate-bucket = true
region-bucket-merge-size-ratio = 0.33

[coprocessor-v2]

[rocksdb]
info-log-level = "info"
wal-recovery-mode = 2
wal-dir = ""
wal-ttl-seconds = 0
wal-size-limit = "0KiB"
max-total-wal-size = "4GiB"
max-background-jobs = 9
max-background-flushes = 3
max-manifest-file-size = "128MiB"
create-if-missing = true
max-open-files = 40960
enable-statistics = true
stats-dump-period = "10m"
compaction-readahead-size = "0KiB"
info-log-max-size = "1GiB"
info-log-roll-time = "0s"
info-log-keep-log-file-num = 10
info-log-dir = ""
rate-bytes-per-sec = "10GiB"
rate-limiter-refill-period = "100ms"
rate-limiter-mode = 2
rate-limiter-auto-tuned = true
bytes-per-sync = "1MiB"
wal-bytes-per-sync = "512KiB"
max-sub-compactions = 3
writable-file-max-buffer-size = "1MiB"
use-direct-io-for-flush-and-compaction = false
enable-pipelined-write = false
enable-unordered-write = false

[rocksdb.defaultcf]
block-size = "64KiB"
block-cache-size = "131960MiB"
disable-block-cache = false
cache-index-and-filter-blocks = true
pin-l0-filter-and-index-blocks = true
use-bloom-filter = true
optimize-filters-for-hits = true
whole-key-filtering = true
bloom-filter-bits-per-key = 10
block-based-bloom-filter = false
read-amp-bytes-per-bit = 0
compression-per-level = ["no", "no", "lz4", "lz4", "lz4", "zstd", "zstd"]
write-buffer-size = "128MiB"
max-write-buffer-number = 5
min-write-buffer-number-to-merge = 1
max-bytes-for-level-base = "512MiB"
target-file-size-base = "8MiB"
level0-file-num-compaction-trigger = 4
level0-slowdown-writes-trigger = 20
level0-stop-writes-trigger = 20
max-compaction-bytes = "2GiB"
compaction-pri = 3
dynamic-level-bytes = true
num-levels = 7
max-bytes-for-level-multiplier = 10
compaction-style = 0
disable-auto-compactions = false
disable-write-stall = true
soft-pending-compaction-bytes-limit = "192GiB"
hard-pending-compaction-bytes-limit = "1TiB"
force-consistency-checks = false
prop-size-index-distance = 4194304
prop-keys-index-distance = 40960
enable-doubly-skiplist = true
enable-compaction-guard = true
compaction-guard-min-output-file-size = "8MiB"
compaction-guard-max-output-file-size = "128MiB"
bottommost-level-compression = "zstd"
bottommost-zstd-compression-dict-size = 0
bottommost-zstd-compression-sample-size = 0
prepopulate-block-cache = "disabled"
format-version = 2
checksum = "crc32c"

[rocksdb.defaultcf.titan]
min-blob-size = "1KiB"
blob-file-compression = "lz4"
blob-cache-size = "0KiB"
min-gc-batch-size = "16MiB"
max-gc-batch-size = "64MiB"
discardable-ratio = 0.5
merge-small-file-threshold = "8MiB"
blob-run-mode = "normal"
level-merge = false
range-merge = true
max-sorted-runs = 20

[rocksdb.writecf]
block-size = "64KiB"
block-cache-size = "79176MiB"
disable-block-cache = false
cache-index-and-filter-blocks = true
pin-l0-filter-and-index-blocks = true
use-bloom-filter = true
optimize-filters-for-hits = false
whole-key-filtering = false
bloom-filter-bits-per-key = 10
block-based-bloom-filter = false
read-amp-bytes-per-bit = 0
compression-per-level = ["no", "no", "lz4", "lz4", "lz4", "zstd", "zstd"]
write-buffer-size = "128MiB"
max-write-buffer-number = 5
min-write-buffer-number-to-merge = 1
max-bytes-for-level-base = "512MiB"
target-file-size-base = "8MiB"
level0-file-num-compaction-trigger = 4
level0-slowdown-writes-trigger = 20
level0-stop-writes-trigger = 20
max-compaction-bytes = "2GiB"
compaction-pri = 3
dynamic-level-bytes = true
num-levels = 7
max-bytes-for-level-multiplier = 10
compaction-style = 0
disable-auto-compactions = false
disable-write-stall = true
soft-pending-compaction-bytes-limit = "192GiB"
hard-pending-compaction-bytes-limit = "1TiB"
force-consistency-checks = false
prop-size-index-distance = 4194304
prop-keys-index-distance = 40960
enable-doubly-skiplist = true
enable-compaction-guard = true
compaction-guard-min-output-file-size = "8MiB"
compaction-guard-max-output-file-size = "128MiB"
bottommost-level-compression = "zstd"
bottommost-zstd-compression-dict-size = 0
bottommost-zstd-compression-sample-size = 0
prepopulate-block-cache = "disabled"
format-version = 2
checksum = "crc32c"

[rocksdb.writecf.titan]
min-blob-size = "1KiB"
blob-file-compression = "lz4"
blob-cache-size = "0KiB"
min-gc-batch-size = "16MiB"
max-gc-batch-size = "64MiB"
discardable-ratio = 0.5
merge-small-file-threshold = "8MiB"
blob-run-mode = "read-only"
level-merge = false
range-merge = true
max-sorted-runs = 20

[rocksdb.lockcf]
block-size = "16KiB"
block-cache-size = "1GiB"
disable-block-cache = false
cache-index-and-filter-blocks = true
pin-l0-filter-and-index-blocks = true
use-bloom-filter = true
optimize-filters-for-hits = false
whole-key-filtering = true
bloom-filter-bits-per-key = 10
block-based-bloom-filter = false
read-amp-bytes-per-bit = 0
compression-per-level = ["no", "no", "no", "no", "no", "no", "no"]
write-buffer-size = "32MiB"
max-write-buffer-number = 5
min-write-buffer-number-to-merge = 1
max-bytes-for-level-base = "128MiB"
target-file-size-base = "8MiB"
level0-file-num-compaction-trigger = 1
level0-slowdown-writes-trigger = 20
level0-stop-writes-trigger = 20
max-compaction-bytes = "2GiB"
compaction-pri = 0
dynamic-level-bytes = true
num-levels = 7
max-bytes-for-level-multiplier = 10
compaction-style = 0
disable-auto-compactions = false
disable-write-stall = true
soft-pending-compaction-bytes-limit = "192GiB"
hard-pending-compaction-bytes-limit = "1TiB"
force-consistency-checks = false
prop-size-index-distance = 4194304
prop-keys-index-distance = 40960
enable-doubly-skiplist = true
enable-compaction-guard = false
compaction-guard-min-output-file-size = "8MiB"
compaction-guard-max-output-file-size = "128MiB"
bottommost-level-compression = "disable"
bottommost-zstd-compression-dict-size = 0
bottommost-zstd-compression-sample-size = 0
prepopulate-block-cache = "disabled"
format-version = 2
checksum = "crc32c"

[rocksdb.lockcf.titan]
min-blob-size = "1KiB"
blob-file-compression = "lz4"
blob-cache-size = "0KiB"
min-gc-batch-size = "16MiB"
max-gc-batch-size = "64MiB"
discardable-ratio = 0.5
merge-small-file-threshold = "8MiB"
blob-run-mode = "read-only"
level-merge = false
range-merge = true
max-sorted-runs = 20

[rocksdb.raftcf]
block-size = "16KiB"
block-cache-size = "128MiB"
disable-block-cache = false
cache-index-and-filter-blocks = true
pin-l0-filter-and-index-blocks = true
use-bloom-filter = true
optimize-filters-for-hits = true
whole-key-filtering = true
bloom-filter-bits-per-key = 10
block-based-bloom-filter = false
read-amp-bytes-per-bit = 0
compression-per-level = ["no", "no", "no", "no", "no", "no", "no"]
write-buffer-size = "128MiB"
max-write-buffer-number = 5
min-write-buffer-number-to-merge = 1
max-bytes-for-level-base = "128MiB"
target-file-size-base = "8MiB"
level0-file-num-compaction-trigger = 1
level0-slowdown-writes-trigger = 20
level0-stop-writes-trigger = 20
max-compaction-bytes = "2GiB"
compaction-pri = 0
dynamic-level-bytes = true
num-levels = 7
max-bytes-for-level-multiplier = 10
compaction-style = 0
disable-auto-compactions = false
disable-write-stall = true
soft-pending-compaction-bytes-limit = "192GiB"
hard-pending-compaction-bytes-limit = "1TiB"
force-consistency-checks = false
prop-size-index-distance = 4194304
prop-keys-index-distance = 40960
enable-doubly-skiplist = true
enable-compaction-guard = false
compaction-guard-min-output-file-size = "8MiB"
compaction-guard-max-output-file-size = "128MiB"
bottommost-level-compression = "disable"
bottommost-zstd-compression-dict-size = 0
bottommost-zstd-compression-sample-size = 0
prepopulate-block-cache = "disabled"
format-version = 2
checksum = "crc32c"

[rocksdb.raftcf.titan]
min-blob-size = "1KiB"
blob-file-compression = "lz4"
blob-cache-size = "0KiB"
min-gc-batch-size = "16MiB"
max-gc-batch-size = "64MiB"
discardable-ratio = 0.5
merge-small-file-threshold = "8MiB"
blob-run-mode = "read-only"
level-merge = false
range-merge = true
max-sorted-runs = 20

[rocksdb.titan]
enabled = false
dirname = ""
disable-gc = false
max-background-gc = 4
purge-obsolete-files-period = "10s"

[raftdb]
wal-recovery-mode = 2
wal-dir = ""
wal-ttl-seconds = 0
wal-size-limit = "0KiB"
max-total-wal-size = "4GiB"
max-background-jobs = 4
max-background-flushes = 1
max-manifest-file-size = "20MiB"
create-if-missing = true
max-open-files = 40960
enable-statistics = true
stats-dump-period = "10m"
compaction-readahead-size = "0KiB"
info-log-max-size = "1GiB"
info-log-roll-time = "0s"
info-log-keep-log-file-num = 10
info-log-dir = ""
info-log-level = "info"
max-sub-compactions = 2
writable-file-max-buffer-size = "1MiB"
use-direct-io-for-flush-and-compaction = false
enable-pipelined-write = true
enable-unordered-write = false
allow-concurrent-memtable-write = true
bytes-per-sync = "1MiB"
wal-bytes-per-sync = "512KiB"

[raftdb.defaultcf]
block-size = "64KiB"
block-cache-size = "2GiB"
disable-block-cache = false
cache-index-and-filter-blocks = true
pin-l0-filter-and-index-blocks = true
use-bloom-filter = false
optimize-filters-for-hits = true
whole-key-filtering = true
bloom-filter-bits-per-key = 10
block-based-bloom-filter = false
read-amp-bytes-per-bit = 0
compression-per-level = ["no", "no", "lz4", "lz4", "lz4", "zstd", "zstd"]
write-buffer-size = "128MiB"
max-write-buffer-number = 5
min-write-buffer-number-to-merge = 1
max-bytes-for-level-base = "512MiB"
target-file-size-base = "8MiB"
level0-file-num-compaction-trigger = 4
level0-slowdown-writes-trigger = 20
level0-stop-writes-trigger = 20
max-compaction-bytes = "2GiB"
compaction-pri = 0
dynamic-level-bytes = true
num-levels = 7
max-bytes-for-level-multiplier = 10
compaction-style = 0
disable-auto-compactions = false
disable-write-stall = false
soft-pending-compaction-bytes-limit = "192GiB"
hard-pending-compaction-bytes-limit = "1TiB"
force-consistency-checks = false
prop-size-index-distance = 4194304
prop-keys-index-distance = 40960
enable-doubly-skiplist = true
enable-compaction-guard = false
compaction-guard-min-output-file-size = "8MiB"
compaction-guard-max-output-file-size = "128MiB"
bottommost-level-compression = "disable"
bottommost-zstd-compression-dict-size = 0
bottommost-zstd-compression-sample-size = 0
prepopulate-block-cache = "disabled"
format-version = 2
checksum = "crc32c"

[raftdb.defaultcf.titan]
min-blob-size = "1KiB"
blob-file-compression = "lz4"
blob-cache-size = "0KiB"
min-gc-batch-size = "16MiB"
max-gc-batch-size = "64MiB"
discardable-ratio = 0.5
merge-small-file-threshold = "8MiB"
blob-run-mode = "normal"
level-merge = false
range-merge = true
max-sorted-runs = 20

[raftdb.titan]
enabled = false
dirname = ""
disable-gc = false
max-background-gc = 4
purge-obsolete-files-period = "10s"

[raft-engine]
enable = true
dir = "/data0/tikv-data/tikv-20160/raft-engine"
recovery-mode = "tolerate-corrupted-tail-records"
recovery-read-block-size = "16KiB"
recovery-threads = 4
batch-compression-threshold = "8KiB"
format-version = 2
target-file-size = "128MiB"
purge-threshold = "10GiB"
purge-rewrite-threshold = "1GiB"
purge-rewrite-garbage-ratio = 0.6
memory-limit = 83022159052
enable-log-recycle = false

[backup]
num-threads = 8
batch-size = 8
sst-max-size = "144MiB"
enable-auto-tune = true
auto-tune-remain-threads = 6
auto-tune-refresh-interval = "1m"
io-thread-size = 2
s3-multi-part-size = "5MiB"

[backup.hadoop]
home = ""
linux-user = ""

[log-backup]
max-flush-interval = "3m"
num-threads = 12
enable = true
temp-path = "/data0/tikv-data/tikv-20160/log-backup-temp"
file-size-limit = "256MiB"
initial-scan-pending-memory-quota = "512MiB"
initial-scan-rate-limit = "60MiB"

[pessimistic-txn]
wait-for-lock-timeout = "1s"
wake-up-delay-duration = "20ms"
pipelined = true
in-memory = true

[gc]
ratio-threshold = 1.1
batch-keys = 512
max-write-bytes-per-sec = "0KiB"
enable-compaction-filter = true
compaction-filter-skip-version-check = false

[split]
qps-threshold = 3000
split-balance-score = 0.25
split-contained-score = 0.5
detect-times = 10
sample-num = 20
sample-threshold = 100
byte-threshold = 31457280
grpc-thread-cpu-overload-threshold-ratio = 0.5
unified-read-pool-thread-cpu-overload-threshold-ratio = 0.8
region-cpu-overload-threshold-ratio = 0.25

[cdc]
min-ts-interval = "1s"
hibernate-regions-compatible = true
incremental-scan-threads = 4
incremental-scan-concurrency = 6
incremental-scan-speed-limit = "128MiB"
incremental-scan-ts-filter-ratio = 0.2
tso-worker-threads = 1
sink-memory-quota = "512MiB"
old-value-cache-memory-quota = "512MiB"

[resolved-ts]
enable = true
advance-ts-interval = "20s"
scan-lock-pool-size = 2

[resource-metering]
receiver-address = ""
report-receiver-interval = "1m"
max-resource-groups = 100
precision = "1s"

[causal-ts]
renew-interval = "100ms"
renew-batch-min-size = 100
renew-batch-max-size = 8192
alloc-ahead-buffer = "3s"

[jwongz]

主要导出来的是04:16开始出故障那几分钟的日志，各个store的日志情况如下：
每个节点都在大量的刷：

[2025/09/06 01:19:14.458 +08:00] [ERROR] [peer.rs:614] ["handle raft message err"] [err_code=KV:Raft:StepLocalMsg] [err="Raft raft: cannot step raft local message"] [peer_id=1112637] [region_id=1088709]

下面的日志是过滤掉上述错误的

107:20180（store-3），这个节点最可疑，能明显的看到他不停的尝试新增leaner节点，但是卡住了

[2025/09/06 04:33:55.994 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:33:55.994 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]

最终重启该节点，恢复了集群，重启前的日志显示有两个peer pending:

[2025/09/06 06:04:30.753 +08:00] [WARN] [store.rs:1035] ["[store 3] handle 2 pending peers include 2 ready, 0 entries, 4 messages and 0 snapshots"] [takes=1629]
[2025/09/06 06:04:31.445 +08:00] [INFO] [batch.rs:648] ["batch system raftstore-3 is stopped."]

store-3的其余日志：

[2025/09/06 04:16:00.011 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=34.891021ms] [snap_key=7251415_183_3287215]
[2025/09/06 04:16:00.011 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:00.016 +08:00] [INFO] [raft_log.rs:627] ["log [committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0] starts to restore snapshot [index: 3287215, term: 183]"] [snapshot_term=183] [snapshot_index=3287215] [log="committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0"] [raft_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.016 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7443775, 7444805, 7439679} }, outgoing: Configuration { voters: {} } }, learners: {7444898}, learners_next: {}, auto_leave: false }"] [raft_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.016 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=183] [snapshot_index=3287215] [last_term=183] [last_index=3287215] [commit=3287215] [raft_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.016 +08:00] [INFO] [raft.rs:2511] ["[commit: 3287215, term: 183] restored snapshot [index: 3287215, term: 183]"] [snapshot_term=183] [snapshot_index=3287215] [commit=3287215] [term=183] [raft_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.016 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.016 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 3287215 truncated_state { index: 3287215 term: 183 }"] [region="id: 7251415 start_key:  region_epoch { conf_ver: 65826 version: 74 } peers { id: 7439679 store_id: 9 } peers { id: 7443775 store_id: 14 } peers { id: 7444805 store_id: 1 } peers { id: 7444898 store_id: 3 role: Learner }"] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.017 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7251415 start_key:  region_epoch { conf_ver: 65826 version: 74 } peers { id: 7439679 store_id: 9 } peers { id: 7443775 store_id: 14 } peers { id: 7444805 store_id: 1 } peers { id: 7444898 store_id: 3 role: Learner }"] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.017 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.024 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:00.085 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=56.582783ms] [region_id=7251415]
[2025/09/06 04:16:00.087 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444898 store_id: 3 } } changes { change_type: AddLearnerNode peer { id: 7443775 store_id: 14 role: Learner } } }"] [index=3287216] [term=183] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.087 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65826 version: 74"] [kind=EnterJoint] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.087 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7251415 start_key:  region_epoch { conf_ver: 65828 version: 74 } peers { id: 7439679 store_id: 9 } peers { id: 7443775 store_id: 14 role: DemotingVoter } peers { id: 7444805 store_id: 1 } peers { id: 7444898 store_id: 3 role: IncomingVoter }"] ["original region"="id: 7251415 start_key:  region_epoch { conf_ver: 65826 version: 74 } peers { id: 7439679 store_id: 9 } peers { id: 7443775 store_id: 14 } peers { id: 7444805 store_id: 1 } peers { id: 7444898 store_id: 3 role: Learner }"] [changes="[peer { id: 7444898 store_id: 3 }, change_type: AddLearnerNode peer { id: 7443775 store_id: 14 role: Learner }]"] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.087 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444805, 7444898, 7439679} }, outgoing: Configuration { voters: {7439679, 7444805, 7443775} } }, learners: {}, learners_next: {7443775}, auto_leave: false }"] [raft_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.091 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=3287217] [term=183] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.091 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65828 version: 74"] [kind=LeaveJoint] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.091 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7251415 start_key:  region_epoch { conf_ver: 65830 version: 74 } peers { id: 7439679 store_id: 9 } peers { id: 7443775 store_id: 14 role: Learner } peers { id: 7444805 store_id: 1 } peers { id: 7444898 store_id: 3 }"] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.092 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444805, 7444898, 7439679} }, outgoing: Configuration { voters: {} } }, learners: {7443775}, learners_next: {}, auto_leave: false }"] [raft_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.093 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7443775 store_id: 14 role: Learner } } }"] [index=3287218] [term=183] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.093 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65830 version: 74"] [kind=Simple] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.093 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7251415 start_key:  region_epoch { conf_ver: 65831 version: 74 } peers { id: 7439679 store_id: 9 } peers { id: 7444805 store_id: 1 } peers { id: 7444898 store_id: 3 }"] ["original region"="id: 7251415 start_key:  region_epoch { conf_ver: 65830 version: 74 } peers { id: 7439679 store_id: 9 } peers { id: 7443775 store_id: 14 role: Learner } peers { id: 7444805 store_id: 1 } peers { id: 7444898 store_id: 3 }"] [changes="[change_type: RemoveNode peer { id: 7443775 store_id: 14 role: Learner }]"] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:00.094 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444805, 7444898, 7439679} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444898] [region_id=7251415]

[2025/09/06 04:16:00.339 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444905 store_id: 1 role: Learner } } }"] [index=1547463] [term=47] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:00.339 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65261 version: 98"] [kind=Simple] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:00.339 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7339530 start_key:  region_epoch { conf_ver: 65262 version: 98 } peers { id: 7443126 store_id: 9 } peers { id: 7443752 store_id: 2 } peers { id: 7444863 store_id: 3 } peers { id: 7444905 store_id: 1 role: Learner }"] ["original region"="id: 7339530 start_key:  region_epoch { conf_ver: 65261 version: 98 } peers { id: 7443126 store_id: 9 } peers { id: 7443752 store_id: 2 } peers { id: 7444863 store_id: 3 }"] [changes="[change_type: AddLearnerNode peer { id: 7444905 store_id: 1 role: Learner }]"] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:00.340 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7443752, 7444863, 7443126} }, outgoing: Configuration { voters: {} } }, learners: {7444905}, learners_next: {}, auto_leave: false }"] [raft_id=7444863] [region_id=7339530]
[2025/09/06 04:16:01.407 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444905 store_id: 1 } } changes { change_type: AddLearnerNode peer { id: 7443752 store_id: 2 role: Learner } } }"] [index=1547464] [term=47] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:01.407 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65262 version: 98"] [kind=EnterJoint] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:01.407 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7339530 start_key:  region_epoch { conf_ver: 65264 version: 98 } peers { id: 7443126 store_id: 9 } peers { id: 7443752 store_id: 2 role: DemotingVoter } peers { id: 7444863 store_id: 3 } peers { id: 7444905 store_id: 1 role: IncomingVoter }"] ["original region"="id: 7339530 start_key:  region_epoch { conf_ver: 65262 version: 98 } peers { id: 7443126 store_id: 9 } peers { id: 7443752 store_id: 2 } peers { id: 7444863 store_id: 3 } peers { id: 7444905 store_id: 1 role: Learner }"] [changes="[peer { id: 7444905 store_id: 1 }, change_type: AddLearnerNode peer { id: 7443752 store_id: 2 role: Learner }]"] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:01.407 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444863, 7444905, 7443126} }, outgoing: Configuration { voters: {7443752, 7443126, 7444863} } }, learners: {}, learners_next: {7443752}, auto_leave: false }"] [raft_id=7444863] [region_id=7339530]
[2025/09/06 04:16:01.409 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=1547465] [term=47] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:01.409 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65264 version: 98"] [kind=LeaveJoint] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:01.409 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7339530 start_key:  region_epoch { conf_ver: 65266 version: 98 } peers { id: 7443126 store_id: 9 } peers { id: 7443752 store_id: 2 role: Learner } peers { id: 7444863 store_id: 3 } peers { id: 7444905 store_id: 1 }"] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:01.409 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444863, 7444905, 7443126} }, outgoing: Configuration { voters: {} } }, learners: {7443752}, learners_next: {}, auto_leave: false }"] [raft_id=7444863] [region_id=7339530]
[2025/09/06 04:16:01.410 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7443752 store_id: 2 role: Learner } } }"] [index=1547466] [term=47] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:01.410 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65266 version: 98"] [kind=Simple] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:01.410 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7339530 start_key:  region_epoch { conf_ver: 65267 version: 98 } peers { id: 7443126 store_id: 9 } peers { id: 7444863 store_id: 3 } peers { id: 7444905 store_id: 1 }"] ["original region"="id: 7339530 start_key: 3638353936343235323031342E706466 region_epoch { conf_ver: 65266 version: 98 } peers { id: 7443126 store_id: 9 } peers { id: 7443752 store_id: 2 role: Learner } peers { id: 7444863 store_id: 3 } peers { id: 7444905 store_id: 1 }"] [changes="[change_type: RemoveNode peer { id: 7443752 store_id: 2 role: Learner }]"] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:01.411 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444863, 7444905, 7443126} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444863] [region_id=7339530]

[2025/09/06 04:16:01.610 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=241764497] [region_id=7218128]
[2025/09/06 04:16:01.767 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=122589] [size=109069377] [region_id=7218128]
[2025/09/06 04:16:01.945 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986966] [region_id=7322408]
[2025/09/06 04:16:02.096 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138172] [size=100663052] [region_id=7322408]
[2025/09/06 04:16:02.276 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201324982] [region_id=7404664]
[2025/09/06 04:16:02.371 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=137978] [size=100662856] [region_id=7404664]
[2025/09/06 04:16:02.982 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=187299442] [region_id=1014895]
[2025/09/06 04:16:03.091 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=103951] [size=74832138] [region_id=1014895]
[2025/09/06 04:16:03.474 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986387] [region_id=7219466]
[2025/09/06 04:16:03.646 +08:00] [INFO] [peer.rs:312] ["replicate peer"] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:03.646 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:03.646 +08:00] [INFO] [raft.rs:1120] ["became follower at term 0"] [term=0] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:03.646 +08:00] [INFO] [raft.rs:384] [newRaft] [peers="Configuration { incoming: Configuration { voters: {} }, outgoing: Configuration { voters: {} } }"] ["last term"=0] ["last index"=0] [applied=0] [commit=0] [term=0] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:03.646 +08:00] [INFO] [raw_node.rs:315] ["RawNode created with id 7444911."] [id=7444911] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:03.646 +08:00] [INFO] [raft.rs:1364] ["received a message with higher term from 7431949"] ["msg type"=MsgHeartbeat] [message_term=52] [term=0] [from=7431949] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:03.646 +08:00] [INFO] [raft.rs:1120] ["became follower at term 52"] [term=52] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:03.655 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129201] [size=100662847] [region_id=7219466]
[2025/09/06 04:16:03.675 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=356298527] [region_id=7221921]
[2025/09/06 04:16:03.909 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128653] [size=100662979] [region_id=7221921]
[2025/09/06 04:16:04.663 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:04.694 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data0/tikv-data/tikv-20160/snap/rev_7337276_52_1620582_(default|lock|write).sst] [snap_key=7337276_52_1620582]
[2025/09/06 04:16:04.712 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=48.187288ms] [snap_key=7337276_52_1620582]
[2025/09/06 04:16:04.715 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:04.719 +08:00] [INFO] [raft_log.rs:627] ["log [committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0] starts to restore snapshot [index: 1620582, term: 52]"] [snapshot_term=52] [snapshot_index=1620582] [log="committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0"] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:04.719 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7431949, 7440501, 7444841} }, outgoing: Configuration { voters: {} } }, learners: {7444911}, learners_next: {}, auto_leave: false }"] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:04.719 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=52] [snapshot_index=1620582] [last_term=52] [last_index=1620582] [commit=1620582] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:04.719 +08:00] [INFO] [raft.rs:2511] ["[commit: 1620582, term: 52] restored snapshot [index: 1620582, term: 52]"] [snapshot_term=52] [snapshot_index=1620582] [commit=1620582] [term=52] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:04.719 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:04.719 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 1620582 truncated_state { index: 1620582 term: 52 }"] [region="id: 7337276 start_key: 6F53937352E706466 region_epoch { conf_ver: 65334 version: 97 } peers { id: 7431949 store_id: 9 } peers { id: 7440501 store_id: 14 } peers { id: 7444841 store_id: 1 } peers { id: 7444911 store_id: 3 role: Learner }"] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:04.737 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7337276 start_key: 6F73735F6F52E706466 region_epoch { conf_ver: 65334 version: 97 } peers { id: 7431949 store_id: 9 } peers { id: 7440501 store_id: 14 } peers { id: 7444841 store_id: 1 } peers { id: 7444911 store_id: 3 role: Learner }"] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:04.737 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:04.743 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:04.870 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=122.861171ms] [region_id=7337276]
[2025/09/06 04:16:05.001 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444916 store_id: 1 role: Learner } } }"] [index=614442] [term=29] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:05.001 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66173 version: 111"] [kind=Simple] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:05.001 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7390716 start_key: 6F7706466 region_epoch { conf_ver: 66174 version: 111 } peers { id: 7444457 store_id: 2 } peers { id: 7444566 store_id: 3 } peers { id: 7444660 store_id: 9 } peers { id: 7444916 store_id: 1 role: Learner }"] ["original region"="id: 7390716 start_key: 6F737392E706466 region_epoch { conf_ver: 66173 version: 111 } peers { id: 7444457 store_id: 2 } peers { id: 7444566 store_id: 3 } peers { id: 7444660 store_id: 9 }"] [changes="[change_type: AddLearnerNode peer { id: 7444916 store_id: 1 role: Learner }]"] [peer_id=7444566] [region_id=7390716]

2025/09/06 04:16:05.005 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444660, 7444457, 7444566} }, outgoing: Configuration { voters: {} } }, learners: {7444916}, learners_next: {}, auto_leave: false }"] [raft_id=7444566] [region_id=7390716]
[2025/09/06 04:16:05.021 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444911 store_id: 3 } } changes { change_type: AddLearnerNode peer { id: 7440501 store_id: 14 role: Learner } } }"] [index=1620604] [term=52] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:05.021 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65334 version: 97"] [kind=EnterJoint] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:05.021 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7337276 start_key: 6F7706466 region_epoch { conf_ver: 65336 version: 97 } peers { id: 7431949 store_id: 9 } peers { id: 7440501 store_id: 14 role: DemotingVoter } peers { id: 7444841 store_id: 1 } peers { id: 7444911 store_id: 3 role: IncomingVoter }"] ["original region"="id: 7337276 start_key: 6F72E706466 region_epoch { conf_ver: 65334 version: 97 } peers { id: 7431949 store_id: 9 } peers { id: 7440501 store_id: 14 } peers { id: 7444841 store_id: 1 } peers { id: 7444911 store_id: 3 role: Learner }"] [changes="[peer { id: 7444911 store_id: 3 }, change_type: AddLearnerNode peer { id: 7440501 store_id: 14 role: Learner }]"] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:05.029 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7431949, 7444911, 7444841} }, outgoing: Configuration { voters: {7431949, 7440501, 7444841} } }, learners: {}, learners_next: {7440501}, auto_leave: false }"] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:05.068 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=1620605] [term=52] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:05.068 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65336 version: 97"] [kind=LeaveJoint] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:05.068 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7337276 start_key: 6F2E706466 region_epoch { conf_ver: 65338 version: 97 } peers { id: 7431949 store_id: 9 } peers { id: 7440501 store_id: 14 role: Learner } peers { id: 7444841 store_id: 1 } peers { id: 7444911 store_id: 3 }"] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:05.075 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7431949, 7444911, 7444841} }, outgoing: Configuration { voters: {} } }, learners: {7440501}, learners_next: {}, auto_leave: false }"] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:05.148 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7440501 store_id: 14 role: Learner } } }"] [index=1620606] [term=52] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:05.149 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65338 version: 97"] [kind=Simple] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:05.149 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7337276 start_key: 6F7372E706466 region_epoch { conf_ver: 65339 version: 97 } peers { id: 7431949 store_id: 9 } peers { id: 7444841 store_id: 1 } peers { id: 7444911 store_id: 3 }"] ["original region"="id: 7337276 start_key: 6F737466 region_epoch { conf_ver: 65338 version: 97 } peers { id: 7431949 store_id: 9 } peers { id: 7440501 store_id: 14 role: Learner } peers { id: 7444841 store_id: 1 } peers { id: 7444911 store_id: 3 }"] [changes="[change_type: RemoveNode peer { id: 7440501 store_id: 14 role: Learner }]"] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:05.157 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7431949, 7444911, 7444841} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444911] [region_id=7337276]
[2025/09/06 04:16:07.050 +08:00] [INFO] [raft.rs:2341] ["[term 29] received MsgTimeoutNow from 7444457 and starts an election to get leadership."] [from=7444457] [term=29] [raft_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.050 +08:00] [INFO] [raft.rs:1550] ["starting a new election"] [term=29] [raft_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.050 +08:00] [INFO] [raft.rs:1144] ["became candidate at term 30"] [term=30] [raft_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.050 +08:00] [INFO] [raft.rs:1299] ["broadcasting vote request"] [to="[7444660, 7444457]"] [log_index=614467] [log_term=29] [term=30] [type=MsgRequestVote] [raft_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.128 +08:00] [INFO] [raft.rs:2212] ["received votes response"] [term=30] [type=MsgRequestVoteResponse] [approvals=2] [rejections=0] [from=7444457] [vote=true] [raft_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.128 +08:00] [INFO] [raft.rs:1228] ["became leader at term 30"] [term=30] [raft_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.128 +08:00] [INFO] [peer.rs:5387] ["require updating max ts"] [initial_status=128849019102] [region_id=7390716]
[2025/09/06 04:16:07.129 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444916 store_id: 1 }, peer { id: 7444457 store_id: 2 role: Learner } change_type: AddLearnerNode]"] [region_id=7390716]
[2025/09/06 04:16:07.129 +08:00] [INFO] [pd.rs:1701] ["succeed to update max timestamp"] [region_id=7390716]
[2025/09/06 04:16:07.131 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.192 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444916 store_id: 1 }, peer { id: 7444457 store_id: 2 role: Learner } change_type: AddLearnerNode]"] [region_id=7390716]
[2025/09/06 04:16:07.195 +08:00] [INFO] [endpoint.rs:338] ["register observe region"] [region="id: 7390716 start_key: 6F7373392E706466 region_epoch { conf_ver: 66174 version: 111 } peers { id: 7444457 store_id: 2 } peers { id: 7444566 store_id: 3 } peers { id: 7444660 store_id: 9 } peers { id: 7444916 store_id: 1 role: Learner }"]
[2025/09/06 04:16:07.195 +08:00] [WARN] [peer.rs:4648] ["failed to propose confchange"] [error="Other(\"[components/raftstore/src/store/util.rs:1002]: [peer { id: 7444916 store_id: 1 }, change_type: AddLearnerNode peer { id: 7444457 store_id: 2 role: Learner }]: before: voters: 7444660 voters: 7444457 voters: 7444566 learners: 7444916, after: voters: 7444660 voters: 7444566 voters: 7444916 voters_outgoing: 7444660 voters_outgoing: 7444457 voters_outgoing: 7444566 learners_next: 7444457, first index 614440, promoted commit index 0\")"]
[2025/09/06 04:16:07.195 +08:00] [INFO] [endpoint.rs:255] ["Resolver initialized"] [pending_data_index=0] [snapshot_index=614468] [observe_id=ObserveId(425044)] [region=7390716]
[2025/09/06 04:16:07.255 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444916 store_id: 1 }, peer { id: 7444457 store_id: 2 role: Learner } change_type: AddLearnerNode]"] [region_id=7390716]
[2025/09/06 04:16:07.256 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=EnterJoint] [changes="[peer { id: 7444916 store_id: 1 }, change_type: AddLearnerNode peer { id: 7444457 store_id: 2 role: Learner }]"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.451 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444916 store_id: 1 } } changes { change_type: AddLearnerNode peer { id: 7444457 store_id: 2 role: Learner } } }"] [index=614469] [term=30] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.451 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66174 version: 111"] [kind=EnterJoint] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.451 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7390716 start_key: 6F7373506466 region_epoch { conf_ver: 66176 version: 111 } peers { id: 7444457 store_id: 2 role: DemotingVoter } peers { id: 7444566 store_id: 3 } peers { id: 7444660 store_id: 9 } peers { id: 7444916 store_id: 1 role: IncomingVoter }"] ["original region"="id: 7390716 start_key: 6F73735392E706466 region_epoch { conf_ver: 66174 version: 111 } peers { id: 7444457 store_id: 2 } peers { id: 7444566 store_id: 3 } peers { id: 7444660 store_id: 9 } peers { id: 7444916 store_id: 1 role: Learner }"] [changes="[peer { id: 7444916 store_id: 1 }, change_type: AddLearnerNode peer { id: 7444457 store_id: 2 role: Learner }]"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.480 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444660, 7444566, 7444916} }, outgoing: Configuration { voters: {7444660, 7444457, 7444566} } }, learners: {}, learners_next: {7444457}, auto_leave: false }"] [raft_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.480 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7390716 start_key: 6F7373392E706466 region_epoch { conf_ver: 66176 version: 111 } peers { id: 7444457 store_id: 2 role: DemotingVoter } peers { id: 7444566 store_id: 3 } peers { id: 7444660 store_id: 9 } peers { id: 7444916 store_id: 1 role: IncomingVoter }"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.492 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7390716]
[2025/09/06 04:16:07.500 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=LeaveJoint] [changes="[]"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.567 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=422462433] [region_id=7218236]
[2025/09/06 04:16:07.601 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7390716]
[2025/09/06 04:16:07.607 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:07.866 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=117886] [size=100662566] [region_id=7218236]
[2025/09/06 04:16:07.888 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7390716]
[2025/09/06 04:16:07.892 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:08.660 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7444368] [request_index=0] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:16:09.653 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138610, total size 100662989. raw_size_per_file 104857600, total takes 992.444367ms"]
[2025/09/06 04:16:09.678 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662989] [key_count=138610] [cf=default] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7217841_289_9197254_(default|lock|write).sst] [region_id=7217841]
[2025/09/06 04:16:09.678 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7217841_289_9197254_(default|lock|write).sst] [region_id=7217841]
[2025/09/06 04:16:09.678 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7217841_289_9197254_(default|lock|write).sst] [region_id=7217841]
[2025/09/06 04:16:09.681 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=1.020364401s] [size=7818471] [key_count=138610] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7217841_289_9197254_(default|lock|write).sst] [region_id=7217841]
[2025/09/06 04:16:09.694 +08:00] [INFO] [peer.rs:312] ["replicate peer"] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:09.694 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:09.694 +08:00] [INFO] [raft.rs:1120] ["became follower at term 0"] [term=0] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:09.694 +08:00] [INFO] [raft.rs:384] [newRaft] [peers="Configuration { incoming: Configuration { voters: {} }, outgoing: Configuration { voters: {} } }"] ["last term"=0] ["last index"=0] [applied=0] [commit=0] [term=0] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:09.694 +08:00] [INFO] [raw_node.rs:315] ["RawNode created with id 7444921."] [id=7444921] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:09.701 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=396511169] [region_id=7249928]
[2025/09/06 04:16:09.704 +08:00] [INFO] [raft.rs:1364] ["received a message with higher term from 7443351"] ["msg type"=MsgHeartbeat] [message_term=39] [term=0] [from=7443351] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:09.705 +08:00] [INFO] [raft.rs:1120] ["became follower at term 39"] [term=39] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:09.732 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:09.788 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=53.357959ms] [size=7818471] [snap_key=7217841_289_9197254] [region_id=7217841]
[2025/09/06 04:16:09.797 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7444368 store_id: 8"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:16:09.972 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138454] [size=100662602] [region_id=7249928]
[2025/09/06 04:16:09.996 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7390716]
[2025/09/06 04:16:09.998 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:11.157 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:11.195 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data0/tikv-data/tikv-20160/snap/rev_7351569_39_1262558_(default|lock|write).sst] [snap_key=7351569_39_1262558]
[2025/09/06 04:16:11.211 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=54.412403ms] [snap_key=7351569_39_1262558]
[2025/09/06 04:16:11.221 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:11.226 +08:00] [INFO] [raft_log.rs:627] ["log [committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0] starts to restore snapshot [index: 1262558, term: 39]"] [snapshot_term=39] [snapshot_index=1262558] [log="committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0"] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.226 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444649, 7442238, 7443351} }, outgoing: Configuration { voters: {} } }, learners: {7444921}, learners_next: {}, auto_leave: false }"] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.226 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=39] [snapshot_index=1262558] [last_term=39] [last_index=1262558] [commit=1262558] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.226 +08:00] [INFO] [raft.rs:2511] ["[commit: 1262558, term: 39] restored snapshot [index: 1262558, term: 39]"] [snapshot_term=39] [snapshot_index=1262558] [commit=1262558] [term=39] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.226 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.226 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 1262558 truncated_state { index: 1262558 term: 39 }"] [region="id: 7351569 start_key: 6F73735F6F626706466 region_epoch { conf_ver: 65412 version: 102 } peers { id: 7442238 store_id: 9 } peers { id: 7443351 store_id: 2 } peers { id: 7444649 store_id: 14 } peers { id: 7444921 store_id: 3 role: Learner }"] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.268 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7351569 start_key: 6F7373506466 end_key: 6F7373392E706466 region_epoch { conf_ver: 65412 version: 102 } peers { id: 7442238 store_id: 9 } peers { id: 7443351 store_id: 2 } peers { id: 7444649 store_id: 14 } peers { id: 7444921 store_id: 3 role: Learner }"] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.268 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.273 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:11.398 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=123.0721ms] [region_id=7351569]
[2025/09/06 04:16:11.496 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444921 store_id: 3 } } changes { change_type: AddLearnerNode peer { id: 7444649 store_id: 14 role: Learner } } }"] [index=1262571] [term=39] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.496 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65412 version: 102"] [kind=EnterJoint] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.496 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7351569 start_key: 6F706466 region_epoch { conf_ver: 65414 version: 102 } peers { id: 7442238 store_id: 9 } peers { id: 7443351 store_id: 2 } peers { id: 7444649 store_id: 14 role: DemotingVoter } peers { id: 7444921 store_id: 3 role: IncomingVoter }"] ["original region"="id: 7351569 start_key: 6F7392E706466 region_epoch { conf_ver: 65412 version: 102 } peers { id: 7442238 store_id: 9 } peers { id: 7443351 store_id: 2 } peers { id: 7444649 store_id: 14 } peers { id: 7444921 store_id: 3 role: Learner }"] [changes="[peer { id: 7444921 store_id: 3 }, change_type: AddLearnerNode peer { id: 7444649 store_id: 14 role: Learner }]"] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.505 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7443351, 7442238, 7444921} }, outgoing: Configuration { voters: {7444649, 7442238, 7443351} } }, learners: {}, learners_next: {7444649}, auto_leave: false }"] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.533 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=1262572] [term=39] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.533 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65414 version: 102"] [kind=LeaveJoint] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.533 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7351569 start_key: 6F7343330392E706466 region_epoch { conf_ver: 65416 version: 102 } peers { id: 7442238 store_id: 9 } peers { id: 7443351 store_id: 2 } peers { id: 7444649 store_id: 14 role: Learner } peers { id: 7444921 store_id: 3 }"] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.537 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7443351, 7442238, 7444921} }, outgoing: Configuration { voters: {} } }, learners: {7444649}, learners_next: {}, auto_leave: false }"] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:11.612 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=296296762] [region_id=7218128]
[2025/09/06 04:16:11.847 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=122589] [size=109069377] [region_id=7218128]
[2025/09/06 04:16:11.947 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=404811213] [region_id=7322408]
[2025/09/06 04:16:12.163 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138172] [size=100663052] [region_id=7322408]
[2025/09/06 04:16:12.277 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301987108] [region_id=7404664]
[2025/09/06 04:16:12.414 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=137978] [size=100662856] [region_id=7404664]
[2025/09/06 04:16:12.493 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7390716]
[2025/09/06 04:16:12.493 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:12.984 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=241833819] [region_id=1014895]
[2025/09/06 04:16:13.149 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=103951] [size=74832138] [region_id=1014895]

[2025/09/06 04:16:13.458 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7444649 store_id: 14 role: Learner } } }"] [index=1262573] [term=39] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:13.458 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65416 version: 102"] [kind=Simple] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:13.458 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7351569 start_key: 6F7373706466 region_epoch { conf_ver: 65417 version: 102 } peers { id: 7442238 store_id: 9 } peers { id: 7443351 store_id: 2 } peers { id: 7444921 store_id: 3 }"] ["original region"="id: 7351569 start_key: 6F7338343330392E706466 region_epoch { conf_ver: 65416 version: 102 } peers { id: 7442238 store_id: 9 } peers { id: 7443351 store_id: 2 } peers { id: 7444649 store_id: 14 role: Learner } peers { id: 7444921 store_id: 3 }"] [changes="[change_type: RemoveNode peer { id: 7444649 store_id: 14 role: Learner }]"] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:13.459 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7443351, 7442238, 7444921} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444921] [region_id=7351569]
[2025/09/06 04:16:13.476 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=406855451] [region_id=7219466]
[2025/09/06 04:16:13.714 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129201] [size=100662847] [region_id=7219466]
[2025/09/06 04:16:14.451 +08:00] [INFO] [peer.rs:312] ["replicate peer"] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:14.451 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444927] [region_id=7297670]
[2025/09/06 04:16:14.451 +08:00] [INFO] [raft.rs:1120] ["became follower at term 0"] [term=0] [raft_id=7444927] [region_id=7297670]
[2025/09/06 04:16:14.451 +08:00] [INFO] [raft.rs:384] [newRaft] [peers="Configuration { incoming: Configuration { voters: {} }, outgoing: Configuration { voters: {} } }"] ["last term"=0] ["last index"=0] [applied=0] [commit=0] [term=0] [raft_id=7444927] [region_id=7297670]
[2025/09/06 04:16:14.451 +08:00] [INFO] [raw_node.rs:315] ["RawNode created with id 7444927."] [id=7444927] [raft_id=7444927] [region_id=7297670]
[2025/09/06 04:16:14.451 +08:00] [INFO] [raft.rs:1364] ["received a message with higher term from 7440507"] ["msg type"=MsgHeartbeat] [message_term=92] [term=0] [from=7440507] [raft_id=7444927] [region_id=7297670]
[2025/09/06 04:16:14.451 +08:00] [INFO] [raft.rs:1120] ["became follower at term 92"] [term=92] [raft_id=7444927] [region_id=7297670]
[2025/09/06 04:16:14.493 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7390716]
[2025/09/06 04:16:14.494 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:14.544 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=400875302] [region_id=7217841]
[2025/09/06 04:16:14.766 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138610] [size=100662989] [region_id=7217841]
[2025/09/06 04:16:15.082 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301987878] [region_id=7331699]
[2025/09/06 04:16:15.092 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=614470] [term=30] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:15.092 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66176 version: 111"] [kind=LeaveJoint] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:15.092 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7390716 start_key: 6F73735F6F06466 region_epoch { conf_ver: 66178 version: 111 } peers { id: 7444457 store_id: 2 role: Learner } peers { id: 7444566 store_id: 3 } peers { id: 7444660 store_id: 9 } peers { id: 7444916 store_id: 1 }"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:15.092 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7390716]
[2025/09/06 04:16:15.093 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:15.097 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444660, 7444566, 7444916} }, outgoing: Configuration { voters: {} } }, learners: {7444457}, learners_next: {}, auto_leave: false }"] [raft_id=7444566] [region_id=7390716]
[2025/09/06 04:16:15.097 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7390716 start_key: 6F737353035392E706466 region_epoch { conf_ver: 66178 version: 111 } peers { id: 7444457 store_id: 2 role: Learner } peers { id: 7444566 store_id: 3 } peers { id: 7444660 store_id: 9 } peers { id: 7444916 store_id: 1 }"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:15.099 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444457 store_id: 2 role: Learner } change_type: RemoveNode]"] [region_id=7390716]
[2025/09/06 04:16:15.099 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=Simple] [changes="[change_type: RemoveNode peer { id: 7444457 store_id: 2 role: Learner }]"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:15.257 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138181] [size=100662626] [region_id=7331699]
[2025/09/06 04:16:15.434 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:15.472 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data0/tikv-data/tikv-20160/snap/rev_7245948_213_3795558_(default|lock|write).sst] [snap_key=7245948_213_3795558]
[2025/09/06 04:16:15.478 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=43.609559ms] [snap_key=7245948_213_3795558]
[2025/09/06 04:16:15.478 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]

[2025/09/06 04:16:15.483 +08:00] [INFO] [raft_log.rs:627] ["log [committed=3795038, persisted=3795038, applied=3795038, unstable.offset=3795039, unstable.entries.len()=0] starts to restore snapshot [index: 3795558, term: 213]"] [snapshot_term=213] [snapshot_index=3795558] [log="committed=3795038, persisted=3795038, applied=3795038, unstable.offset=3795039, unstable.entries.len()=0"] [raft_id=7444890] [region_id=7245948]
[2025/09/06 04:16:15.483 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444538, 7444890, 7444835} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444890] [region_id=7245948]
[2025/09/06 04:16:15.483 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=213] [snapshot_index=3795558] [last_term=213] [last_index=3795558] [commit=3795558] [raft_id=7444890] [region_id=7245948]
[2025/09/06 04:16:15.483 +08:00] [INFO] [raft.rs:2511] ["[commit: 3795558, term: 213] restored snapshot [index: 3795558, term: 213]"] [snapshot_term=213] [snapshot_index=3795558] [commit=3795558] [term=213] [raft_id=7444890] [region_id=7245948]
[2025/09/06 04:16:15.483 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:15.483 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=26.698µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7245948]
[2025/09/06 04:16:15.483 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 3795558 commit_index: 3795038 commit_term: 213 truncated_state { index: 3795558 term: 213 }"] [region="id: 7245948 start_key: 6F7372E706466 region_epoch { conf_ver: 65801 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 } peers { id: 7444890 store_id: 3 }"] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:15.486 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=213] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:15.486 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7245948]
[2025/09/06 04:16:15.486 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7245948 start_key: 6F73735832393734382E706466 region_epoch { conf_ver: 65801 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 } peers { id: 7444890 store_id: 3 }"] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:15.486 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7245948 start_key: 6F73733734382E706466 region_epoch { conf_ver: 65801 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 } peers { id: 7444890 store_id: 3 }"] [prev_region="id: 7245948 start_key: 6F73735F6F6E706466 region_epoch { conf_ver: 65801 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 } peers { id: 7444890 store_id: 3 }"] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:15.487 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:15.681 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:15.782 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=99.022639ms] [region_id=7245948]
[2025/09/06 04:16:16.014 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:16.040 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data0/tikv-data/tikv-20160/snap/rev_7265961_150_2925030_(default|lock|write).sst] [snap_key=7265961_150_2925030]
[2025/09/06 04:16:16.044 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7444457 store_id: 2 role: Learner } } }"] [index=614959] [term=30] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:16.044 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66178 version: 111"] [kind=Simple] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:16.045 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7390716 start_key: 6F73735F6F38373238303035392E706466 region_epoch { conf_ver: 66179 version: 111 } peers { id: 7444566 store_id: 3 } peers { id: 7444660 store_id: 9 } peers { id: 7444916 store_id: 1 }"] ["original region"="id: 7390716 start_key: 6F73735F6F373238303035392E706466 region_epoch { conf_ver: 66178 version: 111 } peers { id: 7444457 store_id: 2 role: Learner } peers { id: 7444566 store_id: 3 } peers { id: 7444660 store_id: 9 } peers { id: 7444916 store_id: 1 }"] [changes="[change_type: RemoveNode peer { id: 7444457 store_id: 2 role: Learner }]"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:16.063 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444660, 7444566, 7444916} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444566] [region_id=7390716]
[2025/09/06 04:16:16.063 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7390716 start_key: 6F73735F6F3238303035392E706466 region_epoch { conf_ver: 66179 version: 111 } peers { id: 7444566 store_id: 3 } peers { id: 7444660 store_id: 9 } peers { id: 7444916 store_id: 1 }"] [peer_id=7444566] [region_id=7390716]
[2025/09/06 04:16:16.064 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=50.321203ms] [snap_key=7265961_150_2925030]
[2025/09/06 04:16:16.064 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:16.070 +08:00] [INFO] [raft_log.rs:627] ["log [committed=2924554, persisted=2924554, applied=2924554, unstable.offset=2924555, unstable.entries.len()=0] starts to restore snapshot [index: 2925030, term: 150]"] [snapshot_term=150] [snapshot_index=2925030] [log="committed=2924554, persisted=2924554, applied=2924554, unstable.offset=2924555, unstable.entries.len()=0"] [raft_id=7444397] [region_id=7265961]
[2025/09/06 04:16:16.070 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444397, 7442004, 7444603} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444397] [region_id=7265961]
[2025/09/06 04:16:16.070 +08:00] [INFO] [raft_log.rs:627] ["log [committed=2924554, persisted=2924554, applied=2924554, unstable.offset=2924555, unstable.entries.len()=0] starts to restore snapshot [index: 2925030, term: 150]"] [snapshot_term=150] [snapshot_index=2925030] [log="committed=2924554, persisted=2924554, applied=2924554, unstable.offset=2924555, unstable.entries.len()=0"] [raft_id=7444397] [region_id=7265961]
[2025/09/06 04:16:16.070 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444397, 7442004, 7444603} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444397] [region_id=7265961]
[2025/09/06 04:16:16.070 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=150] [snapshot_index=2925030] [last_term=150] [last_index=2925030] [commit=2925030] [raft_id=7444397] [region_id=7265961]
[2025/09/06 04:16:16.070 +08:00] [INFO] [raft.rs:2511] ["[commit: 2925030, term: 150] restored snapshot [index: 2925030, term: 150]"] [snapshot_term=150] [snapshot_index=2925030] [commit=2925030] [term=150] [raft_id=7444397] [region_id=7265961]
[2025/09/06 04:16:16.070 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444397] [region_id=7265961]
[2025/09/06 04:16:16.070 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=5.804µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7265961]
[2025/09/06 04:16:16.070 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 2925030 commit_index: 2924554 commit_term: 150 truncated_state { index: 2925030 term: 150 }"] [region="id: 7265961 start_key: 6F73735F6F626A72E706466 region_epoch { conf_ver: 65789 version: 79 } peers { id: 7442004 store_id: 1 } peers { id: 7444397 store_id: 3 } peers { id: 7444603 store_id: 8 }"] [peer_id=7444397] [region_id=7265961]
[2025/09/06 04:16:16.077 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7265961]
[2025/09/06 04:16:16.077 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7265961 start_key: 6F737313436372E706466 region_epoch { conf_ver: 65789 version: 79 } peers { id: 7442004 store_id: 1 } peers { id: 7444397 store_id: 3 } peers { id: 7444603 store_id: 8 }"] [peer_id=7444397] [region_id=7265961]
[2025/09/06 04:16:16.077 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7265961 start_key: 6F73735F313436372E706466 region_epoch { conf_ver: 65789 version: 79 } peers { id: 7442004 store_id: 1 } peers { id: 7444397 store_id: 3 } peers { id: 7444603 store_id: 8 }"] [prev_region="id: 7265961 start_key: 6F73735F63436372E706466 region_epoch { conf_ver: 65789 version: 79 } peers { id: 7442004 store_id: 1 } peers { id: 7444397 store_id: 3 } peers { id: 7444603 store_id: 8 }"] [peer_id=7444397] [region_id=7265961]
[2025/09/06 04:16:16.077 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=150] [peer_id=7444397] [region_id=7265961]
[2025/09/06 04:16:16.911 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:16.944 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data0/tikv-data/tikv-20160/snap/rev_7297670_92_2489525_(default|lock|write).sst] [snap_key=7297670_92_2489525]
[2025/09/06 04:16:16.967 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=56.364218ms] [snap_key=7297670_92_2489525]
[2025/09/06 04:16:16.968 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:16.973 +08:00] [INFO] [raft_log.rs:627] ["log [committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0] starts to restore snapshot [index: 2489525, term: 92]"] [snapshot_term=92] [snapshot_index=2489525] [log="committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0"] [raft_id=7444927] [region_id=7297670]
[2025/09/06 04:16:16.973 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7438221, 7441402, 7440507} }, outgoing: Configuration { voters: {} } }, learners: {7444927}, learners_next: {}, auto_leave: false }"] [raft_id=7444927] [region_id=7297670]
[2025/09/06 04:16:16.973 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=92] [snapshot_index=2489525] [last_term=92] [last_index=2489525] [commit=2489525] [raft_id=7444927] [region_id=7297670]
[2025/09/06 04:16:16.973 +08:00] [INFO] [raft.rs:2511] ["[commit: 2489525, term: 92] restored snapshot [index: 2489525, term: 92]"] [snapshot_term=92] [snapshot_index=2489525] [commit=2489525] [term=92] [raft_id=7444927] [region_id=7297670]
[2025/09/06 04:16:16.973 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:16.973 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 2489525 truncated_state { index: 2489525 term: 92 }"] [region="id: 7297670 start_key: 6F73735F63833373939362E706466 region_epoch { conf_ver: 65070 version: 85 } peers { id: 7438221 store_id: 14 } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 role: Learner }"] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:16.976 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7297670 start_key: 6F73733833373939362E706466 region_epoch { conf_ver: 65070 version: 85 } peers { id: 7438221 store_id: 14 } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 role: Learner }"] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:17.130 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=314571146] [region_id=7390716]
[2025/09/06 04:16:17.288 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=137978] [size=100662724] [region_id=7390716]
[2025/09/06 04:16:19.343 +08:00] [INFO] [peer.rs:312] ["replicate peer"] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:19.343 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444932] [region_id=7217664]
[2025/09/06 04:16:19.343 +08:00] [INFO] [raft.rs:1120] ["became follower at term 0"] [term=0] [raft_id=7444932] [region_id=7217664]
[2025/09/06 04:16:19.343 +08:00] [INFO] [raft.rs:384] [newRaft] [peers="Configuration { incoming: Configuration { voters: {} }, outgoing: Configuration { voters: {} } }"] ["last term"=0] ["last index"=0] [applied=0] [commit=0] [term=0] [raft_id=7444932] [region_id=7217664]
[2025/09/06 04:16:19.343 +08:00] [INFO] [raw_node.rs:315] ["RawNode created with id 7444932."] [id=7444932] [raft_id=7444932] [region_id=7217664]
[2025/09/06 04:16:19.343 +08:00] [INFO] [raft.rs:1364] ["received a message with higher term from 7442424"] ["msg type"=MsgHeartbeat] [message_term=329] [term=0] [from=7442424] [raft_id=7444932] [region_id=7217664]
[2025/09/06 04:16:19.343 +08:00] [INFO] [raft.rs:1120] ["became follower at term 329"] [term=329] [raft_id=7444932] [region_id=7217664]
[2025/09/06 04:16:19.703 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=402951182] [region_id=7249928]
[2025/09/06 04:16:19.910 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138454] [size=100662602] [region_id=7249928]
[2025/09/06 04:16:20.238 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444397] [region_id=7265961]
[2025/09/06 04:16:20.500 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:20.519 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:20.548 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data0/tikv-data/tikv-20160/snap/rev_7217664_329_9929520_(default|lock|write).sst] [snap_key=7217664_329_9929520]
[2025/09/06 04:16:20.553 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=34.113063ms] [snap_key=7217664_329_9929520]
[2025/09/06 04:16:20.553 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:20.553 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=51.48651ms] [region_id=7265961]
[2025/09/06 04:16:20.559 +08:00] [INFO] [raft_log.rs:627] ["log [committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0] starts to restore snapshot [index: 9929520, term: 329]"] [snapshot_term=329] [snapshot_index=9929520] [log="committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0"] [raft_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.559 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:20.559 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7442424, 7440593, 7443844} }, outgoing: Configuration { voters: {} } }, learners: {7444932}, learners_next: {}, auto_leave: false }"] [raft_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.559 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=329] [snapshot_index=9929520] [last_term=329] [last_index=9929520] [commit=9929520] [raft_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.559 +08:00] [INFO] [raft.rs:2511] ["[commit: 9929520, term: 329] restored snapshot [index: 9929520, term: 329]"] [snapshot_term=329] [snapshot_index=9929520] [commit=9929520] [term=329] [raft_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.559 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.559 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 9929520 truncated_state { index: 9929520 term: 329 }"] [region="id: 7217664 start_key: 6F73735F6F6466 region_epoch { conf_ver: 82062 version: 85 } peers { id: 7440593 store_id: 14 } peers { id: 7442424 store_id: 9 } peers { id: 7443844 store_id: 1 } peers { id: 7444932 store_id: 3 role: Learner }"] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.559 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7217664 start_key: 6F73735F6F626A5F43035302E706466 region_epoch { conf_ver: 82062 version: 85 } peers { id: 7440593 store_id: 14 } peers { id: 7442424 store_id: 9 } peers { id: 7443844 store_id: 1 } peers { id: 7444932 store_id: 3 role: Learner }"] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.560 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:20.621 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=59.77616ms] [region_id=7297670]
[2025/09/06 04:16:20.621 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.622 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:20.623 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444927 store_id: 3 } } changes { change_type: AddLearnerNode peer { id: 7438221 store_id: 14 role: Learner } } }"] [index=2489526] [term=92] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:20.624 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65070 version: 85"] [kind=EnterJoint] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:20.624 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7297670 start_key: 6F73735F313536333833373939362E706466 region_epoch { conf_ver: 65072 version: 85 } peers { id: 7438221 store_id: 14 role: DemotingVoter } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 role: IncomingVoter }"] ["original region"="id: 7297670 start_key: 6F73735F6F62673939362E706466 region_epoch { conf_ver: 65070 version: 85 } peers { id: 7438221 store_id: 14 } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 role: Learner }"] [changes="[peer { id: 7444927 store_id: 3 }, change_type: AddLearnerNode peer { id: 7438221 store_id: 14 role: Learner }]"] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:20.624 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7441402, 7444927, 7440507} }, outgoing: Configuration { voters: {7438221, 7441402, 7440507} } }, learners: {}, learners_next: {7438221}, auto_leave: false }"] [raft_id=7444927] [region_id=7297670]

[2025/09/06 04:16:20.629 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=2489527] [term=92] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:20.629 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65072 version: 85"] [kind=LeaveJoint] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:20.629 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7297670 start_key: 6F73735362E706466 region_epoch { conf_ver: 65074 version: 85 } peers { id: 7438221 store_id: 14 role: Learner } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 }"] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:20.630 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7441402, 7444927, 7440507} }, outgoing: Configuration { voters: {} } }, learners: {7438221}, learners_next: {}, auto_leave: false }"] [raft_id=7444927] [region_id=7297670]
[2025/09/06 04:16:20.634 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7438221 store_id: 14 role: Learner } } }"] [index=2489528] [term=92] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:20.634 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65074 version: 85"] [kind=Simple] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:20.634 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7297670 start_key: 6F737706466 region_epoch { conf_ver: 65075 version: 85 } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 }"] ["original region"="id: 7297670 start_key: 6F706466 region_epoch { conf_ver: 65074 version: 85 } peers { id: 7438221 store_id: 14 role: Learner } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 }"] [changes="[change_type: RemoveNode peer { id: 7438221 store_id: 14 role: Learner }]"] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:20.635 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7441402, 7444927, 7440507} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444927] [region_id=7297670]
[2025/09/06 04:16:20.666 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=42.065188ms] [region_id=7217664]
[2025/09/06 04:16:20.668 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444932 store_id: 3 } } changes { change_type: AddLearnerNode peer { id: 7440593 store_id: 14 role: Learner } } }"] [index=9929521] [term=329] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.668 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 82062 version: 85"] [kind=EnterJoint] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.668 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7217664 start_key: 6F7373E706466 region_epoch { conf_ver: 82064 version: 85 } peers { id: 7440593 store_id: 14 role: DemotingVoter } peers { id: 7442424 store_id: 9 } peers { id: 7443844 store_id: 1 } peers { id: 7444932 store_id: 3 role: IncomingVoter }"] ["original region"="id: 7217664 start_key: 6F73735F635302E706466 region_epoch { conf_ver: 82062 version: 85 } peers { id: 7440593 store_id: 14 } peers { id: 7442424 store_id: 9 } peers { id: 7443844 store_id: 1 } peers { id: 7444932 store_id: 3 role: Learner }"] [changes="[peer { id: 7444932 store_id: 3 }, change_type: AddLearnerNode peer { id: 7440593 store_id: 14 role: Learner }]"] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.669 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7442424, 7443844, 7444932} }, outgoing: Configuration { voters: {7442424, 7440593, 7443844} } }, learners: {}, learners_next: {7440593}, auto_leave: false }"] [raft_id=7444932] [region_id=7217664]

[jwongz]

字数限制，接store-3

[2025/09/06 04:16:20.670 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=9929522] [term=329] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.670 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 82064 version: 85"] [kind=LeaveJoint] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.670 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7217664 start_key: 6F73735F6F66 region_epoch { conf_ver: 82066 version: 85 } peers { id: 7440593 store_id: 14 role: Learner } peers { id: 7442424 store_id: 9 } peers { id: 7443844 store_id: 1 } peers { id: 7444932 store_id: 3 }"] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.670 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7442424, 7443844, 7444932} }, outgoing: Configuration { voters: {} } }, learners: {7440593}, learners_next: {}, auto_leave: false }"] [raft_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.671 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7440593 store_id: 14 role: Learner } } }"] [index=9929523] [term=329] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.671 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 82066 version: 85"] [kind=Simple] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.671 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7217664 start_key: 6F73735706466 region_epoch { conf_ver: 82067 version: 85 } peers { id: 7442424 store_id: 9 } peers { id: 7443844 store_id: 1 } peers { id: 7444932 store_id: 3 }"] ["original region"="id: 7217664 start_key: 6F73735F6302E706466 region_epoch { conf_ver: 82066 version: 85 } peers { id: 7440593 store_id: 14 role: Learner } peers { id: 7442424 store_id: 9 } peers { id: 7443844 store_id: 1 } peers { id: 7444932 store_id: 3 }"] [changes="[change_type: RemoveNode peer { id: 7440593 store_id: 14 role: Learner }]"] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:20.672 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7442424, 7443844, 7444932} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444932] [region_id=7217664]
[2025/09/06 04:16:23.497 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201151635] [region_id=7217620]
[2025/09/06 04:16:23.661 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=140061] [size=100662925] [region_id=7217620]
[2025/09/06 04:16:24.545 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986885] [region_id=7217841]

[2025/09/06 04:16:24.709 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138610] [size=100662989] [region_id=7217841]
[2025/09/06 04:16:24.723 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=401345935] [region_id=7218862]
[2025/09/06 04:16:24.955 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129275] [size=100663070] [region_id=7218862]
[2025/09/06 04:16:26.844 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444938 store_id: 9 role: Learner } } }"] [index=3795560] [term=213] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:26.844 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65801 version: 67"] [kind=Simple] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:26.844 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7245948 start_key: 6F73832393734382E706466 region_epoch { conf_ver: 65802 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 } peers { id: 7444890 store_id: 3 } peers { id: 7444938 store_id: 9 role: Learner }"] ["original region"="id: 7245948 start_key: 6F7393734382E706466 region_epoch { conf_ver: 65801 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 } peers { id: 7444890 store_id: 3 }"] [changes="[change_type: AddLearnerNode peer { id: 7444938 store_id: 9 role: Learner }]"] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:26.844 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444538, 7444890, 7444835} }, outgoing: Configuration { voters: {} } }, learners: {7444938}, learners_next: {}, auto_leave: false }"] [raft_id=7444890] [region_id=7245948]
[2025/09/06 04:16:27.460 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7217664_329_9929520] [peer_id=7444932] [region_id=7217664]
[2025/09/06 04:16:27.460 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7219863_197_6676863] [peer_id=7443473] [region_id=7219863]
[2025/09/06 04:16:27.460 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7245948_213_3795015] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:27.460 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7251415_183_3287215] [peer_id=7444898] [region_id=7251415]
[2025/09/06 04:16:27.460 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7245948_213_3795558] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:27.460 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7252821_182_3212918] [peer_id=7444875] [region_id=7252821]
[2025/09/06 04:16:27.460 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:27.460 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7265961_150_2925030] [peer_id=7444397] [region_id=7265961]
[2025/09/06 04:16:27.460 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7297670_92_2489525] [peer_id=7444927] [region_id=7297670]
[2025/09/06 04:16:27.460 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7337276_52_1620582] [peer_id=7444911] [region_id=7337276]
[2025/09/06 04:16:27.460 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7339530_47_1546953] [peer_id=7444863] [region_id=7339530]
[2025/09/06 04:16:27.460 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7351569_39_1262558] [peer_id=7444921] [region_id=7351569]
[2025/09/06 04:16:27.470 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:27.475 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:27.479 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:27.483 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:27.488 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:27.491 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:27.493 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:27.496 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:27.498 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:27.500 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:28.956 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444938 store_id: 9 } } changes { change_type: AddLearnerNode peer { id: 7444835 store_id: 8 role: Learner } } }"] [index=3795561] [term=213] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:28.956 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65802 version: 67"] [kind=EnterJoint] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:28.956 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7245948 start_key: 6F7372393734382E706466 region_epoch { conf_ver: 65804 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 role: DemotingVoter } peers { id: 7444890 store_id: 3 } peers { id: 7444938 store_id: 9 role: IncomingVoter }"] ["original region"="id: 7245948 start_key: 6F73793734382E706466 region_epoch { conf_ver: 65802 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 } peers { id: 7444890 store_id: 3 } peers { id: 7444938 store_id: 9 role: Learner }"] [changes="[peer { id: 7444938 store_id: 9 }, change_type: AddLearnerNode peer { id: 7444835 store_id: 8 role: Learner }]"] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:28.959 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=3795562] [term=213] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:28.959 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65804 version: 67"] [kind=LeaveJoint] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:28.959 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7245948 start_key: 6F7373593734382E706466 region_epoch { conf_ver: 65806 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 role: Learner } peers { id: 7444890 store_id: 3 } peers { id: 7444938 store_id: 9 }"] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:28.959 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444538, 7444890, 7444938} }, outgoing: Configuration { voters: {7444835, 7444538, 7444890} } }, learners: {}, learners_next: {7444835}, auto_leave: false }"] [raft_id=7444890] [region_id=7245948]

[2025/09/06 04:16:28.961 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444538, 7444890, 7444938} }, outgoing: Configuration { voters: {} } }, learners: {7444835}, learners_next: {}, auto_leave: false }"] [raft_id=7444890] [region_id=7245948]
[2025/09/06 04:16:28.961 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7444835 store_id: 8 role: Learner } } }"] [index=3795563] [term=213] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:28.961 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65806 version: 67"] [kind=Simple] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:28.961 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7245948 start_key: 6F7373382E706466 region_epoch { conf_ver: 65807 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444890 store_id: 3 } peers { id: 7444938 store_id: 9 }"] ["original region"="id: 7245948 start_key: 6F7373734382E706466 region_epoch { conf_ver: 65806 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 role: Learner } peers { id: 7444890 store_id: 3 } peers { id: 7444938 store_id: 9 }"] [changes="[change_type: RemoveNode peer { id: 7444835 store_id: 8 role: Learner }]"] [peer_id=7444890] [region_id=7245948]
[2025/09/06 04:16:28.964 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444538, 7444890, 7444938} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444890] [region_id=7245948]
[2025/09/06 04:16:32.547 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201324982] [region_id=7404664]
[2025/09/06 04:16:32.642 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=137978] [size=100662856] [region_id=7404664]
[2025/09/06 04:16:32.770 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986966] [region_id=7322408]
[2025/09/06 04:16:32.926 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138172] [size=100663052] [region_id=7322408]
[2025/09/06 04:16:33.040 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301985031] [region_id=7218236]
[2025/09/06 04:16:33.045 +08:00] [INFO] [peer.rs:312] ["replicate peer"] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:33.045 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:33.045 +08:00] [INFO] [raft.rs:1120] ["became follower at term 0"] [term=0] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:33.045 +08:00] [INFO] [raft.rs:384] [newRaft] [peers="Configuration { incoming: Configuration { voters: {} }, outgoing: Configuration { voters: {} } }"] ["last term"=0] ["last index"=0] [applied=0] [commit=0] [term=0] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:33.045 +08:00] [INFO] [raw_node.rs:315] ["RawNode created with id 7444946."] [id=7444946] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:33.045 +08:00] [INFO] [raft.rs:1364] ["received a message with higher term from 7443211"] ["msg type"=MsgHeartbeat] [message_term=198] [term=0] [from=7443211] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:33.045 +08:00] [INFO] [raft.rs:1120] ["became follower at term 198"] [term=198] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:33.233 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=117886] [size=100662566] [region_id=7218236]
[2025/09/06 04:16:33.508 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301989729] [region_id=7318761]
[2025/09/06 04:16:33.661 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138124] [size=100663243] [region_id=7318761]
[2025/09/06 04:16:33.680 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301988859] [region_id=7221921]
[2025/09/06 04:16:33.857 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128653] [size=100662979] [region_id=7221921]
[2025/09/06 04:16:33.980 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:34.005 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data0/tikv-data/tikv-20160/snap/rev_7244910_198_3864240_(default|lock|write).sst] [snap_key=7244910_198_3864240]
[2025/09/06 04:16:34.008 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=28.081315ms] [snap_key=7244910_198_3864240]
[2025/09/06 04:16:34.008 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:34.013 +08:00] [INFO] [raft_log.rs:627] ["log [committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0] starts to restore snapshot [index: 3864240, term: 198]"] [snapshot_term=198] [snapshot_index=3864240] [log="committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0"] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.013 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444677, 7444530, 7443211} }, outgoing: Configuration { voters: {} } }, learners: {7444946}, learners_next: {}, auto_leave: false }"] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.013 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=198] [snapshot_index=3864240] [last_term=198] [last_index=3864240] [commit=3864240] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.013 +08:00] [INFO] [raft.rs:2511] ["[commit: 3864240, term: 198] restored snapshot [index: 3864240, term: 198]"] [snapshot_term=198] [snapshot_index=3864240] [commit=3864240] [term=198] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.013 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.014 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 3864240 truncated_state { index: 3864240 term: 198 }"] [region="id: 7244910 start_key: 6F7373832342E706466 region_epoch { conf_ver: 65640 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 } peers { id: 7444946 store_id: 3 role: Learner }"] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.015 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7244910 start_key: 6F7373533832342E706466 region_epoch { conf_ver: 65640 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 } peers { id: 7444946 store_id: 3 role: Learner }"] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.015 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.016 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:34.092 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=71.922741ms] [region_id=7244910]
[2025/09/06 04:16:34.094 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444946 store_id: 3 } } changes { change_type: AddLearnerNode peer { id: 7444677 store_id: 14 role: Learner } } }"] [index=3864241] [term=198] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.094 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65640 version: 66"] [kind=EnterJoint] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.094 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7244910 start_key: 6F737342E706466 region_epoch { conf_ver: 65642 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 role: DemotingVoter } peers { id: 7444946 store_id: 3 role: IncomingVoter }"] ["original region"="id: 7244910 start_key: 6F73735F706466 region_epoch { conf_ver: 65640 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 } peers { id: 7444946 store_id: 3 role: Learner }"] [changes="[peer { id: 7444946 store_id: 3 }, change_type: AddLearnerNode peer { id: 7444677 store_id: 14 role: Learner }]"] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.094 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444530, 7444946, 7443211} }, outgoing: Configuration { voters: {7444677, 7444530, 7443211} } }, learners: {}, learners_next: {7444677}, auto_leave: false }"] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=3864242] [term=198] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65642 version: 66"] [kind=LeaveJoint] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7244910 start_key: 6F737E706466 region_epoch { conf_ver: 65644 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 role: Learner } peers { id: 7444946 store_id: 3 }"] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444530, 7444946, 7443211} }, outgoing: Configuration { voters: {} } }, learners: {7444677}, learners_next: {}, auto_leave: false }"] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.097 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7444677 store_id: 14 role: Learner } } }"] [index=3864243] [term=198] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.097 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65644 version: 66"] [kind=Simple] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.097 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7244910 start_key: 6F737E706466 region_epoch { conf_ver: 65645 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444946 store_id: 3 }"] ["original region"="id: 7244910 start_key: 6F7E706466 region_epoch { conf_ver: 65644 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 role: Learner } peers { id: 7444946 store_id: 3 }"] [changes="[change_type: RemoveNode peer { id: 7444677 store_id: 14 role: Learner }]"] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.097 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444530, 7444946, 7443211} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444946] [region_id=7244910]
[2025/09/06 04:16:34.724 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301987543] [region_id=7218862]
[2025/09/06 04:16:34.902 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129275] [size=100663070] [region_id=7218862]
[2025/09/06 04:16:35.722 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986387] [region_id=7219466]
[2025/09/06 04:16:35.894 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129201] [size=100662847] [region_id=7219466]
[2025/09/06 04:16:36.672 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444957 store_id: 9 role: Learner } } }"] [index=3069663] [term=175] [peer_id=7443852] [region_id=7257488]
[2025/09/06 04:16:36.672 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65813 version: 77"] [kind=Simple] [peer_id=7443852] [region_id=7257488]
[2025/09/06 04:16:36.672 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7257488 start_key: 6F736466 region_epoch { conf_ver: 65814 version: 77 } peers { id: 7440340 store_id: 8 } peers { id: 7443483 store_id: 1 } peers { id: 7443852 store_id: 3 } peers { id: 7444957 store_id: 9 role: Learner }"] ["original region"="id: 7257488 start_key: 6F7382E706466 region_epoch { conf_ver: 65813 version: 77 } peers { id: 7440340 store_id: 8 } peers { id: 7443483 store_id: 1 } peers { id: 7443852 store_id: 3 }"] [changes="[change_type: AddLearnerNode peer { id: 7444957 store_id: 9 role: Learner }]"] [peer_id=7443852] [region_id=7257488]
[2025/09/06 04:16:36.676 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7440340, 7443852, 7443483} }, outgoing: Configuration { voters: {} } }, learners: {7444957}, learners_next: {}, auto_leave: false }"] [raft_id=7443852] [region_id=7257488]
[2025/09/06 04:16:38.101 +08:00] [INFO] [peer.rs:312] ["replicate peer"] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:38.101 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:38.101 +08:00] [INFO] [raft.rs:1120] ["became follower at term 0"] [term=0] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:38.101 +08:00] [INFO] [raft.rs:384] [newRaft] [peers="Configuration { incoming: Configuration { voters: {} }, outgoing: Configuration { voters: {} } }"] ["last term"=0] ["last index"=0] [applied=0] [commit=0] [term=0] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:38.101 +08:00] [INFO] [raw_node.rs:315] ["RawNode created with id 7444964."] [id=7444964] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:38.141 +08:00] [INFO] [raft.rs:1364] ["received a message with higher term from 7444547"] ["msg type"=MsgHeartbeat] [message_term=16] [term=0] [from=7444547] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:38.141 +08:00] [INFO] [raft.rs:1120] ["became follower at term 16"] [term=16] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.341 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:39.364 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data0/tikv-data/tikv-20160/snap/rev_7418340_16_254827_(default|lock|write).sst] [snap_key=7418340_16_254827]
[2025/09/06 04:16:39.368 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=26.208697ms] [snap_key=7418340_16_254827]
[2025/09/06 04:16:39.370 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:39.372 +08:00] [INFO] [raft_log.rs:627] ["log [committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0] starts to restore snapshot [index: 254827, term: 16]"] [snapshot_term=16] [snapshot_index=254827] [log="committed=0, persisted=0, applied=0, unstable.offset=1, unstable.entries.len()=0"] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.372 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444492, 7444713, 7444547} }, outgoing: Configuration { voters: {} } }, learners: {7444964}, learners_next: {}, auto_leave: false }"] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.372 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=16] [snapshot_index=254827] [last_term=16] [last_index=254827] [commit=254827] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.372 +08:00] [INFO] [raft.rs:2511] ["[commit: 254827, term: 16] restored snapshot [index: 254827, term: 16]"] [snapshot_term=16] [snapshot_index=254827] [commit=254827] [term=16] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.372 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.372 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 254827 truncated_state { index: 254827 term: 16 }"] [region="id: 7418340 start_key: 6F737706466 region_epoch { conf_ver: 66612 version: 116 } peers { id: 7444492 store_id: 14 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 role: Learner }"] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.374 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7418340 start_key: 6F73706466 region_epoch { conf_ver: 66612 version: 116 } peers { id: 7444492 store_id: 14 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 role: Learner }"] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.374 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.381 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:39.472 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=87.4308ms] [region_id=7418340]
[2025/09/06 04:16:39.483 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444964 store_id: 3 } } changes { change_type: AddLearnerNode peer { id: 7444492 store_id: 14 role: Learner } } }"] [index=254828] [term=16] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.483 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66612 version: 116"] [kind=EnterJoint] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.483 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7418340 start_key: 6F7373032352E706466 region_epoch { conf_ver: 66614 version: 116 } peers { id: 7444492 store_id: 14 role: DemotingVoter } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 role: IncomingVoter }"] ["original region"="id: 7418340 start_key: 6F73735F6F626A5F632D722352E706466 region_epoch { conf_ver: 66612 version: 116 } peers { id: 7444492 store_id: 14 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 role: Learner }"] [changes="[peer { id: 7444964 store_id: 3 }, change_type: AddLearnerNode peer { id: 7444492 store_id: 14 role: Learner }]"] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.485 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444713, 7444964, 7444547} }, outgoing: Configuration { voters: {7444492, 7444713, 7444547} } }, learners: {}, learners_next: {7444492}, auto_leave: false }"] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.495 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=254829] [term=16] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.495 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66614 version: 116"] [kind=LeaveJoint] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.496 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7418340 start_key: 6F73735F6F626A5F632D7352E706466 region_epoch { conf_ver: 66616 version: 116 } peers { id: 7444492 store_id: 14 role: Learner } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 }"] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.500 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444713, 7444964, 7444547} }, outgoing: Configuration { voters: {} } }, learners: {7444492}, learners_next: {}, auto_leave: false }"] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.503 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7444492 store_id: 14 role: Learner } } }"] [index=254830] [term=16] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.503 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66616 version: 116"] [kind=Simple] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.503 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7418340 start_key: 6F73735F6F626A93132363833323032352E706466 region_epoch { conf_ver: 66617 version: 116 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 }"] ["original region"="id: 7418340 start_key: 6F737306466 region_epoch { conf_ver: 66616 version: 116 } peers { id: 7444492 store_id: 14 role: Learner } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 }"] [changes="[change_type: RemoveNode peer { id: 7444492 store_id: 14 role: Learner }]"] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:16:39.505 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444713, 7444964, 7444547} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444964] [region_id=7418340]
[2025/09/06 04:16:41.431 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=209704114] [region_id=7390716]
[2025/09/06 04:16:41.515 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=137978] [size=100662724] [region_id=7390716]
[2025/09/06 04:16:42.545 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=306191573] [region_id=7404664]
[2025/09/06 04:16:42.714 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=137978] [size=100662856] [region_id=7404664]
[2025/09/06 04:16:42.771 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=318765510] [region_id=7322408]
[2025/09/06 04:16:42.936 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138172] [size=100663052] [region_id=7322408]
[2025/09/06 04:16:43.042 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=402656829] [region_id=7218236]
[2025/09/06 04:16:43.310 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=117886] [size=100662566] [region_id=7218236]
[2025/09/06 04:16:43.509 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=398465861] [region_id=7318761]
[2025/09/06 04:16:43.711 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138124] [size=100663243] [region_id=7318761]
[2025/09/06 04:16:43.866 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:43.897 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data0/tikv-data/tikv-20160/snap/rev_7219863_197_6677891_(default|lock|write).sst] [snap_key=7219863_197_6677891]
[2025/09/06 04:16:43.904 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=37.935754ms] [snap_key=7219863_197_6677891]
[2025/09/06 04:16:43.904 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:43.907 +08:00] [INFO] [raft_log.rs:627] ["log [committed=6677631, persisted=6677631, applied=6677631, unstable.offset=6677632, unstable.entries.len()=0] starts to restore snapshot [index: 6677891, term: 197]"] [snapshot_term=197] [snapshot_index=6677891] [log="committed=6677631, persisted=6677631, applied=6677631, unstable.offset=6677632, unstable.entries.len()=0"] [raft_id=7443473] [region_id=7219863]
[2025/09/06 04:16:43.908 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7437537, 7443406, 7443473} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7443473] [region_id=7219863]
[2025/09/06 04:16:43.908 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=197] [snapshot_index=6677891] [last_term=197] [last_index=6677891] [commit=6677891] [raft_id=7443473] [region_id=7219863]
[2025/09/06 04:16:43.908 +08:00] [INFO] [raft.rs:2511] ["[commit: 6677891, term: 197] restored snapshot [index: 6677891, term: 197]"] [snapshot_term=197] [snapshot_index=6677891] [commit=6677891] [term=197] [raft_id=7443473] [region_id=7219863]
[2025/09/06 04:16:43.908 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7443473] [region_id=7219863]
[2025/09/06 04:16:43.908 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=2.99µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7219863]
[2025/09/06 04:16:43.908 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 6677891 commit_index: 6677631 commit_term: 197 truncated_state { index: 6677891 term: 197 }"] [region="id: 7219863 start_key: 6F73735F335312E6A7067 region_epoch { conf_ver: 68315 version: 62 } peers { id: 7437537 store_id: 8 } peers { id: 7443406 store_id: 2 } peers { id: 7443473 store_id: 3 }"] [peer_id=7443473] [region_id=7219863]
[2025/09/06 04:16:43.908 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7219863 start_key: 6F735312E6A7067 region_epoch { conf_ver: 68315 version: 62 } peers { id: 7437537 store_id: 8 } peers { id: 7443406 store_id: 2 } peers { id: 7443473 store_id: 3 }"] [peer_id=7443473] [region_id=7219863]
[2025/09/06 04:16:43.908 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=197] [peer_id=7443473] [region_id=7219863]
[2025/09/06 04:16:43.908 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7219863]
[2025/09/06 04:16:43.908 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7219863 start_key: 6F73735F53335312E6A7067 region_epoch { conf_ver: 68315 version: 62 } peers { id: 7437537 store_id: 8 } peers { id: 7443406 store_id: 2 } peers { id: 7443473 store_id: 3 }"] [prev_region="id: 7219863 start_key: 6F73735F2E6A7067 region_epoch { conf_ver: 68315 version: 62 } peers { id: 7437537 store_id: 8 } peers { id: 7443406 store_id: 2 } peers { id: 7443473 store_id: 3 }"] [peer_id=7443473] [region_id=7219863]
[2025/09/06 04:16:43.908 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7443473] [region_id=7219863]
[2025/09/06 04:16:44.269 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:44.329 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=56.852618ms] [region_id=7219863]
[2025/09/06 04:16:44.629 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=246603711] [region_id=1014895]
[2025/09/06 04:16:44.777 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=103951] [size=74832138] [region_id=1014895]
[2025/09/06 04:16:45.723 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=402648516] [region_id=7219466]
[2025/09/06 04:16:45.944 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129201] [size=100662847] [region_id=7219466]
[2025/09/06 04:16:46.713 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=402649180] [region_id=7217841]
[2025/09/06 04:16:46.927 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138610] [size=100662989] [region_id=7217841]
[2025/09/06 04:16:51.291 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=305010086] [region_id=7331699]
[2025/09/06 04:16:51.442 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138181] [size=100662626] [region_id=7331699]
[2025/09/06 04:16:52.546 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201324982] [region_id=7404664]
[2025/09/06 04:16:52.649 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=137978] [size=100662856] [region_id=7404664]
[2025/09/06 04:16:52.772 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=417394927] [region_id=7322408]
[2025/09/06 04:16:52.930 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138172] [size=100663052] [region_id=7322408]
[2025/09/06 04:16:53.042 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301985031] [region_id=7218236]
[2025/09/06 04:16:53.234 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=117886] [size=100662566] [region_id=7218236]
[2025/09/06 04:16:53.510 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301989729] [region_id=7318761]
[2025/09/06 04:16:53.663 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138124] [size=100663243] [region_id=7318761]
[2025/09/06 04:16:53.682 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301988859] [region_id=7221921]
[2025/09/06 04:16:53.856 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128653] [size=100662979] [region_id=7221921]
[2025/09/06 04:16:54.630 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=241769385] [region_id=7218128]
[2025/09/06 04:16:54.786 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=122594] [size=109073892] [region_id=7218128]
[2025/09/06 04:16:55.724 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986387] [region_id=7219466]
[2025/09/06 04:16:55.895 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129201] [size=100662847] [region_id=7219466]
[2025/09/06 04:16:55.899 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=200004484] [region_id=7217620]
[2025/09/06 04:16:56.063 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=140061] [size=100662925] [region_id=7217620]
[2025/09/06 04:16:56.714 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986885] [region_id=7217841]
[2025/09/06 04:16:56.873 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138610] [size=100662989] [region_id=7217841]
[2025/09/06 04:16:57.556 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:16:57.556 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=Simple] [changes="[change_type: AddLearnerNode peer { id: 7444996 store_id: 9 role: Learner }]"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:17:01.433 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=209704114] [region_id=7390716]
[2025/09/06 04:17:01.530 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=137978] [size=100662724] [region_id=7390716]
[2025/09/06 04:17:02.565 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301985616] [region_id=7249928]
[2025/09/06 04:17:02.724 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138454] [size=100662602] [region_id=7249928]
[2025/09/06 04:17:02.994 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:17:02.994 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:17:04.631 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=187299442] [region_id=1014895]
[2025/09/06 04:17:04.748 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=103951] [size=74832138] [region_id=1014895]
[2025/09/06 04:17:07.994 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:17:07.994 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:17:09.177 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=199961980] [region_id=4667012]
[2025/09/06 04:17:09.365 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=139363] [size=98754630] [region_id=4667012]
[2025/09/06 04:17:13.494 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:17:13.494 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:17:14.728 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986810] [region_id=7218862]
[2025/09/06 04:17:14.901 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129275] [size=100663070] [region_id=7218862]
[2025/09/06 04:17:18.494 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:17:18.494 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:17:23.994 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:17:23.994 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:17:27.461 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7219863_197_6677891] [peer_id=7443473] [region_id=7219863]
[2025/09/06 04:17:27.461 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7244910_198_3864240] [peer_id=7444946] [region_id=7244910]
[2025/09/06 04:17:27.461 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7418340_16_254827] [peer_id=7444964] [region_id=7418340]
[2025/09/06 04:17:27.461 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:17:27.467 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:17:27.472 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:17:29.494 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:17:29.494 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:17:34.994 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:17:34.994 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:17:39.994 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:17:39.994 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:17:45.494 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:17:45.494 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:17:49.716 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:17:49.716 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:17:50.494 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:17:50.494 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]
[2025/09/06 04:17:55.494 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444996 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7217841]
[2025/09/06 04:17:55.494 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444511] [region_id=7217841]

可疑点，刷盘慢？

[2025/09/06 04:16:09.653 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138610, total size 100662989. raw_size_per_file 104857600, total takes 992.444367ms"]

其他节点出现过2.4秒的情况

[jwongz]

第2个可疑点是106:20181（store-8），不停的尝试transfer-leader，日志如下：

[2025/09/06 04:16:03.399 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201323774] [region_id=7219863]
[2025/09/06 04:16:03.539 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129457] [size=100662676] [region_id=7219863]
[2025/09/06 04:16:07.330 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=223713027] [region_id=7223935]
[2025/09/06 04:16:07.476 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128165] [size=100663103] [region_id=7223935]
[2025/09/06 04:16:07.920 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=310387237] [region_id=7219068]
[2025/09/06 04:16:08.103 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129057] [size=100662628] [region_id=7219068]
[2025/09/06 04:16:08.234 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7443469] [request_index=0] [peer_id=7439234] [region_id=7292992]
[2025/09/06 04:16:08.408 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7442652] [request_index=0] [peer_id=7442274] [region_id=7219068]
[2025/09/06 04:16:09.745 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:09.773 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data1/tikv-data/tikv-20161/snap/rev_7217841_289_9197254_(default|lock|write).sst] [snap_key=7217841_289_9197254]
[2025/09/06 04:16:09.785 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=39.781277ms] [snap_key=7217841_289_9197254]
[2025/09/06 04:16:09.785 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:09.791 +08:00] [INFO] [raft_log.rs:627] ["log [committed=9196793, persisted=9196793, applied=9196793, unstable.offset=9196794, unstable.entries.len()=0] starts to restore snapshot [index: 9197254, term: 289]"] [snapshot_term=289] [snapshot_index=9197254] [log="committed=9196793, persisted=9196793, applied=9196793, unstable.offset=9196794, unstable.entries.len()=0"] [raft_id=7444368] [region_id=7217841]
[2025/09/06 04:16:09.791 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444368, 7444511, 7444641} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444368] [region_id=7217841]
[2025/09/06 04:16:09.791 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=289] [snapshot_index=9197254] [last_term=289] [last_index=9197254] [commit=9197254] [raft_id=7444368] [region_id=7217841]
[2025/09/06 04:16:09.791 +08:00] [INFO] [raft.rs:2511] ["[commit: 9197254, term: 289] restored snapshot [index: 9197254, term: 289]"] [snapshot_term=289] [snapshot_index=9197254] [commit=9197254] [term=289] [raft_id=7444368] [region_id=7217841]
[2025/09/06 04:16:09.791 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444368] [region_id=7217841]
[2025/09/06 04:16:09.791 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=5.929µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7217841]
[2025/09/06 04:16:09.791 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 9197254 commit_index: 9196793 commit_term: 289 truncated_state { index: 9197254 term: 289 }"] [region="id: 7217841 start_key: 6F73735F6F62666 region_epoch { conf_ver: 81647 version: 93 } peers { id: 7444368 store_id: 8 } peers { id: 7444511 store_id: 3 } peers { id: 7444641 store_id: 2 }"] [peer_id=7444368] [region_id=7217841]
[2025/09/06 04:16:09.797 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7217841]
[2025/09/06 04:16:09.797 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7217841 start_key: 6F73735352E706466 region_epoch { conf_ver: 81647 version: 93 } peers { id: 7444368 store_id: 8 } peers { id: 7444511 store_id: 3 } peers { id: 7444641 store_id: 2 }"] [peer_id=7444368] [region_id=7217841]
[2025/09/06 04:16:09.797 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444368] [region_id=7217841]
[2025/09/06 04:16:09.797 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7217841 start_key: 6F73735F6F626A506466 region_epoch { conf_ver: 81647 version: 93 } peers { id: 7444368 store_id: 8 } peers { id: 7444511 store_id: 3 } peers { id: 7444641 store_id: 2 }"] [prev_region="id: 7217841 start_key: 6F73735F6F352E706466 region_epoch { conf_ver: 81647 version: 93 } peers { id: 7444368 store_id: 8 } peers { id: 7444511 store_id: 3 } peers { id: 7444641 store_id: 2 }"] [peer_id=7444368] [region_id=7217841]
[2025/09/06 04:16:09.800 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=289] [peer_id=7444368] [region_id=7217841]
[2025/09/06 04:16:09.912 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138255, total size 100662890. raw_size_per_file 104857600, total takes 1.675410999s"]
[2025/09/06 04:16:09.934 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662890] [key_count=138255] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7292992_98_2562621_(default|lock|write).sst] [region_id=7292992]
[2025/09/06 04:16:09.934 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7292992_98_2562621_(default|lock|write).sst] [region_id=7292992]
[2025/09/06 04:16:09.938 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7292992_98_2562621_(default|lock|write).sst] [region_id=7292992]
[2025/09/06 04:16:09.942 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=1.705274918s] [size=7050791] [key_count=138255] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7292992_98_2562621_(default|lock|write).sst] [region_id=7292992]
[2025/09/06 04:16:09.961 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:10.041 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=77.351035ms] [size=7050791] [snap_key=7292992_98_2562621] [region_id=7292992]
[2025/09/06 04:16:10.041 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7443469 store_id: 1"] [peer_id=7439234] [region_id=7292992]
[2025/09/06 04:16:10.134 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:10.221 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=85.328026ms] [region_id=7217841]
[2025/09/06 04:16:10.638 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 129057, total size 100662628. raw_size_per_file 104857600, total takes 2.229079447s"]
[2025/09/06 04:16:10.674 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662628] [key_count=129057] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219068_175_6951478_(default|lock|write).sst] [region_id=7219068]
[2025/09/06 04:16:10.674 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219068_175_6951478_(default|lock|write).sst] [region_id=7219068]
[2025/09/06 04:16:10.674 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219068_175_6951478_(default|lock|write).sst] [region_id=7219068]
[2025/09/06 04:16:10.679 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=2.269610904s] [size=10277602] [key_count=129057] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219068_175_6951478_(default|lock|write).sst] [region_id=7219068]
[2025/09/06 04:16:10.739 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:11.047 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7444787] [request_index=0] [peer_id=7443766] [region_id=7261583]
[2025/09/06 04:16:11.299 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=556.349279ms] [size=10277602] [snap_key=7219068_175_6951478] [region_id=7219068]
[2025/09/06 04:16:11.303 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7442652 store_id: 1"] [peer_id=7442274] [region_id=7219068]
[2025/09/06 04:16:11.303 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7442652 store_id: 1"] [peer_id=7442274] [region_id=7219068]
[2025/09/06 04:16:11.468 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=329517129] [region_id=7271280]
[2025/09/06 04:16:11.634 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138445] [size=100662619] [region_id=7271280]
[2025/09/06 04:16:11.972 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138431, total size 100663130. raw_size_per_file 104857600, total takes 925.153875ms"]
[2025/09/06 04:16:12.007 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100663130] [key_count=138431] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7261583_172_2997489_(default|lock|write).sst] [region_id=7261583]
[2025/09/06 04:16:12.007 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7261583_172_2997489_(default|lock|write).sst] [region_id=7261583]
[2025/09/06 04:16:12.007 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7261583_172_2997489_(default|lock|write).sst] [region_id=7261583]
[2025/09/06 04:16:12.011 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=964.180963ms] [size=7270656] [key_count=138431] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7261583_172_2997489_(default|lock|write).sst] [region_id=7261583]
[2025/09/06 04:16:12.457 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:12.598 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=139.668066ms] [size=7270656] [snap_key=7261583_172_2997489] [region_id=7261583]
[2025/09/06 04:16:12.599 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7444787 store_id: 1"] [peer_id=7443766] [region_id=7261583]
[2025/09/06 04:16:12.728 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7443618] [request_index=0] [peer_id=7440644] [region_id=7217734]
[2025/09/06 04:16:12.924 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7444530] [request_index=0] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:12.957 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=194926098] [region_id=7257488]
[2025/09/06 04:16:13.077 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138366] [size=100662783] [region_id=7257488]
[2025/09/06 04:16:13.108 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7443538] [request_index=0] [peer_id=7434003] [region_id=7219733]
[2025/09/06 04:16:13.248 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:13.271 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:13.346 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=202725637] [region_id=7261583]
[2025/09/06 04:16:13.372 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data1/tikv-data/tikv-20161/snap/rev_7221523_146_6050108_(default|lock|write).sst] [snap_key=7221523_146_6050108]
[2025/09/06 04:16:13.379 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=131.53466ms] [snap_key=7221523_146_6050108]
[2025/09/06 04:16:13.379 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:13.387 +08:00] [INFO] [raft_log.rs:627] ["log [committed=6049617, persisted=6049617, applied=6049617, unstable.offset=6049618, unstable.entries.len()=0] starts to restore snapshot [index: 6050108, term: 146]"] [snapshot_term=146] [snapshot_index=6050108] [log="committed=6049617, persisted=6049617, applied=6049617, unstable.offset=6049618, unstable.entries.len()=0"] [raft_id=7444440] [region_id=7221523]
[2025/09/06 04:16:13.387 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444152, 7444440, 7440404} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444440] [region_id=7221523]
[2025/09/06 04:16:13.387 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=146] [snapshot_index=6050108] [last_term=146] [last_index=6050108] [commit=6050108] [raft_id=7444440] [region_id=7221523]
[2025/09/06 04:16:13.387 +08:00] [INFO] [raft.rs:2511] ["[commit: 6050108, term: 146] restored snapshot [index: 6050108, term: 146]"] [snapshot_term=146] [snapshot_index=6050108] [commit=6050108] [term=146] [raft_id=7444440] [region_id=7221523]
[2025/09/06 04:16:13.387 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444440] [region_id=7221523]
[2025/09/06 04:16:13.387 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=7.197µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7221523]
[2025/09/06 04:16:13.387 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 6050108 commit_index: 6049105 commit_term: 146 truncated_state { index: 6050108 term: 146 }"] [region="id: 7221523 start_key: 6F73735466 region_epoch { conf_ver: 67763 version: 70 } peers { id: 7440404 store_id: 2 } peers { id: 7444152 store_id: 3 } peers { id: 7444440 store_id: 8 }"] [peer_id=7444440] [region_id=7221523]
[2025/09/06 04:16:13.389 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=146] [peer_id=7444440] [region_id=7221523]
[2025/09/06 04:16:13.389 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7221523 start_key: 6F73735F6F62606466 region_epoch { conf_ver: 67763 version: 70 } peers { id: 7440404 store_id: 2 } peers { id: 7444152 store_id: 3 } peers { id: 7444440 store_id: 8 }"] [peer_id=7444440] [region_id=7221523]
[2025/09/06 04:16:13.389 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7221523]
[2025/09/06 04:16:13.389 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7221523 start_key: 6F73706466 region_epoch { conf_ver: 67763 version: 70 } peers { id: 7440404 store_id: 2 } peers { id: 7444152 store_id: 3 } peers { id: 7444440 store_id: 8 }"] [prev_region="id: 7221523 start_key: 6F73735466 region_epoch { conf_ver: 67763 version: 70 } peers { id: 7440404 store_id: 2 } peers { id: 7444152 store_id: 3 } peers { id: 7444440 store_id: 8 }"] [peer_id=7444440] [region_id=7221523]
[2025/09/06 04:16:13.393 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444440] [region_id=7221523]
[2025/09/06 04:16:13.457 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138431] [size=100663130] [region_id=7261583]
[2025/09/06 04:16:13.496 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data1/tikv-data/tikv-20161/snap/rev_7347689_50_1335024_(default|lock|write).sst] [snap_key=7347689_50_1335024]
[2025/09/06 04:16:13.499 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=228.235957ms] [snap_key=7347689_50_1335024]
[2025/09/06 04:16:13.499 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:13.504 +08:00] [INFO] [raft_log.rs:627] ["log [committed=1334508, persisted=1334508, applied=1334508, unstable.offset=1334509, unstable.entries.len()=0] starts to restore snapshot [index: 1335024, term: 50]"] [snapshot_term=50] [snapshot_index=1335024] [log="committed=1334508, persisted=1334508, applied=1334508, unstable.offset=1334509, unstable.entries.len()=0"] [raft_id=7444131] [region_id=7347689]
[2025/09/06 04:16:13.504 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7443737, 7443801, 7444131} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444131] [region_id=7347689]
[2025/09/06 04:16:13.504 +08:00] [INFO] [raft.rs:2511] ["[commit: 1335024, term: 50] restored snapshot [index: 1335024, term: 50]"] [snapshot_term=50] [snapshot_index=1335024] [commit=1335024] [term=50] [raft_id=7444131] [region_id=7347689]
[2025/09/06 04:16:13.504 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444131] [region_id=7347689]
[2025/09/06 04:16:13.504 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=6.106µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7347689]
[2025/09/06 04:16:13.504 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 1335024 commit_index: 1334508 commit_term: 50 truncated_state { index: 1335024 term: 50 }"] [region="id: 7347689 start_key: 6F73735F6F706466 region_epoch { conf_ver: 65489 version: 101 } peers { id: 7443737 store_id: 2 } peers { id: 7443801 store_id: 14 } peers { id: 7444131 store_id: 8 }"] [peer_id=7444131] [region_id=7347689]
[2025/09/06 04:16:13.506 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=50] [peer_id=7444131] [region_id=7347689]
[2025/09/06 04:16:13.506 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7347689 start_key: 6F73735F83033312E706466 region_epoch { conf_ver: 65489 version: 101 } peers { id: 7443737 store_id: 2 } peers { id: 7443801 store_id: 14 } peers { id: 7444131 store_id: 8 }"] [peer_id=7444131] [region_id=7347689]
[2025/09/06 04:16:13.506 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7347689]
[2025/09/06 04:16:13.506 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7347689 start_key: 6F73735F6F706466 region_epoch { conf_ver: 65489 version: 101 } peers { id: 7443737 store_id: 2 } peers { id: 7443801 store_id: 14 } peers { id: 7444131 store_id: 8 }"] [prev_region="id: 7347689 start_key: 6F73735F6F2E706466 region_epoch { conf_ver: 65489 version: 101 } peers { id: 7443737 store_id: 2 } peers { id: 7443801 store_id: 14 } peers { id: 7444131 store_id: 8 }"] [peer_id=7444131] [region_id=7347689]
[2025/09/06 04:16:13.727 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:13.754 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data1/tikv-data/tikv-20161/snap/rev_7375283_35_832099_(default|lock|write).sst] [snap_key=7375283_35_832099]
[2025/09/06 04:16:13.758 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=31.005543ms] [snap_key=7375283_35_832099]
[2025/09/06 04:16:13.758 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:13.760 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=412864602] [region_id=7220974]
[2025/09/06 04:16:13.760 +08:00] [INFO] [raft_log.rs:627] ["log [committed=831562, persisted=831562, applied=831562, unstable.offset=831563, unstable.entries.len()=0] starts to restore snapshot [index: 832099, term: 35]"] [snapshot_term=35] [snapshot_index=832099] [log="committed=831562, persisted=831562, applied=831562, unstable.offset=831563, unstable.entries.len()=0"] [raft_id=7439159] [region_id=7375283]
[2025/09/06 04:16:13.761 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7441280, 7439159, 7439500} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7439159] [region_id=7375283]
[2025/09/06 04:16:13.761 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=35] [snapshot_index=832099] [last_term=35] [last_index=832099] [commit=832099] [raft_id=7439159] [region_id=7375283]
[2025/09/06 04:16:13.761 +08:00] [INFO] [raft.rs:2511] ["[commit: 832099, term: 35] restored snapshot [index: 832099, term: 35]"] [snapshot_term=35] [snapshot_index=832099] [commit=832099] [term=35] [raft_id=7439159] [region_id=7375283]
[2025/09/06 04:16:13.761 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7439159] [region_id=7375283]
[2025/09/06 04:16:13.761 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=4.789µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7375283]
[2025/09/06 04:16:13.761 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 832099 commit_index: 831562 commit_term: 35 truncated_state { index: 832099 term: 35 }"] [region="id: 7375283 start_key: 6F737E706466 region_epoch { conf_ver: 65861 version: 108 } peers { id: 7439159 store_id: 8 } peers { id: 7439500 store_id: 1 } peers { id: 7441280 store_id: 3 }"] [peer_id=7439159] [region_id=7375283]
[2025/09/06 04:16:13.761 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=35] [peer_id=7439159] [region_id=7375283]
[2025/09/06 04:16:13.761 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7375283 start_key: 6F73706466 region_epoch { conf_ver: 65861 version: 108 } peers { id: 7439159 store_id: 8 } peers { id: 7439500 store_id: 1 } peers { id: 7441280 store_id: 3 }"] [peer_id=7439159] [region_id=7375283]
[2025/09/06 04:16:13.761 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7375283 start_key: 6F732E706466 region_epoch { conf_ver: 65861 version: 108 } peers { id: 7439159 store_id: 8 } peers { id: 7439500 store_id: 1 } peers { id: 7441280 store_id: 3 }"] [prev_region="id: 7375283 start_key: 6F7373E706466 region_epoch { conf_ver: 65861 version: 108 } peers { id: 7439159 store_id: 8 } peers { id: 7439500 store_id: 1 } peers { id: 7441280 store_id: 3 }"] [peer_id=7439159] [region_id=7375283]
[2025/09/06 04:16:13.794 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:13.847 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=50.457319ms] [region_id=7221523]
[2025/09/06 04:16:13.847 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444131] [region_id=7347689]
[2025/09/06 04:16:13.857 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7375283]
[2025/09/06 04:16:13.989 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:13.995 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128600] [size=100662579] [region_id=7220974]
[2025/09/06 04:16:14.040 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=48.507036ms] [region_id=7347689]
[2025/09/06 04:16:14.040 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7439159] [region_id=7375283]
[2025/09/06 04:16:14.225 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:14.314 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=86.860392ms] [region_id=7375283]
[2025/09/06 04:16:14.336 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:14.367 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data1/tikv-data/tikv-20161/snap/rev_7217630_273_9964736_(default|lock|write).sst] [snap_key=7217630_273_9964736]
[2025/09/06 04:16:14.387 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=51.292465ms] [snap_key=7217630_273_9964736]
[2025/09/06 04:16:14.388 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:14.393 +08:00] [INFO] [raft_log.rs:627] ["log [committed=9964391, persisted=9964391, applied=9964391, unstable.offset=9964392, unstable.entries.len()=0] starts to restore snapshot [index: 9964736, term: 273]"] [snapshot_term=273] [snapshot_index=9964736] [log="committed=9964391, persisted=9964391, applied=9964391, unstable.offset=9964392, unstable.entries.len()=0"] [raft_id=7436558] [region_id=7217630]
[2025/09/06 04:16:14.393 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7443904, 7436558, 7443446} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7436558] [region_id=7217630]
[2025/09/06 04:16:14.393 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=273] [snapshot_index=9964736] [last_term=273] [last_index=9964736] [commit=9964736] [raft_id=7436558] [region_id=7217630]
[2025/09/06 04:16:14.393 +08:00] [INFO] [raft.rs:2511] ["[commit: 9964736, term: 273] restored snapshot [index: 9964736, term: 273]"] [snapshot_term=273] [snapshot_index=9964736] [commit=9964736] [term=273] [raft_id=7436558] [region_id=7217630]
[2025/09/06 04:16:14.393 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7436558] [region_id=7217630]
[2025/09/06 04:16:14.393 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=5.268µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7217630]
[2025/09/06 04:16:14.393 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 9964736 commit_index: 9964391 commit_term: 273 truncated_state { index: 9964736 term: 273 }"] [region="id: 7217630 start_key: 6F7373731362E706466 region_epoch { conf_ver: 81263 version: 82 } peers { id: 7436558 store_id: 8 } peers { id: 7443446 store_id: 1 } peers { id: 7443904 store_id: 14 }"] [peer_id=7436558] [region_id=7217630]
[2025/09/06 04:16:14.396 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7217630]
[2025/09/06 04:16:14.396 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7217630 start_key: 6F737353731362E706466 region_epoch { conf_ver: 81263 version: 82 } peers { id: 7436558 store_id: 8 } peers { id: 7443446 store_id: 1 } peers { id: 7443904 store_id: 14 }"] [peer_id=7436558] [region_id=7217630]
[2025/09/06 04:16:14.396 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7217630 start_key: 6F7372E706466 region_epoch { conf_ver: 81263 version: 82 } peers { id: 7436558 store_id: 8 } peers { id: 7443446 store_id: 1 } peers { id: 7443904 store_id: 14 }"] [prev_region="id: 7217630 start_key: 6F737E706466 region_epoch { conf_ver: 81263 version: 82 } peers { id: 7436558 store_id: 8 } peers { id: 7443446 store_id: 1 } peers { id: 7443904 store_id: 14 }"] [peer_id=7436558] [region_id=7217630]
[2025/09/06 04:16:14.396 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=273] [peer_id=7436558] [region_id=7217630]
[2025/09/06 04:16:14.400 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7436558] [region_id=7217630]
[2025/09/06 04:16:14.441 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444927 store_id: 3 role: Learner } change_type: AddLearnerNode]"] [region_id=7297670]
[2025/09/06 04:16:14.441 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=Simple] [changes="[change_type: AddLearnerNode peer { id: 7444927 store_id: 3 role: Learner }]"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:14.442 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444927 store_id: 3 role: Learner } } }"] [index=2489525] [term=92] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:14.442 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65069 version: 85"] [kind=Simple] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:14.442 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7297670 start_key: 6F7373E706466 region_epoch { conf_ver: 65070 version: 85 } peers { id: 7438221 store_id: 14 } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 role: Learner }"] ["original region"="id: 7297670 start_key: 6F73735FE706466 region_epoch { conf_ver: 65069 version: 85 } peers { id: 7438221 store_id: 14 } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 }"] [changes="[change_type: AddLearnerNode peer { id: 7444927 store_id: 3 role: Learner }]"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:14.449 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7438221, 7441402, 7440507} }, outgoing: Configuration { voters: {} } }, learners: {7444927}, learners_next: {}, auto_leave: false }"] [raft_id=7440507] [region_id=7297670]
[2025/09/06 04:16:14.449 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7297670 start_key: 6F72E706466 region_epoch { conf_ver: 65070 version: 85 } peers { id: 7438221 store_id: 14 } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 role: Learner }"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:14.452 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7444927] [request_index=0] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:14.614 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:14.697 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=81.04056ms] [region_id=7217630]
[2025/09/06 04:16:14.852 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138397, total size 100662596. raw_size_per_file 104857600, total takes 1.921592627s"]
[2025/09/06 04:16:14.880 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662596] [key_count=138397] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7244910_198_3864238_(default|lock|write).sst] [region_id=7244910]
[2025/09/06 04:16:14.880 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7244910_198_3864238_(default|lock|write).sst] [region_id=7244910]
[2025/09/06 04:16:14.880 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7244910_198_3864238_(default|lock|write).sst] [region_id=7244910]
[2025/09/06 04:16:14.886 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=1.955580623s] [size=7243930] [key_count=138397] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7244910_198_3864238_(default|lock|write).sst] [region_id=7244910]
[2025/09/06 04:16:14.908 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:14.927 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:14.930 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7443483] [request_index=0] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:14.930 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7443228] [request_index=0] [peer_id=7429386] [region_id=7219580]
[2025/09/06 04:16:15.037 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data1/tikv-data/tikv-20161/snap/rev_7314410_77_2198650_(default|lock|write).sst] [snap_key=7314410_77_2198650]
[2025/09/06 04:16:15.044 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=135.638206ms] [snap_key=7314410_77_2198650]
[2025/09/06 04:16:15.044 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:15.049 +08:00] [INFO] [raft_log.rs:627] ["log [committed=2198275, persisted=2198275, applied=2198275, unstable.offset=2198276, unstable.entries.len()=0] starts to restore snapshot [index: 2198650, term: 77]"] [snapshot_term=77] [snapshot_index=2198650] [log="committed=2198275, persisted=2198275, applied=2198275, unstable.offset=2198276, unstable.entries.len()=0"] [raft_id=7444723] [region_id=7314410]
[2025/09/06 04:16:15.049 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444723, 7444082, 7444667} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444723] [region_id=7314410]
[2025/09/06 04:16:15.049 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=77] [snapshot_index=2198650] [last_term=77] [last_index=2198650] [commit=2198650] [raft_id=7444723] [region_id=7314410]
[2025/09/06 04:16:15.049 +08:00] [INFO] [raft.rs:2511] ["[commit: 2198650, term: 77] restored snapshot [index: 2198650, term: 77]"] [snapshot_term=77] [snapshot_index=2198650] [commit=2198650] [term=77] [raft_id=7444723] [region_id=7314410]
[2025/09/06 04:16:15.049 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:15.049 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=5.359µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7314410]
[2025/09/06 04:16:15.049 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 2198650 commit_index: 2198275 commit_term: 77 truncated_state { index: 2198650 term: 77 }"] [region="id: 7314410 start_key: 6F73735F6F6262353233392E706466 region_epoch { conf_ver: 65359 version: 89 } peers { id: 7444082 store_id: 14 } peers { id: 7444667 store_id: 2 } peers { id: 7444723 store_id: 8 }"] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:15.056 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7314410]
[2025/09/06 04:16:15.056 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7314410 start_key: 6F737392E706466 region_epoch { conf_ver: 65359 version: 89 } peers { id: 7444082 store_id: 14 } peers { id: 7444667 store_id: 2 } peers { id: 7444723 store_id: 8 }"] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:15.056 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7314410 start_key: 6F7373E706466 region_epoch { conf_ver: 65359 version: 89 } peers { id: 7444082 store_id: 14 } peers { id: 7444667 store_id: 2 } peers { id: 7444723 store_id: 8 }"] [prev_region="id: 7314410 start_key: 6F73736466 region_epoch { conf_ver: 65359 version: 89 } peers { id: 7444082 store_id: 14 } peers { id: 7444667 store_id: 2 } peers { id: 7444723 store_id: 8 }"] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:15.057 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=77] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:15.079 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=149.889176ms] [size=7243930] [snap_key=7244910_198_3864238] [region_id=7244910]
[2025/09/06 04:16:15.082 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7444530 store_id: 1"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:15.396 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:15.423 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data1/tikv-data/tikv-20161/snap/rev_7333984_73_1766188_(default|lock|write).sst] [snap_key=7333984_73_1766188]
[2025/09/06 04:16:15.442 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=45.975649ms] [snap_key=7333984_73_1766188]
[2025/09/06 04:16:15.442 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:15.447 +08:00] [INFO] [raft_log.rs:627] ["log [committed=1765726, persisted=1765726, applied=1765726, unstable.offset=1765727, unstable.entries.len()=0] starts to restore snapshot [index: 1766188, term: 73]"] [snapshot_term=73] [snapshot_index=1766188] [log="committed=1765726, persisted=1765726, applied=1765726, unstable.offset=1765727, unstable.entries.len()=0"] [raft_id=7442520] [region_id=7333984]
[2025/09/06 04:16:15.447 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7442520, 7442842, 7441687} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7442520] [region_id=7333984]
[2025/09/06 04:16:15.447 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=73] [snapshot_index=1766188] [last_term=73] [last_index=1766188] [commit=1766188] [raft_id=7442520] [region_id=7333984]
[2025/09/06 04:16:15.447 +08:00] [INFO] [raft.rs:2511] ["[commit: 1766188, term: 73] restored snapshot [index: 1766188, term: 73]"] [snapshot_term=73] [snapshot_index=1766188] [commit=1766188] [term=73] [raft_id=7442520] [region_id=7333984]
[2025/09/06 04:16:15.447 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7442520] [region_id=7333984]
[2025/09/06 04:16:15.447 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=5.385µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7333984]
[2025/09/06 04:16:15.447 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 1766188 commit_index: 1765726 commit_term: 73 truncated_state { index: 1766188 term: 73 }"] [region="id: 7333984 start_key: 6F73736466 region_epoch { conf_ver: 65459 version: 95 } peers { id: 7441687 store_id: 2 } peers { id: 7442520 store_id: 8 } peers { id: 7442842 store_id: 3 }"] [peer_id=7442520] [region_id=7333984]
[2025/09/06 04:16:15.455 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7333984]
[2025/09/06 04:16:15.455 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7333984 start_key: 6F7302E706466 region_epoch { conf_ver: 65459 version: 95 } peers { id: 7441687 store_id: 2 } peers { id: 7442520 store_id: 8 } peers { id: 7442842 store_id: 3 }"] [peer_id=7442520] [region_id=7333984]
[2025/09/06 04:16:15.455 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=73] [peer_id=7442520] [region_id=7333984]
[2025/09/06 04:16:15.455 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7333984 start_key: 6F7335302E706466 region_epoch { conf_ver: 65459 version: 95 } peers { id: 7441687 store_id: 2 } peers { id: 7442520 store_id: 8 } peers { id: 7442842 store_id: 3 }"] [prev_region="id: 7333984 start_key: 6F7373502E706466 region_epoch { conf_ver: 65459 version: 95 } peers { id: 7441687 store_id: 2 } peers { id: 7442520 store_id: 8 } peers { id: 7442842 store_id: 3 }"] [peer_id=7442520] [region_id=7333984]
[2025/09/06 04:16:15.472 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 126860, total size 100663021. raw_size_per_file 104857600, total takes 2.743941358s"]
[2025/09/06 04:16:15.499 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100663021] [key_count=126860] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7217734_168_8698088_(default|lock|write).sst] [region_id=7217734]
[2025/09/06 04:16:15.500 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7217734_168_8698088_(default|lock|write).sst] [region_id=7217734]
[2025/09/06 04:16:15.500 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7217734_168_8698088_(default|lock|write).sst] [region_id=7217734]
[2025/09/06 04:16:15.503 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=2.774310099s] [size=7039940] [key_count=126860] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7217734_168_8698088_(default|lock|write).sst] [region_id=7217734]
[2025/09/06 04:16:15.504 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:15.603 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7444754] [request_index=0] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:15.658 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=153.215579ms] [size=7039940] [snap_key=7217734_168_8698088] [region_id=7217734]
[2025/09/06 04:16:15.658 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7443618 store_id: 2"] [peer_id=7440644] [region_id=7217734]
[2025/09/06 04:16:16.661 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=400845634] [region_id=7217641]
[2025/09/06 04:16:16.877 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=133701] [size=100663011] [region_id=7217641]
[2025/09/06 04:16:16.886 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138149, total size 100663232. raw_size_per_file 104857600, total takes 1.382445718s"]
[2025/09/06 04:16:16.902 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100663232] [key_count=138149] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7297670_92_2489525_(default|lock|write).sst] [region_id=7297670]
[2025/09/06 04:16:16.902 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7297670_92_2489525_(default|lock|write).sst] [region_id=7297670]
[2025/09/06 04:16:16.902 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7297670_92_2489525_(default|lock|write).sst] [region_id=7297670]
[2025/09/06 04:16:16.904 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=1.40125761s] [size=6927041] [key_count=138149] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7297670_92_2489525_(default|lock|write).sst] [region_id=7297670]
[2025/09/06 04:16:16.905 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:16.970 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=63.530754ms] [size=6927041] [snap_key=7297670_92_2489525] [region_id=7297670]
[2025/09/06 04:16:16.970 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7444927 store_id: 3 role: Learner"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:17.049 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 129181, total size 100662888. raw_size_per_file 104857600, total takes 2.162984939s"]
[2025/09/06 04:16:17.085 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662888] [key_count=129181] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219733_152_6707823_(default|lock|write).sst] [region_id=7219733]
[2025/09/06 04:16:17.085 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219733_152_6707823_(default|lock|write).sst] [region_id=7219733]
[2025/09/06 04:16:17.085 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219733_152_6707823_(default|lock|write).sst] [region_id=7219733]
[2025/09/06 04:16:17.088 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=2.202090647s] [size=10276120] [key_count=129181] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219733_152_6707823_(default|lock|write).sst] [region_id=7219733]
[2025/09/06 04:16:17.092 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:17.244 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=148.974573ms] [size=10276120] [snap_key=7219733_152_6707823] [region_id=7219733]
[2025/09/06 04:16:17.244 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7443538 store_id: 1"] [peer_id=7434003] [region_id=7219733]
[2025/09/06 04:16:17.317 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:17.333 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=297803328] [region_id=7223935]
[2025/09/06 04:16:17.500 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:17.505 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128165] [size=100663103] [region_id=7223935]
[2025/09/06 04:16:17.546 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=44.484084ms] [region_id=7314410]
[2025/09/06 04:16:18.176 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=402649520] [region_id=7220062]
[2025/09/06 04:16:18.400 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128967] [size=100662557] [region_id=7220062]
[2025/09/06 04:16:18.547 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7442520] [region_id=7333984]
[2025/09/06 04:16:18.662 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138366, total size 100662783. raw_size_per_file 104857600, total takes 1.75728496s"]
[2025/09/06 04:16:18.687 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662783] [key_count=138366] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7257488_175_3069662_(default|lock|write).sst] [region_id=7257488]
[2025/09/06 04:16:18.687 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7257488_175_3069662_(default|lock|write).sst] [region_id=7257488]
[2025/09/06 04:16:18.687 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7257488_175_3069662_(default|lock|write).sst] [region_id=7257488]
[2025/09/06 04:16:18.688 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=1.783215787s] [size=7274247] [key_count=138366] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7257488_175_3069662_(default|lock|write).sst] [region_id=7257488]
[2025/09/06 04:16:18.688 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:18.753 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:18.766 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=75.519752ms] [size=7274247] [snap_key=7257488_175_3069662] [region_id=7257488]
[2025/09/06 04:16:18.766 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:18.793 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=37.927344ms] [region_id=7333984]
[2025/09/06 04:16:18.797 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201325796] [region_id=7292992]
[2025/09/06 04:16:18.897 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138255] [size=100662890] [region_id=7292992]
[2025/09/06 04:16:19.758 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 129648, total size 100663070. raw_size_per_file 104857600, total takes 2.669285643s"]
[2025/09/06 04:16:19.791 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100663070] [key_count=129648] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219580_226_6808664_(default|lock|write).sst] [region_id=7219580]
[2025/09/06 04:16:19.791 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219580_226_6808664_(default|lock|write).sst] [region_id=7219580]
[2025/09/06 04:16:19.792 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219580_226_6808664_(default|lock|write).sst] [region_id=7219580]
[2025/09/06 04:16:19.795 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=2.706285842s] [size=10118449] [key_count=129648] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219580_226_6808664_(default|lock|write).sst] [region_id=7219580]
[2025/09/06 04:16:19.795 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:19.874 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=75.435039ms] [size=10118449] [snap_key=7219580_226_6808664] [region_id=7219580]
[2025/09/06 04:16:19.874 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7443228 store_id: 1"] [peer_id=7429386] [region_id=7219580]
[2025/09/06 04:16:20.101 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138445, total size 100662619. raw_size_per_file 104857600, total takes 1.413208659s"]
[2025/09/06 04:16:20.120 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662619] [key_count=138445] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7271280_149_2853651_(default|lock|write).sst] [region_id=7271280]
[2025/09/06 04:16:20.120 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7271280_149_2853651_(default|lock|write).sst] [region_id=7271280]
[2025/09/06 04:16:20.120 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7271280_149_2853651_(default|lock|write).sst] [region_id=7271280]
[2025/09/06 04:16:20.124 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=1.43561184s] [size=7215131] [key_count=138445] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7271280_149_2853651_(default|lock|write).sst] [region_id=7271280]
[2025/09/06 04:16:20.124 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:20.187 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=62.442683ms] [size=7215131] [snap_key=7271280_149_2853651] [region_id=7271280]
[2025/09/06 04:16:20.187 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:20.623 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444927 store_id: 3 }, peer { id: 7438221 store_id: 14 role: Learner } change_type: AddLearnerNode]"] [region_id=7297670]
[2025/09/06 04:16:20.623 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=EnterJoint] [changes="[peer { id: 7444927 store_id: 3 }, change_type: AddLearnerNode peer { id: 7438221 store_id: 14 role: Learner }]"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.624 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444927 store_id: 3 } } changes { change_type: AddLearnerNode peer { id: 7438221 store_id: 14 role: Learner } } }"] [index=2489526] [term=92] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.624 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65070 version: 85"] [kind=EnterJoint] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.624 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7297670 start_key: 6F737E706466 region_epoch { conf_ver: 65072 version: 85 } peers { id: 7438221 store_id: 14 role: DemotingVoter } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 role: IncomingVoter }"] ["original region"="id: 7297670 start_key: 6F73735F6F6373939362E706466 region_epoch { conf_ver: 65070 version: 85 } peers { id: 7438221 store_id: 14 } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 role: Learner }"] [changes="[peer { id: 7444927 store_id: 3 }, change_type: AddLearnerNode peer { id: 7438221 store_id: 14 role: Learner }]"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.628 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7441402, 7444927, 7440507} }, outgoing: Configuration { voters: {7438221, 7441402, 7440507} } }, learners: {}, learners_next: {7438221}, auto_leave: false }"] [raft_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.628 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7297670 start_key: 6F7373362E706466 region_epoch { conf_ver: 65072 version: 85 } peers { id: 7438221 store_id: 14 role: DemotingVoter } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 role: IncomingVoter }"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.629 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7297670]
[2025/09/06 04:16:20.629 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7297670]
[2025/09/06 04:16:20.629 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=LeaveJoint] [changes="[]"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.629 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7297670]
[2025/09/06 04:16:20.629 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.629 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.629 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=2489527] [term=92] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.629 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65072 version: 85"] [kind=LeaveJoint] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.629 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7297670 start_key: 6F737373939362E706466 region_epoch { conf_ver: 65074 version: 85 } peers { id: 7438221 store_id: 14 role: Learner } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 }"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.633 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7441402, 7444927, 7440507} }, outgoing: Configuration { voters: {} } }, learners: {7438221}, learners_next: {}, auto_leave: false }"] [raft_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.633 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7297670 start_key: 6F73735F6F626A5F662E706466 region_epoch { conf_ver: 65074 version: 85 } peers { id: 7438221 store_id: 14 role: Learner } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 }"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.634 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7438221 store_id: 14 role: Learner } change_type: RemoveNode]"] [region_id=7297670]
[2025/09/06 04:16:20.634 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=Simple] [changes="[change_type: RemoveNode peer { id: 7438221 store_id: 14 role: Learner }]"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.635 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7438221 store_id: 14 role: Learner } } }"] [index=2489528] [term=92] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.635 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65074 version: 85"] [kind=Simple] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.635 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7297670 start_key: 6F73739362E706466 region_epoch { conf_ver: 65075 version: 85 } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 }"] ["original region"="id: 7297670 start_key: 6F7373706466 region_epoch { conf_ver: 65074 version: 85 } peers { id: 7438221 store_id: 14 role: Learner } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 }"] [changes="[change_type: RemoveNode peer { id: 7438221 store_id: 14 role: Learner }]"] [peer_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.637 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7441402, 7444927, 7440507} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7440507] [region_id=7297670]
[2025/09/06 04:16:20.637 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7297670 start_key: 6F7373E706466 region_epoch { conf_ver: 65075 version: 85 } peers { id: 7440507 store_id: 8 } peers { id: 7441402 store_id: 2 } peers { id: 7444927 store_id: 3 }"] [peer_id=7440507] [region_id=7297670]

[jwongz]

接

[2025/09/06 04:16:21.079 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:21.114 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data1/tikv-data/tikv-20161/snap/rev_7220281_161_6336144_(default|lock|write).sst] [snap_key=7220281_161_6336144]
[2025/09/06 04:16:21.123 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=43.503677ms] [snap_key=7220281_161_6336144]
[2025/09/06 04:16:21.123 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:21.130 +08:00] [INFO] [raft_log.rs:627] ["log [committed=6335691, persisted=6335691, applied=6335691, unstable.offset=6335692, unstable.entries.len()=0] starts to restore snapshot [index: 6336144, term: 161]"] [snapshot_term=161] [snapshot_index=6336144] [log="committed=6335691, persisted=6335691, applied=6335691, unstable.offset=6335692, unstable.entries.len()=0"] [raft_id=7444257] [region_id=7220281]
[2025/09/06 04:16:21.130 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7442620, 7444257, 7440899} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444257] [region_id=7220281]
[2025/09/06 04:16:21.130 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=161] [snapshot_index=6336144] [last_term=161] [last_index=6336144] [commit=6336144] [raft_id=7444257] [region_id=7220281]
[2025/09/06 04:16:21.130 +08:00] [INFO] [raft.rs:2511] ["[commit: 6336144, term: 161] restored snapshot [index: 6336144, term: 161]"] [snapshot_term=161] [snapshot_index=6336144] [commit=6336144] [term=161] [raft_id=7444257] [region_id=7220281]
[2025/09/06 04:16:21.130 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444257] [region_id=7220281]
[2025/09/06 04:16:21.130 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=6.054µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7220281]
[2025/09/06 04:16:21.130 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 6336144 commit_index: 6335691 commit_term: 161 truncated_state { index: 6336144 term: 161 }"] [region="id: 7220281 start_key: 6F73735F6F626A5F632D72776D735F70726F642F626D737765622F32303233303932332F2545352542452541452545342542462541312545352539422542452545372538392538375F3230323330393233313035373138325F313639353433383238333538362E6A7067 end_key: 6F73735F6F626A5F632D72776D735F70726F642F626D737765622F32303233313031362F313639373432313732393630385F30336336663262312D366263382D313165652D393938612D3338393061356264343066632E706466 region_epoch { conf_ver: 67787 version: 66 } peers { id: 7440899 store_id: 2 } peers { id: 7442620 store_id: 3 } peers { id: 7444257 store_id: 8 }"] [peer_id=7444257] [region_id=7220281]
[2025/09/06 04:16:21.138 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=161] [peer_id=7444257] [region_id=7220281]
[2025/09/06 04:16:21.138 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7220281 start_key: 6F73735F6F666 region_epoch { conf_ver: 67787 version: 66 } peers { id: 7440899 store_id: 2 } peers { id: 7442620 store_id: 3 } peers { id: 7444257 store_id: 8 }"] [peer_id=7444257] [region_id=7220281]
[2025/09/06 04:16:21.138 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7220281]
[2025/09/06 04:16:21.138 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444257] [region_id=7220281]
[2025/09/06 04:16:21.138 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7220281 start_key: 6F73735F66 region_epoch { conf_ver: 67787 version: 66 } peers { id: 7440899 store_id: 2 } peers { id: 7442620 store_id: 3 } peers { id: 7444257 store_id: 8 }"] [prev_region="id: 7220281 start_key: 6F73706466 region_epoch { conf_ver: 67787 version: 66 } peers { id: 7440899 store_id: 2 } peers { id: 7442620 store_id: 3 } peers { id: 7444257 store_id: 8 }"] [peer_id=7444257] [region_id=7220281]
[2025/09/06 04:16:21.434 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:21.471 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=233039806] [region_id=7271280]
[2025/09/06 04:16:21.487 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=49.810441ms] [region_id=7220281]
[2025/09/06 04:16:21.578 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138445] [size=100662619] [region_id=7271280]
[2025/09/06 04:16:22.958 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=203098975] [region_id=7257488]
[2025/09/06 04:16:23.059 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138366] [size=100662783] [region_id=7257488]
[2025/09/06 04:16:23.400 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301985661] [region_id=7219863]
[2025/09/06 04:16:23.572 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129457] [size=100662676] [region_id=7219863]
[2025/09/06 04:16:25.504 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=306176560] [region_id=7219733]
[2025/09/06 04:16:25.685 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129181] [size=100662888] [region_id=7219733]
[2025/09/06 04:16:26.662 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986510] [region_id=7217641]
[2025/09/06 04:16:26.821 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=133701] [size=100663011] [region_id=7217641]
[2025/09/06 04:16:26.844 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444938 store_id: 9 role: Learner } } }"] [index=3795560] [term=213] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:26.844 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65801 version: 67"] [kind=Simple] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:26.844 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7245948 start_key: 6F737352393734382E706466 region_epoch { conf_ver: 65802 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 } peers { id: 7444890 store_id: 3 } peers { id: 7444938 store_id: 9 role: Learner }"] ["original region"="id: 7245948 start_key: 6F737393734382E706466 region_epoch { conf_ver: 65801 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 } peers { id: 7444890 store_id: 3 }"] [changes="[change_type: AddLearnerNode peer { id: 7444938 store_id: 9 role: Learner }]"] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:26.845 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444835, 7444538, 7444890} }, outgoing: Configuration { voters: {} } }, learners: {7444938}, learners_next: {}, auto_leave: false }"] [raft_id=7444835] [region_id=7245948]
[2025/09/06 04:16:26.895 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=401152946] [region_id=7219580]
[2025/09/06 04:16:27.118 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129648] [size=100663070] [region_id=7219580]
[2025/09/06 04:16:28.177 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301988596] [region_id=7220062]
[2025/09/06 04:16:28.353 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128967] [size=100662557] [region_id=7220062]
[2025/09/06 04:16:28.956 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444938 store_id: 9 } } changes { change_type: AddLearnerNode peer { id: 7444835 store_id: 8 role: Learner } } }"] [index=3795561] [term=213] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.956 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65802 version: 67"] [kind=EnterJoint] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.956 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7245948 start_key: 6F737382E706466 region_epoch { conf_ver: 65804 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 role: DemotingVoter } peers { id: 7444890 store_id: 3 } peers { id: 7444938 store_id: 9 role: IncomingVoter }"] ["original region"="id: 7245948 start_key: 6F7306466 region_epoch { conf_ver: 65802 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 } peers { id: 7444890 store_id: 3 } peers { id: 7444938 store_id: 9 role: Learner }"] [changes="[peer { id: 7444938 store_id: 9 }, change_type: AddLearnerNode peer { id: 7444835 store_id: 8 role: Learner }]"] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.957 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444538, 7444890, 7444938} }, outgoing: Configuration { voters: {7444890, 7444538, 7444835} } }, learners: {}, learners_next: {7444835}, auto_leave: false }"] [raft_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.958 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=3795562] [term=213] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.958 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65804 version: 67"] [kind=LeaveJoint] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.958 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7245948 start_key: 6F734382E706466 region_epoch { conf_ver: 65806 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 role: Learner } peers { id: 7444890 store_id: 3 } peers { id: 7444938 store_id: 9 }"] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.959 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444538, 7444890, 7444938} }, outgoing: Configuration { voters: {} } }, learners: {7444835}, learners_next: {}, auto_leave: false }"] [raft_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.959 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7444835 store_id: 8 role: Learner } } }"] [index=3795563] [term=213] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.959 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65806 version: 67"] [kind=Simple] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.959 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7245948 start_key: 6F737706466 region_epoch { conf_ver: 65807 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444890 store_id: 3 } peers { id: 7444938 store_id: 9 }"] ["original region"="id: 7245948 start_key: 6F73E706466 region_epoch { conf_ver: 65806 version: 67 } peers { id: 7444538 store_id: 1 } peers { id: 7444835 store_id: 8 role: Learner } peers { id: 7444890 store_id: 3 } peers { id: 7444938 store_id: 9 }"] [changes="[change_type: RemoveNode peer { id: 7444835 store_id: 8 role: Learner }]"] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.959 +08:00] [INFO] [router.rs:345] ["[region 7245948] shutdown mailbox"]
[2025/09/06 04:16:28.959 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444538, 7444890, 7444938} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.959 +08:00] [INFO] [peer.rs:3524] ["delays destroy"] [reason=UnFlushLogGc] [merged_by_target=false] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.960 +08:00] [INFO] [peer.rs:3534] ["starts destroy"] [merged_by_target=false] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.960 +08:00] [INFO] [peer.rs:1402] ["begin to destroy"] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.960 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=9.237µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7245948]
[2025/09/06 04:16:28.960 +08:00] [INFO] [peer.rs:1506] ["peer destroy itself"] [keep_data=false] [clean=true] [takes=326.758µs] [peer_id=7444835] [region_id=7245948]
[2025/09/06 04:16:28.960 +08:00] [INFO] [router.rs:345] ["[region 7245948] shutdown mailbox"]
[2025/09/06 04:16:28.960 +08:00] [INFO] [region.rs:602] ["register deleting data in range"] [end_key=7A6F7377333832342E706466] [region_id=7245948]
[2025/09/06 04:16:28.960 +08:00] [INFO] [region.rs:636] ["delete data in range because of stale"] [end_key=7A6F6303933363637333832342E706466] [region_id=7245948]
[2025/09/06 04:16:32.990 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201325734] [region_id=7297670]
[2025/09/06 04:16:33.044 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444946 store_id: 3 role: Learner } change_type: AddLearnerNode]"] [region_id=7244910]
[2025/09/06 04:16:33.044 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=Simple] [changes="[change_type: AddLearnerNode peer { id: 7444946 store_id: 3 role: Learner }]"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:33.044 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444946 store_id: 3 role: Learner } } }"] [index=3864240] [term=198] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:33.044 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65639 version: 66"] [kind=Simple] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:33.044 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7244910 start_key: 6F73735F6F626A342E706466 region_epoch { conf_ver: 65640 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 } peers { id: 7444946 store_id: 3 role: Learner }"] ["original region"="id: 7244910 start_key: 832342E706466 region_epoch { conf_ver: 65639 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 }"] [changes="[change_type: AddLearnerNode peer { id: 7444946 store_id: 3 role: Learner }]"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:33.045 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444530, 7444677, 7443211} }, outgoing: Configuration { voters: {} } }, learners: {7444946}, learners_next: {}, auto_leave: false }"] [raft_id=7443211] [region_id=7244910]
[2025/09/06 04:16:33.045 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7244910 start_key: 6F73735F6F6342E706466 region_epoch { conf_ver: 65640 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 } peers { id: 7444946 store_id: 3 role: Learner }"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:33.046 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7444946] [request_index=0] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:33.088 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138149] [size=100663232] [region_id=7297670]
[2025/09/06 04:16:33.952 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138397, total size 100662596. raw_size_per_file 104857600, total takes 905.50149ms"]
[2025/09/06 04:16:33.973 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662596] [key_count=138397] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7244910_198_3864240_(default|lock|write).sst] [region_id=7244910]
[2025/09/06 04:16:33.973 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7244910_198_3864240_(default|lock|write).sst] [region_id=7244910]
[2025/09/06 04:16:33.973 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7244910_198_3864240_(default|lock|write).sst] [region_id=7244910]
[2025/09/06 04:16:33.973 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=927.193353ms] [size=7243930] [key_count=138397] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7244910_198_3864240_(default|lock|write).sst] [region_id=7244910]
[2025/09/06 04:16:33.974 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:34.011 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=34.993139ms] [size=7243930] [snap_key=7244910_198_3864240] [region_id=7244910]
[2025/09/06 04:16:34.011 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7444946 store_id: 3 role: Learner"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.093 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444946 store_id: 3 }, peer { id: 7444677 store_id: 14 role: Learner } change_type: AddLearnerNode]"] [region_id=7244910
[2025/09/06 04:16:34.093 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=EnterJoint] [changes="[peer { id: 7444946 store_id: 3 }, change_type: AddLearnerNode peer { id: 7444677 store_id: 14 role: Learner }]"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.094 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444946 store_id: 3 } } changes { change_type: AddLearnerNode peer { id: 7444677 store_id: 14 role: Learner } } }"] [index=3864241] [term=198] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.094 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65640 version: 66"] [kind=EnterJoint] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.094 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7244910 start_key: 6F73735F342E706466 region_epoch { conf_ver: 65642 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 role: DemotingVoter } peers { id: 7444946 store_id: 3 role: IncomingVoter }"] ["original region"="id: 7244910 start_key: 6F737342E706466 region_epoch { conf_ver: 65640 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 } peers { id: 7444946 store_id: 3 role: Learner }"] [changes="[peer { id: 7444946 store_id: 3 }, change_type: AddLearnerNode peer { id: 7444677 store_id: 14 role: Learner }]"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.094 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444530, 7444946, 7443211} }, outgoing: Configuration { voters: {7444677, 7444530, 7443211} } }, learners: {}, learners_next: {7444677}, auto_leave: false }"] [raft_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.094 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7244910 start_key: 6F7342E706466 region_epoch { conf_ver: 65642 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 role: DemotingVoter } peers { id: 7444946 store_id: 3 role: IncomingVoter }"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=LeaveJoint] [changes="[]"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=3864242] [term=198] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65642 version: 66"] [kind=LeaveJoint] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.095 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7244910 start_key: 6F73706466 region_epoch { conf_ver: 65644 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 role: Learner } peers { id: 7444946 store_id: 3 }"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.096 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444530, 7444946, 7443211} }, outgoing: Configuration { voters: {} } }, learners: {7444677}, learners_next: {}, auto_leave: false }"] [raft_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.096 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7244910 start_key: 6F7373506466 region_epoch { conf_ver: 65644 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 role: Learner } peers { id: 7444946 store_id: 3 }"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.096 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444677 store_id: 14 role: Learner } change_type: RemoveNode]"] [region_id=7244910]
[2025/09/06 04:16:34.096 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=Simple] [changes="[change_type: RemoveNode peer { id: 7444677 store_id: 14 role: Learner }]"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.097 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7444677 store_id: 14 role: Learner } } }"] [index=3864243] [term=198] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.097 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65644 version: 66"] [kind=Simple] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.097 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7244910 start_key: 6F7372E706466 region_epoch { conf_ver: 65645 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444946 store_id: 3 }"] ["original region"="id: 7244910 start_key: 6F73706466 region_epoch { conf_ver: 65644 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444677 store_id: 14 role: Learner } peers { id: 7444946 store_id: 3 }"] [changes="[change_type: RemoveNode peer { id: 7444677 store_id: 14 role: Learner }]"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.097 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444530, 7444946, 7443211} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.097 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7244910 start_key: 6F737706466 region_epoch { conf_ver: 65645 version: 66 } peers { id: 7443211 store_id: 8 } peers { id: 7444530 store_id: 1 } peers { id: 7444946 store_id: 3 }"] [peer_id=7443211] [region_id=7244910]
[2025/09/06 04:16:34.097 +08:00] [INFO] [store.rs:600] ["raft message is stale, tell to gc"] [msg_type=MsgAppendResponse] [current_region_epoch="conf_ver: 65645 version: 66"] [region_id=7244910]
[2025/09/06 04:16:35.505 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=205513665] [region_id=7219733]
[2025/09/06 04:16:35.638 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129181] [size=100662888] [region_id=7219733]
[2025/09/06 04:16:35.781 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201323654] [region_id=7219068]
[2025/09/06 04:16:35.887 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129057] [size=100662628] [region_id=7219068]
[2025/09/06 04:16:36.646 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444957 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7257488]
[2025/09/06 04:16:36.650 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=Simple] [changes="[change_type: AddLearnerNode peer { id: 7444957 store_id: 9 role: Learner }]"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:36.661 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444957 store_id: 9 role: Learner } } }"] [index=3069663] [term=175] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:36.662 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65813 version: 77"] [kind=Simple] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:36.662 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7257488 start_key: 6F737706466 region_epoch { conf_ver: 65814 version: 77 } peers { id: 7440340 store_id: 8 } peers { id: 7443483 store_id: 1 } peers { id: 7443852 store_id: 3 } peers { id: 7444957 store_id: 9 role: Learner }"] ["original region"="id: 7257488 start_key: 6F737306466 region_epoch { conf_ver: 65813 version: 77 } peers { id: 7440340 store_id: 8 } peers { id: 7443483 store_id: 1 } peers { id: 7443852 store_id: 3 }"] [changes="[change_type: AddLearnerNode peer { id: 7444957 store_id: 9 role: Learner }]"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:36.672 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7443483, 7440340, 7443852} }, outgoing: Configuration { voters: {} } }, learners: {7444957}, learners_next: {}, auto_leave: false }"] [raft_id=7440340] [region_id=7257488]
[2025/09/06 04:16:36.672 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7257488 start_key: 6F737306466 region_epoch { conf_ver: 65814 version: 77 } peers { id: 7440340 store_id: 8 } peers { id: 7443483 store_id: 1 } peers { id: 7443852 store_id: 3 } peers { id: 7444957 store_id: 9 role: Learner }"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:36.707 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7444957] [request_index=0] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:36.758 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986894] [region_id=7217734]
[2025/09/06 04:16:36.926 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=126860] [size=100663021] [region_id=7217734]
[2025/09/06 04:16:37.699 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138366, total size 100662783. raw_size_per_file 104857600, total takes 990.277536ms"]
[2025/09/06 04:16:37.720 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662783] [key_count=138366] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7257488_175_3069663_(default|lock|write).sst] [region_id=7257488]
[2025/09/06 04:16:37.720 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7257488_175_3069663_(default|lock|write).sst] [region_id=7257488]
[2025/09/06 04:16:37.720 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7257488_175_3069663_(default|lock|write).sst] [region_id=7257488]
[2025/09/06 04:16:37.723 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=1.01453324s] [size=7274247] [key_count=138366] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7257488_175_3069663_(default|lock|write).sst] [region_id=7257488]
[2025/09/06 04:16:37.736 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:37.784 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=45.431422ms] [size=7274247] [snap_key=7257488_175_3069663] [region_id=7257488]
[2025/09/06 04:16:37.792 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7444957 store_id: 9 role: Learner"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:37.982 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201324234] [region_id=7223935]
[2025/09/06 04:16:37.988 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444964 store_id: 3 role: Learner } } }"] [index=254827] [term=16] [peer_id=7444713] [region_id=7418340]
[2025/09/06 04:16:37.988 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66611 version: 116"] [kind=Simple] [peer_id=7444713] [region_id=7418340]
[2025/09/06 04:16:37.988 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7418340 start_key: 6F7373E706466 region_epoch { conf_ver: 66612 version: 116 } peers { id: 7444492 store_id: 14 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 role: Learner }"] ["original region"="id: 7418340 start_key: 6F706466 region_epoch { conf_ver: 66611 version: 116 } peers { id: 7444492 store_id: 14 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 }"] [changes="[change_type: AddLearnerNode peer { id: 7444964 store_id: 3 role: Learner }]"] [peer_id=7444713] [region_id=7418340]
[2025/09/06 04:16:37.991 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444492, 7444713, 7444547} }, outgoing: Configuration { voters: {} } }, learners: {7444964}, learners_next: {}, auto_leave: false }"] [raft_id=7444713] [region_id=7418340]
[2025/09/06 04:16:38.095 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:16:38.102 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128165] [size=100663103] [region_id=7223935]
[2025/09/06 04:16:38.174 +08:00] [INFO] [peer.rs:3263] ["reject to transfer leader"] [last_index=3069968] [index=3069686] [reason="log gap"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:38.178 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201326008] [region_id=7220062]
[2025/09/06 04:16:38.348 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128967] [size=100662557] [region_id=7220062]
[2025/09/06 04:16:39.484 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444964 store_id: 3 } } changes { change_type: AddLearnerNode peer { id: 7444492 store_id: 14 role: Learner } } }"] [index=254828] [term=16] [peer_id=7444713] [region_id=7418340]
[2025/09/06 04:16:39.484 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66612 version: 116"] [kind=EnterJoint] [peer_id=7444713] [region_id=7418340]
[2025/09/06 04:16:39.484 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7418340 start_key: 6F7373032352E706466 region_epoch { conf_ver: 66614 version: 116 } peers { id: 7444492 store_id: 14 role: DemotingVoter } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 role: IncomingVoter }"] ["original region"="id: 7418340 start_key: 6F73735F2352E706466 region_epoch { conf_ver: 66612 version: 116 } peers { id: 7444492 store_id: 14 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 role: Learner }"] [changes="[peer { id: 7444964 store_id: 3 }, change_type: AddLearnerNode peer { id: 7444492 store_id: 14 role: Learner }]"] [peer_id=7444713] [region_id=7418340]

[2025/09/06 04:16:39.491 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444713, 7444964, 7444547} }, outgoing: Configuration { voters: {7444492, 7444713, 7444547} } }, learners: {}, learners_next: {7444492}, auto_leave: false }"] [raft_id=7444713] [region_id=7418340]
[2025/09/06 04:16:39.500 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=254829] [term=16] [peer_id=7444713] [region_id=7418340]
[2025/09/06 04:16:39.500 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66614 version: 116"] [kind=LeaveJoint] [peer_id=7444713] [region_id=7418340]
[2025/09/06 04:16:39.500 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7418340 start_key: 6F7373532352E706466 region_epoch { conf_ver: 66616 version: 116 } peers { id: 7444492 store_id: 14 role: Learner } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 }"] [peer_id=7444713] [region_id=7418340]
[2025/09/06 04:16:39.505 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444713, 7444964, 7444547} }, outgoing: Configuration { voters: {} } }, learners: {7444492}, learners_next: {}, auto_leave: false }"] [raft_id=7444713] [region_id=7418340]
[2025/09/06 04:16:39.505 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7444492 store_id: 14 role: Learner } } }"] [index=254830] [term=16] [peer_id=7444713] [region_id=7418340]
[2025/09/06 04:16:39.506 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66616 version: 116"] [kind=Simple] [peer_id=7444713] [region_id=7418340]
[2025/09/06 04:16:39.506 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7418340 start_key: 6F73735F6F52E706466 region_epoch { conf_ver: 66617 version: 116 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 }"] ["original region"="id: 7418340 start_key: 6F73735F6E706466 region_epoch { conf_ver: 66616 version: 116 } peers { id: 7444492 store_id: 14 role: Learner } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 }"] [changes="[change_type: RemoveNode peer { id: 7444492 store_id: 14 role: Learner }]"] [peer_id=7444713] [region_id=7418340]
[2025/09/06 04:16:39.508 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444713, 7444964, 7444547} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444713] [region_id=7418340]
[2025/09/06 04:16:40.031 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=410175200] [region_id=7220974]
[2025/09/06 04:16:40.281 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128600] [size=100662579] [region_id=7220974]
[2025/09/06 04:16:41.267 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444977 store_id: 9 role: Learner } change_type: AddLearnerNode]"] [region_id=7271280]
[2025/09/06 04:16:41.267 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=Simple] [changes="[change_type: AddLearnerNode peer { id: 7444977 store_id: 9 role: Learner }]"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:41.273 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444977 store_id: 9 role: Learner } } }"] [index=2854167] [term=149] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:41.273 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65603 version: 80"] [kind=Simple] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:41.273 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7271280 start_key: 6F7373353338322E706466 region_epoch { conf_ver: 65604 version: 80 } peers { id: 7442849 store_id: 8 } peers { id: 7444706 store_id: 14 } peers { id: 7444754 store_id: 1 } peers { id: 7444977 store_id: 9 role: Learner }"] ["original region"="id: 7271280 start_key: 6F73735F63338322E706466 region_epoch { conf_ver: 65603 version: 80 } peers { id: 7442849 store_id: 8 } peers { id: 7444706 store_id: 14 } peers { id: 7444754 store_id: 1 }"] [changes="[change_type: AddLearnerNode peer { id: 7444977 store_id: 9 role: Learner }]"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:41.282 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444706, 7444754, 7442849} }, outgoing: Configuration { voters: {} } }, learners: {7444977}, learners_next: {}, auto_leave: false }"] [raft_id=7442849] [region_id=7271280]
[2025/09/06 04:16:41.282 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7271280 start_key: 6F73735322E706466 region_epoch { conf_ver: 65604 version: 80 } peers { id: 7442849 store_id: 8 } peers { id: 7444706 store_id: 14 } peers { id: 7444754 store_id: 1 } peers { id: 7444977 store_id: 9 role: Learner }"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:41.299 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7444977] [request_index=0] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:41.491 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7443473] [request_index=0] [peer_id=7437537] [region_id=7219863]
[2025/09/06 04:16:41.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:16:42.002 +08:00] [INFO] [peer.rs:3718] ["transfer leader"] [peer="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:42.002 +08:00] [INFO] [raft.rs:1904] ["[term 175] starts to transfer leadership to 7443483"] [lead_transferee=7443483] [term=175] [raft_id=7440340] [region_id=7257488]
[2025/09/06 04:16:42.733 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7443469] [request_index=0] [peer_id=7439234] [region_id=7292992]
[2025/09/06 04:16:42.753 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138445, total size 100662619. raw_size_per_file 104857600, total takes 1.454168622s"]
[2025/09/06 04:16:42.782 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662619] [key_count=138445] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7271280_149_2854167_(default|lock|write).sst] [region_id=7271280]
[2025/09/06 04:16:42.782 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7271280_149_2854167_(default|lock|write).sst] [region_id=7271280]
[2025/09/06 04:16:42.782 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7271280_149_2854167_(default|lock|write).sst] [region_id=7271280]
[2025/09/06 04:16:42.783 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=1.483523662s] [size=7215131] [key_count=138445] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7271280_149_2854167_(default|lock|write).sst] [region_id=7271280]
[2025/09/06 04:16:42.783 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7444977] [request_index=0] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:42.783 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:42.832 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=46.907827ms] [size=7215131] [snap_key=7271280_149_2854167] [region_id=7271280]
[2025/09/06 04:16:42.835 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:42.939 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7443228] [request_index=0] [peer_id=7429386] [region_id=7219580]
[2025/09/06 04:16:43.092 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201342320] [region_id=7261583]
[2025/09/06 04:16:43.203 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138431] [size=100663130] [region_id=7261583]
[2025/09/06 04:16:43.486 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201325566] [region_id=7257488]
[2025/09/06 04:16:43.547 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444983 store_id: 2 role: Learner } change_type: AddLearnerNode]"] [region_id=7219580]
[2025/09/06 04:16:43.547 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=Simple] [changes="[change_type: AddLearnerNode peer { id: 7444983 store_id: 2 role: Learner }]"] [peer_id=7429386] [region_id=7219580]
[2025/09/06 04:16:43.563 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444983 store_id: 2 role: Learner } } }"] [index=6809198] [term=226] [peer_id=7429386] [region_id=7219580]
[2025/09/06 04:16:43.563 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 68351 version: 60"] [kind=Simple] [peer_id=7429386] [region_id=7219580]
[2025/09/06 04:16:43.563 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7219580 start_key: 6F73735F65F313638383633353334353932392E6A7067 region_epoch { conf_ver: 68352 version: 60 } peers { id: 7429386 store_id: 8 } peers { id: 7441677 store_id: 14 } peers { id: 7443228 store_id: 1 } peers { id: 7444983 store_id: 2 role: Learner }"] ["original region"="id: 7219580 start_key: 6F73735FE6A7067 region_epoch { conf_ver: 68351 version: 60 } peers { id: 7429386 store_id: 8 } peers { id: 7441677 store_id: 14 } peers { id: 7443228 store_id: 1 }"] [changes="[change_type: AddLearnerNode peer { id: 7444983 store_id: 2 role: Learner }]"] [peer_id=7429386] [region_id=7219580]
[2025/09/06 04:16:43.565 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7441677, 7429386, 7443228} }, outgoing: Configuration { voters: {} } }, learners: {7444983}, learners_next: {}, auto_leave: false }"] [raft_id=7429386] [region_id=7219580]
[2025/09/06 04:16:43.565 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7219580 start_key: 6F7377067 region_epoch { conf_ver: 68352 version: 60 } peers { id: 7429386 store_id: 8 } peers { id: 7441677 store_id: 14 } peers { id: 7443228 store_id: 1 } peers { id: 7444983 store_id: 2 role: Learner }"] [peer_id=7429386] [region_id=7219580]
[2025/09/06 04:16:43.601 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138366] [size=100662783] [region_id=7257488]
[2025/09/06 04:16:43.639 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:43.664 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data1/tikv-data/tikv-20161/snap/rev_7410615_33_329038_(default|lock|write).sst] [snap_key=7410615_33_329038]
[2025/09/06 04:16:43.669 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=29.229209ms] [snap_key=7410615_33_329038]
[2025/09/06 04:16:43.669 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:43.674 +08:00] [INFO] [raft_log.rs:627] ["log [committed=328624, persisted=328624, applied=328624, unstable.offset=328625, unstable.entries.len()=0] starts to restore snapshot [index: 329038, term: 33]"] [snapshot_term=33] [snapshot_index=329038] [log="committed=328624, persisted=328624, applied=328624, unstable.offset=328625, unstable.entries.len()=0"] [raft_id=7444284] [region_id=7410615]
[2025/09/06 04:16:43.674 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444284, 7442761, 7444089} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444284] [region_id=7410615]
[2025/09/06 04:16:43.674 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=33] [snapshot_index=329038] [last_term=33] [last_index=329038] [commit=329038] [raft_id=7444284] [region_id=7410615]
[2025/09/06 04:16:43.674 +08:00] [INFO] [raft.rs:2511] ["[commit: 329038, term: 33] restored snapshot [index: 329038, term: 33]"] [snapshot_term=33] [snapshot_index=329038] [commit=329038] [term=33] [raft_id=7444284] [region_id=7410615]
[2025/09/06 04:16:43.674 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444284] [region_id=7410615]
[2025/09/06 04:16:43.674 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=6.794µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7410615]
[2025/09/06 04:16:43.674 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 329038 commit_index: 328624 commit_term: 33 truncated_state { index: 329038 term: 33 }"] [region="id: 7410615 start_key: 6F73735F602E706466 region_epoch { conf_ver: 66581 version: 115 } peers { id: 7442761 store_id: 14 } peers { id: 7444089 store_id: 2 } peers { id: 7444284 store_id: 8 }"] [peer_id=7444284] [region_id=7410615]
[2025/09/06 04:16:43.674 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=33] [peer_id=7444284] [region_id=7410615]
[2025/09/06 04:16:43.674 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7410615]
[2025/09/06 04:16:43.674 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7410615 start_key: 6F7372E706466 region_epoch { conf_ver: 66581 version: 115 } peers { id: 7442761 store_id: 14 } peers { id: 7444089 store_id: 2 } peers { id: 7444284 store_id: 8 }"] [peer_id=7444284] [region_id=7410615]
[2025/09/06 04:16:43.674 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444284] [region_id=7410615]
[2025/09/06 04:16:43.674 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444284] [region_id=7410615]
[2025/09/06 04:16:43.674 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7410615 start_key: 6F73735F6466 region_epoch { conf_ver: 66581 version: 115 } peers { id: 7442761 store_id: 14 } peers { id: 7444089 store_id: 2 } peers { id: 7444284 store_id: 8 }"] [prev_region="id: 7410615 start_key: 6F73735F02E706466 region_epoch { conf_ver: 66581 version: 115 } peers { id: 7442761 store_id: 14 } peers { id: 7444089 store_id: 2 } peers { id: 7444284 store_id: 8 }"] [peer_id=7444284] [region_id=7410615]
[2025/09/06 04:16:43.831 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 129457, total size 100662676. raw_size_per_file 104857600, total takes 2.339268209s"]
[2025/09/06 04:16:43.856 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662676] [key_count=129457] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219863_197_6677891_(default|lock|write).sst] [region_id=7219863]
[2025/09/06 04:16:43.856 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219863_197_6677891_(default|lock|write).sst] [region_id=7219863]
[2025/09/06 04:16:43.857 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219863_197_6677891_(default|lock|write).sst] [region_id=7219863]
[2025/09/06 04:16:43.857 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=2.365094954s] [size=10176263] [key_count=129457] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219863_197_6677891_(default|lock|write).sst] [region_id=7219863]
[2025/09/06 04:16:43.857 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:43.907 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:43.913 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=52.668613ms] [size=10176263] [snap_key=7219863_197_6677891] [region_id=7219863]
[2025/09/06 04:16:43.913 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7443473 store_id: 3"] [peer_id=7437537] [region_id=7219863]
[2025/09/06 04:16:43.954 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=44.30151ms] [region_id=7410615]
[2025/09/06 04:16:44.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:16:44.554 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:44.940 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138255, total size 100662890. raw_size_per_file 104857600, total takes 2.157187816s"]
[2025/09/06 04:16:44.953 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662890] [key_count=138255] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7292992_98_2563134_(default|lock|write).sst] [region_id=7292992]
[2025/09/06 04:16:44.953 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7292992_98_2563134_(default|lock|write).sst] [region_id=7292992]
[2025/09/06 04:16:44.953 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7292992_98_2563134_(default|lock|write).sst] [region_id=7292992]
[2025/09/06 04:16:44.954 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=2.17149045s] [size=7050791] [key_count=138255] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7292992_98_2563134_(default|lock|write).sst] [region_id=7292992]
[2025/09/06 04:16:44.954 +08:00] [INFO] [region.rs:307] ["generate snap is canceled"] [region_id=7219580]
[2025/09/06 04:16:44.955 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:45.004 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=46.906308ms] [size=7050791] [snap_key=7292992_98_2563134] [region_id=7292992]
[2025/09/06 04:16:45.004 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7443469 store_id: 1"] [peer_id=7439234] [region_id=7292992]
[2025/09/06 04:16:45.248 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138445, total size 100662619. raw_size_per_file 104857600, total takes 1.390585801s"]
[2025/09/06 04:16:45.267 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662619] [key_count=138445] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7271280_149_2854168_(default|lock|write).sst] [region_id=7271280]
[2025/09/06 04:16:45.267 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7271280_149_2854168_(default|lock|write).sst] [region_id=7271280]
[2025/09/06 04:16:45.267 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7271280_149_2854168_(default|lock|write).sst] [region_id=7271280]
[2025/09/06 04:16:45.268 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=1.411162687s] [size=7215131] [key_count=138445] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7271280_149_2854168_(default|lock|write).sst] [region_id=7271280]
[2025/09/06 04:16:45.269 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:45.315 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=44.678796ms] [size=7215131] [snap_key=7271280_149_2854168] [region_id=7271280]
[2025/09/06 04:16:45.315 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7444977 store_id: 9 role: Learner"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:45.415 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:16:45.416 +08:00] [INFO] [peer.rs:3718] ["transfer leader"] [peer="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:45.416 +08:00] [INFO] [raft.rs:1904] ["[term 149] starts to transfer leadership to 7444754"] [lead_transferee=7444754] [term=149] [raft_id=7442849] [region_id=7271280]
[2025/09/06 04:16:45.506 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=224960676] [region_id=7219733]
[2025/09/06 04:16:45.652 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129181] [size=100662888] [region_id=7219733]
[2025/09/06 04:16:45.782 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=326133702] [region_id=7219068]
[2025/09/06 04:16:45.923 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129057] [size=100662628] [region_id=7219068]
[2025/09/06 04:16:46.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:16:46.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:16:46.494 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:46.494 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:46.676 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:16:46.676 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:46.759 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=402660115] [region_id=7217734]
[2025/09/06 04:16:46.931 +08:00] [WARN] [peer_storage.rs:493] ["failed to try generating snapshot"] [request_peer=7443228] [times=1] [peer_id=7429386] [region_id=7219580]
[2025/09/06 04:16:46.931 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7443228] [request_index=0] [peer_id=7429386] [region_id=7219580]
[2025/09/06 04:16:46.966 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=126860] [size=100663021] [region_id=7217734]
[2025/09/06 04:16:47.955 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 129648, total size 100663070. raw_size_per_file 104857600, total takes 1.012351577s"]
[2025/09/06 04:16:47.979 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100663070] [key_count=129648] [cf=default] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219580_226_6809198_(default|lock|write).sst] [region_id=7219580]
[2025/09/06 04:16:47.979 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219580_226_6809198_(default|lock|write).sst] [region_id=7219580]
[2025/09/06 04:16:47.979 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219580_226_6809198_(default|lock|write).sst] [region_id=7219580]
[2025/09/06 04:16:47.983 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=1.040290474s] [size=10118449] [key_count=129648] [snapshot=/data1/tikv-data/tikv-20161/snap/gen_7219580_226_6809198_(default|lock|write).sst] [region_id=7219580]
[2025/09/06 04:16:47.988 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:48.042 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=52.663953ms] [size=10118449] [snap_key=7219580_226_6809198] [region_id=7219580]
[2025/09/06 04:16:48.042 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7443228 store_id: 1"] [peer_id=7429386] [region_id=7219580]
[2025/09/06 04:16:48.179 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=222300201] [region_id=7220062]
[2025/09/06 04:16:48.329 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128967] [size=100662557] [region_id=7220062]
[2025/09/06 04:16:48.744 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:48.771 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data1/tikv-data/tikv-20161/snap/rev_7314410_77_2199182_(default|lock|write).sst] [snap_key=7314410_77_2199182]
[2025/09/06 04:16:48.776 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=31.759033ms] [snap_key=7314410_77_2199182]
[2025/09/06 04:16:48.776 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:48.781 +08:00] [INFO] [raft_log.rs:627] ["log [committed=2198662, persisted=2198662, applied=2198662, unstable.offset=2198663, unstable.entries.len()=0] starts to restore snapshot [index: 2199182, term: 77]"] [snapshot_term=77] [snapshot_index=2199182] [log="committed=2198662, persisted=2198662, applied=2198662, unstable.offset=2198663, unstable.entries.len()=0"] [raft_id=7444723] [region_id=7314410]
[2025/09/06 04:16:48.782 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444723, 7444082, 7444667} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444723] [region_id=7314410]
[2025/09/06 04:16:48.782 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=77] [snapshot_index=2199182] [last_term=77] [last_index=2199182] [commit=2199182] [raft_id=7444723] [region_id=7314410]
[2025/09/06 04:16:48.782 +08:00] [INFO] [raft.rs:2511] ["[commit: 2199182, term: 77] restored snapshot [index: 2199182, term: 77]"] [snapshot_term=77] [snapshot_index=2199182] [commit=2199182] [term=77] [raft_id=7444723] [region_id=7314410]
[2025/09/06 04:16:48.782 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:48.782 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=7.284µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7314410]
[2025/09/06 04:16:48.782 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 2199182 commit_index: 2198662 commit_term: 77 truncated_state { index: 2199182 term: 77 }"] [region="id: 7314410 start_key: 6F7373506466 region_epoch { conf_ver: 65359 version: 89 } peers { id: 7444082 store_id: 14 } peers { id: 7444667 store_id: 2 } peers { id: 7444723 store_id: 8 }"] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:48.786 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=77] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:48.786 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7314410 start_key: 6F737706466 region_epoch { conf_ver: 65359 version: 89 } peers { id: 7444082 store_id: 14 } peers { id: 7444667 store_id: 2 } peers { id: 7444723 store_id: 8 }"] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:48.786 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7314410]
[2025/09/06 04:16:48.786 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:48.786 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7314410 start_key: 6F73732E706466 region_epoch { conf_ver: 65359 version: 89 } peers { id: 7444082 store_id: 14 } peers { id: 7444667 store_id: 2 } peers { id: 7444723 store_id: 8 }"] [prev_region="id: 7314410 start_key: 6F73E706466 region_epoch { conf_ver: 65359 version: 89 } peers { id: 7444082 store_id: 14 } peers { id: 7444667 store_id: 2 } peers { id: 7444723 store_id: 8 }"] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:48.793 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=213909512] [region_id=7292992]
[2025/09/06 04:16:48.899 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138255] [size=100662890] [region_id=7292992]
[2025/09/06 04:16:48.978 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:48.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:16:48.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:16:48.994 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:48.994 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:49.025 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=44.865474ms] [region_id=7314410]
[2025/09/06 04:16:49.503 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=342277463] [region_id=7219863]
[2025/09/06 04:16:49.703 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129457] [size=100662676] [region_id=7219863]
[2025/09/06 04:16:50.032 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=309502530] [region_id=7220974]
[2025/09/06 04:16:50.213 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128600] [size=100662579] [region_id=7220974]
[2025/09/06 04:16:51.493 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:16:51.493 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:16:51.493 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:16:51.494 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:51.494 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:53.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:16:53.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:16:53.494 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:53.494 +08:00] [INFO] [peer.rs:3718] ["transfer leader"] [peer="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:53.494 +08:00] [INFO] [raft.rs:1904] ["[term 175] starts to transfer leadership to 7443483"] [lead_transferee=7443483] [term=175] [raft_id=7440340] [region_id=7257488]
[2025/09/06 04:16:54.183 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7217630_273_9964736] [peer_id=7436558] [region_id=7217630]
[2025/09/06 04:16:54.183 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7217841_289_9197254] [peer_id=7444368] [region_id=7217841]
[2025/09/06 04:16:54.183 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7220281_161_6336144] [peer_id=7444257] [region_id=7220281]
[2025/09/06 04:16:54.183 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7221523_146_6050108] [peer_id=7444440] [region_id=7221523]
[2025/09/06 04:16:54.183 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7314410_77_2198650] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:54.183 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:54.183 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7333984_73_1766188] [peer_id=7442520] [region_id=7333984]
[2025/09/06 04:16:54.183 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7314410_77_2199182] [peer_id=7444723] [region_id=7314410]
[2025/09/06 04:16:54.183 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7347689_50_1335024] [peer_id=7444131] [region_id=7347689]
[2025/09/06 04:16:54.183 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7375283_35_832099] [peer_id=7439159] [region_id=7375283]
[2025/09/06 04:16:54.183 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7410615_33_329038] [peer_id=7444284] [region_id=7410615]
[2025/09/06 04:16:54.191 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:54.201 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:54.207 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:54.213 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:54.217 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:54.221 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:54.223 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:54.226 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:54.228 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:55.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:16:55.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:16:55.995 +08:00] [INFO] [peer.rs:3718] ["transfer leader"] [peer="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:55.995 +08:00] [INFO] [raft.rs:1904] ["[term 149] starts to transfer leadership to 7444754"] [lead_transferee=7444754] [term=149] [raft_id=7442849] [region_id=7271280]
[2025/09/06 04:16:55.995 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:56.760 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986894] [region_id=7217734]
[2025/09/06 04:16:56.921 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=126860] [size=100663021] [region_id=7217734]
[2025/09/06 04:16:56.960 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986510] [region_id=7217641]
[2025/09/06 04:16:57.111 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=133701] [size=100663011] [region_id=7217641]
[2025/09/06 04:16:58.337 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=233039806] [region_id=7271280]
[2025/09/06 04:16:58.443 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138445] [size=100662619] [region_id=7271280]
[2025/09/06 04:16:58.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:16:58.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:16:58.494 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:16:58.494 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:16:59.504 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301985661] [region_id=7219863]
[2025/09/06 04:16:59.677 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129457] [size=100662676] [region_id=7219863]
[2025/09/06 04:17:00.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:17:00.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:17:00.495 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:17:00.495 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:17:02.957 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:17:02.957 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:17:02.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:17:02.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:17:02.994 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:17:02.994 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:17:04.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:17:04.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:17:04.995 +08:00] [INFO] [peer.rs:3718] ["transfer leader"] [peer="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:17:04.995 +08:00] [INFO] [raft.rs:1904] ["[term 175] starts to transfer leadership to 7443483"] [lead_transferee=7443483] [term=175] [raft_id=7440340] [region_id=7257488]
[2025/09/06 04:17:04.995 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:17:06.763 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201323748] [region_id=7244910]
[2025/09/06 04:17:06.886 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138397] [size=100662596] [region_id=7244910]
[2025/09/06 04:17:06.905 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=297802255] [region_id=7219580]
[2025/09/06 04:17:06.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:17:06.995 +08:00] [INFO] [peer.rs:3718] ["transfer leader"] [peer="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:17:06.995 +08:00] [INFO] [raft.rs:1904] ["[term 149] starts to transfer leadership to 7444754"] [lead_transferee=7444754] [term=149] [raft_id=7442849] [region_id=7271280]
[2025/09/06 04:17:07.075 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129648] [size=100663070] [region_id=7219580]
[2025/09/06 04:17:07.493 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:17:07.494 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:17:09.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:17:09.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:17:09.494 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:17:09.494 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:17:11.993 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:17:11.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:17:11.994 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:17:11.994 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:17:13.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:17:13.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:17:13.994 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:17:13.994 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:17:16.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:17:16.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:17:16.495 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:17:16.495 +08:00] [INFO] [peer.rs:3718] ["transfer leader"] [peer="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:17:16.495 +08:00] [INFO] [raft.rs:1904] ["[term 175] starts to transfer leadership to 7443483"] [lead_transferee=7443483] [term=175] [raft_id=7440340] [region_id=7257488]
[2025/09/06 04:17:16.764 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201323950] [region_id=7248308]
[2025/09/06 04:17:16.849 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138517] [size=100662705] [region_id=7248308]
[2025/09/06 04:17:18.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:17:18.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:17:18.495 +08:00] [INFO] [peer.rs:3718] ["transfer leader"] [peer="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:17:18.495 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:17:18.496 +08:00] [INFO] [raft.rs:1904] ["[term 149] starts to transfer leadership to 7444754"] [lead_transferee=7444754] [term=149] [raft_id=7442849] [region_id=7271280]
[2025/09/06 04:17:20.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:17:20.494 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:17:20.495 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:17:20.495 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:17:22.993 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:17:22.993 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]
[2025/09/06 04:17:22.994 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7443483 store_id: 1"] [peer_id=7440340] [region_id=7257488]
[2025/09/06 04:17:22.994 +08:00] [INFO] [peer.rs:3278] ["propose transfer leader command"] [to="id: 7444754 store_id: 1"] [peer_id=7442849] [region_id=7271280]
[2025/09/06 04:17:24.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7444754 store_id: 1"] [from_peer="id: 7442849 store_id: 8"] [region_id=7271280]
[2025/09/06 04:17:24.994 +08:00] [INFO] [pd.rs:1521] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 7443483 store_id: 1"] [from_peer="id: 7440340 store_id: 8"] [region_id=7257488]

[jwongz]

从store-8的日志可疑看到，一直尝试将部分region的leader切到store-1(111:20180)，store-1的日志如下：

[2025/09/06 04:16:34.097 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444530, 7444946, 7443211} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444530] [region_id=7244910]
[2025/09/06 04:16:34.219 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129018] [size=100663043] [region_id=7219307]
[2025/09/06 04:16:35.714 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201324054] [region_id=7221274]
[2025/09/06 04:16:35.881 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128586] [size=100662828] [region_id=7221274]
[2025/09/06 04:16:36.956 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=297801480] [region_id=7424617]
[2025/09/06 04:16:37.117 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138017] [size=100662923] [region_id=7424617]
[2025/09/06 04:16:37.770 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444957 store_id: 9 role: Learner } } }"] [index=3069663] [term=175] [peer_id=7443483] [region_id=7257488]
[2025/09/06 04:16:37.770 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 65813 version: 77"] [kind=Simple] [peer_id=7443483] [region_id=7257488]
[2025/09/06 04:16:37.770 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7257488 start_key: 6F73735313132382E706466 region_epoch { conf_ver: 65814 version: 77 } peers { id: 7440340 store_id: 8 } peers { id: 7443483 store_id: 1 } peers { id: 7443852 store_id: 3 } peers { id: 7444957 store_id: 9 role: Learner }"] ["original region"="id: 7257488 start_key: 6F737382E706466 region_epoch { conf_ver: 65813 version: 77 } peers { id: 7440340 store_id: 8 } peers { id: 7443483 store_id: 1 } peers { id: 7443852 store_id: 3 }"] [changes="[change_type: AddLearnerNode peer { id: 7444957 store_id: 9 role: Learner }]"] [peer_id=7443483] [region_id=7257488]
[2025/09/06 04:16:37.798 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7440340, 7443852, 7443483} }, outgoing: Configuration { voters: {} } }, learners: {7444957}, learners_next: {}, auto_leave: false }"] [raft_id=7443483] [region_id=7257488]
[2025/09/06 04:16:37.945 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444964 store_id: 3 role: Learner } change_type: AddLearnerNode]"] [region_id=7418340]
[2025/09/06 04:16:37.950 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=Simple] [changes="[change_type: AddLearnerNode peer { id: 7444964 store_id: 3 role: Learner }]"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:37.980 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: AddLearnerNode peer { id: 7444964 store_id: 3 role: Learner } } }"] [index=254827] [term=16] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:37.980 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66611 version: 116"] [kind=Simple] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:37.980 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7418340 start_key: 6F73735F6F6266466 region_epoch { conf_ver: 66612 version: 116 } peers { id: 7444492 store_id: 14 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 role: Learner }"] ["original region"="id: 7418340 start_key: 6F73735E706466 region_epoch { conf_ver: 66611 version: 116 } peers { id: 7444492 store_id: 14 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 }"] [changes="[change_type: AddLearnerNode peer { id: 7444964 store_id: 3 role: Learner }]"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:38.015 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444492, 7444713, 7444547} }, outgoing: Configuration { voters: {} } }, learners: {7444964}, learners_next: {}, auto_leave: false }"] [raft_id=7444547] [region_id=7418340]
[2025/09/06 04:16:38.015 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7418340 start_key: 6F73735F62E706466 region_epoch { conf_ver: 66612 version: 116 } peers { id: 7444492 store_id: 14 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 role: Learner }"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:38.209 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201326504] [region_id=7219992]
[2025/09/06 04:16:38.246 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7444964] [request_index=0] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:38.409 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128834] [size=100663261] [region_id=7219992]
[2025/09/06 04:16:39.190 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138056, total size 100662941. raw_size_per_file 104857600, total takes 942.928709ms"]
[2025/09/06 04:16:39.215 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662941] [key_count=138056] [cf=default] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7418340_16_254827_(default|lock|write).sst] [region_id=7418340]
[2025/09/06 04:16:39.216 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7418340_16_254827_(default|lock|write).sst] [region_id=7418340]
[2025/09/06 04:16:39.216 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7418340_16_254827_(default|lock|write).sst] [region_id=7418340]
[2025/09/06 04:16:39.219 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=972.405633ms] [size=6703032] [key_count=138056] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7418340_16_254827_(default|lock|write).sst] [region_id=7418340]
[2025/09/06 04:16:39.322 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:39.373 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=47.647861ms] [size=6703032] [snap_key=7418340_16_254827] [region_id=7418340]
[2025/09/06 04:16:39.373 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7444964 store_id: 3 role: Learner"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.476 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444964 store_id: 3 }, peer { id: 7444492 store_id: 14 role: Learner } change_type: AddLearnerNode]"] [region_id=7418340]
[2025/09/06 04:16:39.476 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=EnterJoint] [changes="[peer { id: 7444964 store_id: 3 }, change_type: AddLearnerNode peer { id: 7444492 store_id: 14 role: Learner }]"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.478 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { peer { id: 7444964 store_id: 3 } } changes { change_type: AddLearnerNode peer { id: 7444492 store_id: 14 role: Learner } } }"] [index=254828] [term=16] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.478 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66612 version: 116"] [kind=EnterJoint] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.478 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7418340 start_key: 6F73735F2352E706466 region_epoch { conf_ver: 66614 version: 116 } peers { id: 7444492 store_id: 14 role: DemotingVoter } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 role: IncomingVoter }"] ["original region"="id: 7418340 start_key: 6F7373E706466 region_epoch { conf_ver: 66612 version: 116 } peers { id: 7444492 store_id: 14 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 role: Learner }"] [changes="[peer { id: 7444964 store_id: 3 }, change_type: AddLearnerNode peer { id: 7444492 store_id: 14 role: Learner }]"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.480 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444713, 7444964, 7444547} }, outgoing: Configuration { voters: {7444492, 7444713, 7444547} } }, learners: {}, learners_next: {7444492}, auto_leave: false }"] [raft_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.480 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7418340 start_key: 6F73735F6F2352E706466 region_epoch { conf_ver: 66614 version: 116 } peers { id: 7444492 store_id: 14 role: DemotingVoter } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 role: IncomingVoter }"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.481 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7418340]
[2025/09/06 04:16:39.481 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=LeaveJoint] [changes="[]"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.484 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7418340]
[2025/09/06 04:16:39.484 +08:00] [INFO] [peer.rs:4612] ["there is a pending conf change, try later"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.494 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 {}"] [index=254829] [term=16] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.494 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66614 version: 116"] [kind=LeaveJoint] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.494 +08:00] [INFO] [apply.rs:2415] ["leave joint state successfully"] [region="id: 7418340 start_key: 6F737352E706466 region_epoch { conf_ver: 66616 version: 116 } peers { id: 7444492 store_id: 14 role: Learner } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 }"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.498 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444713, 7444964, 7444547} }, outgoing: Configuration { voters: {} } }, learners: {7444492}, learners_next: {}, auto_leave: false }"] [raft_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.498 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7418340 start_key: 6F73735F2E706466 region_epoch { conf_ver: 66616 version: 116 } peers { id: 7444492 store_id: 14 role: Learner } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 }"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.498 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[]"] [region_id=7418340]
[2025/09/06 04:16:39.499 +08:00] [INFO] [pd.rs:1508] ["try to change peer"] [changes="[peer { id: 7444492 store_id: 14 role: Learner } change_type: RemoveNode]"] [region_id=7418340]
[2025/09/06 04:16:39.499 +08:00] [INFO] [peer.rs:4682] ["propose conf change peer"] [kind=Simple] [changes="[change_type: RemoveNode peer { id: 7444492 store_id: 14 role: Learner }]"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.504 +08:00] [INFO] [apply.rs:1612] ["execute admin command"] [command="cmd_type: ChangePeerV2 change_peer_v2 { changes { change_type: RemoveNode peer { id: 7444492 store_id: 14 role: Learner } } }"] [index=254830] [term=16] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.504 +08:00] [INFO] [apply.rs:2204] ["exec ConfChangeV2"] [epoch="conf_ver: 66616 version: 116"] [kind=Simple] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.504 +08:00] [INFO] [apply.rs:2385] ["conf change successfully"] ["current region"="id: 7418340 start_key: 6F73735F2352E706466 region_epoch { conf_ver: 66617 version: 116 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 }"] ["original region"="id: 7418340 start_key: 6F73735F62352E706466 region_epoch { conf_ver: 66616 version: 116 } peers { id: 7444492 store_id: 14 role: Learner } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 }"] [changes="[change_type: RemoveNode peer { id: 7444492 store_id: 14 role: Learner }]"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.506 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444713, 7444964, 7444547} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.506 +08:00] [INFO] [peer.rs:3787] ["notify pd with change peer region"] [region="id: 7418340 start_key: 6F7373E706466 region_epoch { conf_ver: 66617 version: 116 } peers { id: 7444547 store_id: 1 } peers { id: 7444713 store_id: 8 } peers { id: 7444964 store_id: 3 }"] [peer_id=7444547] [region_id=7418340]
[2025/09/06 04:16:39.512 +08:00] [INFO] [store.rs:600] ["raft message is stale, tell to gc"] [msg_type=MsgAppendResponse] [current_region_epoch="conf_ver: 66617 version: 116"] [region_id=7418340]
[2025/09/06 04:16:40.001 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=297358659] [region_id=7217630]
[2025/09/06 04:16:40.215 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138592] [size=100663218] [region_id=7217630]
[2025/09/06 04:16:40.417 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=302007423] [region_id=7245948]
[2025/09/06 04:16:40.611 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138462] [size=100663069] [region_id=7245948]
[2025/09/06 04:16:41.376 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=375717304] [region_id=7217857]
[2025/09/06 04:16:41.609 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=137515] [size=100663052] [region_id=7217857]
[2025/09/06 04:16:42.259 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=402520117] [region_id=7254885]
[2025/09/06 04:16:42.479 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138306] [size=100662711] [region_id=7254885]
[2025/09/06 04:16:42.665 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7443018] [request_index=0] [peer_id=7440997] [region_id=7395864]
[2025/09/06 04:16:42.790 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:42.799 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=303558133] [region_id=7371381]
[2025/09/06 04:16:42.826 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data0/tikv-data/tikv-20160/snap/rev_7271280_149_2854167_(default|lock|write).sst] [snap_key=7271280_149_2854167]
[2025/09/06 04:16:42.827 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=36.718632ms] [snap_key=7271280_149_2854167]
[2025/09/06 04:16:42.827 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:42.833 +08:00] [INFO] [raft_log.rs:627] ["log [committed=2853761, persisted=2853761, applied=2853761, unstable.offset=2853762, unstable.entries.len()=0] starts to restore snapshot [index: 2854167, term: 149]"] [snapshot_term=149] [snapshot_index=2854167] [log="committed=2853761, persisted=2853761, applied=2853761, unstable.offset=2853762, unstable.entries.len()=0"] [raft_id=7444754] [region_id=7271280]
[2025/09/06 04:16:42.833 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7444706, 7444754, 7442849} }, outgoing: Configuration { voters: {} } }, learners: {7444977}, learners_next: {}, auto_leave: false }"] [raft_id=7444754] [region_id=7271280]
[2025/09/06 04:16:42.833 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=149] [snapshot_index=2854167] [last_term=149] [last_index=2854167] [commit=2854167] [raft_id=7444754] [region_id=7271280]
[2025/09/06 04:16:42.833 +08:00] [INFO] [raft.rs:2511] ["[commit: 2854167, term: 149] restored snapshot [index: 2854167, term: 149]"] [snapshot_term=149] [snapshot_index=2854167] [commit=2854167] [term=149] [raft_id=7444754] [region_id=7271280]
[2025/09/06 04:16:42.833 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7444754] [region_id=7271280]
[2025/09/06 04:16:42.833 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=7.573µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7271280]
[2025/09/06 04:16:42.833 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 2854167 commit_index: 2853761 commit_term: 149 truncated_state { index: 2854167 term: 149 }"] [region="id: 7271280 start_key: 6F7373522E706466 region_epoch { conf_ver: 65604 version: 80 } peers { id: 7442849 store_id: 8 } peers { id: 7444706 store_id: 14 } peers { id: 7444754 store_id: 1 } peers { id: 7444977 store_id: 9 role: Learner }"] [peer_id=7444754] [region_id=7271280]
[2025/09/06 04:16:42.834 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=149] [peer_id=7444754] [region_id=7271280]
[2025/09/06 04:16:42.834 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7271280 start_key: 6F73735F6F62622E706466 region_epoch { conf_ver: 65604 version: 80 } peers { id: 7442849 store_id: 8 } peers { id: 7444706 store_id: 14 } peers { id: 7444754 store_id: 1 } peers { id: 7444977 store_id: 9 role: Learner }"] [peer_id=7444754] [region_id=7271280]
[2025/09/06 04:16:42.834 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7444754] [region_id=7271280]
[2025/09/06 04:16:42.834 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7271280 start_key: 6F73735F6F626AE706466 region_epoch { conf_ver: 65604 version: 80 } peers { id: 7442849 store_id: 8 } peers { id: 7444706 store_id: 14 } peers { id: 7444754 store_id: 1 } peers { id: 7444977 store_id: 9 role: Learner }"] [prev_region="id: 7271280 start_key: 6F7338322E706466 region_epoch { conf_ver: 65603 version: 80 } peers { id: 7442849 store_id: 8 } peers { id: 7444706 store_id: 14 } peers { id: 7444754 store_id: 1 }"] [peer_id=7444754] [region_id=7271280]
[2025/09/06 04:16:42.853 +08:00] [INFO] [peer_storage.rs:520] ["requesting snapshot"] [request_peer=7444618] [request_index=0] [peer_id=7443256] [region_id=7371381]
[2025/09/06 04:16:42.980 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7271280]
[2025/09/06 04:16:42.984 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138033] [size=100662641] [region_id=7371381]
[2025/09/06 04:16:43.210 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:43.306 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=93.859727ms] [region_id=7271280]
[2025/09/06 04:16:43.463 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=348432046] [region_id=7249279]
[2025/09/06 04:16:43.649 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138547] [size=100662740] [region_id=7249279]
[2025/09/06 04:16:43.709 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=306175371] [region_id=7223053]
[2025/09/06 04:16:43.960 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128434] [size=100663083] [region_id=7223053]
[2025/09/06 04:16:44.080 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=401039943] [region_id=7219307]
[2025/09/06 04:16:44.124 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138075, total size 100662858. raw_size_per_file 104857600, total takes 1.458732311s"]
[2025/09/06 04:16:44.152 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662858] [key_count=138075] [cf=default] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7395864_22_543799_(default|lock|write).sst] [region_id=7395864]
[2025/09/06 04:16:44.153 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7395864_22_543799_(default|lock|write).sst] [region_id=7395864]
[2025/09/06 04:16:44.153 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7395864_22_543799_(default|lock|write).sst] [region_id=7395864]
[2025/09/06 04:16:44.158 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=1.492381922s] [size=6750505] [key_count=138075] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7395864_22_543799_(default|lock|write).sst] [region_id=7395864]
[2025/09/06 04:16:44.242 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:44.287 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=43.61614ms] [size=6750505] [snap_key=7395864_22_543799] [region_id=7395864]
[2025/09/06 04:16:44.332 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129018] [size=100663043] [region_id=7219307]
[2025/09/06 04:16:44.372 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7443018 store_id: 9"] [peer_id=7440997] [region_id=7395864]
[2025/09/06 04:16:44.693 +08:00] [INFO] [io.rs:184] ["build_sst_cf_file_list builds 1 files in cf default. Total keys 138033, total size 100662641. raw_size_per_file 104857600, total takes 1.386149444s"]
[2025/09/06 04:16:44.716 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=100662641] [key_count=138033] [cf=default] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7371381_35_903621_(default|lock|write).sst] [region_id=7371381]
[2025/09/06 04:16:44.716 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7371381_35_903621_(default|lock|write).sst] [region_id=7371381]
[2025/09/06 04:16:44.717 +08:00] [INFO] [snap.rs:916] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7371381_35_903621_(default|lock|write).sst] [region_id=7371381]
[2025/09/06 04:16:44.724 +08:00] [INFO] [snap.rs:1055] ["scan snapshot"] [takes=1.417065615s] [size=6616579] [key_count=138033] [snapshot=/data0/tikv-data/tikv-20160/snap/gen_7371381_35_903621_(default|lock|write).sst] [region_id=7371381]
[2025/09/06 04:16:44.724 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:44.782 +08:00] [INFO] [snap.rs:452] ["sent snapshot"] [duration=55.386816ms] [size=6616579] [snap_key=7371381_35_903621] [region_id=7371381]
[2025/09/06 04:16:44.782 +08:00] [INFO] [peer.rs:1921] ["report snapshot status"] [status=Finish] [to="id: 7444618 store_id: 9"] [peer_id=7443256] [region_id=7371381]
[2025/09/06 04:16:44.961 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:44.990 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data0/tikv-data/tikv-20160/snap/rev_7292992_98_2563134_(default|lock|write).sst] [snap_key=7292992_98_2563134]
[2025/09/06 04:16:44.996 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=34.781697ms] [snap_key=7292992_98_2563134]
[2025/09/06 04:16:44.996 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:45.001 +08:00] [INFO] [raft_log.rs:627] ["log [committed=2562652, persisted=2562652, applied=2562652, unstable.offset=2562653, unstable.entries.len()=0] starts to restore snapshot [index: 2563134, term: 98]"] [snapshot_term=98] [snapshot_index=2563134] [log="committed=2562652, persisted=2562652, applied=2562652, unstable.offset=2562653, unstable.entries.len()=0"] [raft_id=7443469] [region_id=7292992]
[2025/09/06 04:16:45.001 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7443469, 7439234, 7442075} }, outgoing: Configuration { voters: {} } }, learners: {}, learners_next: {}, auto_leave: false }"] [raft_id=7443469] [region_id=7292992]
[2025/09/06 04:16:45.001 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=98] [snapshot_index=2563134] [last_term=98] [last_index=2563134] [commit=2563134] [raft_id=7443469] [region_id=7292992]
[2025/09/06 04:16:45.001 +08:00] [INFO] [raft.rs:2511] ["[commit: 2563134, term: 98] restored snapshot [index: 2563134, term: 98]"] [snapshot_term=98] [snapshot_index=2563134] [commit=2563134] [term=98] [raft_id=7443469] [region_id=7292992]
[2025/09/06 04:16:45.001 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7443469] [region_id=7292992]
[2025/09/06 04:16:45.001 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=7.613µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7292992]
[2025/09/06 04:16:45.001 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 2563134 commit_index: 2562652 commit_term: 98 truncated_state { index: 2563134 term: 98 }"] [region="id: 7292992 start_key: 6F73735F6F62E706466 region_epoch { conf_ver: 65183 version: 84 } peers { id: 7439234 store_id: 8 } peers { id: 7442075 store_id: 14 } peers { id: 7443469 store_id: 1 }"] [peer_id=7443469] [region_id=7292992]
[2025/09/06 04:16:45.007 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7292992]
[2025/09/06 04:16:45.007 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7292992 start_key: 6F73735F6F706466 region_epoch { conf_ver: 65183 version: 84 } peers { id: 7439234 store_id: 8 } peers { id: 7442075 store_id: 14 } peers { id: 7443469 store_id: 1 }"] [peer_id=7443469] [region_id=7292992]
[2025/09/06 04:16:45.007 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7443469] [region_id=7292992]
[2025/09/06 04:16:45.007 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7292992 start_key: 6F73735F6F626A322E706466 region_epoch { conf_ver: 65183 version: 84 } peers { id: 7439234 store_id: 8 } peers { id: 7442075 store_id: 14 } peers { id: 7443469 store_id: 1 }"] [prev_region="id: 7292992 start_key: 6F73735F6FE706466 region_epoch { conf_ver: 65183 version: 84 } peers { id: 7439234 store_id: 8 } peers { id: 7442075 store_id: 14 } peers { id: 7443469 store_id: 1 }"] [peer_id=7443469] [region_id=7292992]
[2025/09/06 04:16:45.008 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=98] [peer_id=7443469] [region_id=7292992]
[2025/09/06 04:16:45.249 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:45.326 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=75.748613ms] [region_id=7292992]
[2025/09/06 04:16:45.415 +08:00] [ERROR] [entry_storage.rs:1122] ["entries are fetched unexpectedly during warming up"]
[2025/09/06 04:16:45.715 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301986081] [region_id=7221274]
[2025/09/06 04:16:45.930 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128586] [size=100662828] [region_id=7221274]
[2025/09/06 04:16:47.997 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:48.029 +08:00] [INFO] [snap.rs:263] ["saving snapshot file"] [file=/data0/tikv-data/tikv-20160/snap/rev_7219580_226_6809198_(default|lock|write).sst] [snap_key=7219580_226_6809198]
[2025/09/06 04:16:48.041 +08:00] [INFO] [snap.rs:271] ["saving all snapshot files"] [takes=43.430833ms] [snap_key=7219580_226_6809198]
[2025/09/06 04:16:48.041 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:48.048 +08:00] [INFO] [raft_log.rs:627] ["log [committed=6808665, persisted=6808665, applied=6808665, unstable.offset=6808666, unstable.entries.len()=0] starts to restore snapshot [index: 6809198, term: 226]"] [snapshot_term=226] [snapshot_index=6809198] [log="committed=6808665, persisted=6808665, applied=6808665, unstable.offset=6808666, unstable.entries.len()=0"] [raft_id=7443228] [region_id=7219580]
[2025/09/06 04:16:48.048 +08:00] [INFO] [raft.rs:2646] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {7441677, 7429386, 7443228} }, outgoing: Configuration { voters: {} } }, learners: {7444983}, learners_next: {}, auto_leave: false }"] [raft_id=7443228] [region_id=7219580]
[2025/09/06 04:16:48.048 +08:00] [INFO] [raft.rs:2630] ["restored snapshot"] [snapshot_term=226] [snapshot_index=6809198] [last_term=226] [last_index=6809198] [commit=6809198] [raft_id=7443228] [region_id=7219580]
[2025/09/06 04:16:48.048 +08:00] [INFO] [raft.rs:2511] ["[commit: 6809198, term: 226] restored snapshot [index: 6809198, term: 226]"] [snapshot_term=226] [snapshot_index=6809198] [commit=6809198] [term=226] [raft_id=7443228] [region_id=7219580]
[2025/09/06 04:16:48.048 +08:00] [INFO] [peer_storage.rs:578] ["begin to apply snapshot"] [peer_id=7443228] [region_id=7219580]
[2025/09/06 04:16:48.048 +08:00] [INFO] [peer_storage.rs:1018] ["finish clear peer meta"] [takes=8.918µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=7219580]
[2025/09/06 04:16:48.048 +08:00] [INFO] [peer_storage.rs:661] ["apply snapshot with state ok"] [state="applied_index: 6809198 commit_index: 6808665 commit_term: 226 truncated_state { index: 6809198 term: 226 }"] [region="id: 7219580 start_key: 6F73735F6F6383037373034395F313638383633353334353932392E6A7067 region_epoch { conf_ver: 68352 version: 60 } peers { id: 7429386 store_id: 8 } peers { id: 7441677 store_id: 14 } peers { id: 7443228 store_id: 1 } peers { id: 7444983 store_id: 2 role: Learner }"] [peer_id=7443228] [region_id=7219580]
[2025/09/06 04:16:48.051 +08:00] [INFO] [apply.rs:3583] ["re-register to apply delegates"] [term=226] [peer_id=7443228] [region_id=7219580]
[2025/09/06 04:16:48.051 +08:00] [INFO] [peer.rs:4734] ["snapshot is persisted"] [region="id: 7219580 start_key: 6F73735F6F62638383633353334353932392E6A7067 region_epoch { conf_ver: 68352 version: 60 } peers { id: 7429386 store_id: 8 } peers { id: 7441677 store_id: 14 } peers { id: 7443228 store_id: 1 } peers { id: 7444983 store_id: 2 role: Learner }"] [peer_id=7443228] [region_id=7219580]
[2025/09/06 04:16:48.051 +08:00] [INFO] [region_info_accessor.rs:294] ["trying to create region but it already exists, try to update it"] [region_id=7219580]
[2025/09/06 04:16:48.051 +08:00] [INFO] [region.rs:441] ["begin apply snap data"] [peer_id=7443228] [region_id=7219580]
[2025/09/06 04:16:48.051 +08:00] [INFO] [peer.rs:4793] ["region changed after persisting snapshot"] [region="id: 7219580 start_key: 6F73735F6F626A5F392E6A7067 region_epoch { conf_ver: 68352 version: 60 } peers { id: 7429386 store_id: 8 } peers { id: 7441677 store_id: 14 } peers { id: 7443228 store_id: 1 } peers { id: 7444983 store_id: 2 role: Learner }"] [prev_region="id: 7219580 start_key: 6F737356A7067 region_epoch { conf_ver: 68351 version: 60 } peers { id: 7429386 store_id: 8 } peers { id: 7441677 store_id: 14 } peers { id: 7443228 store_id: 1 }"] [peer_id=7443228] [region_id=7219580]
[2025/09/06 04:16:48.210 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=297805076] [region_id=7219992]
[2025/09/06 04:16:48.318 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:16:48.382 +08:00] [INFO] [region.rs:488] ["apply new data"] [time_takes=61.510245ms] [region_id=7219580]
[2025/09/06 04:16:48.420 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128834] [size=100663261] [region_id=7219992]
[2025/09/06 04:16:49.226 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=209704064] [region_id=7395864]
[2025/09/06 04:16:49.322 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138075] [size=100662858] [region_id=7395864]
[2025/09/06 04:16:50.144 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=258955770] [region_id=7282292]
[2025/09/06 04:16:50.321 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138263] [size=100662797] [region_id=7282292]
[2025/09/06 04:16:50.418 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=197141375] [region_id=7245948]
[2025/09/06 04:16:50.559 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138462] [size=100663069] [region_id=7245948]
[2025/09/06 04:16:51.377 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301990002] [region_id=7217857]
[2025/09/06 04:16:51.536 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=137515] [size=100663052] [region_id=7217857]
[2025/09/06 04:16:52.260 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301988259] [region_id=7254885]
[2025/09/06 04:16:52.415 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138306] [size=100662711] [region_id=7254885]
[2025/09/06 04:16:52.800 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201326012] [region_id=7371381]
[2025/09/06 04:16:52.924 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138033] [size=100662641] [region_id=7371381]
[2025/09/06 04:16:53.472 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301988220] [region_id=7249279]
[2025/09/06 04:16:53.628 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138547] [size=100662740] [region_id=7249279]
[2025/09/06 04:16:53.710 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=205513087] [region_id=7223053]
[2025/09/06 04:16:53.871 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128434] [size=100663083] [region_id=7223053]
[2025/09/06 04:16:53.915 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=298787499] [region_id=7344862]
[2025/09/06 04:16:54.063 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=137975] [size=100663028] [region_id=7344862]
[2025/09/06 04:16:54.081 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301989856] [region_id=7219307]
[2025/09/06 04:16:54.250 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=129018] [size=100663043] [region_id=7219307]
[2025/09/06 04:16:54.340 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=323929742] [region_id=7218353]
[2025/09/06 04:16:54.552 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=118445] [size=100662405] [region_id=7218353]
[2025/09/06 04:16:55.716 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201324054] [region_id=7221274]
[2025/09/06 04:16:55.877 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128586] [size=100662828] [region_id=7221274]
[2025/09/06 04:16:58.211 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201326504] [region_id=7219992]
[2025/09/06 04:16:58.368 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=128834] [size=100663261] [region_id=7219992]
[2025/09/06 04:17:00.145 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=197139730] [region_id=7282292]
[2025/09/06 04:17:00.286 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138263] [size=100662797] [region_id=7282292]
[2025/09/06 04:17:01.673 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=199042519] [region_id=7217630]
[2025/09/06 04:17:01.820 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138592] [size=100663218] [region_id=7217630]
[2025/09/06 04:17:04.130 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=350386254] [region_id=7355946]
[2025/09/06 04:17:04.295 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138047] [size=100662636] [region_id=7355946]
[2025/09/06 04:17:05.755 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=201326612] [region_id=7418340]
[2025/09/06 04:17:05.878 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138056] [size=100662941] [region_id=7418340]
[2025/09/06 04:17:08.218 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=301988908] [region_id=7265961]
[2025/09/06 04:17:08.374 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138441] [size=100663218] [region_id=7265961]
[2025/09/06 04:17:10.199 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=209703598] [region_id=7329113]
[2025/09/06 04:17:10.298 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138125] [size=100662685] [region_id=7329113]
[2025/09/06 04:17:11.381 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=351141506] [region_id=7243913]
[2025/09/06 04:17:11.580 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=153296] [size=116450859] [region_id=7243913]
[2025/09/06 04:17:14.179 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=308089474] [region_id=7385264]
[2025/09/06 04:17:14.341 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=138192] [size=100662644] [region_id=7385264]
[2025/09/06 04:17:24.024 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7219580_226_6808664] [peer_id=7443228] [region_id=7219580]
[2025/09/06 04:17:24.024 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=758573_332_8379364] [peer_id=7440316] [region_id=758573]
[2025/09/06 04:17:24.024 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7219580_226_6809198] [peer_id=7443228] [region_id=7219580]
[2025/09/06 04:17:24.024 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7257488_175_3069662] [peer_id=7443483] [region_id=7257488]
[2025/09/06 04:17:24.024 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7271280_149_2853651] [peer_id=7444754] [region_id=7271280]
[2025/09/06 04:17:24.024 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7292992_98_2563134] [peer_id=7443469] [region_id=7292992]
[2025/09/06 04:17:24.024 +08:00] [INFO] [peer.rs:1295] ["deleting applied snap file"] [snap_file=7271280_149_2854167] [peer_id=7444754] [region_id=7271280]
[2025/09/06 04:17:24.024 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:17:24.032 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:17:24.041 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:17:24.046 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:17:24.051 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:17:24.055 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:17:24.058 +08:00] [INFO] [snap.rs:670] ["set_snapshot_meta total cf files count: 3"]
[2025/09/06 04:18:54.042 +08:00] [INFO] [util.rs:598] ["connecting to PD endpoint"] [endpoints=http://10.25.0.107:2379]
[2025/09/06 04:19:15.024 +08:00] [INFO] [size.rs:202] ["Run size checker"] [policy=Scan] [threshold=150994944] [size=163853333] [region_id=5776555]
[2025/09/06 04:19:15.124 +08:00] [INFO] [split_check.rs:590] ["update approximate size and keys with accurate value"] [bucket_size=0] [bucket_count=0] [keys=55526] [size=81213586] [region_id=5776555]
[2025/09/06 04:19:38.232 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:19:38.232 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:19:38.232 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:19:38.232 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:19:44.371 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:19:44.371 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:19:44.371 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:19:44.371 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:22:30.588 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:22:30.588 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:22:30.588 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:22:30.588 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:22:30.639 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:22:30.639 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:22:30.639 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:22:30.639 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:24:57.805 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:24:57.805 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:24:57.805 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:24:57.805 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:28:54.045 +08:00] [INFO] [util.rs:598] ["connecting to PD endpoint"] [endpoints=http://10.25.0.107:2379]
[2025/09/06 04:30:31.513 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:31:14.131 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]
[2025/09/06 04:31:14.131 +08:00] [INFO] [kv.rs:853] ["kv rpc failed"] [err=RemoteStopped] [request=batch_commands]

比较新鲜的日志是这个：

[2025/09/06 04:16:45.415 +08:00] [ERROR] [entry_storage.rs:1122] ["entries are fetched unexpectedly during warming up"]

[jwongz]

综上所述，04:16 raw_put出现大量超时，磁盘吞吐瞬间下降， 04:30停止了元数据的导入，但是依然没有恢复tikv集群

[pingyu]

采集一下故障时间点前后各半小时的监控和日志吧。

参考 【SOP 系列 22】TiDB 集群诊断信息收集 Clinic 使用指南&资料大全

[我是火炎焱燚 (Ti D Ber 9 Dh P An J6)]

要不试下表妹说的，升级版本测试下。。。。。