【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】v7.1.0
【复现路径】做过哪些操作出现的问题
【遇到的问题：问题现象及影响】从hdfs上get 128GB左右大小文件导入到tidb，花了6个多小时，请教下大佬们是哪出现了问题
【资源配置】
TIKV:8 (16 vCore) 64g
pd:2 (4 vCore) 16g
TIDB:8 (16 vCore) 64g
【附件：截图/日志/监控】
配置参数：
[lightning]
check-requirements = true
#index-concurrency = 4
#table-concurrency = 8
#region-concurrency = 32
level = “info”
file = “/home/hive/data/cdp_lightning_logs”
max-size = 256 # MB 日志文件大小
max-days = 28
#io-concurrency = 5
max-error = 0
meta-schema-name = “lightning_metadata”
[tikv-importer]
backend = “local”
incremental-import = true
sorted-kv-dir = “/home/hive/data/cdp_lightning_kv”
#range-concurrency = 16
#send-kv-pairs = 98304 #32768
on-duplicate = “replace”
duplicate-resolution = “remove”
compress-kv-pairs = “gz”
[mydumper]
#read-block-size = “256MiB” # 默认值
no-schema = true
取值范围为(0 <= batch-import-ratio < 1)。
batch-import-ratio = 0.75
data-source-dir = “/home/hive/data/cdp_lightning_data”
character-set = “auto”
data-character-set = “binary”
data-invalid-char-replace = “uFFFD”
strict-format = true
max-region-size = “256MiB” # 默认值
[checkpoint]
enable = true
[post-restore]
checksum = “false”
analyze = “false”
[cron]
TiDB Lightning 自动
switch-mode = “5m”
在日志中打印导入进度
log-progress = “5m”
