TiDB-Lightning导入数据太慢

[lazzyfu]

背景

数据量：15TB左右
导入模式：local
tidb version：5.0.4

问题描述

在导入约12.6TB后，出现下面情况，且持续了3天，不清楚为什么这么慢，也看不到在处理什么数据

[2021/10/19 09:52:43.876 +08:00] [INFO] [restore.go:1051] [progress] [total=0.0%] [tables="2008/2009 (100.0%)"] [chunks="0/2 (0.0%)"] [engines="0/2 (0.0%)"] [speed(MiB/s)=16.62947533605344] [state=preparing] []
[2021/10/19 09:52:43.886 +08:00] [INFO] [pd.go:406] ["pause scheduler(configs)"] [name="[balance-region-scheduler,balance-leader-scheduler,balance-hot-region-scheduler]"] [cfg="{\"enable-location-replacement\":\"false\",\"leader-schedule-limit\":40,\"max-merge-region-keys\":0,\"max-merge-region-size\":0,\"max-pending-peer-count\":2147483647,\"max-snapshot-count\":40,\"region-schedule-limit\":40}"]
[2021/10/19 09:54:23.884 +08:00] [INFO] [pd.go:406] ["pause scheduler(configs)"] [name="[balance-region-scheduler,balance-leader-scheduler,balance-hot-region-scheduler]"] [cfg="{\"enable-location-replacement\":\"false\",\"leader-schedule-limit\":40,\"max-merge-region-keys\":0,\"max-merge-region-size\":0,\"max-pending-peer-count\":2147483647,\"max-snapshot-count\":40,\"region-schedule-limit\":40}"]

请问如何解决？我重启导入任务，依然是这种情况

[yilong (yi888long)]

1. 请检查是否 lightning 导入节点 oom 了？
2. 可以尝试修改配置参数，降低参数值。
   index-concurrency = 1 table-concurrency = 1 region-concurrency =5