环境.
3台机器, 每台都是40核, 128G, 3T ssd硬盘。 硬件性能应该不是问题.
需求:
1. 在mysql中通过dumpling 导出600G的逻辑文件（4000多张表）.
2. 使用lightning导入到tidb中. 无TiFlash
问题:
## linghtning 导入极慢.
1. 从日志中来看"region-concurrency":40 io-concurrency":5, “check-requirements”:true
不知道40 对于600G的数据是否够用，建议是给多少呢？
2. 从官方文档中可以看出每lightning一个表，都会checksum一下, 我不想校验，从日志看校验时间太长，请问如何关闭.
3. 从日志来看，基本都是单表循序导入, 能否有配置多线程，多张表同时导入, 请问怎么配置.
4. 以上都是我个人认为导入慢的原因，是否有大神能帮我看出其中的问题. 我给的信息只能这么多.
5. 日志中打印的lightning的配置信息. 是否可能是某些参数配置不对造成的？
cfg = {“id”:1639159374251311561,“lightning”:{“table-concurrency”:6,“index-concurrency”:2,“region-concurrency”:40,“io-concurrency”:5,“check-requirements”:true},“tidb”:{“host”:“127.0.0.1”,“port”:4001,“user”:“root”,“status-port”:10081,“pd-addr”:“127.0.0.1: 3379”,“sql-mode”:“ONLY_FULL_GROUP_BY,
NO_AUTO_CREATE_USER”,“tls”:“false”,“security”:{“ca-path”:"",“cert-path”:"",“key-path”:"",“redact-info-log”:false},“max-allowed-packet”:67108864,“distsql-scan-concurrency”:15,“build-stats-concurrency”:20,“index-serial-scan-concurrency”:20,“checksum-table-concurrency”:2},“checkpoint”:{“schema”:“tidb_lightning_checkpoint”,“driver”:“file”,“enable”:true,“keep-after-success”:false},“mydumper”:{“read-block-size”:65536,“batch-size”:107374182400,“batch-import-ratio”:0,“data-source-dir”:“file: ///alexyang/dumpling”,“character-set”:“auto”,“csv”:{“separator”:",
“,“delimiter”:”"",
“null”: “N”,
“header”: true,
“trim-last-separator”: false,
“not-null”: false,
“backslash-escape”: true
},
“max-region-size”: 268435456,
“filter”: [
“.”,
“!mysql.",
"!sys.”,
“!INFORMATION_SCHEMA.",
"!PERFORMANCE_SCHEMA.”,
“!METRICS_SCHEMA.",
"!INSPECTION_SCHEMA.”
],
“files”: null,
“no-schema”: false,
“case-sensitive”: false,
“strict-format”: false,
“default-file-rules”: true
},
“tikv-importer”: {
“addr”: “”,
“backend”: “local”,
“on-duplicate”: “replace”,
“max-kv-pairs”: 4096,
“send-kv-pairs”: 32768,
“region-split-size”: 100663296,
“sorted-kv-dir”: “/alexyang/sorted-kv-dir”,
“disk-quota”: 140892694380544,
“range-concurrency”: 16,
“engine-mem-cache-size”: 536870912,
“local-writer-mem-cache-size”: 134217728
},
“post-restore”: {
“checksum”: “required”,
“analyze”: “optional”,
“level-1-compact”: false,
“post-process-at-last”: true,
“compact”: false
},
“cron”: {
“switch-mode”: “5m0s”,
“log-progress”: “5m0s”,
“check-disk-quota”: “0s”
},
“routes”: null,
“security”: {
“ca-path”: “”,
“cert-path”: “”,
“key-path”: “”,
“redact-info-log”: false
},
“black-white-list”: {
“do-tables”: null,
“do-dbs”: null,
“ignore-tables”: null,
“ignore-dbs”: null
}
}