tidb版本是4.0
tidb-toolkit-v4.0.7-linux-amd64
我的lightning和tidb不在一台机器上面
tidb@tidb4ser:/tidb/tidb-toolkit-v4.0.7-linux-amd64/bin$ ./tidb-lightning-ctl --checkpoint-remove=all --tidb-host=10.3.87.30
: mydumper dir does not exist
github.com/pingcap/tidb-lightning/lightning/config.(*Config).Adjust
/home/jenkins/agent/workspace/ld_lightning_multi_branch_v4.0.7/go/src/github.com/pingcap/tidb-lightning/lightning/config/config.go:579
main.run
/home/jenkins/agent/workspace/ld_lightning_multi_branch_v4.0.7/go/src/github.com/pingcap/tidb-lightning/cmd/tidb-lightning-ctl/main.go:71
main.main
/home/jenkins/agent/workspace/ld_lightning_multi_branch_v4.0.7/go/src/github.com/pingcap/tidb-lightning/cmd/tidb-lightning-ctl/main.go:36
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
tidb@tidb4ser:/tidb/tidb-toolkit-v4.0.7-linux-amd64/bin$ cat tidb-lightning.log
[2020/10/19 10:10:34.911 +08:00] [WARN] [config.go:376] ["currently only per-task configuration can be applied, global configuration changes can only be made on startup"] ["global config changes"="[lightning.level,lightning.file]"]
[2020/10/19 10:10:34.912 +08:00] [INFO] [version.go:49] ["Welcome to lightning"] ["Release Version"=v4.0.7] ["Git Commit Hash"=e4de23b08259514ca20797696b6cd021a6a6de1a] ["Git Branch"=heads/refs/tags/v4.0.7] ["UTC Build Time"="2020-09-29 06:52:27"] ["Go Version"="go version go1.13 linux/amd64"]
[2020/10/19 10:10:34.912 +08:00] [INFO] [lightning.go:181] [cfg] [cfg="{\"id\":1603073434912357400,\"lightning\":{\"table-concurrency\":6,\"index-concurrency\":2,\"region-concurrency\":48,\"io-concurrency\":5,\"check-requirements\":true},\"tidb\":{\"host\":\"10.3.87.30\",\"port\":4000,\"user\":\"root\",\"status-port\":10080,\"pd-addr\":\"10.3.87.30:2379\",\"sql-mode\":\"ONLY_FULL_GROUP_BY,NO_AUTO_CREATE_USER\",\"tls\":\"false\",\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\"},\"max-allowed-packet\":67108864,\"distsql-scan-concurrency\":100,\"build-stats-concurrency\":20,\"index-serial-scan-concurrency\":20,\"checksum-table-concurrency\":16},\"checkpoint\":{\"enable\":true,\"schema\":\"tidb_lightning_checkpoint\",\"driver\":\"file\",\"keep-after-success\":false},\"mydumper\":{\"read-block-size\":65536,\"batch-size\":10737418240,\"batch-import-ratio\":0,\"data-source-dir\":\"file:///tidb/bk\",\"no-schema\":false,\"character-set\":\"auto\",\"csv\":{\"separator\":\",\",\"delimiter\":\"\\\"\",\"header\":true,\"trim-last-separator\":false,\"not-null\":false,\"null\":\"\\\\N\",\"backslash-escape\":true},\"case-sensitive\":false,\"strict-format\":false,\"max-region-size\":268435456,\"filter\":[\"*.*\"],\"files\":null,\"default-file-rules\":true},\"tikv-importer\":{\"addr\":\"\",\"backend\":\"local\",\"on-duplicate\":\"replace\",\"max-kv-pairs\":32,\"send-kv-pairs\":32768,\"region-split-size\":100663296,\"sorted-kv-dir\":\"/tmp\",\"range-concurrency\":16},\"post-restore\":{\"level-1-compact\":false,\"compact\":false,\"checksum\":true,\"analyze\":true},\"cron\":{\"switch-mode\":\"5m0s\",\"log-progress\":\"5m0s\"},\"routes\":null,\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\"},\"black-white-list\":{\"do-tables\":null,\"do-dbs\":null,\"ignore-tables\":null,\"ignore-dbs\":null}}"]
[2020/10/19 10:10:34.913 +08:00] [INFO] [lightning.go:221] ["load data source start"]
[2020/10/19 10:10:34.913 +08:00] [INFO] [lightning.go:224] ["load data source completed"] [takeTime=428.135µs] []
[2020/10/19 10:10:34.922 +08:00] [INFO] [restore.go:282] ["the whole procedure start"]
[2020/10/19 10:10:34.936 +08:00] [INFO] [restore.go:329] ["restore table schema start"] [db=mdb1]
[2020/10/19 10:10:34.937 +08:00] [INFO] [tidb.go:139] ["create tables start"] [db=mdb1]
[2020/10/19 10:10:34.944 +08:00] [INFO] [tidb.go:157] ["create tables completed"] [db=mdb1] [takeTime=6.159818ms] []
[2020/10/19 10:10:34.944 +08:00] [INFO] [restore.go:337] ["restore table schema completed"] [db=mdb1] [takeTime=7.868438ms] []
[2020/10/19 10:10:34.957 +08:00] [INFO] [restore.go:673] ["restore all tables data start"]
[2020/10/19 10:10:34.957 +08:00] [ERROR] [restore.go:771] ["TiDB Lightning has detected tables with illegal checkpoints. To prevent data mismatch, this run will stop now. Please remove these checkpoints first"] [count=1]
[2020/10/19 10:10:34.957 +08:00] [INFO] [restore.go:777] [-] [table=`mdb1`.`fd_supplysalesagency`] [recommendedAction="./tidb-lightning-ctl --checkpoint-remove='`mdb1`.`fd_supplysalesagency`' --config=..."]
[2020/10/19 10:10:34.957 +08:00] [INFO] [restore.go:783] ["You may also run `./tidb-lightning-ctl --checkpoint-remove=all --config=...` to start from scratch"]
[2020/10/19 10:10:34.957 +08:00] [ERROR] [restore.go:301] ["run failed"] [step=3] [error="TiDB Lightning has detected tables with illegal checkpoints; please remove these checkpoints first"]
[2020/10/19 10:10:34.957 +08:00] [ERROR] [restore.go:312] ["the whole procedure failed"] [takeTime=35.295362ms] [error="TiDB Lightning has detected tables with illegal checkpoints; please remove these checkpoints first"]
[2020/10/19 10:10:34.958 +08:00] [ERROR] [main.go:83] ["tidb lightning encountered error stack info"] [error="TiDB Lightning has detected tables with illegal checkpoints; please remove these checkpoints first"] [errorVerbose="TiDB Lightning has detected tables with illegal checkpoints; please remove these checkpoints first\
github.com/pingcap/tidb-lightning/lightning/restore.(*RestoreController).restoreTables\
\t/home/jenkins/agent/workspace/ld_lightning_multi_branch_v4.0.7/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:785\
github.com/pingcap/tidb-lightning/lightning/restore.(*RestoreController).Run\
\t/home/jenkins/agent/workspace/ld_lightning_multi_branch_v4.0.7/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:288\
github.com/pingcap/tidb-lightning/lightning.(*Lightning).run\
\t/home/jenkins/agent/workspace/ld_lightning_multi_branch_v4.0.7/go/src/github.com/pingcap/tidb-lightning/lightning/lightning.go:251\
github.com/pingcap/tidb-lightning/lightning.(*Lightning).RunOnce\
\t/home/jenkins/agent/workspace/ld_lightning_multi_branch_v4.0.7/go/src/github.com/pingcap/tidb-lightning/lightning/lightning.go:154\
main.main\
\t/home/jenkins/agent/workspace/ld_lightning_multi_branch_v4.0.7/go/src/github.com/pingcap/tidb-lightning/cmd/tidb-lightning/main.go:80\
runtime.main\
\t/usr/local/go/src/runtime/proc.go:203\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
[2020/10/19 10:10:34.958 +08:00] [ERROR] [main.go:84] ["tidb lightning encountered error"] [error="TiDB Lightning has detected tables with illegal checkpoints; please remove these checkpoints first"]
[lightning]
转换数据的并发数，默认为逻辑 CPU 数量，不需要配置。
混合部署的情况下可以配置为逻辑 CPU 的 75% 大小。
region-concurrency =
日志
level = “info”
file = “tidb-lightning.log”
[tikv-importer]
backend 设置为 local 模式
backend = “local”
设置本地临时存储路径
sorted-kv-dir = “/tmp”
[mydumper]
Mydumper 源数据目录。
data-source-dir = “/tidb/bk”
[tidb]
目标集群的信息。tidb-server 的监听地址，填一个即可。
host = “10.3.87.30”
port = 4000
user = “root”
password = “1234”
表架构信息在从 TiDB 的“状态端口”获取。
status-port = 10080
pd-server 的地址，填一个即可
pd-addr = “10.3.87.30:2379”
/tidb/bk这个目录我确定是存在的在linghtning这台机器上
不在tidb服务器上