tidb-lightning 导入数据

javacai · 2021 年8 月 11 日 06:03

[2021/08/11 14:02:22.670 +08:00] [WARN] [config.go:518] [“currently only per-task configuration can be applied, global configuration changes can only be made on startup”] [“global config changes”="[lightning.level,lightning.file]"]
[2021/08/11 14:02:22.671 +08:00] [INFO] [info.go:40] [“Welcome to TiDB-Lightning”] [release-version=v5.1.0] [git-hash=f2fcbfa5a9783e33ff44b696920d8bbe6a19eaa6] [git-branch=heads/refs/tags/v5.1.0] [go-version=go1.16.4] [utc-build-time=“2021-06-24 07:09:26”] [race-enabled=false]
[2021/08/11 14:02:22.671 +08:00] [INFO] [lightning.go:219] [cfg] [cfg="{“id”:1628661742671112366,“lightning”:{“table-concurrency”:6,“index-concurrency”:2,“region-concurrency”:8,“io-concurrency”:5,“check-requirements”:true,“meta-schema-name”:“lightning_metadata”},“tidb”:{“host”:“127.0.0.1”,“port”:4000,“user”:“root”,“status-port”:10080,“pd-addr”:“10.139.9.44:2379”,“sql-mode”:“ONLY_FULL_GROUP_BY,NO_AUTO_CREATE_USER”,“tls”:“false”,“security”:{“ca-path”:"",“cert-path”:"",“key-path”:"",“redact-info-log”:false},“max-allowed-packet”:67108864,“distsql-scan-concurrency”:15,“build-stats-concurrency”:20,“index-serial-scan-concurrency”:20,“checksum-table-concurrency”:2},“checkpoint”:{“schema”:“tidb_lightning_checkpoint”,“driver”:“file”,“enable”:true,“keep-after-success”:false},“mydumper”:{“read-block-size”:65536,“batch-size”:107374182400,“batch-import-ratio”:0,“data-source-dir”:“file:///data/restore/1”,“character-set”:“auto”,“csv”:{“separator”:",",“delimiter”:"\"",“null”:"\\N",“header”:true,“trim-last-separator”:false,“not-null”:false,“backslash-escape”:true},“max-region-size”:268435456,“filter”:[".","!mysql.","!sys.","!INFORMATION_SCHEMA.","!PERFORMANCE_SCHEMA.","!METRICS_SCHEMA.","!INSPECTION_SCHEMA."],“files”:null,“no-schema”:false,“case-sensitive”:false,“strict-format”:false,“default-file-rules”:true,“ignore-data-columns”:null},“tikv-importer”:{“addr”:"",“backend”:“local”,“on-duplicate”:“replace”,“max-kv-pairs”:4096,“send-kv-pairs”:32768,“region-split-size”:100663296,“sorted-kv-dir”:"/data/cqhtmp",“disk-quota”:951261667328,“range-concurrency”:16,“engine-mem-cache-size”:536870912,“local-writer-mem-cache-size”:134217728},“post-restore”:{“checksum”:“required”,“analyze”:“optional”,“level-1-compact”:false,“post-process-at-last”:true,“compact”:false},“cron”:{“switch-mode”:“5m0s”,“log-progress”:“5m0s”,“check-disk-quota”:“1m0s”},“routes”:null,“security”:{“ca-path”:"",“cert-path”:"",“key-path”:"",“redact-info-log”:false},“black-white-list”:{“do-tables”:null,“do-dbs”:null,“ignore-tables”:null,“ignore-dbs”:null}}"]
[2021/08/11 14:02:22.768 +08:00] [INFO] [lightning.go:282] [“load data source start”]
[2021/08/11 14:02:22.883 +08:00] [INFO] [loader.go:287] ["[loader] file is filtered by file router"] [path=metadata]
[2021/08/11 14:02:22.906 +08:00] [INFO] [lightning.go:285] [“load data source completed”] [takeTime=137.854107ms] []
[2021/08/11 14:02:22.908 +08:00] [ERROR] [lightning.go:307] [“restore failed”] [error=“config ‘mydumper.sorted-kv-dir’ value ‘/data/cqhtmp’ different from checkpoint value ‘/data/restore/tmp’. You may set ‘check-requirements = false’ to skip this check or delete the file ‘/tmp/tidb_lightning_checkpoint.pb’ and remove all restored tables and try again”]
[2021/08/11 14:02:22.908 +08:00] [ERROR] [main.go:90] [“tidb lightning encountered error stack info”] [error=“config ‘mydumper.sorted-kv-dir’ value ‘/data/cqhtmp’ different from checkpoint value ‘/data/restore/tmp’. You may set ‘check-requirements = false’ to skip this check or delete the file ‘/tmp/tidb_lightning_checkpoint.pb’ and remove all restored tables and try again”] [errorVerbose=“config ‘mydumper.sorted-kv-dir’ value ‘/data/cqhtmp’ different from checkpoint value ‘/data/restore/tmp’. You may set ‘check-requirements = false’ to skip this check or delete the file ‘/tmp/tidb_lightning_checkpoint.pb’ and remove all restored tables and try again\ngithub.com/pingcap/br/pkg/lightning/restore.verifyCheckpoint\ \tgithub.com/pingcap/br/pkg/lightning/restore/restore.go:758\ github.com/pingcap/br/pkg/lightning/restore.NewRestoreControllerWithPauser\ \tgithub.com/pingcap/br/pkg/lightning/restore/restore.go:245\ github.com/pingcap/br/pkg/lightning/restore.NewRestoreController\ \tgithub.com/pingcap/br/pkg/lightning/restore/restore.go:220\ github.com/pingcap/br/pkg/lightning.(*Lightning).run\ \tgithub.com/pingcap/br/pkg/lightning/lightning.go:305\ github.com/pingcap/br/pkg/lightning.(*Lightning).RunOnce\ \tgithub.com/pingcap/br/pkg/lightning/lightning.go:192\ main.main.func2\ \tcommand-line-arguments/main.go:86\ main.main\ \tcommand-line-arguments/main.go:87\ runtime.main\ \truntime/proc.go:225\ runtime.goexit\ \truntime/asm_amd64.s:1371”]
[2021/08/11 14:02:22.908 +08:00] [ERROR] [main.go:91] [“tidb lightning encountered error”] [error=“config ‘mydumper.sorted-kv-dir’ value ‘/data/cqhtmp’ different from checkpoint value ‘/data/restore/tmp’. You may set ‘check-requirements = false’ to skip this check or delete the file ‘/tmp/tidb_lightning_checkpoint.pb’ and remove all restored tables and try again”]

javacai · 2021 年8 月 11 日 06:03

这个是我没加 -check-requirements false 这个参数之前的日志，

songxuecheng · 2021 年8 月 11 日 06:04

/tmp/tidb_lightning_checkpoint.pb’这个文件删掉然后去掉参数重新执行

songxuecheng · 2021 年8 月 11 日 06:06

tidb-lightning-ctl --checkpoint-remove=all -config tidb-lightning.toml 这个也执行一下

javacai · 2021 年8 月 11 日 06:08

songxuecheng · 2021 年8 月 11 日 06:09

换一个不在同一目录或者加参数

songxuecheng · 2021 年8 月 11 日 06:10

不要在/data目录下

javacai · 2021 年8 月 11 日 06:14

换了一个目录；

songxuecheng · 2021 年8 月 11 日 06:15

ok 然后还有问题吗

javacai · 2021 年8 月 11 日 06:17

还没有动，是不是就代表程序在执行中呢。

songxuecheng · 2021 年8 月 11 日 06:17

看下日志

songxuecheng · 2021 年8 月 11 日 06:18

javacai · 2021 年8 月 11 日 06:20

嗯嗯，应该是好了，日志一直在滚动；都是INFO级的日志，非常感谢

system · 2022 年10 月 31 日 19:08

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。