tidb-lightning 3.0.5；tidb 4.0.0-rc，执行 switch-mode 失败。

zhengjunbo · 2020 年5 月 25 日 02:23

为提高效率，提问时请提供以下信息，问题描述清晰可优先响应。

【TiDB 版本】：4.0.0-RC
【问题描述】：lighting失败之后按官网重置命令报错

直接使用官网命令bin/tidb-lightning-ctl -switch-mode=normal 没有配置

加上配置bin/tidb-lightning-ctl -config conf/tidb-lightning.toml -switch-mode=normal

有没有更加完整的命令呢。

yilong · 2020 年5 月 25 日 02:44

请执行./tidb-lightning -V 查看使用的版本是否和集群版本匹配

zhengjunbo · 2020 年5 月 25 日 03:17

感谢，发现lightning的版本比较低导致

来了老弟 · 2020 年5 月 25 日 03:20

，有问题辛苦开新帖继续讨论

zhengjunbo · 2020 年5 月 25 日 03:26

刚测了下，依然是这个问题，应该是我在跑导入数据导致没返回值一直卡着

来了老弟 · 2020 年5 月 25 日 04:07

你好，

描述下目前是个什么状态，switch-mode=normal 是在导入失败，tikv 没有正确交换 mode 才会执行。

mode 可以通过 grafana - lightning 查看下当前值为多少。

yilong · 2020 年5 月 25 日 04:14

之前导入的数据使用的 lightning 3.0.5 对吧？
当前数据如果还没有导入完成，能否尝试清理数据，重新使用匹配的lightning工具导入，多谢

zhengjunbo · 2020 年5 月 25 日 05:45

之前通过ightning 3.0.5 导入成功过一次，后面由于BR也出现问题，又删了库重新导。这次同样用这个lightning继续导就有问题了。

zhengjunbo · 2020 年5 月 25 日 05:53

这个是手动启动的lightning没啥监控

zhengjunbo · 2020 年5 月 25 日 05:56

tidb-lightning的日志发现有switch mode failed 目前在采用loader的方式导入，发现速度很慢，在怀疑是不是mode没对

[2020/05/24 00:08:13.711 +08:00] [INFO] [backend.go:283] ["import completed"] [engineTag=`pay`.`pay_sys_bill`:-1] [engineUUID=0b5f48e9-bece-5df9-b8d1-de9ddff30bea] [retryCnt=0] [takeTime=6m8.9239588s] []
[2020/05/24 00:08:13.711 +08:00] [INFO] [backend.go:295] ["cleanup start"] [engineTag=`pay`.`pay_sys_bill`:-1] [engineUUID=0b5f48e9-bece-5df9-b8d1-de9ddff30bea]
[2020/05/24 00:08:34.835 +08:00] [INFO] [backend.go:297] ["cleanup completed"] [engineTag=`pay`.`pay_sys_bill`:-1] [engineUUID=0b5f48e9-bece-5df9-b8d1-de9ddff30bea] [takeTime=21.12412104s] []
[2020/05/24 00:08:34.835 +08:00] [INFO] [restore.go:1355] ["import and cleanup engine completed"] [engineTag=`pay`.`pay_sys_bill`:-1] [engineUUID=0b5f48e9-bece-5df9-b8d1-de9ddff30bea] [takeTime=6m30.048399318s] []
[2020/05/24 00:08:34.837 +08:00] [INFO] [tidb.go:232] ["alter table auto_increment start"] [table=`pay`.`pay_sys_bill`] [auto_increment=347017291]
[2020/05/24 00:08:34.912 +08:00] [INFO] [tidb.go:234] ["alter table auto_increment completed"] [table=`pay`.`pay_sys_bill`] [auto_increment=347017291] [takeTime=74.821501ms] []
[2020/05/24 00:08:34.912 +08:00] [INFO] [restore.go:958] ["local checksum"] [table=`pay`.`pay_sys_bill`] [checksum="{cksum=12816832026587917942,size=226136567317,kvs=2070449983}"]
[2020/05/24 00:08:34.916 +08:00] [INFO] [restore.go:1431] ["remote checksum start"] [table=`pay`.`pay_sys_bill`]
[2020/05/24 00:09:05.367 +08:00] [WARN] [tikv.go:142] ["switch mode failed"] [mode=Import] [takeTime=5.01451ms] [error="rpc error: code = Unimplemented desc = "]
[2020/05/24 00:09:05.367 +08:00] [INFO] [restore.go:473] [progress] [files="1097/6227 (17.6%)"] [tables="46/51 (90.2%)"] [speed(MiB/s)=41.860691939228516] [state=writing] [remaining=8h11m1s]
[2020/05/24 00:09:41.309 +08:00] [INFO] [restore.go:1445] ["remote checksum completed"] [table=`pay`.`pay_sys_bill`] [takeTime=1m6.393107742s] []
[2020/05/24 00:09:41.313 +08:00] [INFO] [restore.go:1385] ["checksum pass"] [table=`pay`.`pay_sys_bill`] [local="{cksum=12816832026587917942,size=226136567317,kvs=2070449983}"]
[2020/05/24 00:09:41.313 +08:00] [INFO] [restore.go:975] ["skip analyze"] [table=`pay`.`pay_sys_bill`]
[2020/05/24 00:09:41.313 +08:00] [INFO] [restore.go:569] ["restore table completed"] [table=`pay`.`pay_sys_bill`] [takeTime=1h14m26.92234743s] []
[2020/05/24 00:09:41.313 +08:00] [ERROR] [restore.go:615] ["restore all tables data failed"] [takeTime=1h45m35.955145865s] [error="[`pay`.`rep_deposit_bill`] write rows reach max retry 3 and still failed: rpc error: code = Unknown desc = RocksDB(\"IO error: While open a file for appending: /data32/ssd/data.import/.temp/2f633eef-155f-512d-8b26-407861700978/000034.sst: No such file or directory\")"]
[2020/05/24 00:09:41.313 +08:00] [INFO] [restore.go:442] ["everything imported, stopping periodic actions"]
[2020/05/24 00:09:41.313 +08:00] [ERROR] [restore.go:260] ["run failed"] [step=2] [error="[`pay`.`rep_deposit_bill`] write rows reach max retry 3 and still failed: rpc error: code = Unknown desc = RocksDB(\"IO error: While open a file for appending: /data32/ssd/data.import/.temp/2f633eef-155f-512d-8b26-407861700978/000034.sst: No such file or directory\")"]
[2020/05/24 00:09:41.314 +08:00] [ERROR] [restore.go:266] ["the whole procedure failed"] [takeTime=1h45m40.133596638s] [error="[`pay`.`rep_deposit_bill`] write rows reach max retry 3 and still failed: rpc error: code = Unknown desc = RocksDB(\"IO error: While open a file for appending: /data32/ssd/data.import/.temp/2f633eef-155f-512d-8b26-407861700978/000034.sst: No such file or directory\")"]
[2020/05/24 00:09:41.314 +08:00] [ERROR] [restore.go:115] ["tables failed to be imported"] [count=1]
[2020/05/24 00:09:41.314 +08:00] [ERROR] [restore.go:117] [-] [table=`pay`.`rep_deposit_bill`] [status=written] [error="[`pay`.`rep_deposit_bill`] write rows reach max retry 3 and still failed: rpc error: code = Unknown desc = RocksDB(\"IO error: While open a file for appending: /data32/ssd/data.import/.temp/2f633eef-155f-512d-8b26-407861700978/000034.sst: No such file or directory\")"]
[2020/05/24 00:09:41.314 +08:00] [ERROR] [main.go:59] ["tidb lightning encountered error"] [error="[`pay`.`rep_deposit_bill`] write rows reach max retry 3 and still failed: rpc error: code = Unknown desc = RocksDB(\"IO error: While open a file for appending: /data32/ssd/data.import/.temp/2f633eef-155f-512d-8b26-407861700978/000034.sst: No such file or directory\")"] [errorVerbose="rpc error: code = Unknown desc = RocksDB(\"IO error: While open a file for appending: /data32/ssd/data.import/.temp/2f633eef-155f-512d-8b26-407861700978/000034.sst: No such file or directory\")\
github.com/pingcap/errors.AddStack\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/pkg/mod/github.com/pingcap/errors@v0.11.4/errors.go:174\
github.com/pingcap/errors.Trace\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/pkg/mod/github.com/pingcap/errors@v0.11.4/juju_adaptor.go:15\
github.com/pingcap/tidb-lightning/lightning/backend.(*importer).WriteRows.func1\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-lightning/lightning/backend/importer.go:170\
github.com/pingcap/tidb-lightning/lightning/backend.(*importer).WriteRows\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-lightning/lightning/backend/importer.go:212\
github.com/pingcap/tidb-lightning/lightning/backend.(*OpenedEngine).WriteRows\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-lightning/lightning/backend/backend.go:225\
github.com/pingcap/tidb-lightning/lightning/restore.(*chunkRestore).deliverLoop\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:1556\
github.com/pingcap/tidb-lightning/lightning/restore.(*chunkRestore).restore.func2\
\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:1716\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1337\
[`pay`.`rep_deposit_bill`] write rows reach max retry 3 and still failed"]

yilong · 2020 年5 月 25 日 06:21

mode值可以查看Prometheus 表達式 min(tikv_config_rocksdb{name=“hard_pending_compaction_bytes_limit”}) by (instance) 0 = import mode, ≥1 = normal mode
导入报错的问题，稍后答复，多谢

zhengjunbo · 2020 年5 月 25 日 06:23

看起来是没问题的

WangXiangUSTC · 2020 年5 月 25 日 06:36

请问下是否使用了 TiFlash？以及 TiFlash 的版本号是多少？

zhengjunbo · 2020 年5 月 25 日 06:37

使用了TiFlash，TiFlash的版本号也是4.0.0.RC

WangXiangUSTC · 2020 年5 月 25 日 06:44

这是一个已知 bug，集群能否都升级到 4.0.0-rc.2，在这个版本里修复了这个问题

issue 链接：https://github.com/pingcap/tidb-lightning/issues/273

zhengjunbo · 2020 年5 月 25 日 06:45

好的，我试试

yilong · 2020 年5 月 25 日 06:48

有问题再继续反馈，多谢

system · 2022 年10 月 31 日 19:06

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。