[FAQ] tidb-lightning importer 模式导入数据失败， lightning-err 日志报错：403 Forbidden

🌌 运维指南 TiDB 常见 FAQ
1

【问题澄清】

  • tidb v3.0.15 & tools v3.0.15
  • 用户使用 tidb-lightning importer 模式导入数据失败，import log 中没有明显报错，仅为启动日志，lightning err log 中有如下报错信息：

[2020/06/23 10:50:14.318 +08:00] [INFO] [lightning.go:202] [“load data source start”]
[2020/06/23 10:50:14.318 +08:00] [INFO] [lightning.go:205] [“load data source completed”] [takeTime=222.867µs]
[2020/06/23 10:50:14.318 +08:00] [INFO] [checkpoints.go:770] [“open checkpoint file failed, going to create a new one”] [path=/tmp/tidb_lightning_checkpoint.pb] [error=“open /tmp/tidb_lightning_checkpoint.pb: no such file or directory”]
[2020/06/23 10:50:14.325 +08:00] [INFO] [restore.go:255] [“the whole procedure start”]
[2020/06/23 10:50:14.336 +08:00] [INFO] [restore.go:293] [“restore table schema start”] [db=demo]
[2020/06/23 10:50:14.419 +08:00] [INFO] [tidb.go:104] [“create tables start”] [db=demo]
[2020/06/23 10:50:14.506 +08:00] [INFO] [tidb.go:122] [“create tables completed”] [db=demo] [takeTime=87.126301ms]
[2020/06/23 10:50:14.506 +08:00] [INFO] [restore.go:301] [“restore table schema completed”] [db=demo] [takeTime=170.950998ms]
[2020/06/23 10:50:14.510 +08:00] [INFO] [restore.go:575] [“restore all tables data start”]
[2020/06/23 10:50:14.510 +08:00] [INFO] [restore.go:596] [“restore table start”] [table=demo.A]
[2020/06/23 10:50:14.510 +08:00] [INFO] [restore.go:1261] [“load engines and files start”] [table=demo.A]
[2020/06/23 10:50:14.510 +08:00] [INFO] [restore.go:1290] [“load engines and files completed”] [table=demo.A] [enginesCnt=2] [filesCnt=7] [takeTime=34.869µs]
[2020/06/23 10:50:14.511 +08:00] [ERROR] [restore.go:600] [“restore table failed”] [table=demo.A] [takeTime=564.429µs] [error="restore table demo.A failed: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing failed to do connect handshake, response: \"HTTP/1.1 403 Forbidden\\r\\
Content-Length: 3490\\r\\
Connection: keep-alive\\r\\
Content-Language: en\\r\\
Content-Type: text/html;charset=utf-8\\r\\
Date: Tue, 23 Jun 2020 02:50:14 GMT\\r\\
Mime-Version: 1.0\\r\\

【解决方案】

  • Nginx 代理配置错误，临时将其去掉解决。

【排查思路】

  • err log 中存在 403 Forbidden 类型报错并伴随 rpc error，一般为无法通信问题，可以从当前网络和链路上进行分析。
  • 如果使用 backend 模式，403 错误一般为无法连接到 TiKV，可以从此入手进行排查

【最佳案例】

【更多链接】