lightning进行迁移的日志的process并没有到100%，只到75%左右就停止了。无法立刻知道是否完整

zhengjunbo · 2020 年6 月 2 日 08:12

为提高效率，提问时请提供以下信息，问题描述清晰可优先响应。

【TiDB 版本】：4.0.0
【问题描述】：lightning 无法准确判断是否导入完整

反馈一个lightning的问题，目前通过process的进度无法判断是否导完整。晚点进行全量比对来确定数据是否完整。

这个版本默认开启了analyze table。但是pay_purchase没进行analyze，目前手工在跑

在lightning中的最后的日志

[2020/06/02 14:51:47.818 +08:00] [INFO] [restore.go:1357] ["import and cleanup engine completed"] [engineTag=`pay`.`pay_bill`:-1] [engineUUID=1bcc341d-07ce-5529-9aae-f00881124834] [takeTime=19m9.082370848s] []
[2020/06/02 14:51:47.820 +08:00] [INFO] [tidb.go:238] ["alter table auto_increment start"] [table=`pay`.`pay_bill`] [auto_increment=1]
[2020/06/02 14:51:48.832 +08:00] [INFO] [tidb.go:240] ["alter table auto_increment completed"] [table=`pay`.`pay_bill`] [auto_increment=1] [takeTime=1.011917343s] []
[2020/06/02 14:51:48.832 +08:00] [INFO] [restore.go:1058] ["local checksum"] [table=`pay`.`pay_bill`] [checksum="{cksum=2218256298104834937,size=1549389486622,kvs=17525022690}"]
[2020/06/02 14:51:48.839 +08:00] [INFO] [restore.go:1424] ["remote checksum start"] [table=`pay`.`pay_bill`]
[2020/06/02 14:56:11.193 +08:00] [INFO] [restore.go:503] [progress] [files="5109/6641 (76.9%)"] [tables="49/51 (96.1%)"] [speed(MiB/s)=34.04884868177228] [state=writing] [remaining=3h2m55s]
[2020/06/02 14:59:59.381 +08:00] [INFO] [restore.go:1438] ["remote checksum completed"] [table=`pay`.`pay_bill`] [takeTime=8m10.541698186s] []
[2020/06/02 14:59:59.387 +08:00] [INFO] [restore.go:1387] ["checksum pass"] [table=`pay`.`pay_bill`] [local="{cksum=2218256298104834937,size=1549389486622,kvs=17525022690}"]
[2020/06/02 14:59:59.387 +08:00] [INFO] [restore.go:1392] ["analyze start"] [table=`pay`.`pay_bill`]
[2020/06/02 15:01:11.193 +08:00] [INFO] [restore.go:503] [progress] [files="5109/6641 (76.9%)"] [tables="49/51 (96.1%)"] [speed(MiB/s)=33.772028705470156] [state=writing] [remaining=3h4m25s]
[2020/06/02 15:06:11.202 +08:00] [INFO] [restore.go:503] [progress] [files="5109/6641 (76.9%)"] [tables="49/51 (96.1%)"] [speed(MiB/s)=33.49966523015206] [state=writing] [remaining=3h5m55s]
[2020/06/02 15:11:11.197 +08:00] [INFO] [restore.go:503] [progress] [files="5109/6641 (76.9%)"] [tables="49/51 (96.1%)"] [speed(MiB/s)=33.23167263310829] [state=writing] [remaining=3h7m25s]
[2020/06/02 15:16:11.200 +08:00] [INFO] [restore.go:503] [progress] [files="5109/6641 (76.9%)"] [tables="49/51 (96.1%)"] [speed(MiB/s)=32.967926756948486] [state=writing] [remaining=3h8m55s]
[2020/06/02 15:21:11.197 +08:00] [INFO] [restore.go:503] [progress] [files="5109/6641 (76.9%)"] [tables="49/51 (96.1%)"] [speed(MiB/s)=32.7083393744271] [state=writing] [remaining=3h10m25s]
[2020/06/02 15:26:11.194 +08:00] [INFO] [restore.go:503] [progress] [files="5109/6641 (76.9%)"] [tables="49/51 (96.1%)"] [speed(MiB/s)=32.45280768558197] [state=writing] [remaining=3h11m55s]
[2020/06/02 15:28:14.248 +08:00] [INFO] [restore.go:1395] ["analyze completed"] [table=`pay`.`pay_bill`] [takeTime=28m14.861250134s] []
[2020/06/02 15:28:14.248 +08:00] [INFO] [restore.go:600] ["restore table completed"] [table=`pay`.`pay_bill`] [takeTime=8h17m3.25048472s] []
[2020/06/02 15:28:14.249 +08:00] [INFO] [restore.go:266] ["user terminated"] [step=2] [error="restore table `pay`.`pay_purchase` failed: rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 15:28:14.249 +08:00] [INFO] [restore.go:276] ["the whole procedure completed"] [takeTime=10h42m55.297718533s] []
[2020/06/02 15:28:14.248 +08:00] [INFO] [restore.go:472] ["everything imported, stopping periodic actions"]
[2020/06/02 15:28:14.262 +08:00] [INFO] [main.go:82] ["tidb lightning exit"]

在tikv-import中的日志

[2020/06/02 14:50:51.203 +08:00] [INFO] [import.rs:320] ["import sst"] [sst="SSTFile { uuid: 81ab4e40-78ae-425b-8d71-80d7f6101cdb, range: 7480000000000003FFBE5F698000000000FF0000010419A4D6{E0FF67000000038633C0FF5F6C837000000000FCFFFFFFFFA12A749F..E1FFCD000000038633C1FFA581C34001000000FCFFFFFFFFA12A749F}, length: 26358, cf_name: \"write\" }"] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:15:3183]"]
[2020/06/02 14:50:51.204 +08:00] [INFO] [import.rs:363] ["upload and ingest sst"] [region="RegionInfo { region: Region { id: 105661, range: 7480000000000003FFBE5F698000000000FF0000010419A4D6{13FF3A00000003863310FF6F56038000000000FC..E1FFCD000000038633C1FFA584036000000000FC}, region_epoch: RegionEpoch { conf_ver: 5, version: 865 }, peers: [Peer { id: 105662, store_id: 1, is_learner: false }, Peer { id: 105663, store_id: 4, is_learner: false }, Peer { id: 105664, store_id: 5, is_learner: false }] }, leader: Some(Peer { id: 105664, store_id: 5, is_learner: false }) }"] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:15:3183]"]
[2020/06/02 14:50:51.221 +08:00] [INFO] [import.rs:320] ["import sst"] [sst="SSTFile { uuid: ebbf9c8d-e959-4739-ae97-4d927a3faf53, range: 7480000000000003FFBE5F698000000000FF0000010419A4D6{13FF3A00000003863310FF6F56038000000000FCFFFFFFFFA12A749F..E0FF67000000038633C0FF5F5BC37000000000FCFFFFFFFFA12A749F}, length: 70818313, cf_name: \"write\" }"] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:16:3184]"]
[2020/06/02 14:50:51.221 +08:00] [INFO] [import.rs:363] ["upload and ingest sst"] [region="RegionInfo { region: Region { id: 105661, range: 7480000000000003FFBE5F698000000000FF0000010419A4D6{13FF3A00000003863310FF6F56038000000000FC..E1FFCD000000038633C1FFA584036000000000FC}, region_epoch: RegionEpoch { conf_ver: 5, version: 865 }, peers: [Peer { id: 105662, store_id: 1, is_learner: false }, Peer { id: 105663, store_id: 4, is_learner: false }, Peer { id: 105664, store_id: 5, is_learner: false }] }, leader: Some(Peer { id: 105664, store_id: 5, is_learner: false }) }"] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:16:3184]"]
[2020/06/02 14:50:51.235 +08:00] [INFO] [import.rs:478] ["ingest completed"] [region_id=105661] [store=5] [takes=24.141059ms] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:15:3183]"]
[2020/06/02 14:50:51.235 +08:00] [INFO] [import.rs:346] ["import sst completed"] [takes=32.025218ms] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:15:3183]"]
[2020/06/02 14:50:51.236 +08:00] [INFO] [engine.rs:220] ["cleanup SST completed"] [file_path="\"/data32/ssd/data.import/1bcc341d-07ce-5529-9aae-f00881124834/.cc0e2277-43c0-4272-acbf-c48c91595d1b:write\""]
[2020/06/02 14:50:51.599 +08:00] [INFO] [import.rs:478] ["ingest completed"] [region_id=105661] [store=5] [takes=34.448676ms] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:16:3184]"]
[2020/06/02 14:50:51.599 +08:00] [INFO] [import.rs:346] ["import sst completed"] [takes=377.606861ms] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:16:3184]"]
[2020/06/02 14:50:51.599 +08:00] [INFO] [engine.rs:220] ["cleanup SST completed"] [file_path="\"/data32/ssd/data.import/1bcc341d-07ce-5529-9aae-f00881124834/.1cb7f4cc-10b3-43da-b7e6-621f8ce422be:write\""]
[2020/06/02 14:50:52.075 +08:00] [INFO] [import.rs:168] ["dump sst completed"] [sst="[LazySSTInfo { file_path: \"/data32/ssd/data.import/1bcc341d-07ce-5529-9aae-f00881124834/.8c82e7cf-156f-4530-b5e7-1b638302c464:write\", file_size: 69769948, range: Range { start: [116, 128, 0, 0, 0, 0, 0, 3, 255, 190, 95, 105, 128, 0, 0, 0, 0, 255, 0, 0, 3, 3, 133, 211, 212, 20, 255, 197, 131, 80, 0, 4, 25, 164, 208, 255, 42, 231, 0, 0, 0, 3, 134, 47, 255, 72, 107, 240, 67, 64, 0, 0, 0, 253, 255, 255, 255, 255, 161, 42, 116, 159], end: [116, 128, 0, 0, 0, 0, 0, 3, 255, 190, 95, 105, 128, 0, 0, 0, 0, 255, 0, 0, 3, 3, 133, 213, 108, 240, 255, 95, 3, 80, 0, 4, 25, 164, 152, 255, 17, 157, 0, 0, 0, 3, 134, 13, 255, 184, 215, 230, 3, 96, 0, 0, 0, 253, 255, 255, 255, 255, 161, 42, 116, 159] }, cf_name: \"write\" }]"] [range=7480000000000003FFBE5F698000000000FF0000030385{D3D414FFC58350000419A4D0FF2AE700000003862FFF486BF04340000000FDFFFFFFFFA12A749F..D56CF0FF5F0350000419A498FF15E700000003860DFFBCC5C00330000000FDFFFFFFFFA12A749F}] [takes=5.369s]
[2020/06/02 14:50:52.077 +08:00] [INFO] [import.rs:182] ["dispatch-job done"] [ranges_handled=1] [id=9] [tag="[ImportJob 1bcc341d-07ce-5529-9aae-f00881124834]"]
[2020/06/02 14:50:52.097 +08:00] [INFO] [import.rs:320] ["import sst"] [sst="SSTFile { uuid: 7eea01cb-9263-4745-92fa-0f9cadc244b3, range: 7480000000000003FFBE5F698000000000FF0000030385{D3D414FFC58350000419A4D0FF2AE700000003862FFF486BF04340000000FDFFFFFFFFA12A749F..D56CF0FF5F0350000419A498FF119D00000003860DFFB8D7E60360000000FDFFFFFFFFA12A749F}, length: 69769948, cf_name: \"write\" }"] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:18:3185]"]
[2020/06/02 14:50:52.098 +08:00] [INFO] [import.rs:363] ["upload and ingest sst"] [region="RegionInfo { region: Region { id: 114237, range: 7480000000000003FFBE5F698000000000FF0000030385{D3D414FFC58350000419A4D0FF2AE700000003862FFF486BF04340000000FD..D5C80EFF428360000419A5F0FF13780000000386B8FFF5FF920CB0000000FD}, region_epoch: RegionEpoch { conf_ver: 5, version: 1560 }, peers: [Peer { id: 114238, store_id: 1, is_learner: false }, Peer { id: 114239, store_id: 4, is_learner: false }, Peer { id: 114240, store_id: 5, is_learner: false }] }, leader: Some(Peer { id: 114240, store_id: 5, is_learner: false }) }"] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:18:3185]"]
[2020/06/02 14:50:52.438 +08:00] [INFO] [import.rs:478] ["ingest completed"] [region_id=114237] [store=5] [takes=25.405719ms] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:18:3185]"]
[2020/06/02 14:50:52.438 +08:00] [INFO] [import.rs:346] ["import sst completed"] [takes=341.104051ms] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:18:3185]"]
[2020/06/02 14:50:52.438 +08:00] [INFO] [engine.rs:220] ["cleanup SST completed"] [file_path="\"/data32/ssd/data.import/1bcc341d-07ce-5529-9aae-f00881124834/.8c82e7cf-156f-4530-b5e7-1b638302c464:write\""]
[2020/06/02 14:50:59.955 +08:00] [INFO] [import.rs:168] ["dump sst completed"] [sst="[LazySSTInfo { file_path: \"/data32/ssd/data.import/1bcc341d-07ce-5529-9aae-f00881124834/.da8d12e7-92f4-41f7-8fad-f8b2713b3596:write\", file_size: 70675318, range: Range { start: [116, 128, 0, 0, 0, 0, 0, 3, 255, 190, 95, 105, 128, 0, 0, 0, 0, 255, 0, 0, 1, 4, 25, 164, 214, 225, 255, 205, 0, 0, 0, 3, 134, 51, 193, 255, 165, 132, 3, 96, 0, 0, 0, 0, 252, 255, 255, 255, 255, 161, 42, 116, 159], end: [116, 128, 0, 0, 0, 0, 0, 3, 255, 190, 95, 105, 128, 0, 0, 0, 0, 255, 0, 0, 1, 4, 25, 164, 216, 141, 255, 40, 0, 0, 0, 3, 134, 52, 195, 255, 61, 207, 131, 160, 0, 0, 0, 0, 252, 255, 255, 255, 255, 161, 42, 116, 159] }, cf_name: \"write\" }]"] [range=7480000000000003FFBE5F698000000000FF0000010419A4{D6E1FFCD000000038633C1FFA584036000000000FCFFFFFFFFA12A749F..D88DFF28000000038634C3FF3DD2836000000000FCFFFFFFFFA12A749F}] [takes=8.752s]
[2020/06/02 14:50:59.976 +08:00] [INFO] [import.rs:320] ["import sst"] [sst="SSTFile { uuid: ef11f571-d27b-416b-8ceb-f71558c9a4b7, range: 7480000000000003FFBE5F698000000000FF0000010419A4{D6E1FFCD000000038633C1FFA584036000000000FCFFFFFFFFA12A749F..D88DFF28000000038634C3FF3DCF83A000000000FCFFFFFFFFA12A749F}, length: 70675318, cf_name: \"write\" }"] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:2:3186]"]
[2020/06/02 14:50:59.977 +08:00] [INFO] [import.rs:363] ["upload and ingest sst"] [region="RegionInfo { region: Region { id: 105665, range: 7480000000000003FFBE5F698000000000FF0000010419A4{D6E1FFCD000000038633C1FFA584036000000000FC..D891FF26000000038634C5FFFB36037000000000FC}, region_epoch: RegionEpoch { conf_ver: 5, version: 866 }, peers: [Peer { id: 105666, store_id: 1, is_learner: false }, Peer { id: 105667, store_id: 4, is_learner: false }, Peer { id: 105668, store_id: 5, is_learner: false }] }, leader: Some(Peer { id: 105667, store_id: 4, is_learner: false }) }"] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:2:3186]"]
[2020/06/02 14:51:00.097 +08:00] [INFO] [import.rs:320] ["import sst"] [sst="SSTFile { uuid: 9d9fb034-88cd-4103-8ef4-9c5cfeadeba2, range: 7480000000000003FFBE5F698000000000FF0000010419A4D8{8DFF28000000038634C3FF3DD2836000000000FCFFFFFFFFA12A749F..91FF26000000038634C5FFFB35C34000000000FCFFFFFFFFA12A749F}, length: 1210366, cf_name: \"write\" }"] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:12:3187]"]
[2020/06/02 14:51:00.097 +08:00] [INFO] [import.rs:363] ["upload and ingest sst"] [region="RegionInfo { region: Region { id: 105665, range: 7480000000000003FFBE5F698000000000FF0000010419A4{D6E1FFCD000000038633C1FFA584036000000000FC..D891FF26000000038634C5FFFB36037000000000FC}, region_epoch: RegionEpoch { conf_ver: 5, version: 866 }, peers: [Peer { id: 105666, store_id: 1, is_learner: false }, Peer { id: 105667, store_id: 4, is_learner: false }, Peer { id: 105668, store_id: 5, is_learner: false }] }, leader: Some(Peer { id: 105667, store_id: 4, is_learner: false }) }"] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:12:3187]"]
[2020/06/02 14:51:00.163 +08:00] [INFO] [import.rs:478] ["ingest completed"] [region_id=105665] [store=4] [takes=44.287082ms] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:12:3187]"]
[2020/06/02 14:51:00.163 +08:00] [INFO] [import.rs:346] ["import sst completed"] [takes=66.81356ms] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:12:3187]"]
[2020/06/02 14:51:00.163 +08:00] [INFO] [engine.rs:220] ["cleanup SST completed"] [file_path="\"/data32/ssd/data.import/1bcc341d-07ce-5529-9aae-f00881124834/.8f5487b7-e771-4c59-b5b0-c1139ead6f15:write\""]
[2020/06/02 14:51:00.355 +08:00] [INFO] [import.rs:478] ["ingest completed"] [region_id=105665] [store=4] [takes=33.62579ms] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:2:3186]"]
[2020/06/02 14:51:00.355 +08:00] [INFO] [import.rs:346] ["import sst completed"] [takes=378.918897ms] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:2:3186]"]
[2020/06/02 14:51:00.355 +08:00] [INFO] [engine.rs:220] ["cleanup SST completed"] [file_path="\"/data32/ssd/data.import/1bcc341d-07ce-5529-9aae-f00881124834/.da8d12e7-92f4-41f7-8fad-f8b2713b3596:write\""]
[2020/06/02 14:51:05.135 +08:00] [INFO] [import.rs:168] ["dump sst completed"] [sst="[LazySSTInfo { file_path: \"/data32/ssd/data.import/1bcc341d-07ce-5529-9aae-f00881124834/.0a220854-55b4-4c38-a2f7-c25c373fbdeb:write\", file_size: 41356931, range: Range { start: [116, 128, 0, 0, 0, 0, 0, 3, 255, 190, 95, 105, 128, 0, 0, 0, 0, 255, 0, 0, 1, 4, 25, 164, 216, 145, 255, 38, 0, 0, 0, 3, 134, 52, 197, 255, 251, 54, 3, 112, 0, 0, 0, 0, 252, 255, 255, 255, 255, 161, 42, 116, 159], end: [116, 128, 0, 0, 0, 0, 0, 3, 255, 190, 95, 105, 128, 0, 0, 0, 0, 255, 0, 0, 1, 4, 25, 164, 218, 41, 255, 215, 0, 0, 0, 3, 134, 53, 183, 255, 100, 166, 67, 80, 0, 0, 0, 0, 252, 255, 255, 255, 255, 161, 42, 116, 159] }, cf_name: \"write\" }]"] [range=7480000000000003FFBE5F698000000000FF0000010419A4{D891FF26000000038634C5FFFB36037000000000FCFFFFFFFFA12A749F..DA29FFD7000000038635B7FF64A6837000000000FCFFFFFFFFA12A749F}] [takes=5.039s]
[2020/06/02 14:51:05.149 +08:00] [INFO] [import.rs:320] ["import sst"] [sst="SSTFile { uuid: 226cfb41-4de9-4afd-acd4-c1c455ac291e, range: 7480000000000003FFBE5F698000000000FF0000010419A4{D891FF26000000038634C5FFFB36037000000000FCFFFFFFFFA12A749F..DA29FFD7000000038635B7FF64A6435000000000FCFFFFFFFFA12A749F}, length: 41356931, cf_name: \"write\" }"] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:9:3188]"]
[2020/06/02 14:51:05.149 +08:00] [INFO] [import.rs:363] ["upload and ingest sst"] [region="RegionInfo { region: Region { id: 120053, range: 7480000000000003FFBE5F698000000000FF0000010419A4{D891FF26000000038634C5FFFB36037000000000FC..DA32FFA3000000038635BEFF888B036000000000FC}, region_epoch: RegionEpoch { conf_ver: 5, version: 870 }, peers: [Peer { id: 120054, store_id: 1, is_learner: false }, Peer { id: 120055, store_id: 4, is_learner: false }, Peer { id: 120056, store_id: 5, is_learner: false }] }, leader: Some(Peer { id: 120056, store_id: 5, is_learner: false }) }"] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:9:3188]"]
[2020/06/02 14:51:05.160 +08:00] [INFO] [import.rs:182] ["dispatch-job done"] [ranges_handled=1] [id=44] [tag="[ImportJob 1bcc341d-07ce-5529-9aae-f00881124834]"]
[2020/06/02 14:51:05.378 +08:00] [INFO] [import.rs:478] ["ingest completed"] [region_id=120053] [store=5] [takes=25.756177ms] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:9:3188]"]
[2020/06/02 14:51:05.378 +08:00] [INFO] [import.rs:346] ["import sst completed"] [takes=228.898297ms] [tag="[ImportSSTJob 1bcc341d-07ce-5529-9aae-f00881124834:9:3188]"]
[2020/06/02 14:51:05.378 +08:00] [INFO] [engine.rs:220] ["cleanup SST completed"] [file_path="\"/data32/ssd/data.import/1bcc341d-07ce-5529-9aae-f00881124834/.0a220854-55b4-4c38-a2f7-c25c373fbdeb:write\""]
[2020/06/02 14:51:05.380 +08:00] [INFO] [import.rs:99] ["import engine completed"] [takes=1105.683466176s] [tag="[ImportJob 1bcc341d-07ce-5529-9aae-f00881124834]"]
[2020/06/02 14:51:05.381 +08:00] [INFO] [kv_importer.rs:145] ["import completed"] [uuid=1bcc341d-07ce-5529-9aae-f00881124834]
[2020/06/02 14:51:47.814 +08:00] [INFO] [kv_importer.rs:179] ["cleanup completed"] [uuid=1bcc341d-07ce-5529-9aae-f00881124834]

yilong · 2020 年6 月 2 日 08:25

抱歉不是很明白您的问题，75%之后，可以继续导入，支持断点续传。导入完成后，会进行checksum的比较，来验证完整性。

zhengjunbo · 2020 年6 月 2 日 08:39

目前看起来是断开了。

在日志中看到只运行到76.9%的时候就直接断开了说是用户断开的。

[progress] [files="5109/6641 (76.9%)"]

由于有断点续传，再次导入的时候也是报错。

准确的报错应该是这几条，写入失败导致整个退出了

[2020/06/02 16:28:49.545 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=5] [path=/data33/pay-data_all/pay.pay_purchase.000001156.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.547 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=3] [path=/data33/pay-data_all/pay.pay_purchase.000000771.sql:0] [task=deliver] [error=EOF]
[2020/06/02 16:28:49.547 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=10] [path=/data33/pay-data_all/pay.pay_purchase.000001161.sql:0] [task=deliver] [error=EOF]
[2020/06/02 16:28:49.547 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=9] [path=/data33/pay-data_all/pay.pay_purchase.000000011.sql:0] [task=deliver] [error=EOF]

完整的报错

[2020/06/02 16:28:49.511 +08:00] [INFO] [restore.go:1737] ["restore file start"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=11] [path=/data33/pay-data_all/pay.pay_purchase.000000396.sql:0]
[2020/06/02 16:28:49.522 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=2] [path=/data33/pay-data_all/pay.pay_purchase.000000387.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=3] [path=/data33/pay-data_all/pay.pay_purchase.000000005.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=8] [path=/data33/pay-data_all/pay.pay_purchase.000000010.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=0] [path=/data33/pay-data_all/pay.pay_purchase.000001151.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=0] [path=/data33/pay-data_all/pay.pay_purchase.000000385.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=9] [path=/data33/pay-data_all/pay.pay_purchase.000001160.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=4] [path=/data33/pay-data_all/pay.pay_purchase.000001155.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=0] [path=/data33/pay-data_all/pay.pay_purchase.000000768.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=2] [path=/data33/pay-data_all/pay.pay_purchase.000000770.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=1] [path=/data33/pay-data_all/pay.pay_purchase.000001152.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=6] [path=/data33/pay-data_all/pay.pay_purchase.000000774.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=2] [path=/data33/pay-data_all/pay.pay_purchase.000001153.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=4] [path=/data33/pay-data_all/pay.pay_purchase.000000389.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.523 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=5] [path=/data33/pay-data_all/pay.pay_purchase.000000390.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.524 +08:00] [INFO] [restore.go:1737] ["restore file start"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=15] [path=/data33/pay-data_all/pay.pay_purchase.000000017.sql:0]
[2020/06/02 16:28:49.524 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=3] [path=/data33/pay-data_all/pay.pay_purchase.000001154.sql:0] [task=deliver] [error=EOF]
[2020/06/02 16:28:49.525 +08:00] [INFO] [restore.go:1737] ["restore file start"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=13] [path=/data33/pay-data_all/pay.pay_purchase.000000781.sql:0]
[2020/06/02 16:28:49.525 +08:00] [INFO] [restore.go:1737] ["restore file start"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=16] [path=/data33/pay-data_all/pay.pay_purchase.000001167.sql:0]
[2020/06/02 16:28:49.525 +08:00] [INFO] [restore.go:1737] ["restore file start"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=16] [path=/data33/pay-data_all/pay.pay_purchase.000000018.sql:0]
[2020/06/02 16:28:49.525 +08:00] [INFO] [restore.go:1737] ["restore file start"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=14] [path=/data33/pay-data_all/pay.pay_purchase.000000782.sql:0]
[2020/06/02 16:28:49.526 +08:00] [INFO] [restore.go:1737] ["restore file start"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=15] [path=/data33/pay-data_all/pay.pay_purchase.000000783.sql:0]
[2020/06/02 16:28:49.539 +08:00] [INFO] [restore.go:1737] ["restore file start"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=12] [path=/data33/pay-data_all/pay.pay_purchase.000000397.sql:0]
[2020/06/02 16:28:49.545 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=5] [path=/data33/pay-data_all/pay.pay_purchase.000001156.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.547 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=3] [path=/data33/pay-data_all/pay.pay_purchase.000000771.sql:0] [task=deliver] [error=EOF]
[2020/06/02 16:28:49.547 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=10] [path=/data33/pay-data_all/pay.pay_purchase.000001161.sql:0] [task=deliver] [error=EOF]
[2020/06/02 16:28:49.547 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=9] [path=/data33/pay-data_all/pay.pay_purchase.000000011.sql:0] [task=deliver] [error=EOF]
[2020/06/02 16:28:49.554 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=3] [path=/data33/pay-data_all/pay.pay_purchase.000000388.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.556 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=2] [path=/data33/pay-data_all/pay.pay_purchase.000000004.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.559 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=1] [path=/data33/pay-data_all/pay.pay_purchase.000000769.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.560 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=4] [path=/data33/pay-data_all/pay.pay_purchase.000000006.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.560 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=4] [path=/data33/pay-data_all/pay.pay_purchase.000000772.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.560 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=10] [path=/data33/pay-data_all/pay.pay_purchase.000000778.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.560 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=5] [path=/data33/pay-data_all/pay.pay_purchase.000000007.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.561 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=9] [path=/data33/pay-data_all/pay.pay_purchase.000000394.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.561 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=6] [path=/data33/pay-data_all/pay.pay_purchase.000000008.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.561 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=8] [path=/data33/pay-data_all/pay.pay_purchase.000001159.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.561 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=1] [path=/data33/pay-data_all/pay.pay_purchase.000000386.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.561 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=7] [path=/data33/pay-data_all/pay.pay_purchase.000001158.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.561 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=8] [path=/data33/pay-data_all/pay.pay_purchase.000000776.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.562 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=7] [path=/data33/pay-data_all/pay.pay_purchase.000000009.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.562 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=11] [path=/data33/pay-data_all/pay.pay_purchase.000001162.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.562 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=8] [path=/data33/pay-data_all/pay.pay_purchase.000000393.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.562 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=11] [path=/data33/pay-data_all/pay.pay_purchase.000000779.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.562 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=7] [path=/data33/pay-data_all/pay.pay_purchase.000000775.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.562 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=6] [path=/data33/pay-data_all/pay.pay_purchase.000001157.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.562 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=10] [path=/data33/pay-data_all/pay.pay_purchase.000000395.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.562 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=6] [path=/data33/pay-data_all/pay.pay_purchase.000000391.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.563 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=10] [path=/data33/pay-data_all/pay.pay_purchase.000000012.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.563 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=7] [path=/data33/pay-data_all/pay.pay_purchase.000000392.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.563 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=5] [path=/data33/pay-data_all/pay.pay_purchase.000000773.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.563 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=14] [path=/data33/pay-data_all/pay.pay_purchase.000001165.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.563 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=13] [path=/data33/pay-data_all/pay.pay_purchase.000000015.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.563 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=13] [path=/data33/pay-data_all/pay.pay_purchase.000001164.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.563 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=0] [path=/data33/pay-data_all/pay.pay_purchase.000000002.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.563 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=12] [path=/data33/pay-data_all/pay.pay_purchase.000001163.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.565 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=14] [path=/data33/pay-data_all/pay.pay_purchase.000000016.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.565 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=15] [path=/data33/pay-data_all/pay.pay_purchase.000001166.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.565 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=11] [path=/data33/pay-data_all/pay.pay_purchase.000000013.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.565 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=12] [path=/data33/pay-data_all/pay.pay_purchase.000000780.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.565 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=3] [fileIndex=16] [path=/data33/pay-data_all/pay.pay_purchase.000001167.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.565 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=9] [path=/data33/pay-data_all/pay.pay_purchase.000000777.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.565 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=14] [path=/data33/pay-data_all/pay.pay_purchase.000000782.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.565 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=16] [path=/data33/pay-data_all/pay.pay_purchase.000000018.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.565 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=13] [path=/data33/pay-data_all/pay.pay_purchase.000000781.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.566 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=2] [fileIndex=15] [path=/data33/pay-data_all/pay.pay_purchase.000000783.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.566 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=12] [path=/data33/pay-data_all/pay.pay_purchase.000000397.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.567 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=1] [fileIndex=11] [path=/data33/pay-data_all/pay.pay_purchase.000000396.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.575 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=15] [path=/data33/pay-data_all/pay.pay_purchase.000000017.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.577 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=1] [path=/data33/pay-data_all/pay.pay_purchase.000000003.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.687 +08:00] [ERROR] [restore.go:1547] ["write to data engine failed"] [table=`pay`.`pay_purchase`] [engineNumber=0] [fileIndex=12] [path=/data33/pay-data_all/pay.pay_purchase.000000014.sql:0] [task=deliver] [error="rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.694 +08:00] [INFO] [restore.go:266] ["user terminated"] [step=2] [error="restore table `pay`.`pay_purchase` failed: rpc error: code = Canceled desc = Cancelled"]
[2020/06/02 16:28:49.694 +08:00] [INFO] [restore.go:276] ["the whole procedure completed"] [takeTime=447.421115ms] []
[2020/06/02 16:28:49.694 +08:00] [INFO] [restore.go:472] ["everything imported, stopping periodic actions"]
[2020/06/02 16:28:49.699 +08:00] [INFO] [main.go:82] ["tidb lightning exit"]

zhengjunbo · 2020 年6 月 2 日 08:59

根据早上断开的记录貌似找到了点痕迹。看起来是pay_purchase已经完成导入了并且已经结束的情况下，又再次导入

这里说明下，该表确实没导全数据

[2020/06/02 08:12:22.973 +08:00] [INFO] [restore.go:1746] ["restore file completed"] [table=`pay`.`pay_purchase`] [engineNumber=4] [fileIndex=379] [path=/data33/pay-data_all/pay.pay
_purchase.000001913.sql:0] [readDur=7.290139311s] [encodeDur=19.944463295s] [deliverDur=5m52.38484117s] [checksum="{cksum=17768335871634491036,size=441016341,kvs=4054500}"] [takeTim
e=5m54.818120862s] []
[2020/06/02 08:12:29.915 +08:00] [INFO] [restore.go:1746] ["restore file completed"] [table=`pay`.`pay_purchase`] [engineNumber=4] [fileIndex=380] [path=/data33/pay-data_all/pay.pay
_purchase.000001914.sql:0] [readDur=7.35839168s] [encodeDur=20.264928836s] [deliverDur=5m54.475007835s] [checksum="{cksum=1859929105998255960,size=441840521,kvs=4052550}"] [takeTime
=5m56.918418402s] []
[2020/06/02 08:12:41.031 +08:00] [INFO] [restore.go:1746] ["restore file completed"] [table=`pay`.`pay_purchase`] [engineNumber=4] [fileIndex=381] [path=/data33/pay-data_all/pay.pay
_purchase.000001915.sql:0] [readDur=7.261757701s] [encodeDur=19.903188575s] [deliverDur=5m39.800742722s] [checksum="{cksum=17971687218294566055,size=443038856,kvs=4023948}"] [takeTi
me=5m42.119534499s] []
[2020/06/02 08:15:01.520 +08:00] [INFO] [restore.go:1746] ["restore file completed"] [table=`pay`.`pay_purchase`] [engineNumber=4] [fileIndex=382] [path=/data33/pay-data_all/pay.pay
_purchase.000001916.sql:0] [readDur=7.239765422s] [encodeDur=20.502989914s] [deliverDur=7m16.815081159s] [checksum="{cksum=10253989701777825923,size=447876600,kvs=4038660}"] [takeTi
me=7m19.184714572s] []
[2020/06/02 08:15:01.520 +08:00] [INFO] [restore.go:975] ["encode kv data and write completed"] [table=`pay`.`pay_purchase`] [engineNumber=4] [read=18243588416] [written=16885505943
7] [takeTime=2h41m29.729795334s] []
[2020/06/02 08:15:01.520 +08:00] [INFO] [backend.go:292] ["engine close start"] [engineTag=`pay`.`pay_purchase`:4] [engineUUID=cc9832fd-7e66-54e0-829e-dc206d8ceb9a]
[2020/06/02 08:15:02.654 +08:00] [INFO] [backend.go:294] ["engine close completed"] [engineTag=`pay`.`pay_purchase`:4] [engineUUID=cc9832fd-7e66-54e0-829e-dc206d8ceb9a] [takeTime=1.
133909984s] []
[2020/06/02 08:15:02.654 +08:00] [INFO] [restore.go:831] ["restore engine completed"] [table=`pay`.`pay_purchase`] [engineNumber=4] [takeTime=2h41m30.864224348s] []
[2020/06/02 08:15:28.889 +08:00] [INFO] [backend.go:309] ["import completed"] [engineTag=`pay`.`pay_purchase`:5] [engineUUID=9d77cdbf-27a3-5066-b5bc-cbf82a30f99f] [retryCnt=0] [take
Time=5m28.016017028s] []
[2020/06/02 08:15:28.889 +08:00] [INFO] [backend.go:321] ["cleanup start"] [engineTag=`pay`.`pay_purchase`:5] [engineUUID=9d77cdbf-27a3-5066-b5bc-cbf82a30f99f]
[2020/06/02 08:15:38.220 +08:00] [INFO] [backend.go:323] ["cleanup completed"] [engineTag=`pay`.`pay_purchase`:5] [engineUUID=9d77cdbf-27a3-5066-b5bc-cbf82a30f99f] [takeTime=9.33007
6095s] []
[2020/06/02 08:15:38.220 +08:00] [INFO] [restore.go:1357] ["import and cleanup engine completed"] [engineTag=`pay`.`pay_purchase`:5] [engineUUID=9d77cdbf-27a3-5066-b5bc-cbf82a30f99f
] [takeTime=5m37.346361288s] []
[2020/06/02 08:15:38.220 +08:00] [INFO] [restore.go:1350] ["import and cleanup engine start"] [engineTag=`pay`.`pay_purchase`:4] [engineUUID=cc9832fd-7e66-54e0-829e-dc206d8ceb9a]
[2020/06/02 08:15:38.220 +08:00] [INFO] [backend.go:306] ["import start"] [engineTag=`pay`.`pay_purchase`:4] [engineUUID=cc9832fd-7e66-54e0-829e-dc206d8ceb9a] [retryCnt=0]
[2020/06/02 08:19:11.983 +08:00] [INFO] [backend.go:309] ["import completed"] [engineTag=`pay`.`pay_purchase`:4] [engineUUID=cc9832fd-7e66-54e0-829e-dc206d8ceb9a] [retryCnt=0] [take
Time=3m33.763251s] []
[2020/06/02 08:19:11.983 +08:00] [INFO] [backend.go:321] ["cleanup start"] [engineTag=`pay`.`pay_purchase`:4] [engineUUID=cc9832fd-7e66-54e0-829e-dc206d8ceb9a]
[2020/06/02 08:19:20.535 +08:00] [INFO] [backend.go:323] ["cleanup completed"] [engineTag=`pay`.`pay_purchase`:4] [engineUUID=cc9832fd-7e66-54e0-829e-dc206d8ceb9a] [takeTime=8.55171
6704s] []
[2020/06/02 08:19:20.535 +08:00] [INFO] [restore.go:1357] ["import and cleanup engine completed"] [engineTag=`pay`.`pay_purchase`:4] [engineUUID=cc9832fd-7e66-54e0-829e-dc206d8ceb9a
] [takeTime=3m42.315304641s] []
[2020/06/02 15:28:14.249 +08:00] [INFO] [restore.go:266] ["user terminated"] [step=2] [error="restore table `pay`.`pay_purchase` failed: rpc error: code = Canceled desc = Cancelled"
]

csuzhangxc-PingCAP · 2020 年6 月 2 日 10:36

看起来是pay_purchase已经完成导入了并且已经结束的情况下，又再次导入

看上面的 log ，是早上只导入完成了这个表的一部分文件。

从早上到中午报错之间，有过其它什么操作吗？

能否提供一下完整的 lightning 的日志。

zhengjunbo · 2020 年6 月 2 日 11:16

没有进行任何操作，都是在等待导完。和查看日志。

目前在进行重新导入。

百度网盘链接: https://pan.baidu.com/s/13E7QWkGjqM-og8b-lKAUeA 提取码: 3tc5

csuzhangxc-PingCAP · 2020 年6 月 2 日 13:49

不好意思，能再给一下 lightning 报 “rpc error: code = Canceled desc = Cancelled” 时间范围内对应的 tikv-importor 的 log 吗？比如 2020/06/02 05:35:00 到 2020/06/02 05:40:00 的 tikv-importor 的 log。

因为看上面的 tikv importor 的 log 不太对应得上时间。

zhengjunbo · 2020 年6 月 3 日 02:42

这里是到2号8左右的。早上又进行了一次导入，文件看起来到100%了但是还差几个表没完成。

链接: https://pan.baidu.com/s/1kXlF7yPkgdAtnO45ZowQDA 提取码: 7zgg

csuzhangxc-PingCAP · 2020 年6 月 3 日 06:24

在 tikv-importor 的 log 中看到如下的错误信息。

[2020/06/02 05:33:31.791 +08:00] [ERROR] [kv_importer.rs:74] [“open engine failed”] [err=“File "/data32/ssd/data.import/f31ed652-167c-5565-87af-be87f1a8c377" exists”] [uuid=f31ed652-167c-5565-87af-be87f1a8c377]

请问之前是否有过使用 lightning 导入，但是导入中断后，只清理了 lightning 的 checkpoint，而没有清理 tikv-imporoter 的缓存数据？

如果有过重启 tikv-importer 的话，需要把 tikv-importor 的缓存数据都清理了。

zhengjunbo · 2020 年6 月 3 日 06:26

OK，应该是这个问题当时有2个tikv-importer。忘记关了。

yilong · 2020 年6 月 3 日 07:01

后来是重新清空重新导入了吗？如果没有，现在应该可以清理继续导入。

zhengjunbo · 2020 年6 月 3 日 07:15

已经清空导入了。感谢

yilong · 2020 年6 月 3 日 07:23