DM迁移时报错init checkpoint error

【 TiDB 使用环境】生产环境 / Poc
【 TiDB 版本】v8.1.0
【遇到的问题:问题现象及影响】
dm任务出错暂停,报错信息:

[Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1

进入对应dm worker 下查看日志,如下:

[2024/06/05 09:05:10.667 +08:00] [INFO] [import.go:903] ["restore all schema completed"] [task=saas_to_tidb] [unit=lightning-load] [takeTime=285.991013ms] []
[2024/06/05 09:05:10.668 +08:00] [INFO] [version.go:440] ["detect server version"] [type=TiDB] [version=8.1.0]
[2024/06/05 09:05:11.959 +08:00] [INFO] [tidb.go:295] ["skipping check requirements for tidb backend"] [task=saas_to_tidb] [unit=lightning-load]
[2024/06/05 09:05:11.959 +08:00] [INFO] [s3.go:407] ["succeed to get bucket region from s3"] ["bucket region"=]
[2024/06/05 09:05:12.044 +08:00] [WARN] [util.go:201] ["insert checkpoints failed with no retry"] [task=saas_to_tidb] [unit=lightning-load] [retryCnt=0] [error="Error 1406 (22001): Data too long for column 'source_dir' at row 1"]
[2024/06/05 09:05:12.044 +08:00] [ERROR] [import.go:569] ["run failed"] [task=saas_to_tidb] [unit=lightning-load] [step=3] [error="[Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1"]
[2024/06/05 09:05:12.044 +08:00] [ERROR] [import.go:579] ["the whole procedure failed"] [task=saas_to_tidb] [unit=lightning-load] [takeTime=1.662899038s] [error="[Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1"]
[2024/06/05 09:05:12.044 +08:00] [ERROR] [lightning.go:484] ["failed to runlightning"] [task=saas_to_tidb] [unit=lightning-load] [error="[code=34019:class=load-unit:scope=internal:level=high], RawCause: [Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1"] [errorVerbose="[code=34019:class=load-unit:scope=internal:level=high], RawCause: [Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:337\ngithub.com/pingcap/tiflow/dm/loader.convertLightningError\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:318\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).runLightning\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:299\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).restore\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:475\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).Process\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:543\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).Resume\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:611\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/06/05 09:05:12.044 +08:00] [ERROR] [lightning.go:544] ["process error"] [task=saas_to_tidb] [unit=lightning-load] [error="[code=34019:class=load-unit:scope=internal:level=high], RawCause: [Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1"] [errorVerbose="[code=34019:class=load-unit:scope=internal:level=high], RawCause: [Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Delegate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:337\ngithub.com/pingcap/tiflow/dm/loader.convertLightningError\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:318\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).runLightning\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:299\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).restore\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:475\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).Process\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:543\ngithub.com/pingcap/tiflow/dm/loader.(*LightningLoader).Resume\n\tgithub.com/pingcap/tiflow/dm/loader/lightning.go:611\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/06/05 09:05:12.044 +08:00] [INFO] [lightning.go:632] ["progress status of lightning"] [task=saas_to_tidb] [unit=lightning-load] [finished_bytes=0] [total_bytes=2595403032923] [progress="0.00 %"] ["current speed (bytes / seconds)"=0]
[2024/06/05 09:05:12.044 +08:00] [INFO] [lightning.go:556] ["lightning load end"] [task=saas_to_tidb] [unit=lightning-load] [IsCanceled=false] [finished_bytes=0] [total_bytes=2595403032923] [progress="0.00 %"]
[2024/06/05 09:05:12.044 +08:00] [INFO] [lightning.go:632] ["progress status of lightning"] [task=saas_to_tidb] [unit=lightning-load] [finished_bytes=0] [total_bytes=2595403032923] [progress="0.00 %"] ["current speed (bytes / seconds)"=0]
[2024/06/05 09:05:12.044 +08:00] [INFO] [subtask.go:338] ["unit process returned"] [subtask=saas_to_tidb] [unit=Load] [stage=Paused] [status="{\"totalBytes\":2595403032923,\"progress\":\"0.00 %\",\"metaBinlog\":\"(mysql-bin.200129, 56390260)\",\"metaBinlogGTID\":\"06aea9a9-3254-11ee-84d6-b8cef6abafc4:1-9866426949,073f658f-9cfb-11ed-ad7a-b8cef6286318:1-73380394,0b060c54-9732-11ed-84f9-b8cef6204434:1-11950259,1138f250-84b7-11ed-9bad-b8cef6204434:1-44644505,1c044756-822e-11ed-b901-b8cef620662c:1-78939241,2a66430c-9f55-11ed-8ca9-b8cef620662c:1-371343241,4526add7-861f-11ed-bbca-b8cef620662c:1-5001207,48634626-1a20-11ed-abfd-b8cef6204434:1-1366722044,4e92d873-ec4e-11ec-a38f-b8cef620662c:1-253880812,54760216-437b-11ed-9719-b8cef6ab7ca4:1-801633677,69a4c9e7-5906-11ed-bcae-b8cef6205b35:1-391191583,79da58f2-aa65-11ed-867a-b8cef6286318:1-129656734,7b67f9c5-aee1-11ed-af56-b8cef62080dc:1-4288664626,861e55bf-9460-11ed-8ae7-b8cef6204434:1-38063549,86e0231b-9595-11ed-ad05-b8cef620662c:1-54130804,a9715f72-9787-11ed-94fc-b8cef620662c:1-172830361,af2e25f1-919a-11ed-a60a-b8cef620662c:1-26121580:26121582-26121584:26121587:26121589-26121651,b6b31d19-87ae-11ed-b33e-b8cef620662c:1-41855135,bc4ee0b4-8641-11ed-b7b7-b8cef6204434:1-51934417,bdd89ef1-0d88-11ed-a756-b8cef62080e4:1-290031999,c0d30793-6217-11ed-b390-b8cef6202f94:1-1299616952,c6484d6f-8df5-11ed-8caf-b8cef6204434:1-131050974:131050976-131051192:131051194-131051305:131051307-131051359,c78369a2-8bb4-11ed-92ee-b8cef620662c:1-79347384,cab28f12-9317-11ed-92bc-b8cef620662c:1-43069687,ce72cd92-9252-11ed-afa4-b8cef6204434:1-26135774:26135776-26135783,fd014043-88e3-11ed-ac48-b8cef6204434:1-87597635\"}"]
[2024/06/05 09:05:12.044 +08:00] [ERROR] [subtask.go:359] ["unit process error"] [subtask=saas_to_tidb] [unit=Load] ["error information"="ErrCode:34019 ErrClass:\"load-unit\" ErrScope:\"internal\" ErrLevel:\"high\" RawCause:\"[Lightning:Checkpoint:ErrInitCheckpoint]init checkpoint error: insert checkpoints failed: Error 1406 (22001): Data too long for column 'source_dir' at row 1\" "]
[2024/06/05 09:05:12.044 +08:00] [INFO] [subtask.go:361] [paused] [subtask=saas_to_tidb] [unit=Load]
[2024/06/05 09:05:13.066 +08:00] [WARN] [task_checker.go:352] ["task can't auto resume"] [component="task checker"] [task=saas_to_tidb] ["paused duration"=9h30m19.999908942s]
[2024/06/05 09:05:18.066 +08:00] [WARN] [task_checker.go:352] ["task can't auto resume"] [component="task checker"] [task=saas_to_tidb] ["paused duration"=9h30m24.999698406s]
[2024/06/05 09:05:23.067 +08:00] [WARN] [task_checker.go:352] ["task can't auto resume"] [component="task checker"] [task=saas_to_tidb] ["paused duration"=9h30m30.00058815s]

Data too long for column ‘source_dir’

检查下长度拉

在目标库里面,有个临时库saas_to_tidb_1107585871_tidb_lightning_checkpoint,里面有个表task_v2, 我将source_dir 长度由vchar 256 增加到1024了。就正常了。
之前一直在dm_meta 和 lightning_task_info 里面找 source_dir的表。

报错很明显,好解决。

Data too long for column ‘source_dir’

这个错还是比较明显,好处理

Data too long for column ‘source_dir’

:clap:解决了就好 :call_me_hand:

检查下字段的长度是不是超了,这个属于表结构设计不完善的了