使用dumpling备份MySQL遇到报错

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:

【TiDB 版本】4.0.10
MySQL 版本 5.6.22
备份目标库比较大
报错信息 [writer_util.go:202] [“fail to scan from sql.Row”]
一开始是32线程,后来调整到8线程,还是报错 信息内容相同

if err = fileRowIter.Decode(row); err != nil {
log.Error(“fail to scan from sql.Row”, zap.Error(err))
return errors.Trace(err)
}
解码失败导致的?
具体日志信息如下
[root@YZ-222-90-222 bin]# ./dumpling -u xx -p xx -P 3306 -h 127.0.0.1 -B xx --filetype sql --threads 32 -o /export/tidb-data/backup/dazhihui -F 512MiB
Release version: v4.0.10
Git commit hash: 5e6981efa173ff525abb31c6e62a50ecd6da6a09
Git branch: heads/refs/tags/v4.0.10
Build timestamp: 2021-01-14 12:39:07Z
Go version: go version go1.13 linux/amd64

[2021/02/19 15:55:55.814 +08:00] [INFO] [config.go:596] [“detect server type”] [type=MySQL]
[2021/02/19 15:55:55.814 +08:00] [INFO] [config.go:615] [“detect server version”] [version=5.6.16-log]
[2021/02/19 15:57:18.655 +08:00] [INFO] [dump.go:173] [“All the dumping transactions have started. Start to unlock tables”]
[2021/02/19 15:59:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“273/1015 (26.9%)”] [“finished rows”=63246806] [“finished size”=12.85GB] [“average speed(MiB/s)”=102.13631200167188]
[2021/02/19 16:01:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“299/1015 (29.5%)”] [“finished rows”=134388953] [“finished size”=28.42GB] [“average speed(MiB/s)”=123.7023331049533]
[2021/02/19 16:03:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“327/1015 (32.2%)”] [“finished rows”=194548953] [“finished size”=43.82GB] [“average speed(MiB/s)”=122.4302868485057]
[2021/02/19 16:05:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“421/1015 (41.5%)”] [“finished rows”=248285580] [“finished size”=59.09GB] [“average speed(MiB/s)”=121.36668877604895]
[2021/02/19 16:07:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“512/1015 (50.4%)”] [“finished rows”=311942934] [“finished size”=75.08GB] [“average speed(MiB/s)”=127.06049677014298]
[2021/02/19 16:09:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“525/1015 (51.7%)”] [“finished rows”=384954932] [“finished size”=90.9GB] [“average speed(MiB/s)”=125.73231293889278]
[2021/02/19 16:11:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“560/1015 (55.2%)”] [“finished rows”=451870542] [“finished size”=106.9GB] [“average speed(MiB/s)”=127.10284187433716]
[mysql] 2021/02/19 16:12:18 packets.go:36: read tcp 10.222.90.222:58916->172.25.29.26:3306: i/o timeout
[2021/02/19 16:12:18.678 +08:00] [ERROR] [writer_util.go:202] [“fail to scan from sql.Row”] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/juju_adaptor.go:15\ngithub.com/pingcap/dumpling/v4/export.decodeFromRows\n\tgithub.com/pingcap/dumpling@/v4/export/ir.go:67\ngithub.com/pingcap/dumpling/v4/export.(*rowIter).Decode\n\tgithub.com/pingcap/dumpling@/v4/export/ir_impl.go:37\ngithub.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:201\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1357”] [stack=“github.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:202\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57”]
[2021/02/19 16:12:18.682 +08:00] [ERROR] [writer_util.go:202] [“fail to scan from sql.Row”] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/juju_adaptor.go:15\ngithub.com/pingcap/dumpling/v4/export.decodeFromRows\n\tgithub.com/pingcap/dumpling@/v4/export/ir.go:67\ngithub.com/pingcap/dumpling/v4/export.(*rowIter).Decode\n\tgithub.com/pingcap/dumpling@/v4/export/ir_impl.go:37\ngithub.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:201\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1357”]

你好,
请检查下 /export/tidb-data/backup/dazhihui 路径的存储空间是否充足?

充足
/dev/sda3 22T 350G 21T 2% /export

这个时间点看下上游 mysql 是否存活,并确认下 mysql 慢日志中,是否存在 dumpling 在这个时间点的语句。

从 io timeout 报错来看,是 dumpling 在 15min 中内没有收到 mysql 导出的那条语句,可能是 sql 太慢了,辛苦在看下。

如果急于备份请使用 -r 参数进行导出数据的限制,加快大表导出。

如果日志还保存着,可以发下最后几行的日志输出。确定下问题。

[2021/02/19 15:55:55.814 +08:00] [INFO] [config.go:596] [“detect server type”] [type=MySQL]
[2021/02/19 15:55:55.814 +08:00] [INFO] [config.go:615] [“detect server version”] [version=5.6.16-log]
[2021/02/19 15:57:18.655 +08:00] [INFO] [dump.go:173] [“All the dumping transactions have started. Start to unlock tables”]
[2021/02/19 15:59:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“273/1015 (26.9%)”] [“finished rows”=63246806] [“finished size”=12.85GB] [“average speed(MiB/s)”=102.13631200167188]
[2021/02/19 16:01:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“299/1015 (29.5%)”] [“finished rows”=134388953] [“finished size”=28.42GB] [“average speed(MiB/s)”=123.7023331049533]
[2021/02/19 16:03:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“327/1015 (32.2%)”] [“finished rows”=194548953] [“finished size”=43.82GB] [“average speed(MiB/s)”=122.4302868485057]
[2021/02/19 16:05:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“421/1015 (41.5%)”] [“finished rows”=248285580] [“finished size”=59.09GB] [“average speed(MiB/s)”=121.36668877604895]
[2021/02/19 16:07:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“512/1015 (50.4%)”] [“finished rows”=311942934] [“finished size”=75.08GB] [“average speed(MiB/s)”=127.06049677014298]
[2021/02/19 16:09:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“525/1015 (51.7%)”] [“finished rows”=384954932] [“finished size”=90.9GB] [“average speed(MiB/s)”=125.73231293889278]
[2021/02/19 16:11:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“560/1015 (55.2%)”] [“finished rows”=451870542] [“finished size”=106.9GB] [“average speed(MiB/s)”=127.10284187433716]
[mysql] 2021/02/19 16:12:18 packets.go:36: read tcp 10.222.90.222:58916->172.25.29.26:3306: i/o timeout
[2021/02/19 16:12:18.678 +08:00] [ERROR] [writer_util.go:202] [“fail to scan from sql.Row”] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/juju_adaptor.go:15\ngithub.com/pingcap/dumpling/v4/export.decodeFromRows\n\tgithub.com/pingcap/dumpling@/v4/export/ir.go:67\ngithub.com/pingcap/dumpling/v4/export.(*rowIter).Decode\n\tgithub.com/pingcap/dumpling@/v4/export/ir_impl.go:37\ngithub.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:201\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1357”] [stack=“github.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:202\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57”]
[2021/02/19 16:12:18.682 +08:00] [ERROR] [writer_util.go:202] [“fail to scan from sql.Row”] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/juju_adaptor.go:15\ngithub.com/pingcap/dumpling/v4/export.decodeFromRows\n\tgithub.com/pingcap/dumpling@/v4/export/ir.go:67\ngithub.com/pingcap/dumpling/v4/export.(*rowIter).Decode\n\tgithub.com/pingcap/dumpling@/v4/export/ir_impl.go:37\ngithub.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:201\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1357”] [stack=“github.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:202\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57”]
[2021/02/19 16:12:18.679 +08:00] [ERROR] [writer_util.go:202] [“fail to scan from sql.Row”] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/juju_adaptor.go:15\ngithub.com/pingcap/dumpling/v4/export.decodeFromRows\n\tgithub.com/pingcap/dumpling@/v4/export/ir.go:67\ngithub.com/pingcap/dumpling/v4/export.(*rowIter).Decode\n\tgithub.com/pingcap/dumpling@/v4/export/ir_impl.go:37\ngithub.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:201\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1357”] [stack=“github.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:202\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57”]
[2021/02/19 16:12:18.682 +08:00] [ERROR] [writer_util.go:202] [“fail to scan from sql.Row”] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/juju_adaptor.go:15\ngithub.com/pingcap/dumpling/v4/export.decodeFromRows\n\tgithub.com/pingcap/dumpling@/v4/export/ir.go:67\ngithub.com/pingcap/dumpling/v4/export.(*rowIter).Decode\n\tgithub.com/pingcap/dumpling@/v4/export/ir_impl.go:37\ngithub.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:201\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1357”] [stack=“github.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:202\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57”]
[2021/02/19 16:12:18.682 +08:00] [ERROR] [writer_util.go:202] [“fail to scan from sql.Row”] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/juju_adaptor.go:15\ngithub.com/pingcap/dumpling/v4/export.decodeFromRows\n\tgithub.com/pingcap/dumpling@/v4/export/ir.go:67\ngithub.com/pingcap/dumpling/v4/export.(*rowIter).Decode\n\tgithub.com/pingcap/dumpling@/v4/export/ir_impl.go:37\ngithub.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:201\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1357”] [stack=“github.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:202\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57”]
[2021/02/19 16:12:18.682 +08:00] [ERROR] [writer_util.go:202] [“fail to scan from sql.Row”] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/juju_adaptor.go:15\ngithub.com/pingcap/dumpling/v4/export.decodeFromRows\n\tgithub.com/pingcap/dumpling@/v4/export/ir.go:67\ngithub.com/pingcap/dumpling/v4/export.(*rowIter).Decode\n\tgithub.com/pingcap/dumpling@/v4/export/ir_impl.go:37\ngithub.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:201\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1357”] [stack=“github.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:202\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57”]
[2021/02/19 16:12:18.679 +08:00] [ERROR] [writer_util.go:202] [“fail to scan from sql.Row”] [error=“context canceled”] [errorVerbose=“context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/juju_adaptor.go:15\ngithub.com/pingcap/dumpling/v4/export.decodeFromRows\n\tgithub.com/pingcap/dumpling@/v4/export/ir.go:67\ngithub.com/pingcap/dumpling/v4/export.(*rowIter).Decode\n\tgithub.com/pingcap/dumpling@/v4/export/ir_impl.go:37\ngithub.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:201\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1357”] [stack=“github.com/pingcap/dumpling/v4/export.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:202\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\tgithub.com/pingcap/dumpling@/v4/export/writer_util.go:593\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:204\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:190\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\tgithub.com/pingcap/br@v0.0.0-20201119111016-600102357a27/pkg/utils/retry.go:34\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:104\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling@/v4/export/writer.go:86\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling@/v4/export/dump.go:253\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57”]
[2021/02/19 16:13:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=63.7167663960822]
[2021/02/19 16:15:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:17:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:19:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:21:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:23:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:25:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:27:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:29:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:31:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:33:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:35:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:37:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:39:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:41:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:43:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:45:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:47:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:49:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]
[2021/02/19 16:51:18.655 +08:00] [INFO] [status.go:37] [progress] [tables=“565/1015 (55.7%)”] [“finished rows”=480620919] [“finished size”=114.9GB] [“average speed(MiB/s)”=0]

上游的MySQL是存活的 这个是测试环境 没得慢查询监控 我过滤一下看一下。感谢老板

-r 参数哦

从上面日志看的意思,目前 dumpling 还在跑,只是中间会有 error 是吗

嗯 是的 看日志是这样的

好的 我加一下-r试试 感谢老板的帮助

昨天也试过 加-r参数

-t 32 -r 100
这样设置,高并发+小批次。

嗯 好的 我试试

请问当前该问题是否已解决?

稍等 还在验证

麻烦等验证完,有空反馈下,多谢。

好用的 已解决

:+1: