tidb v7.1.5 dumpling数据到minio异常报错

🪐 TiDB 技术问题
1

【 TiDB 使用环境】生产环境
【 TiDB 版本】
v7.1.5
【复现路径】做过哪些操作出现的问题
用dumpling备份一个表的匹配数据到s3，遇到了报错如下：

[tidb@b67 ~]$ export AWS_ACCESS_KEY_ID=tidb-snapshots-user
[tidb@b67 ~]$ export AWS_SECRET_ACCESS_KEY=Udx5xxRXNWZbxxx
[tidb@b67 ~]$ 
[tidb@b67 ~]$ ~/tidb-toolkit/dumpling  -u root -p 'PEsCePtMxxxxxxxx'  -P 4000 -h 192.168.241.85  -T bsppr.xpost --where "update_time < '2023-03-01 00:00:00'" -o "s3://tidb-snapshots/dumpling?endpoint=http://oss.api.private"
Release version: v7.1.5
Git commit hash: caa60c0917a886933a525d25e17057faac5b4da2
Git branch:      HEAD
Build timestamp: 2024-04-24 11:09:23Z
Go version:      go version go1.20.10 linux/amd64

[2024/11/12 11:42:23.046 +08:00] [INFO] [versions.go:54] ["Welcome to dumpling"] ["Release Version"=v7.1.5] ["Git Commit Hash"=caa60c0917a886933a525d25e17057faac5b4da2] ["Git Branch"=HEAD] ["Build timestamp"="2024-04-24 11:09:23"] ["Go Version"="go version go1.20.10 linux/amd64"]
[2024/11/12 11:42:23.106 +08:00] [INFO] [s3.go:403] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1]
[2024/11/12 11:42:23.109 +08:00] [INFO] [version.go:434] ["detect server version"] [type=TiDB] [version=7.1.5]
[2024/11/12 11:42:23.124 +08:00] [INFO] [client.go:311] ["[pd] create pd client with endpoints and keyspace"] [pd-address="[192.168.241.60:2379,192.168.241.59:2379,192.168.241.61:2379]"] [keyspace-id=0]
[2024/11/12 11:42:23.131 +08:00] [INFO] [pd_service_discovery.go:521] ["[pd] update member urls"] [old-urls="[http://192.168.241.60:2379,http://192.168.241.59:2379,http://192.168.241.61:2379]"] [new-urls="[http://192.168.241.59:2379,http://192.168.241.60:2379,http://192.168.241.61:2379]"]
[2024/11/12 11:42:23.131 +08:00] [INFO] [pd_service_discovery.go:546] ["[pd] switch leader"] [new-leader=http://192.168.241.61:2379] [old-leader=]
[2024/11/12 11:42:23.131 +08:00] [INFO] [pd_service_discovery.go:177] ["[pd] init cluster id"] [cluster-id=7210697100626091534]
[2024/11/12 11:42:23.132 +08:00] [INFO] [client.go:386] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2024/11/12 11:42:23.132 +08:00] [INFO] [tso_client.go:231] ["[tso] switch dc tso allocator serving address"] [dc-location=global] [new-address=http://192.168.241.61:2379]
[2024/11/12 11:42:23.133 +08:00] [INFO] [tso_dispatcher.go:296] ["[tso] tso dispatcher created"] [dc-location=global]
[2024/11/12 11:42:23.133 +08:00] [INFO] [client.go:428] ["[pd] service mode changed"] [old-mode=PD_SVC_MODE] [new-mode=PD_SVC_MODE]
[2024/11/12 11:42:23.134 +08:00] [INFO] [dump.go:1519] ["generate dumpling gc safePoint id"] [id=dumpling_1731382943134361831]
[2024/11/12 11:42:23.143 +08:00] [INFO] [dump.go:151] ["begin to run Dump"] [conf="{\"s3\":{\"endpoint\":\"http://oss.api.private\",\"region\":\"\",\"storage-class\":\"\",\"sse\":\"\",\"sse-kms-key-id\":\"\",\"acl\":\"\",\"access-key\":\"\",\"secret-access-key\":\"\",\"session-token\":\"\",\"provider\":\"\",\"force-path-style\":true,\"use-accelerate-endpoint\":false,\"role-arn\":\"\",\"external-id\":\"\",\"object-lock-enabled\":false},\"gcs\":{\"endpoint\":\"\",\"storage-class\":\"\",\"predefined-acl\":\"\",\"credentials-file\":\"\"},\"azblob\":{\"endpoint\":\"\",\"account-name\":\"\",\"account-key\":\"\",\"access-tier\":\"\"},\"AllowCleartextPasswords\":false,\"SortByPk\":true,\"NoViews\":true,\"NoSequences\":true,\"NoHeader\":false,\"NoSchemas\":false,\"NoData\":false,\"CompleteInsert\":false,\"TransactionalConsistency\":true,\"EscapeBackslash\":true,\"DumpEmptyDatabase\":true,\"PosAfterConnect\":false,\"CompressType\":0,\"Host\":\"192.168.241.85\",\"Port\":4000,\"Threads\":4,\"User\":\"root\",\"Security\":{\"CAPath\":\"\",\"CertPath\":\"\",\"KeyPath\":\"\"},\"LogLevel\":\"info\",\"LogFile\":\"\",\"LogFormat\":\"text\",\"OutputDirPath\":\"s3://tidb-snapshots/dumpling?endpoint=http://oss.api.private\",\"StatusAddr\":\":8281\",\"Snapshot\":\"453871650240462857\",\"Consistency\":\"snapshot\",\"CsvNullValue\":\"\\\\N\",\"SQL\":\"\",\"CsvSeparator\":\",\",\"CsvDelimiter\":\"\\\"\",\"Databases\":[],\"Where\":\"update_time \\u003c '2023-03-01 00:00:00'\",\"FileType\":\"sql\",\"ServerInfo\":{\"ServerType\":3,\"ServerVersion\":\"7.1.5\",\"HasTiKV\":true},\"Rows\":0,\"ReadTimeout\":900000000000,\"TiDBMemQuotaQuery\":0,\"FileSize\":0,\"StatementSize\":1000000,\"SessionParams\":{\"tidb_snapshot\":\"453871650240462857\"},\"Tables\":{\"bsppr\":[{\"Name\":\"xpost\",\"AvgRowLength\":0,\"Type\":0}]},\"CollationCompatible\":\"loose\",\"IOTotalBytes\":null,\"Net\":\"\"}"]
[2024/11/12 11:42:23.169 +08:00] [WARN] [s3.go:1059] ["failed to request s3, retrying"] [error="SerializationError: failed to unmarshal error message\n\tstatus code: 403, request id: , host id: \ncaused by: UnmarshalError: failed to unmarshal error message\n\t00000000  3c 68 74 6d 6c 3e 0d 0a  3c 68 65 61 64 3e 3c 74  |<html>..<head><t|\n00000010  69 74 6c 65 3e 34 30 33  20 46 6f 72 62 69 64 64  |itle>403 Forbidd|\n00000020  65 6e 3c 2f 74 69 74 6c  65 3e 3c 2f 68 65 61 64  |en</title></head|\n00000030  3e 0d 0a 3c 62 6f 64 79  3e 0d 0a 3c 63 65 6e 74  |>..<body>..<cent|\n00000040  65 72 3e 3c 68 31 3e 34  30 33 20 46 6f 72 62 69  |er><h1>403 Forbi|\n00000050  64 64 65 6e 3c 2f 68 31  3e 3c 2f 63 65 6e 74 65  |dden</h1></cente|\n00000060  72 3e 0d 0a 3c 68 72 3e  3c 63 65 6e 74 65 72 3e  |r>..<hr><center>|\n00000070  6f 70 65 6e 72 65 73 74  79 2f 31 2e 31 35 2e 38  |openresty/1.15.8|\n00000080  2e 31 3c 2f 63 65 6e 74  65 72 3e 0d 0a 3c 2f 62  |.1</center>..</b|\n00000090  6f 64 79 3e 0d 0a 3c 2f  68 74 6d 6c 3e 0d 0a     |ody>..</html>..|\n\ncaused by: expected element type <Error> but have <html>"] [backoff=1.892388137s]
[2024/11/12 11:42:25.064 +08:00] [WARN] [s3.go:1059] ["failed to request s3, retrying"] [error="SerializationError: failed to unmarshal error message\n\tstatus code: 403, request id: , host id: \ncaused by: UnmarshalError: failed to unmarshal error message\n\t00000000  3c 68 74 6d 6c 3e 0d 0a  3c 68 65 61 64 3e 3c 74  |<html>..<head><t|\n00000010  69 74 6c 65 3e 34 30 33  20 46 6f 72 62 69 64 64  |itle>403 Forbidd|\n00000020  65 6e 3c 2f 74 69 74 6c  65 3e 3c 2f 68 65 61 64  |en</title></head|\n00000030  3e 0d 0a 3c 62 6f 64 79  3e 0d 0a 3c 63 65 6e 74  |>..<body>..<cent|\n00000040  65 72 3e 3c 68 31 3e 34  30 33 20 46 6f 72 62 69  |er><h1>403 Forbi|\n00000050  64 64 65 6e 3c 2f 68 31  3e 3c 2f 63 65 6e 74 65  |dden</h1></cente|\n00000060  72 3e 0d 0a 3c 68 72 3e  3c 63 65 6e 74 65 72 3e  |r>..<hr><center>|\n00000070  6f 70 65 6e 72 65 73 74  79 2f 31 2e 31 35 2e 38  |openresty/1.15.8|\n00000080  2e 31 3c 2f 63 65 6e 74  65 72 3e 0d 0a 3c 2f 62  |.1</center>..</b|\n00000090  6f 64 79 3e 0d 0a 3c 2f  68 74 6d 6c 3e 0d 0a     |ody>..</html>..|\n\ncaused by: expected element type <Error> but have <html>"] [backoff=2.160264182s]
[2024/11/12 11:42:27.226 +08:00] [WARN] [s3.go:1059] ["failed to request s3, retrying"] [error="SerializationError: failed to unmarshal error message\n\tstatus code: 403, request id: , host id: \ncaused by: UnmarshalError: failed to unmarshal error message\n\t00000000  3c 68 74 6d 6c 3e 0d 0a  3c 68 65 61 64 3e 3c 74  |<html>..<head><t|\n00000010  69 74 6c 65 3e 34 30 33  20 46 6f 72 62 69 64 64  |itle>403 Forbidd|\n00000020  65 6e 3c 2f 74 69 74 6c  65 3e 3c 2f 68 65 61 64  |en</title></head|\n00000030  3e 0d 0a 3c 62 6f 64 79  3e 0d 0a 3c 63 65 6e 74  |>..<body>..<cent|\n00000040  65 72 3e 3c 68 31 3e 34  30 33 20 46 6f 72 62 69  |er><h1>403 Forbi|\n00000050  64 64 65 6e 3c 2f 68 31  3e 3c 2f 63 65 6e 74 65  |dden</h1></cente|\n00000060  72 3e 0d 0a 3c 68 72 3e  3c 63 65 6e 74 65 72 3e  |r>..<hr><center>|\n00000070  6f 70 65 6e 72 65 73 74  79 2f 31 2e 31 35 2e 38  |openresty/1.15.8|\n00000080  2e 31 3c 2f 63 65 6e 74  65 72 3e 0d 0a 3c 2f 62  |.1</center>..</b|\n00000090  6f 64 79 3e 0d 0a 3c 2f  68 74 6d 6c 3e 0d 0a     |ody>..</html>..|\n\ncaused by: expected element type <Error> but have <html>"] [backoff=7.354509796s]
2

403了，网络通么，有没有防火墙啥的

3

endpoint对吗？看着不像是region地址信息呢

4

网络是通的，都是内网通讯，内网没限制

5

没错的，我这不是s3，我这是备份到minio的
endpoint是生产的地址，内网IP 加hosts的

6

我想说我这命令没问题吧？ 我好像不备份到minio，备份到本地也卡着，导不出数据。

我这个表数据量很大80亿左右，dumpling不支持导这么大数据的表吗？

7

从你贴的这个报错信息看不到更多的了，你再巴拉巴拉tidb的日志啥的看看呢

8

这是执行命令后 连接的tidb的日志报错

[2024/11/12 16:37:46.391 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203684681] [SnapshotTS=453876296284110872]
[2024/11/12 16:37:46.396 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203684683] [SnapshotTS=453876296284110872]
[2024/11/12 16:37:46.401 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203684685] [SnapshotTS=453876296284110872]
[2024/11/12 16:37:46.406 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203684687] [SnapshotTS=453876296284110872]
[2024/11/12 16:37:46.410 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203684689] [SnapshotTS=453876296284110872]
[2024/11/12 16:37:46.415 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203684691] [SnapshotTS=453876296284110872]
[2024/11/12 16:37:46.419 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203684693] [SnapshotTS=453876296284110872]
[2024/11/12 16:37:46.421 +08:00] [INFO] [session.go:4048] ["use snapshot schema"] [conn=8441874356203684685] [schemaVersion=1750]
[2024/11/12 16:37:46.423 +08:00] [WARN] [session.go:2241] ["compile SQL failed"] [conn=8441874356203684685] [error="[planner:1054]Unknown column '_tidb_rowid' in 'field list'"] [SQL="SELECT _tidb_rowid from `bsppr`.`xpost` LIMIT 1"]
[2024/11/12 16:37:46.423 +08:00] [INFO] [conn.go:1200] ["command dispatched failed"] [conn=8441874356203684685] [connInfo="id:8441874356203684685, addr:192.168.241.67:12960 status:11, collation:utf8mb4_general_ci, user:root"] [command=Query] [status="inTxn:1, autocommit:1"] [sql="SELECT _tidb_rowid from `bsppr`.`xpost` LIMIT 1"] [txn_mode=OPTIMISTIC] [timestamp=453876296284110898] [err="[planner:1054]Unknown column '_tidb_rowid' in 'field list'\ngithub.com/pingcap/errors.AddStack\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20231212100244-799fae176cfb/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20231212100244-799fae176cfb/normalize.go:164\ngithub.com/pingcap/tidb/planner/core.(*expressionRewriter).toColumn\n\t/workspace/source/tidb/planner/core/expression_rewriter.go:2074\ngithub.com/pingcap/tidb/planner/core.(*expressionRewriter).Leave\n\t/workspace/source/tidb/planner/core/expression_rewriter.go:1185\ngithub.com/pingcap/tidb/parser/ast.(*ColumnName).Accept\n\t/workspace/source/tidb/parser/ast/expressions.go:534\ngithub.com/pingcap/tidb/parser/ast.(*ColumnNameExpr).Accept\n\t/workspace/source/tidb/parser/ast/expressions.go:596\ngithub.com/pingcap/tidb/planner/core.(*PlanBuilder).rewriteExprNode\n\t/workspace/source/tidb/planner/core/expression_rewriter.go:200\ngithub.com/pingcap/tidb/planner/core.(*PlanBuilder).rewriteWithPreprocess\n\t/workspace/source/tidb/planner/core/expression_rewriter.go:146\ngithub.com/pingcap/tidb/planner/core.(*PlanBuilder).buildProjection\n\t/workspace/source/tidb/planner/core/logical_plan_builder.go:1557\ngithub.com/pingcap/tidb/planner/core.(*PlanBuilder).buildSelect\n\t/workspace/source/tidb/planner/core/logical_plan_builder.go:4314\ngithub.com/pingcap/tidb/planner/core.(*PlanBuilder).Build\n\t/workspace/source/tidb/planner/core/planbuilder.go:852\ngithub.com/pingcap/tidb/planner.buildLogicalPlan\n\t/workspace/source/tidb/planner/optimize.go:560\ngithub.com/pingcap/tidb/planner.optimize\n\t/workspace/source/tidb/planner/optimize.go:479\ngithub.com/pingcap/tidb/planner.Optimize\n\t/workspace/source/tidb/planner/optimize.go:337\ngithub.com/pingcap/tidb/executor.(*Compiler).Compile\n\t/workspace/source/tidb/executor/compiler.go:98\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/workspace/source/tidb/session/session.go:2211\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt\n\t/workspace/source/tidb/server/driver_tidb.go:294\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt\n\t/workspace/source/tidb/server/conn.go:2133\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery\n\t/workspace/source/tidb/server/conn.go:1901\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/workspace/source/tidb/server/conn.go:1388\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/workspace/source/tidb/server/conn.go:1169\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/workspace/source/tidb/server/server.go:718\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"]
[2024/11/12 16:38:46.850 +08:00] [WARN] [expensivequery.go:118] [expensive_query] [cost_time=60.039109927s] [conn=8441874356203684685] [user=root] [txn_start_ts=453876296284110872] [mem_max="46344228 Bytes (44.2 MB)"] [sql="SELECT `postid` FROM `bsppr`.`xpost` TABLESAMPLE REGIONS() ORDER BY `postid`"]
9

后三行内有expected element type <Error> but have <html> 这个错误信息，在minio的 github 的issue 查来下好像有类似报错但是没有前置。issue里面有提及nginx这类负载，可以参考下
Error when copying files - expected element type but have · Issue #18536 · minio/minio · GitHub

10

image
image1128×802 54.6 KB

他这个是说用错了endpoint用成了console地址，但我用的是没问题的，因为我这个endpoint现在用在了br备份上，正常使用挺长时间了没有问题的~
其他：因为br没法满足备份匹配条件的表数据，所以才想用dumpling备份条件数据的

11

看不出，有其他备份存储么，试一试

12

不用远程存储，即使是存到本地硬盘上，都有问题。

[tidb@b67 ~]$ ~/tidb-toolkit/dumpling  -u root -p 'xxxxx!pE'  -P 4000 -h 192.168.241.85  -T bsppr.xpost --where "update_time < '2023-03-10 00:00:00'" -o /opt/dumpling/before-2023-03-01/
Release version: v7.1.5
Git commit hash: caa60c0917a886933a525d25e17057faac5b4da2
Git branch:      HEAD
Build timestamp: 2024-04-24 11:09:23Z
Go version:      go version go1.20.10 linux/amd64

[2024/11/13 10:59:38.629 +08:00] [INFO] [versions.go:54] ["Welcome to dumpling"] ["Release Version"=v7.1.5] ["Git Commit Hash"=caa60c0917a886933a525d25e17057faac5b4da2] ["Git Branch"=HEAD] ["Build timestamp"="2024-04-24 11:09:23"] ["Go Version"="go version go1.20.10 linux/amd64"]
[2024/11/13 10:59:38.635 +08:00] [INFO] [version.go:434] ["detect server version"] [type=TiDB] [version=7.1.5]
[2024/11/13 10:59:38.650 +08:00] [INFO] [client.go:311] ["[pd] create pd client with endpoints and keyspace"] [pd-address="[192.168.241.60:2379,192.168.241.59:2379,192.168.241.61:2379]"] [keyspace-id=0]
[2024/11/13 10:59:38.660 +08:00] [INFO] [pd_service_discovery.go:521] ["[pd] update member urls"] [old-urls="[http://192.168.241.60:2379,http://192.168.241.59:2379,http://192.168.241.61:2379]"] [new-urls="[http://192.168.241.59:2379,http://192.168.241.60:2379,http://192.168.241.61:2379]"]
[2024/11/13 10:59:38.660 +08:00] [INFO] [pd_service_discovery.go:546] ["[pd] switch leader"] [new-leader=http://192.168.241.61:2379] [old-leader=]
[2024/11/13 10:59:38.660 +08:00] [INFO] [pd_service_discovery.go:177] ["[pd] init cluster id"] [cluster-id=7210697100626091534]
[2024/11/13 10:59:38.661 +08:00] [INFO] [client.go:386] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2024/11/13 10:59:38.661 +08:00] [INFO] [tso_client.go:231] ["[tso] switch dc tso allocator serving address"] [dc-location=global] [new-address=http://192.168.241.61:2379]
[2024/11/13 10:59:38.661 +08:00] [INFO] [tso_dispatcher.go:296] ["[tso] tso dispatcher created"] [dc-location=global]
[2024/11/13 10:59:38.661 +08:00] [INFO] [client.go:428] ["[pd] service mode changed"] [old-mode=PD_SVC_MODE] [new-mode=PD_SVC_MODE]
[2024/11/13 10:59:38.663 +08:00] [INFO] [dump.go:1519] ["generate dumpling gc safePoint id"] [id=dumpling_1731466778663079714]
[2024/11/13 10:59:38.675 +08:00] [INFO] [dump.go:151] ["begin to run Dump"] [conf="{\"s3\":{\"endpoint\":\"\",\"region\":\"\",\"storage-class\":\"\",\"sse\":\"\",\"sse-kms-key-id\":\"\",\"acl\":\"\",\"access-key\":\"\",\"secret-access-key\":\"\",\"session-token\":\"\",\"provider\":\"\",\"force-path-style\":true,\"use-accelerate-endpoint\":false,\"role-arn\":\"\",\"external-id\":\"\",\"object-lock-enabled\":false},\"gcs\":{\"endpoint\":\"\",\"storage-class\":\"\",\"predefined-acl\":\"\",\"credentials-file\":\"\"},\"azblob\":{\"endpoint\":\"\",\"account-name\":\"\",\"account-key\":\"\",\"access-tier\":\"\"},\"AllowCleartextPasswords\":false,\"SortByPk\":true,\"NoViews\":true,\"NoSequences\":true,\"NoHeader\":false,\"NoSchemas\":false,\"NoData\":false,\"CompleteInsert\":false,\"TransactionalConsistency\":true,\"EscapeBackslash\":true,\"DumpEmptyDatabase\":true,\"PosAfterConnect\":false,\"CompressType\":0,\"Host\":\"192.168.241.85\",\"Port\":4000,\"Threads\":4,\"User\":\"root\",\"Security\":{\"CAPath\":\"\",\"CertPath\":\"\",\"KeyPath\":\"\"},\"LogLevel\":\"info\",\"LogFile\":\"\",\"LogFormat\":\"text\",\"OutputDirPath\":\"/opt/dumpling/before-2023-03-01/\",\"StatusAddr\":\":8281\",\"Snapshot\":\"453893627213774876\",\"Consistency\":\"snapshot\",\"CsvNullValue\":\"\\\\N\",\"SQL\":\"\",\"CsvSeparator\":\",\",\"CsvDelimiter\":\"\\\"\",\"Databases\":[],\"Where\":\"update_time \\u003c '2023-03-10 00:00:00'\",\"FileType\":\"sql\",\"ServerInfo\":{\"ServerType\":3,\"ServerVersion\":\"7.1.5\",\"HasTiKV\":true},\"Rows\":0,\"ReadTimeout\":900000000000,\"TiDBMemQuotaQuery\":0,\"FileSize\":0,\"StatementSize\":1000000,\"SessionParams\":{\"tidb_snapshot\":\"453893627213774876\"},\"Tables\":{\"bsppr\":[{\"Name\":\"xpost\",\"AvgRowLength\":0,\"Type\":0}]},\"CollationCompatible\":\"loose\",\"IOTotalBytes\":null,\"Net\":\"\"}"]
[2024/11/13 11:01:38.704 +08:00] [INFO] [status.go:37] [progress] [tables="0/1 (0.0%)"] ["finished rows"=0] ["estimate total rows"=7400524299] ["finished size"=0B] ["average speed(MiB/s)"=0] ["recent speed bps"=0] ["chunks progress"=]
[2024/11/13 11:03:38.704 +08:00] [INFO] [status.go:37] [progress] [tables="0/1 (0.0%)"] ["finished rows"=0] ["estimate total rows"=7400524299] ["finished size"=0B] ["average speed(MiB/s)"=0] ["recent speed bps"=0] ["chunks progress"=]


[2024/11/13 11:05:38.703 +08:00] [INFO] [status.go:37] [progress] [tables="0/1 (0.0%)"] ["finished rows"=0] ["estimate total rows"=7400524299] ["finished size"=0B] ["average speed(MiB/s)"=0] ["recent speed bps"=0] ["chunks progress"=]

[2024/11/13 11:07:38.703 +08:00] [INFO] [status.go:37] [progress] [tables="0/1 (0.0%)"] ["finished rows"=0] ["estimate total rows"=7400524299] ["finished size"=0B] ["average speed(MiB/s)"=0] ["recent speed bps"=0] ["chunks progress"=]


[2024/11/13 10:58:03.143 +08:00] [INFO] [job_manager.go:489] ["try lock new job"] [ttl-worker=job-manager] [tableID=1929]
[2024/11/13 10:58:03.146 +08:00] [WARN] [job_manager.go:496] ["fail to create new job"] [ttl-worker=job-manager] [error="couldn't schedule ttl job"]
[2024/11/13 10:59:38.671 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203685091] [SnapshotTS=453893627213774876]
[2024/11/13 10:59:38.676 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203685093] [SnapshotTS=453893627213774876]
[2024/11/13 10:59:38.682 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203685095] [SnapshotTS=453893627213774876]
[2024/11/13 10:59:38.689 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203685097] [SnapshotTS=453893627213774876]
[2024/11/13 10:59:38.694 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203685099] [SnapshotTS=453893627213774876]
[2024/11/13 10:59:38.699 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203685101] [SnapshotTS=453893627213774876]
[2024/11/13 10:59:38.704 +08:00] [INFO] [set.go:309] ["load snapshot info schema"] [conn=8441874356203685103] [SnapshotTS=453893627213774876]
[2024/11/13 10:59:38.706 +08:00] [INFO] [session.go:4048] ["use snapshot schema"] [conn=8441874356203685095] [schemaVersion=1750]
[2024/11/13 10:59:38.708 +08:00] [WARN] [session.go:2241] ["compile SQL failed"] [conn=8441874356203685095] [error="[planner:1054]Unknown column '_tidb_rowid' in 'field list'"] [SQL="SELECT _tidb_rowid from `bsppr`.`xpost` LIMIT 1"]
[2024/11/13 10:59:38.708 +08:00] [INFO] [conn.go:1200] ["command dispatched failed"] [conn=8441874356203685095] [connInfo="id:8441874356203685095, addr:192.168.241.67:28607 status:11, collation:utf8mb4_general_ci, user:root"] [command=Query] [status="inTxn:1, autocommit:1"] [sql="SELECT _tidb_rowid from `bsppr`.`xpost` LIMIT 1"] [txn_mode=OPTIMISTIC] [timestamp=453893627226882060] [err="[planner:1054]Unknown column '_tidb_rowid' in 'field list'\ngithub.com/pingcap/errors.AddStack\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20231212100244-799fae176cfb/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20231212100244-799fae176cfb/normalize.go:164\ngithub.com/pingcap/tidb/planner/core.(*expressionRewriter).toColumn\n\t/workspace/source/tidb/planner/core/expression_rewriter.go:2074\ngithub.com/pingcap/tidb/planner/core.(*expressionRewriter).Leave\n\t/workspace/source/tidb/planner/core/expression_rewriter.go:1185\ngithub.com/pingcap/tidb/parser/ast.(*ColumnName).Accept\n\t/workspace/source/tidb/parser/ast/expressions.go:534\ngithub.com/pingcap/tidb/parser/ast.(*ColumnNameExpr).Accept\n\t/workspace/source/tidb/parser/ast/expressions.go:596\ngithub.com/pingcap/tidb/planner/core.(*PlanBuilder).rewriteExprNode\n\t/workspace/source/tidb/planner/core/expression_rewriter.go:200\ngithub.com/pingcap/tidb/planner/core.(*PlanBuilder).rewriteWithPreprocess\n\t/workspace/source/tidb/planner/core/expression_rewriter.go:146\ngithub.com/pingcap/tidb/planner/core.(*PlanBuilder).buildProjection\n\t/workspace/source/tidb/planner/core/logical_plan_builder.go:1557\ngithub.com/pingcap/tidb/planner/core.(*PlanBuilder).buildSelect\n\t/workspace/source/tidb/planner/core/logical_plan_builder.go:4314\ngithub.com/pingcap/tidb/planner/core.(*PlanBuilder).Build\n\t/workspace/source/tidb/planner/core/planbuilder.go:852\ngithub.com/pingcap/tidb/planner.buildLogicalPlan\n\t/workspace/source/tidb/planner/optimize.go:560\ngithub.com/pingcap/tidb/planner.optimize\n\t/workspace/source/tidb/planner/optimize.go:479\ngithub.com/pingcap/tidb/planner.Optimize\n\t/workspace/source/tidb/planner/optimize.go:337\ngithub.com/pingcap/tidb/executor.(*Compiler).Compile\n\t/workspace/source/tidb/executor/compiler.go:98\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/workspace/source/tidb/session/session.go:2211\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt\n\t/workspace/source/tidb/server/driver_tidb.go:294\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt\n\t/workspace/source/tidb/server/conn.go:2133\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery\n\t/workspace/source/tidb/server/conn.go:1901\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/workspace/source/tidb/server/conn.go:1388\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/workspace/source/tidb/server/conn.go:1169\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/workspace/source/tidb/server/server.go:718\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"]
[2024/11/13 11:00:12.638 +08:00] [INFO] [domain.go:2708] ["refreshServerIDTTL succeed"] [serverID=3838920] ["lease id"=53a48fb8aeb520d1]
[2024/11/13 11:00:38.750 +08:00] [WARN] [expensivequery.go:118] [expensive_query] [cost_time=60.041214578s] [conn=8441874356203685095] [user=root] [txn_start_ts=453893627213774876] [mem_max="0 Bytes (0 Bytes)"] [sql="SELECT * FROM `bsppr`.`xpost` LIMIT 1"]
[2024/11/13 11:00:57.488 +08:00] [WARN] [coprocessor.go:431] ["buildCopTasks takes too much time"] [elapsed=1m18.777688667s] ["range len"=1] ["task len"=189030]
[2024/11/13 11:00:57.491 +08:00] [INFO] [controller.go:430] ["[resource group controller] create resource group cost controller"] [name=default]```

以上是我备份到本地的日志 上面是控制台
下面是tidb
就一直在那卡着不动，  好像dumpling就不太适合备份特别大的表数据呀
13

你这应该是s3服务认证问题，你试着用官方工具，或s3链接工具上传下文件，看存不存在权限问题

14

传个文件上去试试，先确认minio上传下载文件没有问题

15

这个minio是正在使用的每天 br备份不同的库表数据，同一个用户密码一样的endpoint

16

本地都不行吗？不应该啊，可以卡住的时候，看下当前的会话列表。

17

看过，都是一些整表的查询，， 可能是我表太大了，其实也说不太通，我的条件字段是有索引的，手动查询数据都是挺快的，不知道为啥这个这么慢

18

看来虽然有索引，但是还是在扫全表 :thinking:

19

这个说明了啥？

20

我用的也是这个命令，没有问题。
1、你可以先用个小的表验证下备份命令是否正确在来反向验证是不是大表的问题
2、建议你dumpling的时候添加 --logfile 和 --loglevel 参数， level设置为debug再试试
3、看到你后面的报错有个 Unknown column '_tidb_rowid' in 'field list'"]

[2024/11/12 16:37:46.423 +08:00] [WARN] [session.go:2241] ["compile SQL failed"] [conn=8441874356203684685] [error="[planner:1054]Unknown column '_tidb_rowid' in 'field list'"] [SQL="SELECT _tidb_rowid from `bsppr`.`xpost` LIMIT 1"]

确认下对应的表不是有了主键，有主键的好像就没有这个字段了吧？