关于tidb导出大量数据,出现 connect: connection refused的问题

【 TiDB 使用环境】测试
【 TiDB 版本】v6.5.0
【复现路径】

tiup dumpling -u root -p'HGe34545e9' -P 4000 -h 10.18.104.156 --filetype sql -t 4 -L ./baklog.log -o ./bak -r 200000 -F256MiB



求教下,我在进行数据导出的时候 汇报connect: connection refused。无法连接数据据从而导致备份失败,求教下是什么原因导致的。以下为运行日志
过程中我监控了cpu 和内存 cpu拉起20% 内存90%
baklog.log (33.8 KB)

看一下tidb的状态,是不是重启了。。。

123

怎么查看tidb 是否重启过? 我只能在监控上看到一段时间tidb是没有性能值的。
我在对应tidb服务器 156机器上日志中也没有看到报错。

但是,这样的导出中断的情况,我重复试验了6-8次,看监控样子不像是tidb 重启了。

2379站点的Dashboard里,可以看。tiup cluster display命令也可以看

我研究了下,发现在进行tiup dumpling 备份的时候响应tidb 节点会Down掉。但是我查看tidb节点日志后发发现节点没有重启日志。而是在备份过程中服务器无法登陆了,想着是不是io或者open_file打开过多导致服务端口假死。这也是猜测,大神帮忙指导下这种情况怎么验证啊。
tiup dumpling -u root -p’HGe34545e9’ -P 4000 -h 10.18.104.162 --filetype sql -t 1 -L ./baklog.log -o ./bak -r 2000 -F32MiB

这个里面看看呢?
http://10.18.104.162:2379/dashboard/#/overview

http://10.18.104.162:2379/dashboard 无法访问。

162为什么down了,看了日志吗?

操作系统日志也可以看一下的

现在能看到的错误日志就这个了, 这会服务也挂了 拉不起来
[2023/05/18 09:21:47.987 +08:00] [INFO] [sst_importer.rs:442] [“shrink cache by tick”] [“retain size”=0] [“shrink size”=0]
[2023/05/18 09:21:52.691 +08:00] [WARN] [errors.rs:155] [“backup stream meet error”] [verbose_err=“Etcd(GRpcStatus(Status { code: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", source: None }))”] [err=“Etcd meet error grpc request error: status: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", details: [], metadata: MetadataMap { headers: {} }”] [context=“failed to get backup stream task”]
[2023/05/18 09:21:57.693 +08:00] [WARN] [errors.rs:155] [“backup stream meet error”] [verbose_err=“Etcd(GRpcStatus(Status { code: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", source: None }))”] [err=“Etcd meet error grpc request error: status: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", details: [], metadata: MetadataMap { headers: {} }”] [context=“failed to get backup stream task”]
[2023/05/18 09:21:57.988 +08:00] [INFO] [sst_importer.rs:442] [“shrink cache by tick”] [“retain size”=0] [“shrink size”=0]
[2023/05/18 09:22:02.694 +08:00] [WARN] [errors.rs:155] [“backup stream meet error”] [verbose_err=“Etcd(GRpcStatus(Status { code: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", source: None }))”] [err=“Etcd meet error grpc request error: status: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", details: [], metadata: MetadataMap { headers: {} }”] [context=“failed to get backup stream task”]
[2023/05/18 09:22:07.695 +08:00] [WARN] [errors.rs:155] [“backup stream meet error”] [verbose_err=“Etcd(GRpcStatus(Status { code: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", source: None }))”] [err=“Etcd meet error grpc request error: status: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", details: [], metadata: MetadataMap { headers: {} }”] [context=“failed to get backup stream task”]
[2023/05/18 09:22:07.989 +08:00] [INFO] [sst_importer.rs:442] [“shrink cache by tick”] [“retain size”=0] [“shrink size”=0]
[2023/05/18 09:22:12.696 +08:00] [WARN] [errors.rs:155] [“backup stream meet error”] [verbose_err=“Etcd(GRpcStatus(Status { code: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", source: None }))”] [err=“Etcd meet error grpc request error: status: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", details: [], metadata: MetadataMap { headers: {} }”] [context=“failed to get backup stream task”]
[2023/05/18 09:22:17.697 +08:00] [WARN] [errors.rs:155] [“backup stream meet error”] [verbose_err=“Etcd(GRpcStatus(Status { code: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", source: None }))”] [err=“Etcd meet error grpc request error: status: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", details: [], metadata: MetadataMap { headers: {} }”] [context=“failed to get backup stream task”]
[2023/05/18 09:22:17.991 +08:00] [INFO] [sst_importer.rs:442] [“shrink cache by tick”] [“retain size”=0] [“shrink size”=0]
[2023/05/18 09:22:22.698 +08:00] [WARN] [errors.rs:155] [“backup stream meet error”] [verbose_err=“Etcd(GRpcStatus(Status { code: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", source: None }))”] [err=“Etcd meet error grpc request error: status: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", details: [], metadata: MetadataMap { headers: {} }”] [context=“failed to get backup stream task”]
[2023/05/18 09:22:27.700 +08:00] [WARN] [errors.rs:155] [“backup stream meet error”] [verbose_err=“Etcd(GRpcStatus(Status { code: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", source: None }))”] [err=“Etcd meet error grpc request error: status: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", details: [], metadata: MetadataMap { headers: {} }”] [context=“failed to get backup stream task”]
[2023/05/18 09:22:27.991 +08:00] [INFO] [sst_importer.rs:442] [“shrink cache by tick”] [“retain size”=0] [“shrink size”=0]
[2023/05/18 09:22:32.701 +08:00] [WARN] [errors.rs:155] [“backup stream meet error”] [verbose_err=“Etcd(GRpcStatus(Status { code: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", source: None }))”] [err=“Etcd meet error grpc request error: status: Unknown, message: "Service was not ready: buffered service failed: load balancer discovery error: transport error: transport error", details: [], metadata: MetadataMap { headers: {} }”] [context=“failed to get backup stream task”]

您好

可以看下 TiDB - Server - Uptime 中断开连接的 tidb-server 是否发生重启。
可以在 操作系统 执行 dmesg -T | grep tidb-server 看是否发生 OOM。
服务器是多大内存的?按经验 -t 4 是不会跑挂 tidb-server 的。

您好 我按照您的方法进行了查看,但是没有搜索到任何结果如下图:
image

我备份数据使用的语句如下:
mydumper -u root -p ‘HGYe9’ -P 4000 -h 10.18.104.156 --regex ‘^(?!(mysql|test|information_schema|METRICS_SCHEMA|performance_schema|sys))’ -G -R -E -c -K -r 20000 -t 1 -F 32 --no-schemas -o ./bak -L ./bak.log


另外我发现 156这台服务器像是挂掉了。

我的服务器内存是8G 是台虚拟机。

我发现一旦进行备份服务器都连接不上了。