drainer同步时丢失数据

🪐 TiDB 技术问题 备份&数据迁移
1

【 TiDB 使用环境】测试
【 TiDB 版本】v4.0.4
【复现路径】
运行大事务操作
【遇到的问题：问题现象及影响】
有一批数据需要初始化，考虑到这批数据涉及的库已经在drainer的syncer.ignore-schemas配置中指明，所以临时将tidb配置performance.txn-total-size-limit从800M调至8G，但drainer还是出事了

从现象看是drainer同步部分数据丢失，同时该时间段有连pump的报错，pump本身也有报错，给人的感觉就是，从9点56分之后，drainer只拉取了部分binlog，我尝试自己用消费者消费kafka里的数据，发现kafka里的数据就是不全的，具体体现在某表的sql的id不连贯

看起来像是drainer从此读不到某个pump的binlog了？我这里是三个pump实例

请问我需要调整pump的什么配置避免这种情况呢？

而且我要如何修复数据呢？

【资源配置】
【附件：截图/日志/监控】
drainer日志
drainer.log (115.2 KB)
pump日志
pump_stderr.log (9.7 KB)
pump.log (54.3 KB)
下游数据库缺失的sql，id不连贯

image
image1379×338 31.2 KB

2

可以看看监控，是否有skip binlog 的tidb server

3

image
image931×296 16.5 KB

确实是有的，而且在我中午重启tidb-server后就恢复正常了，请问这个要如何追查原因呢，下次我想避免这种情况

4

在出现skip binlog 的tidb server 的节点的日志里面搜binlog 关键字，应该记录了skip binlog 的原因，根据你上面说的删除大事务来看，应该是这个删除的大事务导致的。可以根据日志里面的startts 到慢日志里面找一下具体的sql

5

看到tidb-server中有大量的连不上pump端口的报错

[2023/06/09 09:34:36.803 +08:00] [INFO] [main.go:332] [tidb-server] ["create pumps client success, ignore binlog error"=true]
[2023/06/09 09:55:44.237 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442046981399904258] ["commit ts"=0] [length=1905219679] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068575891462] ["commit ts"=0] [length=138] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068615213059] ["commit ts"=0] [length=145] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068431712258] ["commit ts"=0] [length=276] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068313747464] ["commit ts"=0] [length=140] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068195782659] ["commit ts"=0] [length=145] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068339961865] ["commit ts"=0] [length=138] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068405497876] ["commit ts"=0] [length=143] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068130246662] ["commit ts"=0] [length=143] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068707225602] ["commit ts"=0] [length=246] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068366176258] ["commit ts"=0] [length=188] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.809 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068313747464] ["commit ts"=0] [length=140] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2023/06/09 09:55:44.809 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068195782659] ["commit ts"=0] [length=145] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2023/06/09 09:55:44.809 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068339961865] ["commit ts"=0] [length=138] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2023/06/09 09:55:45.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:46.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:47.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:48.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:49.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:50.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:51.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:52.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:53.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:54.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:55.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:56.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:57.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:58.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:59.338 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.18.27:8250] ["binlog type"=Prewrite] ["start ts"=442046981399904258] ["commit ts"=0] [length=1905219679] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"]

请问是tidb主动往pump写binlog的吗？我一直以为是pump主动去tidb-server取

目前观察pump只有一个实例在那个时间点内存稍微大了些，但远没到主机内存瓶颈，磁盘也是够的，不知道该怎么往下查了

6

你的pump 有几个节点？ 再看看有没有binlog ERROR相关的日志

7

你看下你的配置参数
binlog.ignore-error
是不是tidb写pump失败，然后跳过写pump的阶段了

8

看着像命中了啥呢

9

是开启的, 而且我看文档, 确实是触发这个错误后, 就再也不写binlog了, 但现在感觉问题在为什么会写binlog失败, 后面想避免写binlog失败这个情况

10

看日志, 像是大事务导致pump异常, 而后tidb-server连不上pump触发binlog.ignore-error, 从此这个tidb-server就再也不写binlog了, 由于我看不懂go语言, 所以不知道这个大事务为啥为导致pump异常


goroutine 12626829 [semacquire]:
sync.runtime_Semacquire(0xc0041cc708)
        /usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc0041cc700)
        /usr/local/go/src/sync/waitgroup.go:130 +0x64
github.com/pingcap/tidb-binlog/pump/storage.(*Append).writeBinlog(0xc000194340, 0xc000b26480, 0x0)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/pump/storage/storage.go:848 +0x2cc
github.com/pingcap/tidb-binlog/pump/storage.(*Append).WriteBinlog(0xc000194340, 0xc000b26480, 0x8a, 0x90)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/pump/storage/storage.go:813 +0x5f
github.com/pingcap/tidb-binlog/pump.(*Server).writeBinlog(0xc00033c500, 0x13920c0, 0xc00371a060, 0xc002fa4e20, 0x106d700, 0x0, 0x0, 0x0)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/pump/server.go:242 +0x713
github.com/pingcap/tidb-binlog/pump.(*Server).WriteBinlog(0xc00033c500, 0x13920c0, 0xc00371a060, 0xc002fa4e20, 0xc00033c500, 0xc00371a060, 0xc0005c2a80)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/pump/server.go:192 +0x5c
github.com/pingcap/tipb/go-binlog._Pump_WriteBinlog_Handler(0x11ae1a0, 0xc00033c500, 0x13920c0, 0xc00371a060, 0xc0041cc660, 0x0, 0x13920c0, 0xc00371a060, 0xc000b10b40, 0x97)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/github.com/pingcap/tipb@v0.0.0-20200212061130-c4d518eb1d60/go-binlog/pump.pb.go:330 +0x217
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000372180, 0x139fc40, 0xc002e96000, 0xc00313ac00, 0xc0005a4e40, 0x1c6d220, 0x0, 0x0, 0x0)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/google.golang.org/grpc@v1.25.1/server.go:1007 +0x460
google.golang.org/grpc.(*Server).handleStream(0xc000372180, 0x139fc40, 0xc002e96000, 0xc00313ac00, 0x0)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/google.golang.org/grpc@v1.25.1/server.go:1287 +0xd97
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0037db370, 0xc000372180, 0x139fc40, 0xc002e96000, 0xc00313ac00)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/google.golang.org/grpc@v1.25.1/server.go:722 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/google.golang.org/grpc@v1.25.1/server.go:720 +0xa1

goroutine 12621185 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000b63950, 0x1, 0x0, 0x0, 0x0, 0x0)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/google.golang.org/grpc@v1.25.1/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc003515ec0, 0x0, 0x0)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/google.golang.org/grpc@v1.25.1/internal/transport/controlbuf.go:513 +0x1e3
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc002e96180)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/google.golang.org/grpc@v1.25.1/internal/transport/http2_server.go:290 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/google.golang.org/grpc@v1.25.1/internal/transport/http2_server.go:287 +0xfe4

goroutine 2049585 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f7b4c564988, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc002b04318, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc002b04300, 0xc000bba000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc002b04300, 0xc000bba000, 0x8000, 0x8000, 0x0, 0x800010601, 0x0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000337198, 0xc000bba000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc00f9c3800, 0xc000472738, 0x9, 0x9, 0xc0000f4e00, 0x7f7b4e84d6d0, 0x0)
        /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x136cce0, 0xc00f9c3800, 0xc000472738, 0x9, 0x9, 0x9, 0x926265, 0xc003421aec, 0xc000706de8)
        /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc000472738, 0x9, 0x9, 0x136cce0, 0xc00f9c3800, 0x0, 0xc118bf8700000000, 0x3005de341db3e, 0x1caf0e0)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/golang.org/x/net@v0.0.0-20200301022130-244492dfa37a/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000472700, 0xc003421ae0, 0x1caf0e0, 0x0, 0x0)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/golang.org/x/net@v0.0.0-20200301022130-244492dfa37a/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0001f9340)
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/google.golang.org/grpc@v1.25.1/internal/transport/http2_client.go:1250 +0x183
created by google.golang.org/grpc/internal/transport.newHTTP2Client
        /home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/pkg/mod/google.golang.org/grpc@v1.25.1/internal/transport/http2_client.go:293 +0xd23
11

三个节点, 三个节点的日志都差不多,我放在附件里了, 并没发现binlog相关的异常报错, 但是有其他报错

12

在, pump_stderr.log里, 结合pump.log里的"start pump…", 分析他是触发了什么然后挂了重启

13

中间是否做过节点的扩容缩容、重启过tidb？

14

当天没有, 只是放开了事务大小限制, 初始化了一大批数据, 但是初始化数据相关的表已经在drainer中做了过滤

15

我之前drainer也发生过丢数据 不过原因是在扩容缩容节点后tidb的binlog配置被自动改成关闭状态

16

应该和我这个现象不一样, 我这里是进行大事务前同步正常的, 事发后重启tidb-server后也恢复正常

17

下游集群达到一致的状态是指：下游集群的数据等同于上游设置了 tidb_snapshot = ts 的快照。

checkpoint 状态一致性是指：Drainer checkpoint 通过 consistent 保存了同步的一致性状态。Drainer 运行时 consistent 为 false，Drainer 正常退出后 consistent 更新为 true。

18

额…这个貌似不能解释pump异常的原因吧?

19

看下相对应时间段tidb.log 看里面有什么信息

20

tidb里面也是连不上pump的

[2023/06/09 09:34:36.803 +08:00] [INFO] [main.go:332] [tidb-server] ["create pumps client success, ignore binlog error"=true]
[2023/06/09 09:55:44.237 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442046981399904258] ["commit ts"=0] [length=1905219679] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068575891462] ["commit ts"=0] [length=138] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068615213059] ["commit ts"=0] [length=145] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068431712258] ["commit ts"=0] [length=276] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068313747464] ["commit ts"=0] [length=140] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068195782659] ["commit ts"=0] [length=145] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068339961865] ["commit ts"=0] [length=138] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068405497876] ["commit ts"=0] [length=143] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068130246662] ["commit ts"=0] [length=143] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068707225602] ["commit ts"=0] [length=246] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.707 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068366176258] ["commit ts"=0] [length=188] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/06/09 09:55:44.809 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068313747464] ["commit ts"=0] [length=140] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2023/06/09 09:55:44.809 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068195782659] ["commit ts"=0] [length=145] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2023/06/09 09:55:44.809 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Prewrite] ["start ts"=442047068339961865] ["commit ts"=0] [length=138] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2023/06/09 09:55:45.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:46.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:47.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:48.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:49.711 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:50.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:51.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:52.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:53.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:54.712 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:55.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:56.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:57.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:58.713 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.17.65:8250] ["binlog type"=Commit] ["start ts"=442047068130246665] ["commit ts"=442047068588998659] [length=47] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.16.17.65:8250: connect: connection refused\""]
[2023/06/09 09:55:59.338 +08:00] [WARN] [client.go:295] ["[pumps client] write binlog to pump failed"] [NodeID=172.16.18.27:8250] ["binlog type"=Prewrite] ["start ts"=442046981399904258] ["commit ts"=0] [length=1905219679] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"]