tikv缩容，影响tidb连接中断

TiDB_C罗 · 2023 年7 月 4 日 02:50

【 TiDB 使用环境】生产环境
【 TiDB 版本】v5.4.0
【复现路径】tikv磁盘缩容
【遇到的问题：问题现象及影响】
【附件：截图/日志/监控】
tikv磁盘缩容步骤：
1.先关闭tikv
2.拷贝数据
3.再启动tikv

1.先关闭tikv

tiup cluster stop tidb-risk -N 10.0.0.17:20160

查看集群状态，显示Disconnected

ID                    Role          Host            Ports        OS/Arch       Status        Data Dir                           Deploy Dir
--                    ----          ----            -----        -------       ------        --------                           ----------
10.0.0.10:9093   alertmanager  10.0.0.10  9093/9094    linux/x86_64  Up            /data/tidb-data/alertmanager-9093  /data/tidb-deploy/alertmanager-9093
10.0.0.10:3000   grafana       10.0.0.10  3000         linux/x86_64  Up            -                                  /data/tidb-deploy/grafana-3000
10.0.0.11:2379   pd            10.0.0.11  2379/2380    linux/x86_64  Up|L          /data/tidb-data/pd-2379            /data/tidb-deploy/pd-2379
10.0.0.12:2379   pd            10.0.0.12  2379/2380    linux/x86_64  Up            /data/tidb-data/pd-2379            /data/tidb-deploy/pd-2379
10.0.0.13:2379    pd            10.0.0.13   2379/2380    linux/x86_64  Up|UI         /data/tidb-data/pd-2379            /data/tidb-deploy/pd-2379
10.0.0.10:9090   prometheus    10.0.0.10  9090/12020   linux/x86_64  Up            /data/tidb-data/prometheus-9090    /data/tidb-deploy/prometheus-9090
10.0.0.14:4000   tidb          10.0.0.14  4000/10080   linux/x86_64  Up            -                                  /data/tidb-deploy/tidb-4000
10.0.0.15:4000   tidb          10.0.0.15  4000/10080   linux/x86_64  Up            -                                  /data/tidb-deploy/tidb-4000
10.0.0.16:4000    tidb          10.0.0.16   4000/10080   linux/x86_64  Up            -                                  /data/tidb-deploy/tidb-4000
10.0.0.17:20160  tikv          10.0.0.17  20160/20180  linux/x86_64  Disconnected  /data/tidb-data/tikv-20160         /data/tidb-deploy/tikv-20160
10.0.0.18:20160  tikv          10.0.0.18  20160/20180  linux/x86_64  Up            /data/tidb-data/tikv-20160         /data/tidb-deploy/tikv-20160
10.0.0.19:20160   tikv          10.0.0.19   20160/20180  linux/x86_64  Up            /data/tidb-data/tikv-20160         /data/tidb-deploy/tikv-20160

2.拷贝数据
cp -a /data /data1

3.启动tikv
tiup cluster start tidb-risk -N 10.0.0.17:20160

查看tikv-detail

关闭一个tikv，leader 发生选举，恢复后各节点又均衡leader

这时业务收到告警：
com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

The last packet successfully received from the server was 10,674 milliseconds ago. The last packet sent successfully to the server was 10,674 milliseconds ago.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
at com.mysql.cj.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:370)
at com.opay.realtime.etl.util.JdbcUtil.riskSink(JdbcUtil.scala:142)
at com.opay.risk.features.sink.TableSinkMappingBroadcastProcessExt$TableSinkMappingProcess.doProcess(TableSinkMappingBroadcastProcessExt.scala:34)
at com.opay.risk.features.sink.TableSinkMappingBroadcastProcessFunction$$anonfun$processElement$1.apply(TableSinkMappingBroadcastProcessFunction.scala:64)
at com.opay.risk.features.sink.TableSinkMappingBroadcastProcessFunction$$anonfun$processElement$1.apply(TableSinkMappingBroadcastProcessFunction.scala:63)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
at com.opay.risk.features.sink.TableSinkMappingBroadcastProcessFunction.processElement(TableSinkMappingBroadcastProcessFunction.scala:63)
at com.opay.risk.features.sink.TableSinkMappingBroadcastProcessFunction.processElement(TableSinkMappingBroadcastProcessFunction.scala:24)
at org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperator.processElement1(CoBroadcastWithNonKeyedOperator.java:110)
at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessorFactory.processRecord1(StreamTwoInputProcessorFactory.java:213)
at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessorFactory.lambda$create$0(StreamTwoInputProcessorFactory.java:178)
at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessorFactory$StreamTaskNetworkOutput.emitRecord(StreamTwoInputProcessorFactory.java:291)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66)
at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:96)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.lang.Thread.run(Thread.java:750)
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure

The last packet successfully received from the server was 10,674 milliseconds ago. The last packet sent successfully to the server was 10,674 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:546)
at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:710)
at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:649)
at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:948)
at com.mysql.cj.NativeSession.execSQL(NativeSession.java:1075)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:930)
… 25 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:67)
at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:63)
at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:45)
at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:52)
at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:41)
at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:54)
at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:44)
at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:540)
… 30 more

连接参数 jdbc:mysql://10.0.0.11:3306/orders?useSSL=false&rewriteBatchedStatements=true&autoReconnect=true
这个3306是在pd节点上起的haproxy，代理后面三个tidb-4000节点

我的理解是操作tikv，发生了leader选举，请求到新的store上，不应该影响到tidb上的连接报错，为什么这里会影响到tidb的连接中断？

tidb菜鸟一只 · 2023 年7 月 4 日 03:41

缩容是存储缩容了？

有猫万事足 · 2023 年7 月 4 日 05:54

应该是你的tikv短时不满足3副本的最低要求。

xingzhenxiang · 2023 年7 月 4 日 06:06

tidb 默认三副本，你这缩成2个tikv了

h5n1 · 2023 年7 月 4 日 06:10

先看下报错时的那3个tidb server log里有啥

TiDB_C罗 · 2023 年7 月 4 日 08:00

就是停掉实例，换了一个小些的磁盘

TiDB_C罗 · 2023 年7 月 4 日 08:02

部署是至少三个节点，可以容忍一个节点不可用的情况，我正是这种情况

TiDB_C罗 · 2023 年7 月 4 日 08:48

TiKV的错误信息

[2023/07/04 01:33:23.989 +00:00] [INFO] [signal_handler.rs:19] ["receive signal 15, stopping server..."]
[2023/07/04 01:33:24.043 +00:00] [INFO] [<unknown>] ["ipv4:10.0.0.14:39746: Sending goaway err={\"created\":\"@1688434404.043286523\",\"description\":\"Server shutdown\",\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/surface/server.cc\",\"file_line\":480,\"grpc_status\":0}"]
[2023/07/04 01:33:24.043 +00:00] [INFO] [<unknown>] ["ipv4:10.0.0.14:39742: Sending goaway err={\"created\":\"@1688434404.043293155\",\"description\":\"Server shutdown\",\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/surface/server.cc\",\"file_line\":480,\"grpc_status\":0}"]
[2023/07/04 01:33:24.044 +00:00] [INFO] [<unknown>] ["ipv4:10.0.0.14:39740: Sending goaway err={\"created\":\"@1688434404.043294237\",\"description\":\"Server shutdown\",\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/surface/server.cc\",\"file_line\":480,\"grpc_status\":0}"]
[2023/07/04 01:33:24.044 +00:00] [INFO] [<unknown>] ["ipv4:10.0.0.14:39744: Sending goaway err={\"created\":\"@1688434404.043296301\",\"description\":\"Server shutdown\",\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/surface/server.cc\",\"file_line\":480,\"grpc_status\":0}"]
[2023/07/04 01:33:24.044 +00:00] [INFO] [<unknown>] ["ipv4:10.0.0.15:60918: Sending goaway err={\"created\":\"@1688434404.043297724\",\"description\":\"Server shutdown\",\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/surface/server.cc\",\"file_line\":480,\"grpc_status\":0}"]
[2023/07/04 01:33:24.044 +00:00] [INFO] [<unknown>] ["ipv4:10.0.0.15:60912: Sending goaway err={\"created\":\"@1688434404.043299056\",\"description\":\"Server shutdown\",\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/surface/server.cc\",\"file_line\":480,\"grpc_status\":0}"]
[2023/07/04 01:33:24.044 +00:00] [INFO] [<unknown>] ["ipv4:10.0.0.15:60916: Sending goaway err={\"created\":\"@1688434404.043299958\",\"description\":\"Server shutdown\",\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/surface/server.cc\",\"file_line\":480,\"grpc_status\":0}"]
[2023/07/04 01:33:24.044 +00:00] [INFO] [<unknown>] ["ipv4:10.0.0.15:60914: Sending goaway err={\"created\":\"@1688434404.043311610\",\"description\":\"Server shutdown\",\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/surface/server.cc\",\"file_line\":480,\"grpc_status\":0}"]
[2023/07/04 01:33:24.044 +00:00] [INFO] [<unknown>] ["ipv4:10.0.0.16:41874: Sending goaway err={\"created\":\"@1688434404.043312832\",\"description\":\"Server shutdown\",\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/surface/server.cc\",\"file_line\":480,\"grpc_status\":0}"]
[2023/07/04 01:33:24.044 +00:00] [INFO] [<unknown>] ["ipv4:10.0.0.16:41869: Sending goaway err={\"created\":\"@1688434404.043313994\",\"description\":\"Server shutdown\",\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/surface/server.cc\",\"file_line\":480,\"grpc_status\":0}"]

TiDB的错误信息：

[2023/07/04 01:33:24.057 +00:00] [INFO] [client_batch.go:609] ["batchRecvLoop fails when receiving, needs to reconnect"] [target=10.0.0.17:20160] [forwardedHost=] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/07/04 01:33:24.057 +00:00] [INFO] [client_batch.go:609] ["batchRecvLoop fails when receiving, needs to reconnect"] [target=10.0.0.17:20160] [forwardedHost=] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/07/04 01:33:24.057 +00:00] [INFO] [client_batch.go:609] ["batchRecvLoop fails when receiving, needs to reconnect"] [target=10.0.0.17:20160] [forwardedHost=] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/07/04 01:33:24.057 +00:00] [INFO] [client_batch.go:609] ["batchRecvLoop fails when receiving, needs to reconnect"] [target=10.0.0.17:20160] [forwardedHost=] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/07/04 01:33:24.364 +00:00] [WARN] [client_batch.go:365] ["no available connections"] [target=10.0.0.17:20160]
[2023/07/04 01:33:24.366 +00:00] [INFO] [region_cache.go:2199] ["[health check] check health error"] [store=10.0.0.17:20160] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 10.0.0.17:20160: connect: connection refused\""]
[2023/07/04 01:33:24.366 +00:00] [INFO] [region_request.go:785] ["mark store's regions need be refill"] [id=297684] [addr=10.0.0.17:20160] [error="no available connections"]

tidb菜鸟一只 · 2023 年7 月 4 日 09:11

tidb的三副本如果挂掉1个只是不会导致完全不可用，但是并不能保持tidb连接不断开，例如你正在连着10.0.0.14:4000这个tidb节点执行sql，它查询的是10.0.0.17:20160这个tikv节点的leader数据，这是如果10.0.0.17:20160忽然下线，这个sql必然会报错，然后重新连接，这时候其他节点的follower会转化为leader继续对tidb提供服务，还能提供服务，但是肯定需要重新断开连接的。

redgame · 2023 年7 月 4 日 09:17

失联了。。。再试试

buptzhoutian · 2023 年7 月 4 日 09:29

这种方式有点暴力，应该在整个集群都停掉的时候操作。

如果想换磁盘的过程对客户端透明的话，官方推荐的做法是用 TiUP 的 scale-in/scale-out 吧

TiDB_C罗 · 2023 年7 月 4 日 09:32

那为啥要断开服务到tidb之间的连接呢，我理解只需要tidb重新连接到其它的tikv节点，服务最多请求返回的慢些，另外对于这种情况，有什么办法或者能否手工把这个要操作的tikv节点上的leader转移到其它的tikv节点？

tidb菜鸟一只 · 2023 年7 月 4 日 09:51

是可以先把tikv对应的节点上的leader都驱逐到其他tikv节点的，通过pd-ctl 命令
scheduler add evict-leader-scheduler 1 ：添加移除 Store 1 的所有 Leader 的调度器

TiDB_C罗 · 2023 年7 月 4 日 09:52

这种scale-in/scale-out的方式是可以，缺点就是周期太长

TiDB_C罗 · 2023 年7 月 4 日 09:55

你这个是最佳实践，但是我还得觉得服务与tidb没有断开的必要。

system · 2023 年9 月 2 日 09:55

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。