一个tikv节点宕机导致集群不可用

其他kv期间都报这个错

[2023/07/20 12:47:26.961 +08:00] [INFO] [<unknown>] ["Connect failed: {\"created\":\"@1689828446.961442851\",\"description\":\"Failed to connect to remote host: Connection timed out\",\"errno\":110,\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.5.3/grpc/src/core/lib/iomgr/tcp_client_posix.cc\",\"file_line\":205,\"os_error\":\"Connection timed out\",\"syscall\":\"getsockopt(SO_ERROR)\",\"target_address\":\"ipv4:10.0.1.11:10000\"}"]
[2023/07/20 12:47:26.961 +08:00] [INFO] [<unknown>] ["Subchannel 0x7f5daeeb9b80: Retry immediately"]
[2023/07/20 12:47:26.961 +08:00] [INFO] [<unknown>] ["Failed to connect to channel, retrying"]
[2023/07/20 12:47:26.961 +08:00] [WARN] [raft_client.rs:296] ["RPC batch_raft fail"] [err="Some(RpcFailure(RpcStatus { status: 14-UNAVAILABLE, details: Some(\"failed to connect to all addresses\") }))"] [sink_err="Some(RpcFinished(Some(RpcStatus { status: 14-UNAVAILABLE, details: Some(\"failed to connect to all addresses\") })))"] [to_addr=10.0.1.11:10000]
[2023/07/20 12:47:26.961 +08:00] [WARN] [raft_client.rs:199] ["send to 10.0.1.11:10000 nnnfail, the gRPC connection could be broken"]
[2023/07/20 12:47:26.961 +08:00] [ERROR] [transport.rs:163] ["send raft msg err"] [err="Other(\"[src/server/raft_client.rs:208]: RaftClient send fail\")"]

tidb期间的报错

[2023/07/20 04:47:20.901 +00:00] [INFO] [client_batch.go:348] ["batchRecvLoop fails when receiving, needs to reconnect"] [target=10.0.1.11:10000] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/07/20 04:47:20.917 +00:00] [INFO] [client_batch.go:348] ["batchRecvLoop fails when receiving, needs to reconnect"] [target=10.0.1.11:10000] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/07/20 04:47:20.917 +00:00] [INFO] [client_batch.go:348] ["batchRecvLoop fails when receiving, needs to reconnect"] [target=10.0.1.11:10000] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/07/20 04:47:20.917 +00:00] [INFO] [client_batch.go:348] ["batchRecvLoop fails when receiving, needs to reconnect"] [target=10.0.1.11:10000] [error="rpc error: code = Unavailable desc = transport is closing"]
[2023/07/20 04:47:20.952 +00:00] [WARN] [client_batch.go:530] ["no available connections"] [target=10.0.1.11:10000]
[2023/07/20 04:47:20.953 +00:00] [WARN] [client_batch.go:530] ["no available connections"] [target=10.0.1.11:10000]
[2023/07/20 04:47:20.955 +00:00] [WARN] [client_batch.go:530] ["no available connections"] [target=10.0.1.11:10000]
[2023/07/20 04:47:20.957 +00:00] [WARN] [client_batch.go:530] ["no available connections"] [target=10.0.1.11:10000]
[2023/07/20 04:47:20.959 +00:00] [WARN] [client_batch.go:530] ["no available connections"] [target=10.0.1.11:10000]

其中10.0.1.11:10000就是 那个宕机的kv