【 TiDB 使用环境】生产
【 TiDB 版本】6.3.0
【遇到的问题】手动配置TLS证书后reload，tikv、tidb启动失败
【复现路径】手动创建证书，tiup cluster edit-cofig
【问题现象及影响】tiup cluster reload 失败，部分tikv、tidb status变为 disconnected
【附件】：
- 相关日志
Log from tikv
[2022/11/02 01:41:38.893 +00:00] [INFO] [util.rs:551] [“PD failed to respond”] [err=“Grpc(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: "failed to connect to all addresses", details: }))”] [endpoints=10.250.87.38:2378]
[2022/11/02 01:41:38.893 +00:00] [INFO] [util.rs:589] [“connecting to PD endpoint”] [endpoints=10.250.87.122:2378]
[2022/11/02 01:41:38.895 +00:00] [INFO] [] ["subchannel 0x7faa87d35000 {address=ipv4:10.250.87.122:2378, args=grpc.client_channel_factory=0x7faa87c9c1f8, grpc.default_authority=10.250.87.122:2378, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x7faa875c7c60, grpc.internal.security_connector=0x7faa7fe719b0…
Log from PD
[2022/11/03 02:21:45.797 +00:00] [INFO] [server.go:1406] [“start to watch pd leader”] [pd-leader="name:"pd-10.250.87.38-2378" member_id:16326710579290257846 peer_urls:"http://10.250.87.38:2380" client_urls:"http://10.250.87.38:2378" "]
[2022/11/03 02:21:45.799 +00:00] [WARN] [grpclog.go:60] [“grpc: addrConn.createTransport failed to connect to {10.250.87.38:2378 0 }. Err :connection error: desc = "transport: authentication handshake failed: EOF". Reconnecting…”]
[2022/11/03 02:21:45.833 +00:00] [WARN] [leadership.go:194] [“required revision has been compacted, use the compact revision”] [required-revision=305224] [compact-revision=713535]
[2022/11/03 02:21:46.800 +00:00] [WARN] [grpclog.go:60] [“grpc: addrConn.createTransport failed to connect to {10.250.87.38:2378 0 }. Err :connection error: desc = "transport: authentication handshake failed: EOF". Reconnecting…”]
[2022/11/03 02:21:48.461 +00:00] [WARN] [stream.go:436] [“lost TCP streaming connection with remote peer”] [stream-reader-type=“stream MsgApp v2”] [local-member-id=f5476fe9c527f5b9] [remote-peer-id=3310c14e027b4f49] [error=EOF]
[2022/11/03 02:21:48.463 +00:00] [WARN] [stream.go:436] [“lost TCP streaming connection with remote peer”] [stream-reader-type=“stream Message”] [local-member-id=f5476fe9c527f5b9] [remote-peer-id=3310c14e027b4f49] [error=EOF]
[2022/11/03 02:21:48.464 +00:00] [WARN] [peer_status.go:68] [“peer became inactive (message send to peer failed)”] [peer-id=3310c14e027b4f49] [error=“failed to dial 3310c14e027b4f49 on stream Message (peer 3310c14e027b4f49 failed to find local node f5476fe9c527f5b9)”]
[2022/11/03 02:21:48.566 +00:00] [WARN] [grpclog.go:60] [“grpc: addrConn.createTransport failed to connect to {10.250.87.38:2378 0 }. Err :connection error: desc = "transport: authentication handshake failed: EOF". Reconnecting…”]
- 配置文件
server_configs:
tidb:
binlog.enable: false
binlog.ignore-error: false
log.file.max-days: 7
log.slow-threshold: 300
mem-quota-query: 524288000
oom-action: cancel
performance.txn-total-size-limit: 10730418240
security.cluster-ssl-ca: /vdb/tidb-certs/idtrca.cer
security.cluster-ssl-cert: /vdb/tidb-certs/server-cert.pem
security.cluster-ssl-key: /vdb/tidb-certs/server-key.pem
security.ssl-ca: /vdb/tidb-certs/idtrca.cer
security.ssl-cert: /vdb/tidb-certs/server-cert.pem
security.ssl-key: /vdb/tidb-certs/server-key.pem
tikv-client.copr-cache.enable: false
tikv:
readpool.coprocessor.use-unified-pool: true
readpool.storage.use-unified-pool: false
security.ca-path: /vdb/tidb-certs/idtrca.cer
security.cert-path: /vdb/tidb-certs/server-cert.pem
security.key-path: /vdb/tidb-certs/server-key.pem
pd:
auto-compaction-retention: 5m
log.file.max-days: 7
quota-backend-bytes: 17179869184
schedule.leader-schedule-limit: 4
schedule.region-schedule-limit: 2048
schedule.replica-schedule-limit: 64
security.cacert-path: /vdb/tidb-certs/idtrca.cer
security.cert-path: /vdb/tidb-certs/server-cert.pem
security.key-path: /vdb/tidb-certs/server-key.pem