TiDB集群运行过程中突然宕机然后又自己启动回来了。

【 TiDB 使用环境】
【概述】:TiDB集群运行过程中突然宕机然后又自己启动回来了。
【背景】:做过哪些操作
【现象】:dashboard里有显示10几20+条大于5秒的慢查询。
【问题】:TiDB集群突然挂了,过了几分钟后自己又恢复正常了。
【业务影响】:
【TiDB 版本】:
【附件】:
12c2b91aacd2e4386821eb0a14f151d

  • 相关日志
    tidb error log
    {“level”:“warn”,“ts”:“2022-01-12T09:59:37.793+0800”,“caller”:“clientv3/retry_interceptor.go:61”,“msg”:“retrying of unary invoker failed”,“target”:“endpoint://client-39c4e526-d6bc-445e-bb67-400b056592e8/168.18.163.100:2379”,“attempt”:0,“error”:“rpc error: code = Unavailable desc = transport is closing”}
    [2022/01/12 09:59:38.226 +08:00] [ERROR] [domain.go:870] [“load privilege loop watch channel closed”]
    [2022/01/12 10:00:17.807 +08:00] [WARN] [base_client.go:194] ["[pd] cannot update leader"] [address=http://192.18.163.106:2379] [error="[PD:client:ErrClientGetMember]error:rpc error: code = DeadlineExceeded desc = context deadline exceeded target:192.18.163.106:2379 status:READY"]
    [2022/01/12 10:00:56.236 +08:00] [INFO] [region_cache.go:839] [“switch region leader to specific leader due to kv return NotLeader”] [regionID=135201] [currIdx=1] [leaderStoreID=7]

tikv.log
[2022/01/12 09:57:49.973 +08:00] [WARN] [endpoint.rs:530] [error-response] [err=“Region error (will back off and retry) message: “peer is not leader for region 125117, leader may Some(id: 125119 store_id: 2)” not_leader { region_id: 125117 leader { id: 125119 store_id: 2 } }”]
[2022/01/12 09:59:49.050 +08:00] [ERROR] [] [“ipv4:192.18.163.106:45616: Keepalive watchdog fired. Closing transport.”]

pd.log
[2022/01/12 09:59:42.262 +08:00] [WARN] [retry_interceptor.go:61] [“retrying of unary invoker failed”] [target=endpoint://client-2fe2dba4-3c53-4d0d-b259-5c489ff81db2/192.16.163.100:2379] [attempt=0] [error=“rpc error: code = Unavailable desc = transport is closing”]

endpoint://client-39c4e526-d6bc-445e-bb67-400b056592e8/168.18.163.100:2379

检查下这个 PD,是否正常

群集正常运行。我查看了这个PD的日志:
[2022/01/12 09:59:42.262 +08:00] [WARN] [retry_interceptor.go:61] [“retrying of unary invoker failed”] [target=endpoint://client-2fe2dba4-3c53-4d0d-b259-5c489ff81db2/192.16.163.100:2379] [attempt=0] [error=“rpc error: code = Unavailable desc = transport is closing”]

endpoint://client-39c4e526-d6bc-445e-bb67-400b056592e8/168.18.163.100:2379

检查下这个 PD,是否正常

tidb的报错信息可以参考一下这里