【 TiDB 使用环境`】生产
【 TiDB 版本】v5.3.0
【遇到的问题】TiDB集群中TiKV是集群3台部署,其中1台误删除/data文件后无法启动
【复现路径】查看了/tidb-deploy/tikv-20160/log
【问题现象及影响】tikv日志报错如下:
[2022/05/20 14:16:52.559 +08:00] [ERROR] [util.rs:460] [“request failed”] [err_code=KV:PD:gRPC] [err=“Grpc(RpcFailure(RpcStatus { code: 2-UNKNOWN, message: "duplicated store address: id:1144223 address:\"10.2.236.21:20160\" version:\"5.3.0\" status_address:\"10.2.236.21:20180\" git_hash:\"6c1424706f3d5885faa668233f34c9f178302f36\" start_timestamp:1653027410 deploy_path:\"/home/tidb-deploy/tikv-20160/bin\" , already registered by id:5 address:\"10.2.236.21:20160\" version:\"5.3.0\" status_address:\"10.2.236.21:20180\" git_hash:\"6c1424706f3d5885faa668233f34c9f178302f36\" start_timestamp:1653011590 deploy_path:\"/home/tidb-deploy/tikv-20160/bin\" last_heartbeat:1653012976022099210 ", details: [] }))”]
[2022/05/20 14:16:52.559 +08:00] [ERROR] [util.rs:469] [“reconnect failed”] [err_code=KV:PD:Unknown] [err=“Other("[components/pd_client/src/util.rs:306]: cancel reconnection due to too small interval")”]
按照这篇文档做的(恢复步骤):
【SOP 系列 18】TiUP 环境恢复 TiKV 副本 - 运维指南 / TiDB 运维手册 - TiDB 的问答社区 (asktug.com)
9. 删除之前的 store id
10. 重启 tikv 集群后正常
运行 第9-10步后,状态变成offline.
缩容使用–force后,使用tiup cluster display tidb ,故障节点已经消失。
使用tiup ctl:v5.3.0 pd -u pdip:2379 store查询,故障tikv节点的store id仍在,执行扩容提示重复的ID
[2022/05/23 17:07:48.598 +08:00] [ERROR] [util.rs:460] [“request failed”] [err_code=KV:PD:gRPC] [err=“Grpc(RpcFailure(RpcStatus { code: 2-UNKNOWN, message: “duplicated store address: id:1151064 address:\“10.2.236.21:20160\” version:\“5.3.0\” status_address:\“10.2.236.21:20180\” git_hash:\“6c1424706f3d5885faa668233f34c9f178302f36\” start_timestamp:1653296863 deploy_path:\”/home/tidb-deploy/tikv-20160/bin\” , already registered by id:5 address:\“10.2.236.21:20160\” state:Offline version:\“5.3.0\” status_address:\“10.2.236.21:20180\” git_hash:\“6c1424706f3d5885faa668233f34c9f178302f36\” start_timestamp:1653011590 deploy_path:"/home/tidb-deploy/tikv-20160/bin" last_heartbeat:1653012730545870133 “, details: [] }))”]
[2022/05/23 17:07:48.598 +08:00] [FATAL] [server.rs:843] [“failed to start node: Grpc(RpcFailure(RpcStatus { code: 2-UNKNOWN, message: “duplicated store address: id:1151064 address:\“10.2.236.21:20160\” version:\“5.3.0\” status_address:\“10.2.236.21:20180\” git_hash:\“6c1424706f3d5885faa668233f34c9f178302f36\” start_timestamp:1653296863 deploy_path:\”/home/tidb-deploy/tikv-20160/bin\” , already registered by id:5 address:\“10.2.236.21:20160\” state:Offline version:\“5.3.0\” status_address:\“10.2.236.21:20180\” git_hash:\“6c1424706f3d5885faa668233f34c9f178302f36\” start_timestamp:1653011590 deploy_path:"/home/tidb-deploy/tikv-20160/bin" last_heartbeat:1653012730545870133 “, details: [] }))”]