故障排查 重启tikv 节点后 id 号变了,日志报地址占用

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。

  • 【TiDB 版本】: 测试环境,版本是3.0.11

  • 【问题描述】: 由于是测试环境,没有ssd 盘,所以手工创建pd、tikv、tidb

ps -ef |grep tikv ,杀死进程,修改参数配置文件,在启动服务的时候,日志报如下错误:

[2020/03/18 10:39:08.459 +08:00] [ERROR] [util.rs:327] [“request failed”] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some("duplicated store address: id:111 address:\“10.10.103.113:20160\” version:\“3.0.11\” , already registered by id:1 address:\“10.10.103.113:20160\” version:\“3.0.11\” “) }))”] [2020/03/18 10:39:08.460 +08:00] [ERROR] [util.rs:327] [“request failed”] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some("duplicated store address: id:111 address:\“10.10.103.113:20160\” version:\“3.0.11\” , already registered by id:1 address:\“10.10.103.113:20160\” version:\“3.0.11\” “) }))”] [2020/03/18 10:39:08.462 +08:00] [ERROR] [util.rs:327] [“request failed”] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some("duplicated store address: id:111 address:\“10.10.103.113:20160\” version:\“3.0.11\” , already registered by id:1 address:\“10.10.103.113:20160\” version:\“3.0.11\” “) }))”] [2020/03/18 10:39:08.463 +08:00] [ERROR] [util.rs:327] [“request failed”] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some("duplicated store address: id:111 address:\“10.10.103.113:20160\” version:\“3.0.11\” , already registered by id:1 address:\“10.10.103.113:20160\” version:\“3.0.11\” “) }))”] [2020/03/18 10:39:08.464 +08:00] [ERROR] [util.rs:327] [“request failed”] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some("duplicated store address: id:111 address:\“10.10.103.113:20160\” version:\“3.0.11\” , already registered by id:1 address:\“10.10.103.113:20160\” version:\“3.0.11\” “) }))”] [2020/03/18 10:39:08.465 +08:00] [ERROR] [util.rs:327] [“request failed”] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some("duplicated store address: id:111 address:\“10.10.103.113:20160\” version:\“3.0.11\” , already registered by id:1 address:\“10.10.103.113:20160\” version:\“3.0.11\” “) }))”] [2020/03/18 10:39:08.466 +08:00] [ERROR] [util.rs:327] [“request failed”] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some("duplicated store address: id:111 address:\“10.10.103.113:20160\” version:\“3.0.11\” , already registered by id:1 address:\“10.10.103.113:20160\” version:\“3.0.11\” “) }))”] [2020/03/18 10:39:08.467 +08:00] [ERROR] [util.rs:327] [“request failed”] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some("duplicated store address: id:111 address:\“10.10.103.113:20160\” version:\“3.0.11\” , already registered by id:1 address:\“10.10.103.113:20160\” version:\“3.0.11\” “) }))”] [2020/03/18 10:39:08.468 +08:00] [ERROR] [util.rs:327] [“request failed”] [err="Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some("duplicated store address: id:111 address:\“10.10.103.113:20160\” version:\“3.0.11\” , already registered by id:1 address:\“10.10.103.113:20160\” version:\“3.0.11\” “) }))”] [2020/03/18 10:39:08.470 +08:00] [ERROR] [util.rs:327] [“request failed”] [err=“Grpc(RpcFailure(RpcStatus { status: Unknown, details: Some(“duplicated store address: id:111 address:\“10.10.103.113:20160\” version:\“3.0.11\” , already registered by id:1 address:\“10.10.103.113:20160\” version:\“3.0.11\” “) }))”] [2020/03/18 10:39:08.470 +08:00] [FATAL] [server.rs:264] [“failed to start node: Other(”[src/pd/util.rs:335]: fail to request”)”]

说明tikv 在启动的时候,已经使用的一个新的id 和之前的id 不一样了,所以说地址已经占用,这个如何解决

» store { “count”: 3, “stores”: [ { “store”: { “id”: 1, “address”: “10.10.103.113:20160”, “version”: “3.0.11”, “state_name”: “Down” }, “status”: { “leader_weight”: 1, “region_weight”: 1, “start_ts”: “1970-01-01T08:00:00+08:00” } }, { “store”: { “id”: 4, “address”: “10.10.103.114:20160”, “version”: “3.0.11”, “state_name”: “Up” }, “status”: { “capacity”: “139.6GiB”, “available”: “139.3GiB”, “leader_count”: 12, “leader_weight”: 1, “leader_score”: 12, “leader_size”: 12, “region_count”: 24, “region_weight”: 1, “region_score”: 26, “region_size”: 26, “start_ts”: “2020-03-18T10:19:48+08:00”, “last_heartbeat_ts”: “2020-03-18T10:36:48.249721861+08:00”, “uptime”: “17m0.249721861s” } }, { “store”: { “id”: 5, “address”: “10.10.103.88:20160”, “version”: “3.0.11”, “state_name”: “Up” }, “status”: { “capacity”: “99.95GiB”, “available”: “86.8GiB”, “leader_count”: 12, “leader_weight”: 1, “leader_score”: 14, “leader_size”: 14, “region_count”: 24, “region_weight”: 1, “region_score”: 26, “region_size”: 26, “start_ts”: “2020-03-18T10:20:07+08:00”, “last_heartbeat_ts”: “2020-03-18T10:36:47.510896971+08:00”, “uptime”: “16m40.510896971s” } } ] }

若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

duplicated store address 表示同样的地址和端口已经有 tikv 注册了

1、确认下数据目录是不是清理过

2、通过 pd-ctl > store 查看之前的 store 是否存在

3、可以执行 store delete 下线掉冲突的 tikv 后再启动重新加入集群