集群部分tikv节点疯狂连接tombstone状态的tiflash

[peng-xin]

对，可以降级到v4.0.2吧

[rleungx (Ryan Leung)]

可以，不过最好下周直接使用 v4.0.5，如果需要降级 v4.0.2 可以参考 https://docs.pingcap.com/zh/tidb/stable/tiup-cluster#更新组件。
具体步骤应该是：
wget https://tiup-mirrors.pingcap.com/pd-v4.0.2-linux-amd64.tar.gz
tiup cluster patch $(pwd)/pd-v4.0.2-linux-amd64.tar.gz -R pd --overwrite

[peng-xin]

好的，目前来看还不影响使用，那就下周等升级吧。

[peng-xin]

您好，还有两个小问题
1.tidb滚动重启时，region leader驱逐会超时

2.store limit里面存在已经被删掉的store，如何删掉这部分store limit

[来了老弟]

@peng-xin

1.tidb滚动重启时，region leader驱逐会超时

增加 --ssh-timeout int --wait-timeout int 参数，设置一个超大的值，可能 leader 太多，或者迁移速度较慢导致超时。

@peng-xin

2.store limit里面存在已经被删掉的store，如何删掉这部分store limit

先用 store 看下，确定节点状态，在进行处理

[peng-xin]

节点都是UP的

{
“count”: 4,
“stores”: [
{
“store”: {
“id”: 2,
“address”: “172.16.116.153:20171”,
“labels”: [
{
“key”: “host”,
“value”: “tikv4”
}
],
“version”: “4.0.4”,
“status_address”: “172.16.116.153:20181”,
“git_hash”: “28e3d44b00700137de4fa933066ab83e5f8306cf”,
“start_timestamp”: 1596468242,
“deploy_path”: “/data/soft/tidb/1/deploy/bin”,
“last_heartbeat”: 1596787397200040923,
“state_name”: “Up”
},
“status”: {
“capacity”: “1TiB”,
“available”: “483.6GiB”,
“used_size”: “540.4GiB”,
“leader_count”: 640,
“leader_weight”: 1,
“leader_score”: 640,
“leader_size”: 102673,
“region_count”: 14572,
“region_weight”: 1,
“region_score”: 2122823,
“region_size”: 2122823,
“start_ts”: “2020-08-03T23:24:02+08:00”,
“last_heartbeat_ts”: “2020-08-07T16:03:17.200040923+08:00”,
“uptime”: “88h39m15.200040923s”
}
},
{
“store”: {
“id”: 91,
“address”: “172.16.116.213:20171”,
“labels”: [
{
“key”: “host”,
“value”: “tikv3”
}
],
“version”: “4.0.4”,
“status_address”: “172.16.116.213:20181”,
“git_hash”: “28e3d44b00700137de4fa933066ab83e5f8306cf”,
“start_timestamp”: 1596467965,
“deploy_path”: “/date/soft/tidb/1/deploy/bin”,
“last_heartbeat”: 1596787393901129038,
“state_name”: “Up”
},
“status”: {
“capacity”: “1TiB”,
“available”: “484.7GiB”,
“used_size”: “539.3GiB”,
“leader_count”: 4560,
“leader_weight”: 1,
“leader_score”: 4560,
“leader_size”: 659063,
“region_count”: 14572,
“region_weight”: 1,
“region_score”: 2122823,
“region_size”: 2122823,
“start_ts”: “2020-08-03T23:19:25+08:00”,
“last_heartbeat_ts”: “2020-08-07T16:03:13.901129038+08:00”,
“uptime”: “88h43m48.901129038s”
}
},
{
“store”: {
“id”: 37876005,
“address”: “172.16.117.77:3930”,
“labels”: [
{
“key”: “engine”,
“value”: “tiflash”
}
],
“version”: “v4.0.4”,
“peer_address”: “172.16.117.77:20170”,
“status_address”: “172.16.117.77:20292”,
“git_hash”: “bfa9128f59cf800e129152f06b12480ad78adafd”,
“start_timestamp”: 1596468769,
“deploy_path”: “/data/soft/tidb/tiflash/bin/tiflash”,
“last_heartbeat”: 1596787398655560644,
“state_name”: “Up”
},
“status”: {
“capacity”: “3.161TiB”,
“available”: “2.609TiB”,
“used_size”: “335.8GiB”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 9583,
“region_weight”: 1,
“region_score”: 1485376,
“region_size”: 1485376,
“start_ts”: “2020-08-03T23:32:49+08:00”,
“last_heartbeat_ts”: “2020-08-07T16:03:18.655560644+08:00”,
“uptime”: “88h30m29.655560644s”
}
},
{
“store”: {
“id”: 90,
“address”: “172.16.117.177:20171”,
“labels”: [
{
“key”: “host”,
“value”: “tikv2”
}
],
“version”: “4.0.4”,
“status_address”: “172.16.117.177:20181”,
“git_hash”: “28e3d44b00700137de4fa933066ab83e5f8306cf”,
“start_timestamp”: 1596467638,
“deploy_path”: “/data/soft/tidb/1/deploy/bin”,
“last_heartbeat”: 1596787398413725529,
“state_name”: “Up”
},
“status”: {
“capacity”: “1TiB”,
“available”: “484.8GiB”,
“used_size”: “539.2GiB”,
“leader_count”: 9372,
“leader_weight”: 1,
“leader_score”: 9372,
“leader_size”: 1361087,
“region_count”: 14572,
“region_weight”: 1,
“region_score”: 2122823,
“region_size”: 2122823,
“start_ts”: “2020-08-03T23:13:58+08:00”,
“last_heartbeat_ts”: “2020-08-07T16:03:18.413725529+08:00”,
“uptime”: “88h49m20.413725529s”
}
}
]
}

[lhy1024-PingCAP]

是正常的，只是 tombstone 并不会从 PD 中删掉，如果需要删掉，可以使用 pd-ctl 通过 store remove-tombstone进行移除。

[peng-xin]

多谢老师的命令

[peng-xin]

谢谢老师的命令

[yilong (yi888long)]