如何删除已下线实例在information_schema中信息

🛰️ 应用开发者专区
1

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：

【TiDB 版本】
5.0.1
【问题描述】
如图:

image
image1915×777 161 KB

tiup 显示只有两个tikv节点（开始有三个，其中一个我下线了）
但是:
image
image1785×492 117 KB

查询Information_schema的表时，确显示有三个tikv, 我重启集群多次仍然是上面这种情况，我想问一下如何将Information_schema中不存在的tikv信息删掉?

若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出的打印结果，请务必全选并复制粘贴上传。

2

下线是如何操作的？tiup cluster scale-in 节点，然后 tiup cluster prune 清理 tombstone 节点么？

3

理论上在 v4.0.2 版本修复了一个类似的问题，在 cluster_info 表中不再显示状态为 tombstone 的 tikv 以及 tiflash 节点：

https://docs.pingcap.com/zh/tidb/stable/release-4.0.2#兼容性

但当前是 v5.0 的版本，所以辛苦再拿下下面的信息：

1、http://127.0.0.1:2379/pd/api/v1/stores
2、pd-ctl store
3、select * from TIKV_STORE_STATUS;

上面的信息收集后辛苦上传下～

4

执行 http://pd_address:2379/pd/api/v1/stores 结果如下:
{
“count”: 6,
“stores”: [
{
“store”: {
“id”: 2748,
“address”: “10.53.x.x:20160”,
“state”: 1,
“labels”: [
{
“key”: “zone”,
“value”: “bj”
},
{
“key”: “rack”,
“value”: “r1”
},
{
“key”: “host”,
“value”: “tikv1”
},
{
“key”: “dc”,
“value”: “bj7”
}
],
“version”: “5.0.0”,
“status_address”: “10.53.x.x:20260”,
“git_hash”: “7706b9634bd901c9fe8dbe6a556025abbfd0793d”,
“start_timestamp”: 1621246167,
“deploy_path”: “/home/work/tidb/tikv-20160/bin”,
“last_heartbeat”: 1621411463776847922,
“state_name”: “Offline”
},
“status”: {
“capacity”: “0B”,
“available”: “0B”,
“used_size”: “0B”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 345278,
“region_weight”: 1,
“region_score”: 891962,
“region_size”: 891962,
“start_ts”: “2021-05-17T18:09:27+08:00”,
“last_heartbeat_ts”: “2021-05-19T16:04:23.776847922+08:00”,
“uptime”: “45h54m56.776847922s”
}
},
{
“store”: {
“id”: 2749,
“address”: “10.53.x.x:20160”,
“labels”: [
{
“key”: “zone”,
“value”: “bj”
},
{
“key”: “rack”,
“value”: “r1”
},
{
“key”: “host”,
“value”: “tikv3”
},
{
“key”: “dc”,
“value”: “bj8”
}
],
“version”: “5.0.1”,
“status_address”: “10.53.x.x:20260”,
“git_hash”: “e26389a278116b2f61addfa9f15ca25ecf38bc80”,
“start_timestamp”: 1621583805,
“deploy_path”: “/home/work/tidb/tikv-20160/bin”,
“last_heartbeat”: 1621994741861730525,
“state_name”: “Up”
},
“status”: {
“capacity”: “937.3GiB”,
“available”: “498.1GiB”,
“used_size”: “149.7GiB”,
“leader_count”: 170519,
“leader_weight”: 1,
“leader_score”: 170519,
“leader_size”: 437265,
“region_count”: 345278,
“region_weight”: 1,
“region_score”: 891962,
“region_size”: 891962,
“start_ts”: “2021-05-21T15:56:45+08:00”,
“last_heartbeat_ts”: “2021-05-26T10:05:41.861730525+08:00”,
“uptime”: “114h8m56.861730525s”
}
},
{
“store”: {
“id”: 24700863,
“address”: “10.53.x.x:4130”,
“labels”: [
{
“key”: “engine”,
“value”: “tiflash”
}
],
“version”: “v5.0.1”,
“peer_address”: “10.53.x.x:20370”,
“status_address”: “10.53.x.x:20492”,
“git_hash”: “1821cf655bc90e1fab6e6154cfe994c19c75d377”,
“start_timestamp”: 1621583849,
“deploy_path”: “/home/work/tidb/tiflash-10000/bin/tiflash”,
“last_heartbeat”: 1621994749124741240,
“state_name”: “Up”
},
“status”: {
“capacity”: “937.3GiB”,
“available”: “891.1GiB”,
“used_size”: “46.18GiB”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 5281,
“region_weight”: 1,
“region_score”: 146358,
“region_size”: 146358,
“start_ts”: “2021-05-21T15:57:29+08:00”,
“last_heartbeat_ts”: “2021-05-26T10:05:49.12474124+08:00”,
“uptime”: “114h8m20.12474124s”
}
},
{
“store”: {
“id”: 5519,
“address”: “10.53.x.x:20160”,
“labels”: [
{
“key”: “zone”,
“value”: “bj”
},
{
“key”: “rack”,
“value”: “r1”
},
{
“key”: “host”,
“value”: “tikv1”
},
{
“key”: “dc”,
“value”: “bj10”
}
],
“version”: “5.0.1”,
“status_address”: “10.53.x.x:20260”,
“git_hash”: “e26389a278116b2f61addfa9f15ca25ecf38bc80”,
“start_timestamp”: 1621583804,
“deploy_path”: “/home/work/tidb/tikv-20160/bin”,
“last_heartbeat”: 1621994746787285801,
“state_name”: “Up”
},
“status”: {
“capacity”: “937.3GiB”,
“available”: “546.3GiB”,
“used_size”: “149.7GiB”,
“leader_count”: 174759,
“leader_weight”: 1,
“leader_score”: 174759,
“leader_size”: 454697,
“region_count”: 345278,
“region_weight”: 1,
“region_score”: 891962,
“region_size”: 891962,
“start_ts”: “2021-05-21T15:56:44+08:00”,
“last_heartbeat_ts”: “2021-05-26T10:05:46.787285801+08:00”,
“uptime”: “114h9m2.787285801s”
}
},
{
“store”: {
“id”: 8731869,
“address”: “10.53.x.x:4130”,
“labels”: [
{
“key”: “engine”,
“value”: “tiflash”
}
],
“version”: “v5.0.1”,
“peer_address”: “10.53.x.x:20370”,
“status_address”: “10.53.x.x:20492”,
“git_hash”: “1821cf655bc90e1fab6e6154cfe994c19c75d377”,
“start_timestamp”: 1621583849,
“deploy_path”: “/home/work/tidb/tiflash-10000/bin/tiflash”,
“last_heartbeat”: 1621994748771621501,
“state_name”: “Up”
},
“status”: {
“capacity”: “937.3GiB”,
“available”: “880GiB”,
“used_size”: “57.34GiB”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 6096,
“region_weight”: 1,
“region_score”: 182848,
“region_size”: 182848,
“start_ts”: “2021-05-21T15:57:29+08:00”,
“last_heartbeat_ts”: “2021-05-26T10:05:48.771621501+08:00”,
“uptime”: “114h8m19.771621501s”
}
},
{
“store”: {
“id”: 8731871,
“address”: “10.53.x.x:4130”,
“labels”: [
{
“key”: “engine”,
“value”: “tiflash”
}
],
“version”: “v5.0.1”,
“peer_address”: “10.53.x.x:20370”,
“status_address”: “10.53.x.x:20492”,
“git_hash”: “1821cf655bc90e1fab6e6154cfe994c19c75d377”,
“start_timestamp”: 1621583850,
“deploy_path”: “/home/work/tidb/tiflash-10000/bin/tiflash”,
“last_heartbeat”: 1621994750119651475,
“state_name”: “Up”
},
“status”: {
“capacity”: “937.3GiB”,
“available”: “879.9GiB”,
“used_size”: “57.38GiB”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 7837,
“region_weight”: 1,
“region_score”: 179318,
“region_size”: 179318,
“start_ts”: “2021-05-21T15:57:30+08:00”,
“last_heartbeat_ts”: “2021-05-26T10:05:50.119651475+08:00”,
“uptime”: “114h8m20.119651475s”
}
}
]
}
select * from TIKV_STORE_STATUS 结果如下:
±---------±-------------------±------------±-----------------±------------------------------------------------------------------------------------------------------------------------------------±--------±---------±----------±-------------±--------------±-------------±------------±-------------±--------------±-------------±------------±--------------------±--------------------±---------------------+
| STORE_ID | ADDRESS | STORE_STATE | STORE_STATE_NAME | LABEL | VERSION | CAPACITY | AVAILABLE | LEADER_COUNT | LEADER_WEIGHT | LEADER_SCORE | LEADER_SIZE | REGION_COUNT | REGION_WEIGHT | REGION_SCORE | REGION_SIZE | START_TS | LAST_HEARTBEAT_TS | UPTIME |
±---------±-------------------±------------±-----------------±------------------------------------------------------------------------------------------------------------------------------------±--------±---------±----------±-------------±--------------±-------------±------------±-------------±--------------±-------------±------------±--------------------±--------------------±---------------------+
| 2748 | 10.53.18.214:20160 | 1 | Offline | [{“key”: “zone”, “value”: “bj”}, {“key”: “rack”, “value”: “r1”}, {“key”: “host”, “value”: “tikv1”}, {“key”: “dc”, “value”: “bj7”}] | 5.0.0 | 0B | 0B | 0 | 1 | 0 | 0 | 345278 | 1 | 891962 | 891962 | 2021-05-17 18:09:27 | 2021-05-19 16:04:23 | 45h54m56.776847922s |
| 2749 | 10.53.65.90:20160 | 0 | Up | [{“key”: “zone”, “value”: “bj”}, {“key”: “rack”, “value”: “r1”}, {“key”: “host”, “value”: “tikv3”}, {“key”: “dc”, “value”: “bj8”}] | 5.0.1 | 937.3GiB | 498.1GiB | 170519 | 1 | 170519 | 437265 | 345278 | 1 | 891962 | 891962 | 2021-05-21 15:56:45 | 2021-05-26 10:18:33 | 114h21m48.762637216s |
| 24700863 | 10.53.18.214:4130 | 0 | Up | [{“key”: “engine”, “value”: “tiflash”}] | v5.0.1 | 937.3GiB | 891.1GiB | 0 | 1 | 0 | 0 | 5281 | 1 | 146358 | 146358 | 2021-05-21 15:57:29 | 2021-05-26 10:18:29 | 114h21m0.230769432s |
| 8731869 | 10.53.74.172:4130 | 0 | Up | [{“key”: “engine”, “value”: “tiflash”}] | v5.0.1 | 937.3GiB | 880GiB | 0 | 1 | 0 | 0 | 6096 | 1 | 182848 | 182848 | 2021-05-21 15:57:29 | 2021-05-26 10:18:28 | 114h20m59.933316763s |
| 8731871 | 10.53.65.90:4130 | 0 | Up | [{“key”: “engine”, “value”: “tiflash”}] | v5.0.1 | 937.3GiB | 879.9GiB | 0 | 1 | 0 | 0 | 7837 | 1 | 179318 | 179318 | 2021-05-21 15:57:30 | 2021-05-26 10:18:30 | 114h21m0.226639806s |
| 5519 | 10.53.74.172:20160 | 0 | Up | [{“key”: “zone”, “value”: “bj”}, {“key”: “rack”, “value”: “r1”}, {“key”: “host”, “value”: “tikv1”}, {“key”: “dc”, “value”: “bj10”}] | 5.0.1 | 937.3GiB | 546.3GiB | 174759 | 1 | 174759 | 454697 | 345278 | 1 | 891962 | 891962 | 2021-05-21 15:56:44 | 2021-05-26 10:18:28 | 114h21m44.795760573s |
±---------±-------------------±------------±-----------------±------------------------------------------------------------------------------------------------------------------------------------±--------±---------±----------±-------------±--------------±-------------±------------±-------------±--------------±-------------±------------±--------------------±--------------------±---------------------+
6 rows in set (0.01 sec)

tiup ctl pd 执行结果如下:

{
  "count": 6,
  "stores": [
    {
      "store": {
        "id": 8731871,
        "address": "10.53.65.90:4130",
        "labels": [
          {
            "key": "engine",
            "value": "tiflash"
          }
        ],
        "version": "v5.0.1",
        "peer_address": "10.53.65.90:20370",
        "status_address": "10.53.65.90:20492",
        "git_hash": "1821cf655bc90e1fab6e6154cfe994c19c75d377",
        "start_timestamp": 1621583850,
        "deploy_path": "/home/work/tidb/tiflash-10000/bin/tiflash",
        "last_heartbeat": 1621995620242148138,
        "state_name": "Up"
      },
      "status": {
        "capacity": "937.3GiB",
        "available": "879.9GiB",
        "used_size": "57.38GiB",
        "leader_count": 0,
        "leader_weight": 1,
        "leader_score": 0,
        "leader_size": 0,
        "region_count": 7837,
        "region_weight": 1,
        "region_score": 179318,
        "region_size": 179318,
        "start_ts": "2021-05-21T15:57:30+08:00",
        "last_heartbeat_ts": "2021-05-26T10:20:20.242148138+08:00",
        "uptime": "114h22m50.242148138s"
      }
    },
    {
      "store": {
        "id": 5519,
        "address": "10.53.74.172:20160",
        "labels": [
          {
            "key": "zone",
            "value": "bj"
          },
          {
            "key": "rack",
            "value": "r1"
          },
          {
            "key": "host",
            "value": "tikv1"
          },
          {
            "key": "dc",
            "value": "bj10"
          }
        ],
        "version": "5.0.1",
        "status_address": "10.53.74.172:20260",
        "git_hash": "e26389a278116b2f61addfa9f15ca25ecf38bc80",
        "start_timestamp": 1621583804,
        "deploy_path": "/home/work/tidb/tikv-20160/bin",
        "last_heartbeat": 1621995628990826384,
        "state_name": "Up"
      },
      "status": {
        "capacity": "937.3GiB",
        "available": "546.3GiB",
        "used_size": "149.7GiB",
        "leader_count": 174759,
        "leader_weight": 1,
        "leader_score": 174759,
        "leader_size": 454697,
        "region_count": 345278,
        "region_weight": 1,
        "region_score": 891962,
        "region_size": 891962,
        "start_ts": "2021-05-21T15:56:44+08:00",
        "last_heartbeat_ts": "2021-05-26T10:20:28.990826384+08:00",
        "uptime": "114h23m44.990826384s"
      }
    },
    {
      "store": {
        "id": 8731869,
        "address": "10.53.74.172:4130",
        "labels": [
          {
            "key": "engine",
            "value": "tiflash"
          }
        ],
        "version": "v5.0.1",
        "peer_address": "10.53.74.172:20370",
        "status_address": "10.53.74.172:20492",
        "git_hash": "1821cf655bc90e1fab6e6154cfe994c19c75d377",
        "start_timestamp": 1621583849,
        "deploy_path": "/home/work/tidb/tiflash-10000/bin/tiflash",
        "last_heartbeat": 1621995628958534394,
        "state_name": "Up"
      },
      "status": {
        "capacity": "937.3GiB",
        "available": "880GiB",
        "used_size": "57.34GiB",
        "leader_count": 0,
        "leader_weight": 1,
        "leader_score": 0,
        "leader_size": 0,
        "region_count": 6096,
        "region_weight": 1,
        "region_score": 182848,
        "region_size": 182848,
        "start_ts": "2021-05-21T15:57:29+08:00",
        "last_heartbeat_ts": "2021-05-26T10:20:28.958534394+08:00",
        "uptime": "114h22m59.958534394s"
      }
    },
    {
      "store": {
        "id": 24700863,
        "address": "10.53.18.214:4130",
        "labels": [
          {
            "key": "engine",
            "value": "tiflash"
          }
        ],
        "version": "v5.0.1",
        "peer_address": "10.53.18.214:20370",
        "status_address": "10.53.18.214:20492",
        "git_hash": "1821cf655bc90e1fab6e6154cfe994c19c75d377",
        "start_timestamp": 1621583849,
        "deploy_path": "/home/work/tidb/tiflash-10000/bin/tiflash",
        "last_heartbeat": 1621995629247694667,
        "state_name": "Up"
      },
      "status": {
        "capacity": "937.3GiB",
        "available": "891.1GiB",
        "used_size": "46.18GiB",
        "leader_count": 0,
        "leader_weight": 1,
        "leader_score": 0,
        "leader_size": 0,
        "region_count": 5281,
        "region_weight": 1,
        "region_score": 146358,
        "region_size": 146358,
        "start_ts": "2021-05-21T15:57:29+08:00",
        "last_heartbeat_ts": "2021-05-26T10:20:29.247694667+08:00",
        "uptime": "114h23m0.247694667s"
      }
    },
    {
      "store": {
        "id": 2748,
        "address": "10.53.18.214:20160",
        "state": 1,
        "labels": [
          {
            "key": "zone",
            "value": "bj"
          },
          {
            "key": "rack",
            "value": "r1"
          },
          {
            "key": "host",
            "value": "tikv1"
          },
          {
            "key": "dc",
            "value": "bj7"
          }
        ],
        "version": "5.0.0",
        "status_address": "10.53.18.214:20260",
        "git_hash": "7706b9634bd901c9fe8dbe6a556025abbfd0793d",
        "start_timestamp": 1621246167,
        "deploy_path": "/home/work/tidb/tikv-20160/bin",
        "last_heartbeat": 1621411463776847922,
        "state_name": "Offline"
      },
      "status": {
        "capacity": "0B",
        "available": "0B",
        "used_size": "0B",
        "leader_count": 0,
        "leader_weight": 1,
        "leader_score": 0,
        "leader_size": 0,
        "region_count": 345278,
        "region_weight": 1,
        "region_score": 891962,
        "region_size": 891962,
        "start_ts": "2021-05-17T18:09:27+08:00",
        "last_heartbeat_ts": "2021-05-19T16:04:23.776847922+08:00",
        "uptime": "45h54m56.776847922s"
      }
    },
    {
      "store": {
        "id": 2749,
        "address": "10.53.65.90:20160",
        "labels": [
          {
            "key": "zone",
            "value": "bj"
          },
          {
            "key": "rack",
            "value": "r1"
          },
          {
            "key": "host",
            "value": "tikv3"
          },
          {
            "key": "dc",
            "value": "bj8"
          }
        ],
        "version": "5.0.1",
        "status_address": "10.53.65.90:20260",
        "git_hash": "e26389a278116b2f61addfa9f15ca25ecf38bc80",
        "start_timestamp": 1621583805,
        "deploy_path": "/home/work/tidb/tikv-20160/bin",
        "last_heartbeat": 1621995624027129030,
        "state_name": "Up"
      },
      "status": {
        "capacity": "937.3GiB",
        "available": "498.1GiB",
        "used_size": "149.7GiB",
        "leader_count": 170519,
        "leader_weight": 1,
        "leader_score": 170519,
        "leader_size": 437265,
        "region_count": 345278,
        "region_weight": 1,
        "region_score": 891962,
        "region_size": 891962,
        "start_ts": "2021-05-21T15:56:45+08:00",
        "last_heartbeat_ts": "2021-05-26T10:20:24.02712903+08:00",
        "uptime": "114h23m39.02712903s"
      }
    }
  ]
}

从以上三个结果结果都没有问题: 确实是有一个tikv 处于offline 状况

5

是这样的， 其中有一台TikV 挂了有一段时间了， 无法正常起启，于是我用
tiup cluster scale-in --force 缩容

6

offline 状态的节点是之前下线的那个节点么？

7

是的， 这种情况下， 求问一下如何把PD中下线的那个TiKV信息给删掉啊

8

curl -X POST http://127.0.0.1:2379/pd/api/v1/store/${store_id}/state?state=Tombstone

将对应节点的状态修改为 Tombstone ，然后 pd-ctl 执行 remove-tombstone 命令，看能不能把 pd 中的节点信息清理掉。
操作完可以通过 pd-ctl 执行 store 命令确认一下结果。

9

好多， 我试试，多谢!!!

10

image
image1743×76 17.2 KB

我看了一下PD的代码，似乎不能设置成 Tombstone

image
image1241×692 108 KB

11

操作的时候报错了么

12

curl -X POST http://127.0.0.1:2379/pd/api/v1/store/${store_id}/state?state=Tombstone
执行这个 返回 “invalid state Tombstone”
看代码，似乎不支持设置成Tombstone

13

在 5.0 及以上版本中，该接口只支持更改 state 为 Up 或者 Offline，废弃了直接更改为 Tombstone 这个功能。这是由于直接更改为 Tombstone 总是引起操作者意料之外的结果。之后我们有计划后续对相关的场景进行一些针对性的开发。

在您的这个例子中，虽然该 TiKV 实例已经不存在，但由于使用 --force 直接下线机器，因此在逻辑上该 TiKV 上仍存在 Peer，PD 需要将这部分 Peer 全部搬运到其他节点上，这个 Store 才能正常转变为 Tombstone 并从 Information_schema 中消失。但是由于当前集群配置为 3 副本，但只有 2 个可用的 TiKV 节点，这部分 Peer 并没有目标可以搬运，所以下线流程只能阻塞。您这边需要增加一个 TiKV 实例，让这部分 Peer 有目标可以移动。然后只需等待旧 TiKV 的下线流程结束，Information_schema 中该 TiKV 就会消失。

如果原 IP 的机器上数据已经丢失，您希望在原机器上直接启动新的 TiKV 实例，此时会提示地址冲突而失败。目前推荐的做法是，在原机器上使用另外一个端口进行上线新 TiKV 的操作。

1 个赞
14

好的， 我再试试

15

:+1::+1::+1:

16

通过重新上线一个TiKV实例，问题已解决， 多谢各位大佬的解答与帮助~

17

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。