没有region的store状态不能从offline转成tombstone

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:
【 TiDB 使用环境】
k8s
【概述】场景+问题概述
对tiflash进行缩容之后,region数量为0的store状态一直在offline,无法转成tombstone
【背景】做过哪些操作
缩容,store delete 2004
【现象】业务和数据库现象

» store delete 2004
Success!
» store 2004
{
  "store": {
    "id": 2004,
    "address": "c134a-tiflash-1.c134a-tiflash-peer.tidb-prod.svc:3930",
    "state": 1,
    "labels": [
      {
        "key": "engine",
        "value": "tiflash"
      }
    ],
    "version": "v4.0.12",
    "peer_address": "c134a-tiflash-1.c134a-tiflash-peer.tidb-prod.svc:20170",
    "status_address": "c134a-tiflash-1.c134a-tiflash-peer.tidb-prod.svc:20292",
    "git_hash": "e5e3fcf251bae181b9b6c8676f57c51eb22106d8",
    "start_timestamp": 1642051324,
    "deploy_path": "/tiflash",
    "last_heartbeat": 1642056195387010018,
    "state_name": "Offline"
  },
  "status": {
    "capacity": "0B",
    "available": "0B",
    "used_size": "0B",
    "leader_count": 0,
    "leader_weight": 1,
    "leader_score": 0,
    "leader_size": 0,
    "region_count": 1568,
    "region_weight": 1,
    "region_score": 165593,
    "region_size": 165593,
    "start_ts": "2022-01-13T05:22:04Z",
    "last_heartbeat_ts": "2022-01-13T06:43:15.387010018Z",
    "uptime": "1h21m11.387010018s"
  }
}

» region store 2004
{
  "count": 0,
  "regions": []
}
» config show
{
  "replication": {
    "enable-placement-rules": "true",
    "location-labels": "",
    "max-replicas": 5,
    "strictly-match-label": "false"
  },
  "schedule": {
    "enable-cross-table-merge": "false",
    "enable-debug-metrics": "false",
    "enable-location-replacement": "true",
    "enable-make-up-replica": "true",
    "enable-one-way-merge": "false",
    "enable-remove-down-replica": "true",
    "enable-remove-extra-replica": "true",
    "enable-replace-offline-replica": "true",
    "high-space-ratio": 0.7,
    "hot-region-cache-hits-threshold": 3,
    "hot-region-schedule-limit": 4,
    "leader-schedule-limit": 4,
    "leader-schedule-policy": "count",
    "low-space-ratio": 0.8,
    "max-merge-region-keys": 200000,
    "max-merge-region-size": 80,
    "max-pending-peer-count": 16,
    "max-snapshot-count": 3,
    "max-store-down-time": "30m0s",
    "merge-schedule-limit": 8,
    "patrol-region-interval": "100ms",
    "region-schedule-limit": 2048,
    "replica-schedule-limit": 64,
    "scheduler-max-waiting-operator": 5,
    "split-merge-interval": "1h0m0s",
    "store-limit-mode": "manual",
    "tolerant-size-ratio": 20
  }
}

【业务影响】
删除tiflash节点后无法在pd内删除对应store节点
【TiDB 版本】
v4.0.12
【附件】

  1. TiUP Cluster Display 信息

  2. TiUP Cluster Edit Config 信息

  3. TiDB- Overview 监控

  • 对应模块日志(包含问题前后1小时日志)
1赞

asktug 上有个类似的帖子,可以通过 官网的 pd-ctl region 命令,通过 jq 找到 region peer在 这个 store 上的region,然后通过 remove-region 命令移除一下 命令参考:>> region --jq=".regions[] | {id: .id, peer_stores: [.peers[].store_id]}" https://docs.pingcap.com/zh/tidb/v5.2/pd-control/#简化-store-的输出

这个store上的region count已经是0了,但是state一直是offline,后来大概过了一天才下线