Tikv节点异常下线，怎么有损恢复

TiDBer_WZnVC06e · 2022 年3 月 28 日 10:03

【 TiDB 使用环境】
v5.2.2，使用TiDB Operator部署。
【概述】场景+问题概述
之前做集群迁移时，一个TiKV和一个TiFlash节点数据没有迁移完成就把机器给下线了。目前集群在pd-ctl下查看存在两个 Offline 状态的 store，因为涉及的region数据不多，请问有什么办法可以将这些store和异常的region移除(可以接受某些数据丢失的情况下)

【现象】业务和数据库现象
dashboard面板显示信息:

pd-ctl
store:
{
“count”: 6,
“stores”: [
{
“store”: {
“id”: 30432003,
“address”: “stg-bigdata-tidb-tikv-0.stg-bigdata-tidb-tikv-peer.stg-bigdata-tidb.svc:20160”,
“version”: “5.2.2”,
“status_address”: “stg-bigdata-tidb-tikv-0.stg-bigdata-tidb-tikv-peer.stg-bigdata-tidb.svc:20180”,
“git_hash”: “7acaec5d9c809439b9b0017711f114b44ffd9a49”,
“start_timestamp”: 1645958544,
“deploy_path”: “/”,
“last_heartbeat”: 1648461606600619619,
“state_name”: “Up”
},
“status”: {
“capacity”: “196.7GiB”,
“available”: “115.8GiB”,
“used_size”: “16.43GiB”,
“leader_count”: 12353,
“leader_weight”: 1,
“leader_score”: 12353,
“leader_size”: 82809,
“region_count”: 12353,
“region_weight”: 1,
“region_score”: 82809,
“region_size”: 82809,
“slow_score”: 1,
“start_ts”: “2022-02-27T10:42:24Z”,
“last_heartbeat_ts”: “2022-03-28T10:00:06.600619619Z”,
“uptime”: “695h17m42.600619619s”
}
},
{
“store”: {
“id”: 44,
“address”: “192.168.2.232:3930”,
“state”: 1,
“labels”: [
{
“key”: “engine”,
“value”: “tiflash”
}
],
“version”: “v5.2.2”,
“peer_address”: “192.168.2.232:20170”,
“status_address”: “192.168.2.232:20292”,
“git_hash”: “8242cf25b1e3f99c2fe3c7dd8909c05f7338c455”,
“start_timestamp”: 1645090158,
“deploy_path”: “/data/tiflash/deploy/bin/tiflash”,
“last_heartbeat”: 1645172439887251443,
“state_name”: “Offline”
},
“status”: {
“capacity”: “0B”,
“available”: “0B”,
“used_size”: “0B”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 0,
“region_weight”: 1,
“region_score”: 0,
“region_size”: 0,
“slow_score”: 0,
“start_ts”: “2022-02-17T09:29:18Z”,
“last_heartbeat_ts”: “2022-02-18T08:20:39.887251443Z”,
“uptime”: “22h51m21.887251443s”
}
},
{
“store”: {
“id”: 111003,
“address”: “stg-bigdata-tidb-tiflash-0.stg-bigdata-tidb-tiflash-peer.stg-bigdata-tidb.svc:3930”,
“labels”: [
{
“key”: “engine”,
“value”: “tiflash”
}
],
“version”: “v5.2.2”,
“peer_address”: “stg-bigdata-tidb-tiflash-0.stg-bigdata-tidb-tiflash-peer.stg-bigdata-tidb.svc:20170”,
“status_address”: “stg-bigdata-tidb-tiflash-0.stg-bigdata-tidb-tiflash-peer.stg-bigdata-tidb.svc:20292”,
“git_hash”: “8242cf25b1e3f99c2fe3c7dd8909c05f7338c455”,
“start_timestamp”: 1645688448,
“deploy_path”: “/tiflash”,
“last_heartbeat”: 1648461612909183625,
“state_name”: “Up”
},
“status”: {
“capacity”: “879.8GiB”,
“available”: “873.3GiB”,
“used_size”: “6.554GiB”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 83,
“region_weight”: 1,
“region_score”: 6076,
“region_size”: 6076,
“slow_score”: 0,
“start_ts”: “2022-02-24T07:40:48Z”,
“last_heartbeat_ts”: “2022-03-28T10:00:12.909183625Z”,
“uptime”: “770h19m24.909183625s”
}
},
{
“store”: {
“id”: 111100,
“address”: “stg-bigdata-tidb-tikv-0.stg-bigdata-tidb-tikv-peer.stg-bigdata-tidb.svc:20160”,
“state”: 1,
“version”: “5.2.2”,
“status_address”: “stg-bigdata-tidb-tikv-0.stg-bigdata-tidb-tikv-peer.stg-bigdata-tidb.svc:20180”,
“git_hash”: “7acaec5d9c809439b9b0017711f114b44ffd9a49”,
“start_timestamp”: 1645066037,
“deploy_path”: “/”,
“last_heartbeat”: 1645066026855935212,
“physically_destroyed”: true,
“state_name”: “Offline”
},
“status”: {
“capacity”: “0B”,
“available”: “0B”,
“used_size”: “0B”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 0,
“region_weight”: 1,
“region_score”: 0,
“region_size”: 0,
“slow_score”: 0,
“start_ts”: “2022-02-17T02:47:17Z”,
“last_heartbeat_ts”: “2022-02-17T02:47:06.855935212Z”
}
},
{
“store”: {
“id”: 30432001,
“address”: “stg-bigdata-tidb-tikv-2.stg-bigdata-tidb-tikv-peer.stg-bigdata-tidb.svc:20160”,
“version”: “5.2.2”,
“status_address”: “stg-bigdata-tidb-tikv-2.stg-bigdata-tidb-tikv-peer.stg-bigdata-tidb.svc:20180”,
“git_hash”: “7acaec5d9c809439b9b0017711f114b44ffd9a49”,
“start_timestamp”: 1645778486,
“deploy_path”: “/”,
“last_heartbeat”: 1648461604416859196,
“state_name”: “Up”
},
“status”: {
“capacity”: “196.7GiB”,
“available”: “115.8GiB”,
“used_size”: “14.93GiB”,
“leader_count”: 12001,
“leader_weight”: 1,
“leader_score”: 12001,
“leader_size”: 71324,
“region_count”: 12001,
“region_weight”: 1,
“region_score”: 71324,
“region_size”: 71324,
“slow_score”: 1,
“start_ts”: “2022-02-25T08:41:26Z”,
“last_heartbeat_ts”: “2022-03-28T10:00:04.416859196Z”,
“uptime”: “745h18m38.416859196s”
}
},
{
“store”: {
“id”: 30432002,
“address”: “stg-bigdata-tidb-tikv-1.stg-bigdata-tidb-tikv-peer.stg-bigdata-tidb.svc:20160”,
“version”: “5.2.2”,
“status_address”: “stg-bigdata-tidb-tikv-1.stg-bigdata-tidb-tikv-peer.stg-bigdata-tidb.svc:20180”,
“git_hash”: “7acaec5d9c809439b9b0017711f114b44ffd9a49”,
“start_timestamp”: 1645868519,
“deploy_path”: “/”,
“last_heartbeat”: 1648461613515972235,
“state_name”: “Up”
},
“status”: {
“capacity”: “281.5GiB”,
“available”: “221.3GiB”,
“used_size”: “27.45GiB”,
“leader_count”: 14236,
“leader_weight”: 1,
“leader_score”: 14236,
“leader_size”: 113304,
“region_count”: 14236,
“region_weight”: 1,
“region_score”: 113304,
“region_size”: 113304,
“slow_score”: 1,
“start_ts”: “2022-02-26T09:41:59Z”,
“last_heartbeat_ts”: “2022-03-28T10:00:13.515972235Z”,
“uptime”: “720h18m14.515972235s”
}
}
]
}

region store 44:
{
“count”: 2,
“regions”: [
{
“id”: 21831,
“start_key”: “7480000000000018FFE85F720000000000FA”,
“end_key”: “7480000000000018FFE900000000000000F8”,
“epoch”: {
“conf_ver”: 8,
“version”: 4734
},
“peers”: [
{
“id”: 21833,
“store_id”: 44,
“role”: 1,
“role_name”: “Learner”,
“is_learner”: true
},
{
“id”: 111134,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 22600,
“start_key”: “748000000000001AFF125F720000000000FA”,
“end_key”: “748000000000001AFF1300000000000000F8”,
“epoch”: {
“conf_ver”: 10,
“version”: 4847
},
“peers”: [
{
“id”: 82727,
“store_id”: 44,
“role”: 1,
“role_name”: “Learner”,
“is_learner”: true
},
{
“id”: 111143,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
}
]
}

region store 111100:
{
“count”: 25,
“regions”: [
{
“id”: 8377,
“start_key”: “748000000000000BFF8500000000000000F8”,
“end_key”: “748000000000000BFF8800000000000000F8”,
“epoch”: {
“conf_ver”: 7,
“version”: 2726
},
“peers”: [
{
“id”: 111146,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 9755,
“start_key”: “748000000000000EFF265F720000000000FA”,
“end_key”: “748000000000000EFF2700000000000000F8”,
“epoch”: {
“conf_ver”: 12,
“version”: 3323
},
“peers”: [
{
“id”: 111139,
“store_id”: 111100,
“role_name”: “Voter”
},
{
“id”: 141613,
“store_id”: 111003,
“role”: 1,
“role_name”: “Learner”,
“is_learner”: true
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 21499,
“start_key”: “7480000000000018FF755F720000000000FA”,
“end_key”: “7480000000000018FF7600000000000000F8”,
“epoch”: {
“conf_ver”: 10,
“version”: 4689
},
“peers”: [
{
“id”: 111147,
“store_id”: 111100,
“role_name”: “Voter”
},
{
“id”: 113206,
“store_id”: 111003,
“role”: 1,
“role_name”: “Learner”,
“is_learner”: true
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 21669,
“start_key”: “7480000000000018FFAD5F720000000000FA”,
“end_key”: “7480000000000018FFAE00000000000000F8”,
“epoch”: {
“conf_ver”: 10,
“version”: 4719
},
“peers”: [
{
“id”: 111106,
“store_id”: 111100,
“role_name”: “Voter”
},
{
“id”: 122269,
“store_id”: 111003,
“role”: 1,
“role_name”: “Learner”,
“is_learner”: true
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 21831,
“start_key”: “7480000000000018FFE85F720000000000FA”,
“end_key”: “7480000000000018FFE900000000000000F8”,
“epoch”: {
“conf_ver”: 8,
“version”: 4734
},
“peers”: [
{
“id”: 21833,
“store_id”: 44,
“role”: 1,
“role_name”: “Learner”,
“is_learner”: true
},
{
“id”: 111134,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 22169,
“start_key”: “7480000000000019FF4900000000000000F8”,
“end_key”: “7480000000000019FF495F720000000000FA”,
“epoch”: {
“conf_ver”: 7,
“version”: 4770
},
“peers”: [
{
“id”: 111121,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 22600,
“start_key”: “748000000000001AFF125F720000000000FA”,
“end_key”: “748000000000001AFF1300000000000000F8”,
“epoch”: {
“conf_ver”: 10,
“version”: 4847
},
“peers”: [
{
“id”: 82727,
“store_id”: 44,
“role”: 1,
“role_name”: “Learner”,
“is_learner”: true
},
{
“id”: 111143,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 83831,
“start_key”: “748000000000001AFFC700000000000000F8”,
“end_key”: “748000000000001AFFC75F720000000000FA”,
“epoch”: {
“conf_ver”: 9,
“version”: 4912
},
“peers”: [
{
“id”: 111140,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 23131,
“start_key”: “748000000000001AFFD45F720000000000FA”,
“end_key”: “748000000000001AFFD500000000000000F8”,
“epoch”: {
“conf_ver”: 10,
“version”: 4918
},
“peers”: [
{
“id”: 111120,
“store_id”: 111100,
“role_name”: “Voter”
},
{
“id”: 111347,
“store_id”: 111003,
“role”: 1,
“role_name”: “Learner”,
“is_learner”: true
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 84531,
“start_key”: “748000000000001AFFD800000000000000F8”,
“end_key”: “748000000000001AFFD85F720000000000FA”,
“epoch”: {
“conf_ver”: 7,
“version”: 4920
},
“peers”: [
{
“id”: 111124,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 23175,
“start_key”: “748000000000001AFFE45F720000000000FA”,
“end_key”: “748000000000001AFFE500000000000000F8”,
“epoch”: {
“conf_ver”: 10,
“version”: 4926
},
“peers”: [
{
“id”: 111125,
“store_id”: 111100,
“role_name”: “Voter”
},
{
“id”: 153350,
“store_id”: 111003,
“role”: 1,
“role_name”: “Learner”,
“is_learner”: true
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 84543,
“start_key”: “748000000000001AFFF000000000000000F8”,
“end_key”: “748000000000001AFFF05F720000000000FA”,
“epoch”: {
“conf_ver”: 7,
“version”: 4932
},
“peers”: [
{
“id”: 111136,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 34193,
“start_key”: “748000000000002DFF9C5F720000000000FA”,
“end_key”: “748000000000002DFF9D00000000000000F8”,
“epoch”: {
“conf_ver”: 10,
“version”: 8751
},
“peers”: [
{
“id”: 111127,
“store_id”: 111100,
“role_name”: “Voter”
},
{
“id”: 155697,
“store_id”: 111003,
“role”: 1,
“role_name”: “Learner”,
“is_learner”: true
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 35257,
“start_key”: “748000000000002FFF4E00000000000000F8”,
“end_key”: “748000000000002FFF4E5F720000000000FA”,
“epoch”: {
“conf_ver”: 7,
“version”: 9080
},
“peers”: [
{
“id”: 111118,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 37271,
“start_key”: “7480000000000032FFB100000000000000F8”,
“end_key”: “7480000000000032FFB15F720000000000FA”,
“epoch”: {
“conf_ver”: 7,
“version”: 9405
},
“peers”: [
{
“id”: 111130,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 38349,
“start_key”: “7480000000000034FF5B00000000000000F8”,
“end_key”: “7480000000000034FF5B5F720000000000FA”,
“epoch”: {
“conf_ver”: 7,
“version”: 9578
},
“peers”: [
{
“id”: 111103,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 38537,
“start_key”: “7480000000000034FF9700000000000000F8”,
“end_key”: “7480000000000034FF975F720000000000FA”,
“epoch”: {
“conf_ver”: 7,
“version”: 9600
},
“peers”: [
{
“id”: 111105,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 84827,
“start_key”: “7480000000000039FFC300000000000000F8”,
“end_key”: “7480000000000039FFC35F720000000000FA”,
“epoch”: {
“conf_ver”: 9,
“version”: 10109
},
“peers”: [
{
“id”: 111117,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 42426,
“start_key”: “748000000000003AFF705F720000000000FA”,
“end_key”: “748000000000003AFF7100000000000000F8”,
“epoch”: {
“conf_ver”: 12,
“version”: 10171
},
“peers”: [
{
“id”: 111144,
“store_id”: 111100,
“role_name”: “Voter”
},
{
“id”: 132300,
“store_id”: 111003,
“role”: 1,
“role_name”: “Learner”,
“is_learner”: true
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 84984,
“start_key”: “748000000000003AFFC100000000000000F8”,
“end_key”: “748000000000003AFFC15F720000000000FA”,
“epoch”: {
“conf_ver”: 7,
“version”: 10200
},
“peers”: [
{
“id”: 111128,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 76258,
“start_key”: “7480000000000074FFAC00000000000000F8”,
“end_key”: “7480000000000074FFAD00000000000000F8”,
“epoch”: {
“conf_ver”: 7,
“version”: 23700
},
“peers”: [
{
“id”: 111129,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 76322,
“start_key”: “7480000000000074FFCC00000000000000F8”,
“end_key”: “7480000000000074FFCD00000000000000F8”,
“epoch”: {
“conf_ver”: 7,
“version”: 23732
},
“peers”: [
{
“id”: 111133,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 76448,
“start_key”: “7480000000000075FF0B00000000000000F8”,
“end_key”: “7480000000000075FF0C00000000000000F8”,
“epoch”: {
“conf_ver”: 7,
“version”: 23795
},
“peers”: [
{
“id”: 111107,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 78961,
“start_key”: “7480000000000078FF7F00000000000000F8”,
“end_key”: “7480000000000078FF8100000000000000F8”,
“epoch”: {
“conf_ver”: 7,
“version”: 24645
},
“peers”: [
{
“id”: 111111,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 79040,
“start_key”: “7480000000000078FF8800000000000000F8”,
“end_key”: “7480000000000078FF8B00000000000000F8”,
“epoch”: {
“conf_ver”: 7,
“version”: 24649
},
“peers”: [
{
“id”: 111102,
“store_id”: 111100,
“role_name”: “Voter”
}
],
“leader”: {
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
}
]
}

h5n1 · 2022 年3 月 28 日 10:30

参考以下几个文章

TiDBer_WZnVC06e · 2022 年3 月 29 日 03:29

您好，我按照这些文档的操作之后，查询不会有问题了。可是store的状态还是无法删除，可以强制去掉吗

h5n1 · 2022 年3 月 29 日 03:30

pd-ctl store看下这个store状态，unsaf-recover后会自动补副本

TiDBer_WZnVC06e · 2022 年3 月 29 日 03:33

» store 111100
{
“store”: {
“id”: 111100,
“address”: “stg-bigdata-tidb-tikv-0.stg-bigdata-tidb-tikv-peer.stg-bigdata-tidb.svc:20160”,
“state”: 1,
“version”: “5.2.2”,
“status_address”: “stg-bigdata-tidb-tikv-0.stg-bigdata-tidb-tikv-peer.stg-bigdata-tidb.svc:20180”,
“git_hash”: “7acaec5d9c809439b9b0017711f114b44ffd9a49”,
“start_timestamp”: 1645066037,
“deploy_path”: “/”,
“last_heartbeat”: 1645066026855935212,
“physically_destroyed”: true,
“state_name”: “Offline”
},
“status”: {
“capacity”: “0B”,
“available”: “0B”,
“used_size”: “0B”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 0,
“region_weight”: 1,
“region_score”: 0,
“region_size”: 0,
“slow_score”: 0,
“start_ts”: “2022-02-17T02:47:17Z”,
“last_heartbeat_ts”: “2022-02-17T02:47:06.855935212Z”
}
}

h5n1 · 2022 年3 月 29 日 03:48

tiup display现在还能看到吗

TiDBer_WZnVC06e · 2022 年3 月 29 日 03:50

集群是部署在K8S中的，不能用tiup。 dashboard上是可以看到的

h5n1 · 2022 年3 月 29 日 04:04

pd-ctl store里显示“region_count”: 0,用下面的看下这store上的region是否还有
pd-ctl region --jq=’.regions[] | {id: .id, peer_stores: [.peers[].store_id] | select(length as $total | map(if .==(store_ID) then . else empty end)) }’

TiDBer_WZnVC06e · 2022 年3 月 29 日 04:09

是能看到的

h5n1 · 2022 年3 月 29 日 04:12

你之前的操作具体是如何执行的？找个前面查出来的region id 试试 pd-ctl operator add remove-peer <region_id> <from_store_id> 命令能从offline store上remove不

TiDBer_WZnVC06e · 2022 年3 月 29 日 06:51

之前使用过 curl -X DELETE http://XXXX:2379/pd/api/v1/store/111100?force=1
强制下线该store，现在该机器已经不可见了，肯定是不能添加调度了。

h5n1 · 2022 年3 月 29 日 07:01

这命令好像并不能强制删除节点，只是让他成为offline状态，然后转移region，之后成为tombone. 前面做unsafe-recover的具体步骤什么样

TiDBer_WZnVC06e · 2022 年3 月 29 日 07:10

是的，所以这个store的状态一直有问题。前面做的 unsafe-recover步骤：

在所有的tikv pod上:

kubectl annotate pod ${pod_name} -n ${namespace} runmode=debug
kubectl exec ${pod_name} -n ${namespace} -c tikv – kill -s TERM
kubectl exec -it ${pod_name} -n ${namespace} – sh
./tikv-ctl --data-dir /var/lib/tikv unsafe-recover remove-fail-stores -s 111100 --all-regions

然后重启 pd和tikv:
kubectl edit tc ${name} -n ${namespace}
给pd 添加annotation:
annotations:
tidb.pingcap.com/restartedAt: 2022-03-29T12:00

kubectl delete pods tikv… -n {namespace}
pd-ctl store 显示异常

在 Kubernetes 上使用 TiKV Control

TiDBer_WZnVC06e · 2022 年3 月 29 日 07:13

这是测试集群，副本数只有一，不太清楚要不要补偿空的region

h5n1 · 2022 年3 月 29 日 07:15

副本数只有一是max-replicas=1吗？

TiDBer_WZnVC06e · 2022 年3 月 29 日 07:16

使用了 tiflash，应该是由placement-rule确定的，副本数是1

TiDBer_WZnVC06e · 2022 年3 月 29 日 08:03

想知道把region对应的表直接truncate对store的状态是否有效

h5n1 · 2022 年3 月 29 日 08:06

tikv默认是3副本，tiflash 根据你SQL设置的副本数设置，按下面看看有多少副本数<3的
pd-ctl region --jq=".regions[] | {id: .id, peer_stores: [.peers[].store_id] | select(length < 3)}

TiDBer_WZnVC06e · 2022 年3 月 29 日 08:10

应该是全部region，出来了一大批。默认的值可能之前改动过

从监控也能看出来

截屏2022-03-29 下午4.12.38

h5n1 · 2022 年3 月 29 日 08:15

pd-ctl config |grep replicas 看下