tikv节点下线卡住

🪐 TiDB 技术问题 部署&运维管理
1

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：

【TiDB 版本】
v.4.0.6
【问题描述】
节点中有一台正在下线的kv，当region还剩25个的时候不再减少，过了一天之后也是如此，此时主要情况如图
集群状况：

WX20210425-104316%402x
WX20210425-104316%402x2500×596 457 KB
WX20210425-104316%402x
WX20210425-104316%402x2500×596 457 KB

store：
store
{
“count”: 7,
“stores”: [
{
“store”: {
“id”: 24590972,
“address”: “10.12.5.240:20160”,
“version”: “4.0.6”,
“status_address”: “10.12.5.240:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1619073191,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619318660054342245,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.952TiB”,
“available”: “3.893TiB”,
“used_size”: “1.94TiB”,
“leader_count”: 29748,
“leader_weight”: 2,
“leader_score”: 14874,
“leader_size”: 1916820,
“region_count”: 85719,
“region_weight”: 2,
“region_score”: 2934732.5,
“region_size”: 5869465,
“start_ts”: “2021-04-22T06:33:11Z”,
“last_heartbeat_ts”: “2021-04-25T02:44:20.054342245Z”,
“uptime”: “68h11m9.054342245s”
}
},
{
“store”: {
“id”: 38833310,
“address”: “10.12.5.147:20160”,
“version”: “4.0.6”,
“status_address”: “10.12.5.147:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1619059119,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619318663215606078,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.952TiB”,
“available”: “3.619TiB”,
“used_size”: “2.124TiB”,
“leader_count”: 29743,
“leader_weight”: 2,
“leader_score”: 14871.5,
“leader_size”: 1771028,
“region_count”: 91434,
“region_weight”: 2,
“region_score”: 2934741.5,
“region_size”: 5869483,
“start_ts”: “2021-04-22T02:38:39Z”,
“last_heartbeat_ts”: “2021-04-25T02:44:23.215606078Z”,
“uptime”: “72h5m44.215606078s”
}
},
{
“store”: {
“id”: 256634687,
“address”: “10.12.5.12:20160”,
“state”: 1,
“version”: “4.0.6”,
“status_address”: “10.12.5.12:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1618814078,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619318655613350203,
“state_name”: “Offline”
},
“status”: {
“capacity”: “2.952TiB”,
“available”: “2.914TiB”,
“used_size”: “2.169GiB”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 25,
“region_weight”: 1,
“region_score”: 0,
“region_size”: 0,
“start_ts”: “2021-04-19T06:34:38Z”,
“last_heartbeat_ts”: “2021-04-25T02:44:15.613350203Z”,
“uptime”: “140h9m37.613350203s”
}
},
{
“store”: {
“id”: 262397455,
“address”: “10.12.5.13:20160”,
“version”: “4.0.6”,
“status_address”: “10.12.5.13:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1618813933,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619318654726043464,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.952TiB”,
“available”: “4.801TiB”,
“used_size”: “1.13TiB”,
“leader_count”: 14873,
“leader_weight”: 1,
“leader_score”: 14873,
“leader_size”: 912239,
“region_count”: 47370,
“region_weight”: 1,
“region_score”: 2934787,
“region_size”: 2934787,
“start_ts”: “2021-04-19T06:32:13Z”,
“last_heartbeat_ts”: “2021-04-25T02:44:14.726043464Z”,
“uptime”: “140h12m1.726043464s”
}
},
{
“store”: {
“id”: 268391998,
“address”: “10.12.5.119:20160”,
“version”: “4.0.6”,
“status_address”: “10.12.5.119:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1618813990,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619318659505151908,
“state_name”: “Up”
},
“status”: {
“capacity”: “320TiB”,
“available”: “314.2TiB”,
“used_size”: “1.195TiB”,
“leader_count”: 14873,
“leader_weight”: 1,
“leader_score”: 14873,
“leader_size”: 825752,
“region_count”: 52465,
“region_weight”: 1,
“region_score”: 2934773,
“region_size”: 2934773,
“start_ts”: “2021-04-19T06:33:10Z”,
“last_heartbeat_ts”: “2021-04-25T02:44:19.505151908Z”,
“uptime”: “140h11m9.505151908s”
}
},
{
“store”: {
“id”: 24478148,
“address”: “10.12.5.236:20160”,
“version”: “4.0.6”,
“status_address”: “10.12.5.236:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1619066222,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619318657961619561,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.952TiB”,
“available”: “2.381TiB”,
“used_size”: “1.918TiB”,
“leader_count”: 29756,
“leader_weight”: 2,
“leader_score”: 14878,
“leader_size”: 2259221,
“region_count”: 82902,
“region_weight”: 2,
“region_score”: 2943703.663598895,
“region_size”: 5812946,
“start_ts”: “2021-04-22T04:37:02Z”,
“last_heartbeat_ts”: “2021-04-25T02:44:17.961619561Z”,
“uptime”: “70h7m15.961619561s”
}
},
{
“store”: {
“id”: 24480822,
“address”: “10.12.5.239:20160”,
“version”: “4.0.6”,
“status_address”: “10.12.5.239:20180”,
“git_hash”: “ca2475bfbcb49a7c34cf783596acb3edd05fc88f”,
“start_timestamp”: 1618917905,
“deploy_path”: “/home/tidb/deploy/bin”,
“last_heartbeat”: 1619318661493542236,
“state_name”: “Up”
},
“status”: {
“capacity”: “5.952TiB”,
“available”: “3.753TiB”,
“used_size”: “2.097TiB”,
“leader_count”: 29751,
“leader_weight”: 2,
“leader_score”: 14875.5,
“leader_size”: 2078368,
“region_count”: 86404,
“region_weight”: 2,
“region_score”: 2934440,
“region_size”: 5868880,
“start_ts”: “2021-04-20T11:25:05Z”,
“last_heartbeat_ts”: “2021-04-25T02:44:21.493542236Z”,
“uptime”: “111h19m16.493542236s”
}
}
]
}

下线节点的region状况:
region store 256634687
{
“count”: 25,
“regions”: [
{
“id”: 244288376,
“start_key”: “7480000000000001FF745F728000000018FF3D9A880000000000FA”,
“end_key”: “7480000000000001FF745F728000000018FF3DA62C0000000000FA”,
“epoch”: {
“conf_ver”: 1646,
“version”: 12144
},
“peers”: [
{
“id”: 244288378,
“store_id”: 38833310
},
{
“id”: 248345049,
“store_id”: 24478148
},
{
“id”: 261952190,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 256965,
“start_key”: “7480000000000001FF745F728000000019FF0A96B30000000000FA”,
“end_key”: “7480000000000001FF745F728000000019FF0B47DB0000000000FA”,
“epoch”: {
“conf_ver”: 1681,
“version”: 11771
},
“peers”: [
{
“id”: 38742994,
“store_id”: 24478148
},
{
“id”: 246642599,
“store_id”: 38833310
},
{
“id”: 262000265,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 6778802,
“start_key”: “7480000000000001FF745F728000000019FFEDD7530000000000FA”,
“end_key”: “7480000000000001FF745F728000000019FFEDFE9C0000000000FA”,
“epoch”: {
“conf_ver”: 1751,
“version”: 12135
},
“peers”: [
{
“id”: 247676313,
“store_id”: 24478148
},
{
“id”: 248584277,
“store_id”: 38833310
},
{
“id”: 262050508,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 80401278,
“start_key”: “7480000000000001FF745F72800000001AFF7F1AE90000000000FA”,
“end_key”: “7480000000000001FF745F72800000001AFF7F24760000000000FA”,
“epoch”: {
“conf_ver”: 1711,
“version”: 17175
},
“peers”: [
{
“id”: 80401281,
“store_id”: 38833310
},
{
“id”: 247373045,
“store_id”: 24478148
},
{
“id”: 261799117,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 9413063,
“start_key”: “7480000000000001FF745F72800000001CFF807A620000000000FA”,
“end_key”: “7480000000000001FF745F72800000001CFF80B7E80000000000FA”,
“epoch”: {
“conf_ver”: 1889,
“version”: 13234
},
“peers”: [
{
“id”: 233744881,
“store_id”: 24478148
},
{
“id”: 246043318,
“store_id”: 38833310
},
{
“id”: 261794408,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 9427705,
“start_key”: “7480000000000001FF745F72800000001CFFE469160000000000FA”,
“end_key”: “7480000000000001FF745F72800000001CFFE4B81C0000000000FA”,
“epoch”: {
“conf_ver”: 1892,
“version”: 13410
},
“peers”: [
{
“id”: 38972805,
“store_id”: 38833310
},
{
“id”: 248225260,
“store_id”: 24478148
},
{
“id”: 261862295,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 9588953,
“start_key”: “7480000000000001FF745F72800000001DFF256BA40000000000FA”,
“end_key”: “7480000000000001FF745F72800000001DFF25B0050000000000FA”,
“epoch”: {
“conf_ver”: 1907,
“version”: 13512
},
“peers”: [
{
“id”: 247656496,
“store_id”: 24478148
},
{
“id”: 248460362,
“store_id”: 38833310
},
{
“id”: 261813488,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 14324204,
“start_key”: “7480000000000001FF745F72800000001DFF507B160000000000FA”,
“end_key”: “7480000000000001FF745F72800000001DFF50E0350000000000FA”,
“epoch”: {
“conf_ver”: 1904,
“version”: 13576
},
“peers”: [
{
“id”: 248027420,
“store_id”: 38833310
},
{
“id”: 248892035,
“store_id”: 24478148
},
{
“id”: 261893968,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 9520702,
“start_key”: “7480000000000001FF745F72800000001DFFA9CDE90000000000FA”,
“end_key”: “7480000000000001FF745F72800000001DFFA9FE2C0000000000FA”,
“epoch”: {
“conf_ver”: 1952,
“version”: 13722
},
“peers”: [
{
“id”: 245474950,
“store_id”: 38833310
},
{
“id”: 248071470,
“store_id”: 24478148
},
{
“id”: 261913970,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 9506276,
“start_key”: “7480000000000001FF745F72800000001EFF24E47A0000000000FA”,
“end_key”: “7480000000000001FF745F72800000001EFF24E6820000000000FA”,
“epoch”: {
“conf_ver”: 1961,
“version”: 13926
},
“peers”: [
{
“id”: 194252073,
“store_id”: 38833310
},
{
“id”: 248819540,
“store_id”: 24478148
},
{
“id”: 261798312,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 23066631,
“start_key”: “7480000000000001FF745F728000000020FF29A9010000000000FA”,
“end_key”: “7480000000000001FF745F728000000020FF29BC580000000000FA”,
“epoch”: {
“conf_ver”: 2002,
“version”: 14976
},
“peers”: [
{
“id”: 247096627,
“store_id”: 38833310
},
{
“id”: 247652952,
“store_id”: 24478148
},
{
“id”: 262066138,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 23073113,
“start_key”: “7480000000000001FF745F728000000020FF8B4A140000000000FA”,
“end_key”: “7480000000000001FF745F728000000020FF8B86360000000000FA”,
“epoch”: {
“conf_ver”: 2029,
“version”: 15863
},
“peers”: [
{
“id”: 247260250,
“store_id”: 38833310
},
{
“id”: 249108661,
“store_id”: 24478148
},
{
“id”: 262088650,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 246923725,
“start_key”: “7480000000000001FF745F728000000021FF8C5E780000000000FA”,
“end_key”: “7480000000000001FF745F728000000021FF8CB6370000000000FA”,
“epoch”: {
“conf_ver”: 1996,
“version”: 17715
},
“peers”: [
{
“id”: 246923726,
“store_id”: 38833310
},
{
“id”: 248146517,
“store_id”: 24478148
},
{
“id”: 261808137,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 24215180,
“start_key”: “7480000000000001FF745F728000000022FF2C36060000000000FA”,
“end_key”: “7480000000000001FF745F728000000022FF2C3FDC0000000000FA”,
“epoch”: {
“conf_ver”: 8686,
“version”: 15833
},
“peers”: [
{
“id”: 245820272,
“store_id”: 24478148
},
{
“id”: 249359537,
“store_id”: 38833310
},
{
“id”: 261980561,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 80335931,
“start_key”: “7480000000000001FF745F728000000022FF8A4F660000000000FA”,
“end_key”: “7480000000000001FF745F728000000022FF8A55020000000000FA”,
“epoch”: {
“conf_ver”: 12018,
“version”: 16391
},
“peers”: [
{
“id”: 194207454,
“store_id”: 38833310
},
{
“id”: 247374608,
“store_id”: 24478148
},
{
“id”: 261819044,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 38453328,
“start_key”: “7480000000000001FF745F728000000022FF99A6820000000000FA”,
“end_key”: “7480000000000001FF745F728000000022FF99B4970000000000FA”,
“epoch”: {
“conf_ver”: 13284,
“version”: 16480
},
“peers”: [
{
“id”: 193959102,
“store_id”: 38833310
},
{
“id”: 247629065,
“store_id”: 24478148
},
{
“id”: 262090281,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 139439976,
“start_key”: “7480000000000001FF745F728000000023FF1529800000000000FA”,
“end_key”: “7480000000000001FF745F728000000023FF1531CB0000000000FA”,
“epoch”: {
“conf_ver”: 24105,
“version”: 17205
},
“peers”: [
{
“id”: 193492372,
“store_id”: 24478148
},
{
“id”: 248631790,
“store_id”: 38833310
},
{
“id”: 261800753,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 195867836,
“start_key”: “7480000000000001FF745F728000000023FF6866E60000000000FA”,
“end_key”: “7480000000000001FF745F728000000023FF686A970000000000FA”,
“epoch”: {
“conf_ver”: 37305,
“version”: 17686
},
“peers”: [
{
“id”: 234794161,
“store_id”: 38833310
},
{
“id”: 249024664,
“store_id”: 24478148
},
{
“id”: 262068060,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 246797073,
“start_key”: “7480000000000001FF745F728000000024FF8497760000000000FA”,
“end_key”: “7480000000000001FF745F728000000024FF84A4140000000000FA”,
“epoch”: {
“conf_ver”: 44565,
“version”: 18482
},
“peers”: [
{
“id”: 246797074,
“store_id”: 24478148
},
{
“id”: 246797075,
“store_id”: 38833310
},
{
“id”: 261952197,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 246838547,
“start_key”: “7480000000000001FF745F728000000024FF84E6CD0000000000FA”,
“end_key”: “7480000000000001FF745F728000000024FF84EB810000000000FA”,
“epoch”: {
“conf_ver”: 44586,
“version”: 18487
},
“peers”: [
{
“id”: 246838549,
“store_id”: 24478148
},
{
“id”: 246838550,
“store_id”: 38833310
},
{
“id”: 261954162,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 248266559,
“start_key”: “7480000000000001FF745F728000000025FF2C08630000000000FA”,
“end_key”: “7480000000000001FF745F728000000025FF2C1CBF0000000000FA”,
“epoch”: {
“conf_ver”: 61062,
“version”: 19588
},
“peers”: [
{
“id”: 248273535,
“store_id”: 24478148
},
{
“id”: 248603891,
“store_id”: 38833310
},
{
“id”: 261899633,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 332451,
“start_key”: “7480000000000001FFF15F728000000000FF3DD8CA0000000000FA”,
“end_key”: “7480000000000001FFF15F728000000000FF3E0F6D0000000000FA”,
“epoch”: {
“conf_ver”: 962,
“version”: 2956
},
“peers”: [
{
“id”: 246929063,
“store_id”: 38833310
},
{
“id”: 248401276,
“store_id”: 24478148
},
{
“id”: 261955797,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 2579214,
“start_key”: “7480000000000001FFF15F728000000000FF7D6BB50000000000FA”,
“end_key”: “7480000000000001FFF15F728000000000FF7DA55B0000000000FA”,
“epoch”: {
“conf_ver”: 950,
“version”: 3074
},
“peers”: [
{
“id”: 245657958,
“store_id”: 24478148
},
{
“id”: 248988410,
“store_id”: 38833310
},
{
“id”: 261980565,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 338610,
“start_key”: “7480000000000001FFF15F728000000001FF4434330000000000FA”,
“end_key”: “7480000000000001FFF15F728000000001FF4468940000000000FA”,
“epoch”: {
“conf_ver”: 971,
“version”: 3419
},
“peers”: [
{
“id”: 38857667,
“store_id”: 38833310
},
{
“id”: 246707490,
“store_id”: 24478148
},
{
“id”: 262039954,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
},
{
“id”: 362351,
“start_key”: “7480000000000001FFF15F728000000005FF57AFA20000000000FA”,
“end_key”: “7480000000000001FFF15F728000000005FF57DDE80000000000FA”,
“epoch”: {
“conf_ver”: 1181,
“version”: 5251
},
“peers”: [
{
“id”: 246636450,
“store_id”: 24478148
},
{
“id”: 247856915,
“store_id”: 38833310
},
{
“id”: 261985228,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
}
]
}

» config show
{
“replication”: {
“enable-placement-rules”: “false”,
“location-labels”: “”,
“max-replicas”: 3,
“strictly-match-label”: “false”
},
“schedule”: {
“enable-cross-table-merge”: “true”,
“enable-debug-metrics”: “false”,
“enable-location-replacement”: “true”,
“enable-make-up-replica”: “true”,
“enable-one-way-merge”: “false”,
“enable-remove-down-replica”: “true”,
“enable-remove-extra-replica”: “true”,
“enable-replace-offline-replica”: “true”,
“high-space-ratio”: 0.6,
“hot-region-cache-hits-threshold”: 3,
“hot-region-schedule-limit”: 4,
“leader-schedule-limit”: 32,
“leader-schedule-policy”: “count”,
“low-space-ratio”: 0.8,
“max-merge-region-keys”: 200000,
“max-merge-region-size”: 20,
“max-pending-peer-count”: 24,
“max-snapshot-count”: 6,
“max-store-down-time”: “30m0s”,
“merge-schedule-limit”: 16,
“patrol-region-interval”: “20ms”,
“region-schedule-limit”: 2500,
“replica-schedule-limit”: 64,
“scheduler-max-waiting-operator”: 3,
“schedulers-payload”: {
“balance-hot-region-scheduler”: null,
“balance-leader-scheduler”: {
“name”: “balance-leader-scheduler”,
“ranges”: [
{
“end-key”: “”,
“start-key”: “”
}
]
},
“balance-region-scheduler”: {
“name”: “balance-region-scheduler”,
“ranges”: [
{
“end-key”: “”,
“start-key”: “”
}
]
},
“evict-leader-scheduler”: {
“store-id-ranges”: {
“256634687”: [
{
“end-key”: “”,
“start-key”: “”
}
]
}
},
“label-scheduler”: {
“name”: “label-scheduler”,
“ranges”: [
{
“end-key”: “”,
“start-key”: “”
}
]
}
},
“schedulers-v2”: [
{
“args”: null,
“args-payload”: “”,
“disable”: false,
“type”: “balance-region”
},
{
“args”: null,
“args-payload”: “”,
“disable”: false,
“type”: “balance-leader”
},
{
“args”: null,
“args-payload”: “”,
“disable”: false,
“type”: “hot-region”
},
{
“args”: null,
“args-payload”: “”,
“disable”: false,
“type”: “label”
},
{
“args”: [
“256634687”
],
“args-payload”: “”,
“disable”: false,
“type”: “evict-leader”
}
],
“split-merge-interval”: “1h0m0s”,
“store-limit”: {
“24478148”: {
“add-peer”: 15,
“remove-peer”: 15
},
“24480822”: {
“add-peer”: 15,
“remove-peer”: 15
},
“24590972”: {
“add-peer”: 15,
“remove-peer”: 15
},
“256634687”: {
“add-peer”: 15,
“remove-peer”: 15
},
“262397455”: {
“add-peer”: 15,
“remove-peer”: 15
},
“268391998”: {
“add-peer”: 15,
“remove-peer”: 15
},
“38546296”: {
“add-peer”: 15,
“remove-peer”: 15
},
“38833310”: {
“add-peer”: 15,
“remove-peer”: 15
}
},
“store-limit-mode”: “manual”,
“tolerant-size-ratio”: 5
}
}

我参考了https://asktug.com/t/topic/32938的内容，但由于下线节点上的region都没有leader，无法执行operator操作

此时balance-leader的情况如图

WX20210425-105438%402x
WX20210425-105438%402x1334×1090 99.5 KB

：
目前的主要问题是查询中有大量region is unavailble，region health状况如图：
WX20210425-105636%402x
WX20210425-105636%402x1264×536 48.4 KB

2

更新一下问题，出现region is unavailable的访问都是针对tx_info这张表的，通过查询未下线节点的25个region所属数据表发现，的确属于tx_info表,由于这25个region无leader，导致无法转移到其他节点，请问针对该具体问题应该如何操作

» region --jq '.regions[]|select(has(“leader”)|not)|{id: .id,peer_stores: [.peers[].store_id]}’

{“id”:14324204,“peer_stores”:[38833310,24478148,256634687]}

{“id”:23073113,“peer_stores”:[38833310,24478148,256634687]}

{“id”:80335931,“peer_stores”:[38833310,24478148,256634687]}

{“id”:261757919,“peer_stores”:[24480822,38833310]}

{“id”:9413063,“peer_stores”:[24478148,38833310,256634687]}

{“id”:9520702,“peer_stores”:[38833310,24478148,256634687]}

{“id”:38453328,“peer_stores”:[38833310,24478148,256634687]}

{“id”:195867836,“peer_stores”:[38833310,24478148,256634687]}

{“id”:9506276,“peer_stores”:[38833310,24478148,256634687]}

{“id”:362351,“peer_stores”:[24478148,38833310,256634687]}

{“id”:9427705,“peer_stores”:[38833310,24478148,256634687]}

{“id”:246923725,“peer_stores”:[38833310,24478148,256634687]}

{“id”:80401278,“peer_stores”:[38833310,24478148,256634687]}

{“id”:338610,“peer_stores”:[38833310,24478148,256634687]}

{“id”:24215180,“peer_stores”:[24478148,38833310,256634687]}

{“id”:139439976,“peer_stores”:[24478148,38833310,256634687]}

{“id”:255117,“peer_stores”:[24480822,38833310,24478148]}

{“id”:246838547,“peer_stores”:[24478148,38833310,256634687]}

{“id”:256965,“peer_stores”:[24478148,38833310,256634687]}

{“id”:23066631,“peer_stores”:[38833310,24478148,256634687]}

{“id”:332451,“peer_stores”:[38833310,24478148,256634687]}

{“id”:246797073,“peer_stores”:[24478148,38833310,256634687]}

{“id”:6778802,“peer_stores”:[24478148,38833310,256634687]}

{“id”:244288376,“peer_stores”:[38833310,24478148,256634687]}

{“id”:9588953,“peer_stores”:[24478148,38833310,256634687]}

{“id”:248266559,“peer_stores”:[24478148,38833310,256634687]}

{“id”:2579214,“peer_stores”:[24478148,38833310,256634687]}

tidb@three : ~ $ curl http://10.176.34.148:10080/regions/23073113

{

“region_id”: 23073113,

“start_key”: “dIAAAAAAAAF0X3KAAAAAIItKFA==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAIIuGNg==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 545999380

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/80335931

{

“region_id”: 80335931,

“start_key”: “dIAAAAAAAAF0X3KAAAAAIopPZg==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAIopVAg==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 579489638

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/261757919

{

“region_id”: 261757919,

“start_key”: “dIAAAAAAAAF0X2mAAAAAAAAAAQE3NTVjODcwOP81NTZhNjEzM/83OWZhY2FhMP9iNGVkMzYzNf8wZTI3Y2FiMf9hNWE5OTg1Yv9lOGM4OWI0Nf9lNDY3ZDhmM/8AAAAAAAAAAPc=”,

“end_key”: “dIAAAAAAAAF0X2mAAAAAAAAAAQE3NWE0NzRlMv85MjBkZjE0MP85YTI2OWQwYf8zMDQ3OTdmY/83NDM0ZDIwOP83MTcwN2ZhYf8wNTU3YmRkYf8xMjUyYzFhNf8AAAAAAAAAAPc=”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: false,

“index_name”: “PRIMARY”,

“index_id”: 1,

“index_values”: [

“755c8708556a613379facaa0b4ed36350e27cab1a5a9985be8c89b45e467d8f3”

]

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/9413063

{

“region_id”: 9413063,

“start_key”: “dIAAAAAAAAF0X3KAAAAAHIB6Yg==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAHIC36A==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 478181986

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/9520702

{

“region_id”: 9520702,

“start_key”: “dIAAAAAAAAF0X3KAAAAAHanN6Q==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAHan+LA==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 497667561

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/38453328

{

“region_id”: 38453328,

“start_key”: “dIAAAAAAAAF0X3KAAAAAIpmmgg==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAIpm0lw==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 580494978

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/195867836

{

“region_id”: 195867836,

“start_key”: “dIAAAAAAAAF0X3KAAAAAI2hm5g==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAI2hqlw==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 594044646

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/9506276

{

“region_id”: 9506276,

“start_key”: “dIAAAAAAAAF0X3KAAAAAHiTkeg==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAHiTmgg==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 505734266

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/362351

{

“region_id”: 362351,

“start_key”: “dIAAAAAAAAHxX3KAAAAABVevog==”,

“end_key”: “dIAAAAAAAAHxX3KAAAAABVfd6A==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_tmp”,

“table_id”: 497,

“is_record”: true,

“record_id”: 89632674

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/9427705

{

“region_id”: 9427705,

“start_key”: “dIAAAAAAAAF0X3KAAAAAHORpFg==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAHOS4HA==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 484731158

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/246923725

{

“region_id”: 246923725,

“start_key”: “dIAAAAAAAAF0X3KAAAAAIYxeeA==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAIYy2Nw==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 562847352

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/80401278

{

“region_id”: 80401278,

“start_key”: “dIAAAAAAAAF0X3KAAAAAGn8a6Q==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAGn8kdg==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 444537577

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/338610

{

“region_id”: 338610,

“start_key”: “dIAAAAAAAAHxX3KAAAAAAUQ0Mw==”,

“end_key”: “dIAAAAAAAAHxX3KAAAAAAURolA==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_tmp”,

“table_id”: 497,

“is_record”: true,

“record_id”: 21247027

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/24215180

{

“region_id”: 24215180,

“start_key”: “dIAAAAAAAAF0X3KAAAAAIiw2Bg==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAIiw/3A==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 573322758

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/139439976

{

“region_id”: 139439976,

“start_key”: “dIAAAAAAAAF0X3KAAAAAIxUpgA==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAIxUxyw==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 588589440

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/255117

{

“region_id”: 255117,

“start_key”: “dIAAAAAAAAF0X2mAAAAAAAAAAQE3NWE0NzRlMv85MjBkZjE0MP85YTI2OWQwYf8zMDQ3OTdmY/83NDM0ZDIwOP83MTcwN2ZhYf8wNTU3YmRkYf8xMjUyYzFhNf8AAAAAAAAAAPc=”,

“end_key”: “dIAAAAAAAAF0X2mAAAAAAAAAAQE3NWM5MTBhOP9kOWVjZTU5MP84NzI5NGY5Zf82YWJiOGM5Nf8yZjM5NTNiZf9lMTkyZTAzNv9iMjE3MDk3NP9hYmU3ODJmY/8AAAAAAAAAAPc=”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: false,

“index_name”: “PRIMARY”,

“index_id”: 1,

“index_values”: [

“75a474e2920df1409a269d0a304797fc7434d20871707faa0557bdda1252c1a5”

]

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/246838547

{

“region_id”: 246838547,

“start_key”: “dIAAAAAAAAF0X3KAAAAAJITmzQ==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAJITrgQ==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 612689613

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/256965

{

“region_id”: 256965,

“start_key”: “dIAAAAAAAAF0X3KAAAAAGQqWsw==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAGQtH2w==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 420124339

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/23066631

{

“region_id”: 23066631,

“start_key”: “dIAAAAAAAAF0X3KAAAAAICmpAQ==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAICm8WA==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 539601153

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/332451

{

“region_id”: 332451,

“start_key”: “dIAAAAAAAAHxX3KAAAAAAD3Yyg==”,

“end_key”: “dIAAAAAAAAHxX3KAAAAAAD4PbQ==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_tmp”,

“table_id”: 497,

“is_record”: true,

“record_id”: 4053194

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/246797073

{

“region_id”: 246797073,

“start_key”: “dIAAAAAAAAF0X3KAAAAAJISXdg==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAJISkFA==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 612669302

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/6778802

{

“region_id”: 6778802,

“start_key”: “dIAAAAAAAAF0X3KAAAAAGe3XUw==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAGe3+nA==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 435017555

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/244288376

{

“region_id”: 244288376,

“start_key”: “dIAAAAAAAAF0X3KAAAAAGD2aiA==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAGD2mLA==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 406690440

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/9588953

{

“region_id”: 9588953,

“start_key”: “dIAAAAAAAAF0X3KAAAAAHSVrpA==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAHSWwBQ==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 488991652

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/248266559

{

“region_id”: 248266559,

“start_key”: “dIAAAAAAAAF0X3KAAAAAJSwIYw==”,

“end_key”: “dIAAAAAAAAF0X3KAAAAAJSwcvw==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_info”,

“table_id”: 372,

“is_record”: true,

“record_id”: 623642723

}

]

} tidb@three : ~ $ curl http://10.176.34.148:10080/regions/2579214

{

“region_id”: 2579214,

“start_key”: “dIAAAAAAAAHxX3KAAAAAAH1rtQ==”,

“end_key”: “dIAAAAAAAAHxX3KAAAAAAH2lWw==”,

“frames”: [

{

“db_name”: “Transaction_Base”,

“table_name”: “tx_tmp”,

“table_id”: 497,

“is_record”: true,

“record_id”: 8219573

}

]

}

3

麻烦提供一下所有 TiKV 节点的 tikv.log 日志，我们分析下看为什么 region 无法选举出 leader

4

一共7个kv的tikv.log，请帮忙看一下，非常感谢
链接: https://pan.baidu.com/s/12EgsTk9_xG2SEV9ijBbN_A 密码: uu9m

5 
6296, not found\") }))"]
[2021/04/25 00:44:54.673 +00:00] [ERROR] [util.rs:347] ["request failed"] [err_code=KV-PD-gRPC] [err="Grpc(RpcFailure(RpcStatus { status: 2-UNKNOWN, details: Some(\"invalid store ID 38546296, not found\") }))"]
[2021/04/25 00:44:54.673 +00:00] [ERROR] [util.rs:347] ["request failed"] [err_code=KV-PD-gRPC] [err="Grpc(RpcFailure(RpcStatus { status: 2-UNKNOWN, details: Some(\"invalid store ID 38546296, not found\") }))"]
[2021/04/25 00:44:54.674 +00:00] [ERROR] [util.rs:347] ["request failed"] [err_code=KV-PD-gRPC] [err="Grpc(RpcFailure(RpcStatus { status: 2-UNKNOWN, details: Some(\"invalid store ID 38546296, not found\") }))"]
[2021/04/25 00:44:54.674 +00:00] [ERROR] [util.rs:347] ["request failed"] [err_code=KV-PD-gRPC] [err="Grpc(RpcFailure(RpcStatus { status: 2-UNKNOWN, details: Some(\"invalid store ID 38546296, not found\") }))"]
[2021/04/25 00:44:54.674 +00:00] [ERROR] [util.rs:347] ["request failed"] [err_code=KV-PD-gRPC] [err="Grpc(RpcFailure(RpcStatus { status: 2-UNKNOWN, details: Some(\"invalid store ID 38546296, not found\") }))"]
[2021/04/25 00:44:54.674 +00:00] [ERROR] [transport.rs:137] ["resolve store address failed"] [err="Other(\"[src/server/resolve.rs:72]: description() is deprecated; use Display\")"] [store_id=38546296]

从下线的 12 节点日志中看到还有有请求发送给 store 38546296 节点的信息，但是因为 38546296 节点之前强制下线了，所以解析有异常。

可以在所有 tikv 节点上都执行一下 unsafe-recover 操作，将残留的 38546296 节点信息都清理一下

tikv-ctl --db /path/to/tikv/db unsafe-recover remove-fail-stores -s 38546296 --all-regions

执行该命令的时候对应的 tikv 节点实例需要关闭
https://docs.pingcap.com/zh/tidb/v4.0/tikv-control#强制-region-从多副本失败状态恢复服务

等所有节点都处理之后，再观察一下情况。
如果有多个强制下线的节点的话，几个强制下线的节点 ID 都类似操作一下吧。

6

已经在kv上都执行过该命令，现在观察到有多个kv的日志出现如下类型的报错
[2021/04/26 03:19:55.359 +00:00] [ ERROR ] [split_observer.rs:149] [“failed to handle split req”] [err="“no valid key found for split.”"] [region_id=38384474]

[2021/04/26 03:20:14.444 +00:00] [ ERROR ] [split_observer.rs:149] [“failed to handle split req”] [err="“no valid key found for split.”"] [region_id=139556264]

[2021/04/26 03:20:18.388 +00:00] [ ERROR ] [split_observer.rs:149] [“failed to handle split req”] [err="“no valid key found for split.”"] [region_id=268570597]

[2021/04/26 03:20:22.263 +00:00] [ ERROR ] [split_observer.rs:149] [“failed to handle split req”] [err="“no valid key found for split.”"] [region_id=248054096]

[2021/04/26 03:20:22.659 +00:00] [ ERROR ] [split_observer.rs:149] [“failed to handle split req”] [err="“no valid key found for split.”"] [region_id=246935125]

[2021/04/26 03:20:42.791 +00:00] [ ERROR ] [split_observer.rs:149] [“failed to handle split req”] [err="“no valid key found for split.”"] [region_id=23753338]

[2021/04/26 03:20:47.518 +00:00] [ ERROR ] [split_observer.rs:149] [“failed to handle split req”] [err="“no valid key found for split.”"] [region_id=246824333]

7
  1. region is unavaible 的现象是否还存在？
  2. 下线的 12 节点上的 region 数量是否有减少？
8

1 region is unavaible现象依然存在

WX20210426-113148%402x
WX20210426-113148%402x702×58 20.2 KB

2、下线节点的region数量没有减少
WX20210426-113108%402x
WX20210426-113108%402x1318×588 65.7 KB

9
  1. 我从提供的 tikv 日志中，搜索这些没有 leader 的 region 相关的信息，你那边方便在更早的日志中匹配下关键字看能匹配到么 region_id=xxx
  2. tx_info 这个表的数据，有办法重新导入一份吗？如果可以重新导入的话，可以考虑drop 这个表然后重新创建表导入数据
10

1、之前上传的已经是每台kv上的日志全部内容了，哪里有更早的日志？
2、tx_info这个表的数据条数有六亿左右，基本不可能重新导入，而且就算tx_info可以访问了，下线节点的问题依然没有解决

11

另外，现在无leader的节点一下子变得很多
region --jq ‘.regions[]|select(has(“leader”)|not)|{id: .id,peer_stores: [.peers[].store_id]}’

{“id”:206890,“peer_stores”:[24478148,38833310,268391998]}

{“id”:15833,“peer_stores”:[24590972,24478148,268391998]}

{“id”:183659,“peer_stores”:[38833310,24478148,24590972]}

{“id”:12462042,“peer_stores”:[38833310,24480822,24478148]}

{“id”:79889304,“peer_stores”:[38833310,24590972,24478148]}

{“id”:249220015,“peer_stores”:[24478148,38833310,24480822]}

{“id”:80660412,“peer_stores”:[24478148,24590972,38833310]}

{“id”:39024222,“peer_stores”:[24480822,268391998,24478148]}

{“id”:246785885,“peer_stores”:[24590972,24478148,38833310]}

{“id”:17114765,“peer_stores”:[24590972,38833310,24478148]}

{“id”:107876,“peer_stores”:[24480822,24590972,24478148]}

{“id”:3737636,“peer_stores”:[38833310,24478148,24590972]}

{“id”:246766286,“peer_stores”:[24478148,38833310,262397455]}

{“id”:1177349,“peer_stores”:[262397455,24480822,24478148]}

{“id”:248749993,“peer_stores”:[24480822,24478148,24590972]}

{“id”:346850,“peer_stores”:[24478148,24480822,38833310]}

{“id”:246833757,“peer_stores”:[24480822,24478148,24590972]}

{“id”:12252913,“peer_stores”:[24478148,24480822,24590972]}

{“id”:38410101,“peer_stores”:[24590972,24478148,24480822]}

{“id”:230342,“peer_stores”:[24480822,24478148,24590972]}

{“id”:194312065,“peer_stores”:[24480822,24478148,24590972]}

{“id”:248821712,“peer_stores”:[268391998,24478148,38833310]}

{“id”:8907858,“peer_stores”:[24478148,262397455,38833310]}

{“id”:23066431,“peer_stores”:[38833310,24478148,24590972]}

{“id”:24824080,“peer_stores”:[38833310,24478148,268391998]}

{“id”:246308434,“peer_stores”:[24478148,38833310,24590972]}

{“id”:708747,“peer_stores”:[38833310,24480822,24478148]}

{“id”:24055410,“peer_stores”:[38833310,268391998,24478148]}

{“id”:39157172,“peer_stores”:[38833310,24478148,268391998]}

{“id”:246175607,“peer_stores”:[24480822,24478148,24590972]}

{“id”:99036329,“peer_stores”:[38833310,24478148,268391998]}

{“id”:246942999,“peer_stores”:[24590972,38833310,24478148]}

{“id”:11954,“peer_stores”:[24480822,38833310,24478148]}

{“id”:247583739,“peer_stores”:[24480822,24590972,24478148]}

{“id”:180654,“peer_stores”:[24480822,24590972,24478148]}

{“id”:80583494,“peer_stores”:[24478148,24480822,38833310]}

{“id”:753010,“peer_stores”:[24478148,24590972,38833310]}

{“id”:9218019,“peer_stores”:[24480822,38833310,24478148]}

{“id”:80625472,“peer_stores”:[24590972,24478148,24480822]}

{“id”:80629095,“peer_stores”:[24480822,38833310,24478148]}

{“id”:247960358,“peer_stores”:[24480822,38833310,24478148]}

{“id”:80950076,“peer_stores”:[24478148,24590972,38833310]}

{“id”:16932248,“peer_stores”:[24480822,262397455,24478148]}

{“id”:80949513,“peer_stores”:[38833310,24480822,24478148]}

{“id”:38492909,“peer_stores”:[24590972,24480822,24478148]}

{“id”:14308477,“peer_stores”:[24478148,24590972,38833310]}

{“id”:248665923,“peer_stores”:[24478148,24590972,262397455]}

{“id”:105642,“peer_stores”:[24480822,24590972,24478148]}

{“id”:120826,“peer_stores”:[38833310,24478148,268391998]}

{“id”:248056446,“peer_stores”:[24480822,38833310,24478148]}

{“id”:7811784,“peer_stores”:[38833310,24590972,24478148]}

{“id”:80394548,“peer_stores”:[24590972,24478148,38833310]}

{“id”:195883,“peer_stores”:[24590972,24478148,262397455]}

{“id”:3905338,“peer_stores”:[24478148,24480822,38833310]}

{“id”:232087,“peer_stores”:[24478148,24590972,38833310]}

{“id”:109578,“peer_stores”:[24480822,38833310,24478148]}

{“id”:161094400,“peer_stores”:[38833310,24478148,24590972]}

{“id”:18524,“peer_stores”:[24478148,38833310,24590972]}

{“id”:246768595,“peer_stores”:[24478148,24590972,38833310]}

{“id”:12488996,“peer_stores”:[38833310,24480822,24478148]}

{“id”:13404764,“peer_stores”:[24590972,24478148,262397455]}

{“id”:222603,“peer_stores”:[24478148,24590972,24480822]}

{“id”:2262564,“peer_stores”:[24480822,24478148,24590972]}

{“id”:139511669,“peer_stores”:[38833310,24480822,24478148]}

{“id”:172315,“peer_stores”:[24478148,24480822,38833310]}

{“id”:2253536,“peer_stores”:[24480822,24478148,24590972]}

{“id”:24776405,“peer_stores”:[38833310,24478148,24590972]}

{“id”:19256538,“peer_stores”:[24478148,24590972,24480822]}

{“id”:136029,“peer_stores”:[24478148,38833310,268391998]}

{“id”:6729767,“peer_stores”:[24480822,38833310,24478148]}

{“id”:80946599,“peer_stores”:[24478148,24480822,38833310]}

{“id”:105816,“peer_stores”:[24478148,38833310,24590972]}

{“id”:220322,“peer_stores”:[24590972,24478148,24480822]}

{“id”:38763659,“peer_stores”:[24480822,38833310,24478148]}

{“id”:80308970,“peer_stores”:[24478148,38833310,262397455]}

{“id”:80618576,“peer_stores”:[24478148,24590972,38833310]}

{“id”:80946564,“peer_stores”:[24590972,38833310,24478148]}

{“id”:193960071,“peer_stores”:[24590972,24480822,24478148]}

{“id”:13266304,“peer_stores”:[24480822,24478148,38833310]}

{“id”:24080714,“peer_stores”:[24480822,38833310,24478148]}

{“id”:80633092,“peer_stores”:[24478148,38833310,262397455]}

{“id”:406705,“peer_stores”:[262397455,24478148,38833310]}

{“id”:121079,“peer_stores”:[24478148,38833310,268391998]}

{“id”:330822,“peer_stores”:[24590972,38833310,24478148]}

{“id”:245835966,“peer_stores”:[38833310,24478148,268391998]}

{“id”:80590669,“peer_stores”:[24478148,38833310,24480822]}

{“id”:80654171,“peer_stores”:[24480822,268391998,24478148]}

{“id”:2383838,“peer_stores”:[24590972,38833310,24478148]}

{“id”:12739418,“peer_stores”:[24478148,38833310,24590972]}

{“id”:263984469,“peer_stores”:[24480822,38833310,24478148]}

{“id”:269049065,“peer_stores”:[24590972,268391998,24478148]}

{“id”:12671141,“peer_stores”:[24590972,24478148,38833310]}

{“id”:12749204,“peer_stores”:[24590972,38833310,24478148]}

{“id”:16582164,“peer_stores”:[24480822,38833310,24478148]}

{“id”:24252559,“peer_stores”:[24478148,38833310,24590972]}

{“id”:12183,“peer_stores”:[38833310,24480822,24478148]}

{“id”:247573738,“peer_stores”:[24480822,24478148,262397455]}

{“id”:79898210,“peer_stores”:[24590972,24478148,38833310]}

{“id”:16010,“peer_stores”:[38833310,24480822,24478148]}

{“id”:80351326,“peer_stores”:[24590972,24480822,24478148]}

{“id”:12935776,“peer_stores”:[24590972,38833310,24478148]}

{“id”:24072068,“peer_stores”:[24478148,38833310,24480822]}

{“id”:3531831,“peer_stores”:[24480822,268391998,24478148]}

{“id”:39358441,“peer_stores”:[24478148,38833310,24480822]}

{“id”:20205872,“peer_stores”:[24480822,268391998,24478148]}

{“id”:80099984,“peer_stores”:[24480822,38833310,24478148]}

{“id”:248984775,“peer_stores”:[38833310,268391998,24478148]}

{“id”:37976070,“peer_stores”:[24478148,268391998,24480822]}

{“id”:7328082,“peer_stores”:[24480822,24478148,38833310]}

{“id”:246896034,“peer_stores”:[24590972,24478148,24480822]}

{“id”:12421375,“peer_stores”:[24590972,38833310,24478148]}

{“id”:16305253,“peer_stores”:[38833310,268391998,24478148]}

{“id”:80253573,“peer_stores”:[24478148,38833310,24590972]}

{“id”:80608389,“peer_stores”:[24478148,24590972,24480822]}

{“id”:194326366,“peer_stores”:[24480822,24590972,24478148]}

{“id”:3556896,“peer_stores”:[24478148,38833310,24480822]}

{“id”:246985281,“peer_stores”:[24478148,38833310,24480822]}

{“id”:9366795,“peer_stores”:[38833310,24480822,24478148]}

{“id”:18749964,“peer_stores”:[24480822,268391998,24478148]}

{“id”:39524357,“peer_stores”:[24590972,24480822,24478148]}

{“id”:247625833,“peer_stores”:[24478148,24590972,24480822]}

{“id”:9770519,“peer_stores”:[24478148,24480822,24590972]}

{“id”:406561,“peer_stores”:[24478148,38833310,262397455]}

{“id”:14946780,“peer_stores”:[24590972,38833310,24478148]}

{“id”:102642,“peer_stores”:[24480822,38833310,24478148]}

{“id”:10309144,“peer_stores”:[24478148,268391998,262397455]}

{“id”:13444127,“peer_stores”:[24480822,262397455,24478148]}

{“id”:2243358,“peer_stores”:[24478148,38833310,268391998]}

{“id”:6621007,“peer_stores”:[24590972,24478148,38833310]}

{“id”:255684,“peer_stores”:[24590972,24478148,24480822]}

{“id”:80479003,“peer_stores”:[24590972,24478148,24480822]}

{“id”:248906373,“peer_stores”:[24480822,24590972,24478148]}

{“id”:246010219,“peer_stores”:[24590972,38833310,24478148]}

{“id”:80561290,“peer_stores”:[24480822,38833310,24478148]}

{“id”:23076634,“peer_stores”:[24478148,38833310,268391998]}

{“id”:39219582,“peer_stores”:[24590972,24478148,38833310]}

{“id”:268816409,“peer_stores”:[24480822,24478148,38833310]}

{“id”:80583615,“peer_stores”:[24478148,24590972,24480822]}

{“id”:225581,“peer_stores”:[24478148,24590972,268391998]}

{“id”:249219358,“peer_stores”:[38833310,24478148,24590972]}

{“id”:139457671,“peer_stores”:[24478148,38833310,268391998]}

{“id”:246037407,“peer_stores”:[24478148,24590972,38833310]}

{“id”:113206,“peer_stores”:[24478148,24590972,38833310]}

{“id”:37977440,“peer_stores”:[24478148,38833310,268391998]}

{“id”:80393361,“peer_stores”:[24478148,24590972,268391998]}

{“id”:245267706,“peer_stores”:[24480822,38833310,24478148]}

{“id”:12803932,“peer_stores”:[24478148,24590972,38833310]}

{“id”:221094572,“peer_stores”:[38833310,24590972,24478148]}

{“id”:16063,“peer_stores”:[24478148,24480822,38833310]}

{“id”:8633315,“peer_stores”:[24480822,38833310,24478148]}

{“id”:410968,“peer_stores”:[24478148,24590972,38833310]}

{“id”:12896353,“peer_stores”:[24478148,24590972,38833310]}

{“id”:18697283,“peer_stores”:[24480822,262397455,24478148]}

{“id”:23108956,“peer_stores”:[24478148,38833310,24590972]}

{“id”:240075778,“peer_stores”:[24478148,24590972,268391998]}

{“id”:248202367,“peer_stores”:[24478148,38833310,24590972]}

{“id”:19018357,“peer_stores”:[268391998,24478148,24480822]}

{“id”:90536,“peer_stores”:[24590972,268391998,24478148]}

{“id”:188101,“peer_stores”:[24480822,24590972,24478148]}

{“id”:17453661,“peer_stores”:[24590972,262397455,24478148]}

{“id”:23747791,“peer_stores”:[24478148,24590972,38833310]}

{“id”:38859924,“peer_stores”:[24478148,262397455,24590972]}

{“id”:247964259,“peer_stores”:[38833310,24590972,24478148]}

{“id”:9161071,“peer_stores”:[24478148,24590972,24480822]}

{“id”:248379675,“peer_stores”:[24478148,262397455,24480822]}

{“id”:23162945,“peer_stores”:[24478148,262397455,268391998]}

{“id”:12985300,“peer_stores”:[24480822,24590972,24478148]}

{“id”:10704192,“peer_stores”:[24478148,38833310,262397455]}

{“id”:80949913,“peer_stores”:[24590972,24480822,24478148]}

{“id”:363734,“peer_stores”:[24480822,24478148,38833310]}

{“id”:6829397,“peer_stores”:[24478148,24480822,38833310]}

{“id”:2240322,“peer_stores”:[38833310,24478148,268391998]}

{“id”:80417854,“peer_stores”:[24590972,24480822,24478148]}

{“id”:18598632,“peer_stores”:[38833310,24590972,24478148]}

{“id”:17694092,“peer_stores”:[24480822,268391998,24478148]}

{“id”:13795935,“peer_stores”:[24590972,24480822,24478148]}

{“id”:247721216,“peer_stores”:[24480822,24478148,268391998]}

{“id”:6820590,“peer_stores”:[24590972,24480822,24478148]}

{“id”:4354968,“peer_stores”:[24480822,24590972,24478148]}

{“id”:105596,“peer_stores”:[24478148,24590972,268391998]}

{“id”:2256310,“peer_stores”:[38833310,24478148,268391998]}

{“id”:115817800,“peer_stores”:[24590972,38833310,24478148]}

{“id”:194902,“peer_stores”:[24480822,24590972,24478148]}

{“id”:246417888,“peer_stores”:[24480822,24478148,38833310]}

{“id”:9448628,“peer_stores”:[24480822,24478148,24590972]}

{“id”:410600,“peer_stores”:[38833310,24480822,24478148]}

{“id”:249273971,“peer_stores”:[24590972,24478148,38833310]}

{“id”:297580,“peer_stores”:[24480822,24478148,38833310]}

{“id”:22724067,“peer_stores”:[24590972,24480822,24478148]}

{“id”:80651567,“peer_stores”:[24480822,268391998,24478148]}

{“id”:247363971,“peer_stores”:[24478148,38833310,24480822]}

{“id”:8067757,“peer_stores”:[24480822,24478148,262397455]}

{“id”:12199827,“peer_stores”:[38833310,24590972,24478148]}

{“id”:246604860,“peer_stores”:[24590972,38833310,24478148]}

{“id”:18103696,“peer_stores”:[38833310,24480822,24478148]}

{“id”:809638,“peer_stores”:[24478148,24480822,38833310]}

{“id”:217883,“peer_stores”:[24590972,24478148,24480822]}

{“id”:80895658,“peer_stores”:[24478148,38833310,24480822]}

{“id”:80648927,“peer_stores”:[24590972,24480822,24478148]}

{“id”:248109037,“peer_stores”:[38833310,24590972,24478148]}

{“id”:512986,“peer_stores”:[24590972,24480822,24478148]}

{“id”:80405241,“peer_stores”:[24478148,24480822,38833310]}

{“id”:9419288,“peer_stores”:[24480822,24478148,24590972]}

{“id”:12483538,“peer_stores”:[24590972,38833310,24478148]}

{“id”:262600284,“peer_stores”:[262397455,24480822,24478148]}

{“id”:2092495,“peer_stores”:[24478148,24480822,268391998]}

{“id”:80682233,“peer_stores”:[38833310,24590972,24478148]}

{“id”:2589259,“peer_stores”:[24480822,24478148,38833310]}

{“id”:16178482,“peer_stores”:[24480822,262397455,24478148]}

{“id”:80447410,“peer_stores”:[24590972,24480822,24478148]}

{“id”:571468,“peer_stores”:[24478148,24480822,38833310]}

{“id”:3688586,“peer_stores”:[24478148,24590972,24480822]}

{“id”:8639349,“peer_stores”:[24478148,38833310,268391998]}

{“id”:268580268,“peer_stores”:[24478148,24590972,262397455]}

{“id”:20271031,“peer_stores”:[24480822,24478148,262397455]}

{“id”:247991912,“peer_stores”:[24478148,24590972,24480822]}

{“id”:297686,“peer_stores”:[24478148,24480822,24590972]}

{“id”:9165849,“peer_stores”:[24480822,24478148,38833310]}

{“id”:80305046,“peer_stores”:[24480822,24590972,24478148]}

{“id”:23571346,“peer_stores”:[38833310,24478148,24590972]}

{“id”:80256045,“peer_stores”:[24478148,24480822,24590972]}

{“id”:247957127,“peer_stores”:[24480822,38833310,24478148]}

{“id”:9371708,“peer_stores”:[24478148,24480822,268391998]}

{“id”:1240735,“peer_stores”:[24478148,24480822,38833310]}

{“id”:1353363,“peer_stores”:[24480822,38833310,24478148]}

{“id”:409527,“peer_stores”:[24590972,24480822,24478148]}

{“id”:202149989,“peer_stores”:[24480822,24590972,24478148]}

{“id”:24097374,“peer_stores”:[38833310,24590972,24478148]}

{“id”:80948783,“peer_stores”:[24478148,268391998,24480822]}

{“id”:12488481,“peer_stores”:[38833310,24478148,24590972]}

{“id”:1244923,“peer_stores”:[24478148,24590972,24480822]}

{“id”:199470,“peer_stores”:[24478148,24480822,38833310]}

{“id”:80355387,“peer_stores”:[24478148,24590972,24480822]}

{“id”:111448,“peer_stores”:[24480822,38833310,24478148]}

{“id”:6649118,“peer_stores”:[24480822,24478148,24590972]}

{“id”:406078,“peer_stores”:[24480822,38833310,24478148]}

{“id”:247420464,“peer_stores”:[24590972,24480822,24478148]}

{“id”:105951,“peer_stores”:[24478148,38833310,268391998]}

{“id”:80565700,“peer_stores”:[38833310,24480822,24478148]}

{“id”:12930884,“peer_stores”:[24590972,38833310,24478148]}

{“id”:38216675,“peer_stores”:[24590972,24478148,24480822]}

{“id”:20147034,“peer_stores”:[24478148,24590972,38833310]}

{“id”:249035586,“peer_stores”:[24480822,24590972,24478148]}

{“id”:6628839,“peer_stores”:[24590972,24478148,24480822]}

{“id”:8132465,“peer_stores”:[24480822,24478148,24590972]}

{“id”:12751730,“peer_stores”:[38833310,24590972,24478148]}

{“id”:18003482,“peer_stores”:[24478148,38833310,24590972]}

{“id”:7859937,“peer_stores”:[38833310,268391998,24478148]}

{“id”:12980525,“peer_stores”:[38833310,24480822,24478148]}

{“id”:3145008,“peer_stores”:[24478148,24480822,38833310]}

{“id”:24430408,“peer_stores”:[24478148,38833310,24480822]}

{“id”:18270188,“peer_stores”:[24478148,24590972,38833310]}

{“id”:23886563,“peer_stores”:[24478148,262397455,24590972]}

{“id”:576277,“peer_stores”:[24590972,24478148,268391998]}

{“id”:194315181,“peer_stores”:[24480822,268391998,24478148]}

{“id”:5594275,“peer_stores”:[24480822,38833310,24478148]}

{“id”:80259549,“peer_stores”:[24480822,24590972,24478148]}

{“id”:249017950,“peer_stores”:[24478148,38833310,262397455]}

{“id”:12681578,“peer_stores”:[38833310,24590972,24478148]}

{“id”:39037628,“peer_stores”:[24478148,24480822,24590972]}

{“id”:247967275,“peer_stores”:[24480822,24478148,24590972]}

{“id”:7575838,“peer_stores”:[24480822,38833310,24478148]}

{“id”:9783474,“peer_stores”:[24478148,262397455,24480822]}

{“id”:248299076,“peer_stores”:[24478148,24590972,38833310]}

{“id”:9421434,“peer_stores”:[24480822,38833310,24478148]}

{“id”:8561396,“peer_stores”:[24478148,24590972,24480822]}

{“id”:262526815,“peer_stores”:[24590972,24478148,38833310]}

{“id”:10726291,“peer_stores”:[24480822,24590972,24478148]}

{“id”:17748589,“peer_stores”:[38833310,24478148,268391998]}

{“id”:80289364,“peer_stores”:[24478148,24590972,24480822]}

{“id”:14651648,“peer_stores”:[38833310,24480822,24478148]}

{“id”:3781213,“peer_stores”:[24590972,38833310,24478148]}

{“id”:23081887,“peer_stores”:[24480822,24590972,24478148]}

{“id”:14103368,“peer_stores”:[38833310,24478148,268391998]}

{“id”:80473718,“peer_stores”:[24590972,24478148,38833310]}

{“id”:18661586,“peer_stores”:[24478148,24590972,38833310]}

{“id”:247863527,“peer_stores”:[24590972,268391998,24478148]}

{“id”:247556465,“peer_stores”:[24478148,24480822,262397455]}

{“id”:80505821,“peer_stores”:[38833310,24480822,24478148]}

{“id”:247640796,“peer_stores”:[24480822,24478148,24590972]}

{“id”:18535990,“peer_stores”:[24480822,262397455,24478148]}

{“id”:80492774,“peer_stores”:[24478148,268391998,24480822]}

{“id”:248609993,“peer_stores”:[24478148,38833310,24590972]}

{“id”:256681,“peer_stores”:[24478148,24480822,38833310]}

{“id”:3225689,“peer_stores”:[24480822,38833310,24478148]}

{“id”:10036638,“peer_stores”:[38833310,24480822,24478148]}

{“id”:246122095,“peer_stores”:[24478148,38833310,24590972]}

{“id”:104604,“peer_stores”:[24480822,24590972,24478148]}

{“id”:6792458,“peer_stores”:[38833310,24478148,268391998]}

{“id”:248817204,“peer_stores”:[24480822,24590972,24478148]}

{“id”:80408898,“peer_stores”:[24478148,24590972,38833310]}

{“id”:80611635,“peer_stores”:[24478148,38833310,24480822]}

{“id”:40633,“peer_stores”:[24480822,24478148,24590972]}

12
  1. tikv 的日志会被归档到 tikv 的 log 目录下，会以 tikv.log.${时间} 的文件名形式存在
  2. 无 leader 的 region 变多，这个期间有做过什么操作么
13

1、尝试匹配了一下region=xx，结果大致如图

WX20210426-150420%402x
WX20210426-150420%402x2808×894 1.33 MB

2、唯一的操作就是把节点stop，执行unsafe recover后重启，不过现在缺leader的region又变少了
» region --jq ‘.regions[]|select(has(“leader”)|not)|{id: .id,peer_stores: [.peers[].store_id]}’

{“id”:248584962,“peer_stores”:[24590972,24478148,38833310]}

{“id”:247022182,“peer_stores”:[38833310,24478148,24590972]}

{“id”:2579214,“peer_stores”:[24478148,38833310,256634687]}

{“id”:3434,“peer_stores”:[24590972,24478148,268391998]}

{“id”:9520702,“peer_stores”:[38833310,24478148,256634687]}

{“id”:139439976,“peer_stores”:[24478148,38833310,256634687]}

{“id”:80365619,“peer_stores”:[24478148,24590972,38833310,24480822]}

{“id”:24215180,“peer_stores”:[24478148,38833310,256634687]}

{“id”:39077233,“peer_stores”:[24480822,24590972,268391998]}

{“id”:24271913,“peer_stores”:[24478148,24590972,24480822]}

{“id”:80401278,“peer_stores”:[38833310,24478148,256634687]}

{“id”:9427705,“peer_stores”:[38833310,24478148,256634687]}

{“id”:245999700,“peer_stores”:[38833310,24480822,24590972]}

{“id”:247419092,“peer_stores”:[38833310,24480822,262397455]}

{“id”:12563787,“peer_stores”:[24478148,24590972,268391998]}

{“id”:261757919,“peer_stores”:[24480822,38833310]}

{“id”:338610,“peer_stores”:[38833310,24478148,256634687]}

{“id”:245913901,“peer_stores”:[24478148,24590972,24480822]}

{“id”:113879901,“peer_stores”:[24590972,38833310,24478148]}

{“id”:23757219,“peer_stores”:[24590972,38833310,268391998]}

{“id”:247020627,“peer_stores”:[24480822,38833310,268391998]}

{“id”:246838547,“peer_stores”:[24478148,38833310,256634687]}

{“id”:234017108,“peer_stores”:[38833310,24480822,24478148]}

{“id”:23073113,“peer_stores”:[38833310,24478148,256634687]}

{“id”:9413063,“peer_stores”:[24478148,38833310,256634687]}

{“id”:9588953,“peer_stores”:[24478148,38833310,256634687]}

{“id”:248156003,“peer_stores”:[24590972,24478148,268391998,24480822]}

{“id”:80335931,“peer_stores”:[38833310,24478148,256634687]}

{“id”:248266559,“peer_stores”:[24478148,38833310,256634687]}

{“id”:9506276,“peer_stores”:[38833310,24478148,256634687]}

{“id”:362351,“peer_stores”:[24478148,38833310,256634687]}

{“id”:6778802,“peer_stores”:[24478148,38833310,256634687]}

{“id”:246923725,“peer_stores”:[38833310,24478148,256634687]}

{“id”:248445984,“peer_stores”:[38833310,24590972,24478148]}

{“id”:23071258,“peer_stores”:[24590972,24478148,268391998]}

{“id”:246797073,“peer_stores”:[24478148,38833310,256634687]}

{“id”:80245963,“peer_stores”:[24590972,268391998,262397455]}

{“id”:256965,“peer_stores”:[24478148,38833310,256634687]}

{“id”:195867836,“peer_stores”:[38833310,24478148,256634687]}

{“id”:244288376,“peer_stores”:[38833310,24478148,256634687]}

{“id”:23066631,“peer_stores”:[38833310,24478148,256634687]}

{“id”:80406817,“peer_stores”:[24590972,24478148,268391998]}

{“id”:80480758,“peer_stores”:[24478148,24590972,268391998]}

{“id”:248677302,“peer_stores”:[38833310,24590972,24480822]}

{“id”:245735969,“peer_stores”:[24590972,24478148,24480822]}

{“id”:332451,“peer_stores”:[38833310,24478148,256634687]}

{“id”:14324204,“peer_stores”:[38833310,24478148,256634687]}

{“id”:38453328,“peer_stores”:[38833310,24478148,256634687]}

»

14

region_id=38384474 这个并不是没有 leader 的 region ，挑几个没有 leader 的 region id 过滤看看有没有信息。

因为我这边的日志匹配不到，所以不明确这些 region 之前发生过什么

15

那麻烦您看一下我2楼贴的那一次的没有leader的region，这些region都是12节点上的，它们到现在为止依然是没有leader的，我这里挑了前三个region id执行了一下结果如图
» region 14324204
{
“id”: 14324204,
“start_key”: “7480000000000001FF745F72800000001DFF507B160000000000FA”,
“end_key”: “7480000000000001FF745F72800000001DFF50E0350000000000FA”,
“epoch”: {
“conf_ver”: 1904,
“version”: 13576
},
“peers”: [
{
“id”: 248027420,
“store_id”: 38833310
},
{
“id”: 248892035,
“store_id”: 24478148
},
{
“id”: 261893968,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
}

» region 23073113
{
“id”: 23073113,
“start_key”: “7480000000000001FF745F728000000020FF8B4A140000000000FA”,
“end_key”: “7480000000000001FF745F728000000020FF8B86360000000000FA”,
“epoch”: {
“conf_ver”: 2029,
“version”: 15863
},
“peers”: [
{
“id”: 247260250,
“store_id”: 38833310
},
{
“id”: 249108661,
“store_id”: 24478148
},
{
“id”: 262088650,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
}

» region 80335931
{
“id”: 80335931,
“start_key”: “7480000000000001FF745F728000000022FF8A4F660000000000FA”,
“end_key”: “7480000000000001FF745F728000000022FF8A55020000000000FA”,
“epoch”: {
“conf_ver”: 12018,
“version”: 16391
},
“peers”: [
{
“id”: 194207454,
“store_id”: 38833310
},
{
“id”: 247374608,
“store_id”: 24478148
},
{
“id”: 261819044,
“store_id”: 256634687
}
],
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 0,
“approximate_keys”: 0
}

16

这里面的 region id 我都过滤了
egrep ‘256965|6778802|80401278|9427705|9588953|14324204|9520702|9506276|23066631|23073113|246923725|24215180|80335931|38453328|139439976|195867836|246797073|246838547|248266559|332451|2579214|338610|362351’ ./*

17

那请问还需要我找什么信息吗

18

你在你那边的 tikv.log 中过滤下这几个 region id 关键字看能过滤到信息么

提供的日志这边最早是到 4-22 号的日志，再之前没有

19

如果在22号之前的也要的话，一个节点就有好多，7个kv的话实在是太多了，您如果确实需要的话，向日葵远程控制协助可以吗

20

你可以在你的环境那边 grep 过滤一下，先看能不能过滤到信息，如果过滤不到信息，那也没用。
如果能过滤到，那将过滤的结果重定向一下，将相关信息拿一下就行
grep ‘region_id=xxx’ ./tikv.log* > region_xxx.log 这样