tidb 集群无法启动

🪐 TiDB 技术问题
21

一样的store id，因为有很多个，我是通过这个找的，/home/tidb/.tiup/components/ctl/v5.2.2/pd-ctl -u http://172.16.120.124:2379 store check offline |grep id

22

前面是15698413 后面就成了34673997 ，能贴个tiup display完整输出吗

23

image
image1373×760 181 KB
刚才有做了一遍，但是看到tidb起来了，但是原本正常的tikv有几个没有起来

24

以前正常的tikv现在起不来，报这样的错误

image
image1367×403 132 KB

image 也无法查询

image
image1073×536 89.2 KB

25

是只处理了这一个store吧，之前有缩容了几个？

26

不是，处理了很多个。34673997

15698413
15698410

15698412 因为这些都是无效的节点上面的。

27

现在正常的无法启动，我可以通过缩容，将他们缩掉吗，缩容是不是他们自己的region就会偏移到其他还有的节点上面,还是说，我还要通过刚才的方式，继续偏移

28

你最初缩容怎么做的？ 你看前面链接文章里找下 查找3个副本都没有的 脚本，然后检查下看看有多少全丢失的region,这些得重建里面数据肯定没了

29

缺失一点数据没有关系，现在我是tikv有6个挂了，导致tidb无法查询数据，无法使用。已经不希望找回所有的数据了。缩容是IP是172.16.120.9和 172.16.120.10 ，我同事是正常缩容，然后offline，很长时间，因为数据量大，所以他又停掉了，然后加了force强制缩容，后面就出现整个集群卡住了，无法使用，然后他重启所有集群，就起不来了

30

看你现在的操作不止是 .9 .10上做了缩容操作 .123上也做了， 在同一主机上的tikv应设置相同的label 保障同一主机上不会有相同region的副本，避免出现多副本失败的情况，你现在一下弄了这么多store 肯定有很多全副本丢失了， 多副本恢复的步骤是在 所有正常的tikv上去unsafe recover 下线的store，你先 按下面
故障节点为store id 1，5，6）： pd-ctl region --jq=".regions\[] | {id: .id, peer\_stores: \[.peers\[].store\_id] | select(length as $total | map(if .==(1,5,6) then . else empty end) | length>=3) }"
检查下你下线的store上的region 都丢的

31

我是根据这个找到我们所有挂掉的store id。/data/tools/tidb-community-server-v5.2.2-linux-amd64/pd-ctl region --jq=’.regions[] | {id: .id, peer_stores: [.peers[].store_id] |
select(length as $total |map(if .==(17625573,15698410,34673997,33515118,17625571,17625570,1,7,15698413,17625576,34784748,15698411,15698412,17625568) then . else empty end)|length>=$total-length)}’

你说的我要根据1 5 6 去找吗

32

store 列表里，你先找那些你下线的 、store delete 操作的节点，之前这些不是都unsafe recover了

33

是的，我在查找。image ，这些都是失败的

34

down的不算，就你做过缩容、store delete 、unsafe recover的，你这些offline 还有没有数据目录没被删除的 ，哟的话 改为up状态停止缩容： curl -X POST http://:/pd/api/v1/store/<store_id>/state?state=Up

35

down 不需要是吗，offline 的机器上面已经没有目录了。都是空的。已经被缩容完。需要更改为up吗

36

down是因为暂时起不来，先不要动呢。 offline没正常缩容完你就删了目录的话，up也不好使了

37

是的，早就没有了，那直接通过pd-ctl找出来，unsafe recover 这样可以吗

38

之前下线的store 你不是已经做过unsafe recover了吗，现在是把那些在3个副本都在offline store里的region，然后把这些region recreate 后看看能否把down的tikv拉起来

39

region recreate 有具体的操作步骤吗。
下面是我根据offline的机器，找到的region

---------±---------------------±-------------±-------------±-----------------+
| STORE_ID | ADDRESS | LEADER_COUNT | REGION_COUNT | STORE_STATE_NAME |
±---------±---------------------±-------------±-------------±-----------------+
| 15698410 | 172.16.120.10:20161 | 0 | 15285 | Offline |
| 34673997 | 172.16.120.123:20162 | 0 | 644 | Offline |
| 1 | 172.16.120.123:20160 | 0 | 20594 | Offline |
| 7 | 172.16.120.123:20161 | 0 | 24350 | Offline |
| 15698413 | 172.16.120.9:20160 | 0 | 26818 | Offline |
| 15698411 | 172.16.120.10:20160 | 0 | 39000 | Offline |
| 15698412 | 172.16.120.9:20161 | 0 | 25010 | Offline |
| 17625571 | 172.16.120.123:20260 | 0 | 5003 | Offline |
±---------±---------------------±-------------±-------------±-----------------+

/data/tools/tidb-community-server-v5.2.2-linux-amd64/pd-ctl region --jq=’.regions[] | {id: .id, peer_stores: [.peers[].store_id] |
select(length as $total |map(if .==(15698410,34673997,1,7,15698413,34784748,15698411,15698412,17625571,33515118) then . else empty end)|length>=5)}’

{“id”:509183,“peer_stores”:[1,15698410,15698412,17625570,17625571,17625576,17625573,17625568,33515118,34673997,34784748]}
{“id”:15342317,“peer_stores”:[1,15698413,15698411,17625571,17625568,17625573,17625576,17625570,33515118,34673997,34784748]}
{“id”:14302574,“peer_stores”:[1,15698411,15698412,17625571,17625576,17625573,17625568,17625570,33515118,34673997,34784748]}
{“id”:535926,“peer_stores”:[1,15698412,15698410,17625573,17625570,17625571,17625568,17625576,33515118,34673997,34923434]}
{“id”:13119726,“peer_stores”:[1,15698412,15698410,17625570,17625571,17625573,17625576,17625568,33515118,34673997,34784748]}
{“id”:1308559,“peer_stores”:[1,15698413,15698410,17625570,17625576,17625571,17625568,17625573,33515118,34673997,34784748]}
{“id”:677725,“peer_stores”:[1,15698410,15698413,17625571,17625573,17625568,17625570,17625576,33515118,34673997,34784748]}
{“id”:1624842,“peer_stores”:[1,15698412,15698410,17625570,17625571,17625568,17625576,17625573,33515118,34673997,34784748]}
{“id”:16059655,“peer_stores”:[1,15698410,15698412,17625573,17625568,17625571,17625576,17625570,33515118,34673997,34784748]}
{“id”:1056199,“peer_stores”:[1,15698413,15698411,17625570,17625571,17625573,17625568,17625576,33515118,34673997,34923434]}
{“id”:1118286,“peer_stores”:[1,15698410,15698412,17625571,17625573,17625570,17625568,17625576,33515118,34673997,34784748]}
{“id”:698627,“peer_stores”:[1,15698411,15698413,17625571,17625573,17625568,17625576,17625570,33515118,34673997]}
{“id”:7692845,“peer_stores”:[1,15698413,15698411,17625573,17625571,17625570,17625568,17625576,33515118,34673997,34784748]}
{“id”:1117939,“peer_stores”:[1,15698411,15698412,17625568,17625573,17625571,17625570,17625576,33515118,34673997,34784748]}
{“id”:1295236,“peer_stores”:[1,15698412,15698410,17625576,17625571,17625573,17625568,17625570,33515118,34673997,34784748,35473206]}
{“id”:15756791,“peer_stores”:[1,15698412,15698410,17625571,17625576,17625568,17625570,17625573,33515118,34673997,34784748]}
{“id”:5205590,“peer_stores”:[1,15698412,15698411,17625571,17625573,17625568,17625570,17625576,33515118,34673997,34923434]}
{“id”:1296909,“peer_stores”:[1,15698410,15698413,17625571,17625573,17625570,17625568,17625576,33515118,34673997,34784748]}
{“id”:1095599,“peer_stores”:[1,15698410,15698412,17625571,17625573,17625570,17625576,17625568,33515118,34673997,34784748]}
{“id”:1272283,“peer_stores”:[1,15698411,15698413,17625570,17625571,17625573,17625568,17625576,33515118,34673997,34784748]}
{“id”:1065135,“peer_stores”:[1,15698412,15698410,17625570,17625576,17625571,17625568,17625573,33515118,34673997,34923434]}
{“id”:13513271,“peer_stores”:[1,15698412,15698410,17625571,17625576,17625568,17625573,17625570,33515118,34673997]}
{“id”:1713194,“peer_stores”:[1,15698410,15698413,17625571,17625576,17625573,17625568,17625570,33515118,34673997,34784748]}
{“id”:13487106,“peer_stores”:[1,15698411,15698413,17625573,17625571,17625570,17625568,17625576,33515118,34673997,34923434]}
{“id”:6079993,“peer_stores”:[1,15698410,15698412,17625571,17625568,17625576,17625573,17625570,33515118,34673997,34784748]}
{“id”:8864482,“peer_stores”:[1,15698412,15698411,17625568,17625571,17625573,17625576,17625570,33515118,34673997,34923434]}
{“id”:8036158,“peer_stores”:[1,15698413,15698410,17625568,17625571,17625570,17625576,17625573,33515118,34673997]}
{“id”:6757242,“peer_stores”:[1,15698413,15698410,17625571,17625570,17625573,17625576,17625568,33515118,34673997,34923434]}
{“id”:15832113,“peer_stores”:[1,15698412,15698411,17625576,17625571,17625570,17625568,17625573,33515118,34673997]}
{“id”:5317880,“peer_stores”:[1,15698412,15698411,17625573,17625571,17625568,17625576,17625570,33515118,34673997]}
{“id”:13236352,“peer_stores”:[1,15698411,15698412,17625576,17625571,17625573,17625568,17625570,33515118,34673997,34784748]}
{“id”:14342325,“peer_stores”:[1,15698411,15698413,17625573,17625571,17625570,17625576,17625568,33515118,34673997,34784748]}
{“id”:737372,“peer_stores”:[1,15698412,15698411,17625570,17625571,17625568,17625576,17625573,33515118,34673997,34784748]}
{“id”:1331604,“peer_stores”:[1,15698412,15698410,17625568,17625573,17625571,17625576,17625570,33515118,34673997,34784748]}
{“id”:444647,“peer_stores”:[1,15698412,15698411,17625570,17625571,17625568,17625573,17625576,33515118,34673997,34923434]}
{“id”:1186646,“peer_stores”:[1,15698411,15698413,17625573,17625571,17625576,17625570,17625568,33515118,34673997,34923434]}
{“id”:7097340,“peer_stores”:[1,15698412,15698410,17625573,17625571,17625570,17625576,17625568,33515118,34673997,34923434]}
{“id”:5473573,“peer_stores”:[1,15698412,15698410,17625571,17625576,17625570,17625568,17625573,33515118,34673997,34923434]}
{“id”:1145774,“peer_stores”:[1,15698410,15698413,17625571,17625570,17625576,17625573,17625568,33515118,34673997]}
{“id”:1215657,“peer_stores”:[1,15698412,15698410,17625573,17625570,17625571,17625576,17625568,33515118,34673997,34923434]}
{“id”:5478475,“peer_stores”:[1,15698411,15698412,17625571,17625570,17625573,17625576,17625568,33515118,34673997,34784748]}
{“id”:4969036,“peer_stores”:[1,15698412,15698411,17625576,17625571,17625573,17625570,17625568,33515118,34673997,34923434]}
{“id”:1448991,“peer_stores”:[1,15698412,15698411,17625576,17625571,17625573,17625570,17625568,33515118,34673997,34784748]}
{“id”:1182107,“peer_stores”:[1,15698410,15698412,17625573,17625568,17625571,17625570,17625576,33515118,34673997,34784748]}
{“id”:5017113,“peer_stores”:[1,15698412,15698411,17625571,17625573,17625570,17625568,17625576,33515118,34673997,34923434]}
{“id”:1347658,“peer_stores”:[1,15698410,15698412,17625568,17625571,17625573,17625576,17625570,33515118,34673997,34923434]}
{“id”:1053645,“peer_stores”:[1,15698413,15698410,17625576,17625571,17625568,17625573,17625570,33515118,34673997,34923434]}
{“id”:1012259,“peer_stores”:[1,15698411,15698412,17625571,17625573,17625570,17625568,17625576,33515118,34673997,34923434]}
{“id”:5019333,“peer_stores”:[1,15698413,15698410,17625571,17625576,17625568,17625570,17625573,33515118,34673997,34923434]}

40

你是配置了5副本吗，把脚本输出结果 找几个region_id 在确认下，pd-ctl region xxxx 看下输出的store_id是不是全是之前处理的那些，是的话就可以 recreate , 参考下面 ：