tidb 集群无法启动

你最初缩容怎么做的? 你看前面链接文章里找下 查找3个副本都没有的 脚本,然后检查下看看有多少全丢失的region,这些得重建里面数据肯定没了

缺失一点数据没有关系,现在我是tikv有6个挂了,导致tidb无法查询数据,无法使用。已经不希望找回所有的数据了。缩容是IP是172.16.120.9和 172.16.120.10 ,我同事是正常缩容,然后offline,很长时间,因为数据量大,所以他又停掉了,然后加了force强制缩容,后面就出现整个集群卡住了,无法使用,然后他重启所有集群,就起不来了

看你现在的操作不止是 .9 .10上做了缩容操作 .123上也做了, 在同一主机上的tikv应设置相同的label 保障同一主机上不会有相同region的副本,避免出现多副本失败的情况,你现在一下弄了这么多store 肯定有很多全副本丢失了, 多副本恢复的步骤是在 所有正常的tikv上去unsafe recover 下线的store,你先 按下面
故障节点为store id 1,5,6): pd-ctl region --jq=".regions\[] | {id: .id, peer\_stores: \[.peers\[].store\_id] | select(length as $total | map(if .==(1,5,6) then . else empty end) | length>=3) }"
检查下你下线的store上的region 都丢的

我是根据这个找到我们所有挂掉的store id。/data/tools/tidb-community-server-v5.2.2-linux-amd64/pd-ctl region --jq=’.regions[] | {id: .id, peer_stores: [.peers[].store_id] |
select(length as $total |map(if .==(17625573,15698410,34673997,33515118,17625571,17625570,1,7,15698413,17625576,34784748,15698411,15698412,17625568) then . else empty end)|length>=$total-length)}’

你说的我要根据1 5 6 去找吗

store 列表里,你先找那些你下线的 、store delete 操作的节点,之前这些不是都unsafe recover了

是的,我在查找。image ,这些都是失败的

down的不算,就你做过缩容、store delete 、unsafe recover的,你这些offline 还有没有数据目录没被删除的 ,哟的话 改为up状态停止缩容: curl -X POST http://:/pd/api/v1/store/<store_id>/state?state=Up

down 不需要是吗,offline 的机器上面已经没有目录了。都是空的。已经被缩容完。需要更改为up吗

down是因为暂时起不来,先不要动呢。 offline没正常缩容完你就删了目录的话,up也不好使了

是的,早就没有了,那直接通过pd-ctl找出来,unsafe recover 这样可以吗

之前下线的store 你不是已经做过unsafe recover了吗,现在是把那些在3个副本都在offline store里的region,然后把这些region recreate 后看看能否把down的tikv拉起来

region recreate 有具体的操作步骤吗。
下面是我根据offline的机器,找到的region

---------±---------------------±-------------±-------------±-----------------+
| STORE_ID | ADDRESS | LEADER_COUNT | REGION_COUNT | STORE_STATE_NAME |
±---------±---------------------±-------------±-------------±-----------------+
| 15698410 | 172.16.120.10:20161 | 0 | 15285 | Offline |
| 34673997 | 172.16.120.123:20162 | 0 | 644 | Offline |
| 1 | 172.16.120.123:20160 | 0 | 20594 | Offline |
| 7 | 172.16.120.123:20161 | 0 | 24350 | Offline |
| 15698413 | 172.16.120.9:20160 | 0 | 26818 | Offline |
| 15698411 | 172.16.120.10:20160 | 0 | 39000 | Offline |
| 15698412 | 172.16.120.9:20161 | 0 | 25010 | Offline |
| 17625571 | 172.16.120.123:20260 | 0 | 5003 | Offline |
±---------±---------------------±-------------±-------------±-----------------+

/data/tools/tidb-community-server-v5.2.2-linux-amd64/pd-ctl region --jq=’.regions[] | {id: .id, peer_stores: [.peers[].store_id] |
select(length as $total |map(if .==(15698410,34673997,1,7,15698413,34784748,15698411,15698412,17625571,33515118) then . else empty end)|length>=5)}’

{“id”:509183,“peer_stores”:[1,15698410,15698412,17625570,17625571,17625576,17625573,17625568,33515118,34673997,34784748]}
{“id”:15342317,“peer_stores”:[1,15698413,15698411,17625571,17625568,17625573,17625576,17625570,33515118,34673997,34784748]}
{“id”:14302574,“peer_stores”:[1,15698411,15698412,17625571,17625576,17625573,17625568,17625570,33515118,34673997,34784748]}
{“id”:535926,“peer_stores”:[1,15698412,15698410,17625573,17625570,17625571,17625568,17625576,33515118,34673997,34923434]}
{“id”:13119726,“peer_stores”:[1,15698412,15698410,17625570,17625571,17625573,17625576,17625568,33515118,34673997,34784748]}
{“id”:1308559,“peer_stores”:[1,15698413,15698410,17625570,17625576,17625571,17625568,17625573,33515118,34673997,34784748]}
{“id”:677725,“peer_stores”:[1,15698410,15698413,17625571,17625573,17625568,17625570,17625576,33515118,34673997,34784748]}
{“id”:1624842,“peer_stores”:[1,15698412,15698410,17625570,17625571,17625568,17625576,17625573,33515118,34673997,34784748]}
{“id”:16059655,“peer_stores”:[1,15698410,15698412,17625573,17625568,17625571,17625576,17625570,33515118,34673997,34784748]}
{“id”:1056199,“peer_stores”:[1,15698413,15698411,17625570,17625571,17625573,17625568,17625576,33515118,34673997,34923434]}
{“id”:1118286,“peer_stores”:[1,15698410,15698412,17625571,17625573,17625570,17625568,17625576,33515118,34673997,34784748]}
{“id”:698627,“peer_stores”:[1,15698411,15698413,17625571,17625573,17625568,17625576,17625570,33515118,34673997]}
{“id”:7692845,“peer_stores”:[1,15698413,15698411,17625573,17625571,17625570,17625568,17625576,33515118,34673997,34784748]}
{“id”:1117939,“peer_stores”:[1,15698411,15698412,17625568,17625573,17625571,17625570,17625576,33515118,34673997,34784748]}
{“id”:1295236,“peer_stores”:[1,15698412,15698410,17625576,17625571,17625573,17625568,17625570,33515118,34673997,34784748,35473206]}
{“id”:15756791,“peer_stores”:[1,15698412,15698410,17625571,17625576,17625568,17625570,17625573,33515118,34673997,34784748]}
{“id”:5205590,“peer_stores”:[1,15698412,15698411,17625571,17625573,17625568,17625570,17625576,33515118,34673997,34923434]}
{“id”:1296909,“peer_stores”:[1,15698410,15698413,17625571,17625573,17625570,17625568,17625576,33515118,34673997,34784748]}
{“id”:1095599,“peer_stores”:[1,15698410,15698412,17625571,17625573,17625570,17625576,17625568,33515118,34673997,34784748]}
{“id”:1272283,“peer_stores”:[1,15698411,15698413,17625570,17625571,17625573,17625568,17625576,33515118,34673997,34784748]}
{“id”:1065135,“peer_stores”:[1,15698412,15698410,17625570,17625576,17625571,17625568,17625573,33515118,34673997,34923434]}
{“id”:13513271,“peer_stores”:[1,15698412,15698410,17625571,17625576,17625568,17625573,17625570,33515118,34673997]}
{“id”:1713194,“peer_stores”:[1,15698410,15698413,17625571,17625576,17625573,17625568,17625570,33515118,34673997,34784748]}
{“id”:13487106,“peer_stores”:[1,15698411,15698413,17625573,17625571,17625570,17625568,17625576,33515118,34673997,34923434]}
{“id”:6079993,“peer_stores”:[1,15698410,15698412,17625571,17625568,17625576,17625573,17625570,33515118,34673997,34784748]}
{“id”:8864482,“peer_stores”:[1,15698412,15698411,17625568,17625571,17625573,17625576,17625570,33515118,34673997,34923434]}
{“id”:8036158,“peer_stores”:[1,15698413,15698410,17625568,17625571,17625570,17625576,17625573,33515118,34673997]}
{“id”:6757242,“peer_stores”:[1,15698413,15698410,17625571,17625570,17625573,17625576,17625568,33515118,34673997,34923434]}
{“id”:15832113,“peer_stores”:[1,15698412,15698411,17625576,17625571,17625570,17625568,17625573,33515118,34673997]}
{“id”:5317880,“peer_stores”:[1,15698412,15698411,17625573,17625571,17625568,17625576,17625570,33515118,34673997]}
{“id”:13236352,“peer_stores”:[1,15698411,15698412,17625576,17625571,17625573,17625568,17625570,33515118,34673997,34784748]}
{“id”:14342325,“peer_stores”:[1,15698411,15698413,17625573,17625571,17625570,17625576,17625568,33515118,34673997,34784748]}
{“id”:737372,“peer_stores”:[1,15698412,15698411,17625570,17625571,17625568,17625576,17625573,33515118,34673997,34784748]}
{“id”:1331604,“peer_stores”:[1,15698412,15698410,17625568,17625573,17625571,17625576,17625570,33515118,34673997,34784748]}
{“id”:444647,“peer_stores”:[1,15698412,15698411,17625570,17625571,17625568,17625573,17625576,33515118,34673997,34923434]}
{“id”:1186646,“peer_stores”:[1,15698411,15698413,17625573,17625571,17625576,17625570,17625568,33515118,34673997,34923434]}
{“id”:7097340,“peer_stores”:[1,15698412,15698410,17625573,17625571,17625570,17625576,17625568,33515118,34673997,34923434]}
{“id”:5473573,“peer_stores”:[1,15698412,15698410,17625571,17625576,17625570,17625568,17625573,33515118,34673997,34923434]}
{“id”:1145774,“peer_stores”:[1,15698410,15698413,17625571,17625570,17625576,17625573,17625568,33515118,34673997]}
{“id”:1215657,“peer_stores”:[1,15698412,15698410,17625573,17625570,17625571,17625576,17625568,33515118,34673997,34923434]}
{“id”:5478475,“peer_stores”:[1,15698411,15698412,17625571,17625570,17625573,17625576,17625568,33515118,34673997,34784748]}
{“id”:4969036,“peer_stores”:[1,15698412,15698411,17625576,17625571,17625573,17625570,17625568,33515118,34673997,34923434]}
{“id”:1448991,“peer_stores”:[1,15698412,15698411,17625576,17625571,17625573,17625570,17625568,33515118,34673997,34784748]}
{“id”:1182107,“peer_stores”:[1,15698410,15698412,17625573,17625568,17625571,17625570,17625576,33515118,34673997,34784748]}
{“id”:5017113,“peer_stores”:[1,15698412,15698411,17625571,17625573,17625570,17625568,17625576,33515118,34673997,34923434]}
{“id”:1347658,“peer_stores”:[1,15698410,15698412,17625568,17625571,17625573,17625576,17625570,33515118,34673997,34923434]}
{“id”:1053645,“peer_stores”:[1,15698413,15698410,17625576,17625571,17625568,17625573,17625570,33515118,34673997,34923434]}
{“id”:1012259,“peer_stores”:[1,15698411,15698412,17625571,17625573,17625570,17625568,17625576,33515118,34673997,34923434]}
{“id”:5019333,“peer_stores”:[1,15698413,15698410,17625571,17625576,17625568,17625570,17625573,33515118,34673997,34923434]}

你是配置了5副本吗,把脚本输出结果 找几个region_id 在确认下,pd-ctl region xxxx 看下输出的store_id是不是全是之前处理的那些,是的话就可以 recreate , 参考下面 :

现在啥情况了

没有恢复成功,可能和我的操作不对有关吧,后面搭建了一个新的环境。后续再继续研究一下

整理了篇文章

好的,感谢

又学了一招

3 个赞

感谢分享,案例学习

:+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: :+1: