Tikv节点region分布不均

  • 系统版本 & kernel 版本: centos6

  • TiDB 版本: 2.1.6

  • 磁盘型号: ssd

  • 集群节点分布: 3pd 5tikv 3tidb

  • 数据量 & region 数量 & 副本数:
    数据3亿
    region数 1350 副本数 3

  • 集群 QPS、.999-Duration、读写比例:

  • 问题描述(我做了什么): 导入数据之后,tikv5个节点只有三个节点有region分布,其他两个节点没有; 后继续导入数据,在晚间没region的节点曾经出现过500多个Region, 但第二天早上看又全部转移到另外三个节点上去了。

    stores信息: { “count”: 5, “stores”: [ { “store”: { “id”: 1, “address”: “server1:20160”, “version”: “2.1.6”, “state_name”: “Up” }, “status”: { “capacity”: “1.8 TiB”, “available”: “831 GiB”, “leader_count”: 396, “leader_weight”: 1, “leader_score”: 46851, “leader_size”: 46851, “region_count”: 1356, “region_weight”: 1, “region_score”: 139112, “region_size”: 139112, “start_ts”: “2019-08-30T17:49:20+08:00”, “last_heartbeat_ts”: “2019-09-03T10:32:07.47088465+08:00”, “uptime”: “88h42m47.47088465s” } }, { “store”: { “id”: 4, “address”: “server2:20160”, “version”: “2.1.6”, “state_name”: “Up” }, “status”: { “capacity”: “1.8 TiB”, “available”: “596 GiB”, “leader_weight”: 1, “region_weight”: 1, “region_score”: 400047722.26514226, “start_ts”: “2019-09-02T14:52:50+08:00”, “last_heartbeat_ts”: “2019-09-03T10:32:12.497016108+08:00”, “uptime”: “19h39m22.497016108s” } }, { “store”: { “id”: 5, “address”: “server3:20160”, “version”: “2.1.6”, “state_name”: “Up” }, “status”: { “capacity”: “1.8 TiB”, “available”: “643 GiB”, “leader_count”: 502, “leader_weight”: 1, “leader_score”: 46368, “leader_size”: 46368, “region_count”: 1356, “region_weight”: 1, “region_score”: 262591912.02775782, “region_size”: 139112, “start_ts”: “2019-08-30T17:48:43+08:00”, “last_heartbeat_ts”: “2019-09-03T10:32:05.497334715+08:00”, “uptime”: “88h43m22.497334715s” } }, { “store”: { “id”: 6, “address”: “server4:20160”, “version”: “2.1.6”, “state_name”: “Up” }, “status”: { “capacity”: “1.8 TiB”, “available”: “487 GiB”, “leader_weight”: 1, “region_weight”: 1, “region_score”: 718598132.8772172, “start_ts”: “2019-08-30T17:48:20+08:00”, “last_heartbeat_ts”: “2019-09-03T10:32:08.574113626+08:00”, “uptime”: “88h43m48.574113626s” } }, { “store”: { “id”: 7, “address”: “server5:20160”, “version”: “2.1.6”, “state_name”: “Up” }, “status”: { “capacity”: “1.8 TiB”, “available”: “613 GiB”, “leader_count”: 458, “leader_weight”: 1, “leader_score”: 45893, “leader_size”: 45893, “region_count”: 1356, “region_weight”: 1, “region_score”: 352086555.48031425, “region_size”: 139112, “start_ts”: “2019-08-30T17:47:43+08:00”, “last_heartbeat_ts”: “2019-09-03T10:32:09.976414906+08:00”, “uptime”: “88h44m26.976414906s” } } ] }

    没region的tikv节点日志: 2019/09/02 16:37:20.496 INFO peer.rs:911: [region 229594] destroy peer id: 245537 store_id: 4 2019/09/02 16:37:20.496 INFO read.rs:484: [region 229594] 245537 destroy ReadDelegate 2019/09/02 16:37:20.496 INFO peer.rs:513: [region 229594] 245537 begin to destroy 2019/09/02 16:37:20.496 INFO pd.rs:633: [region 229594] remove peer statistic record in pd 2019/09/02 16:37:20.496 INFO peer_storage.rs:1257: [region 229594] clear peer 1 meta key, 1 apply key, 1 raft key and 7 raft logs, takes 127.497µs 2019/09/02 16:37:20.496 INFO peer.rs:550: [region 229594] 245537 destroy itself, takes 387.25µs 2019/09/02 16:37:20.496 INFO region.rs:456: [region 229594] register deleting data in [zt200�00�00�00�00�00�01377:_i200�00�00�00�00377�00�00�01�03200�00�00�00377�013241236�03200�00�00377�00�00�00�00�02�03200�00377�00�007P/230�00�00375, zt200�00�00�00�00�00�01377:_i200�00�00�00�00377�00�00�01�03200�00�00�00377�013241237�03200�00�00377�00�00�00�00�02�03200�00377�00�00:<310a�00�00375) 2019/09/02 16:37:24.297 INFO pd.rs:558: [region 35141] try to change peer AddNode id: 247777 store_id: 7 2019/09/02 16:37:24.297 INFO peer.rs:1850: [region 35141] 245552 propose conf change AddNode peer 247777 2019/09/02 16:37:24.372 INFO apply.rs:902: [region 35141] 245552 execute admin command cmd_type: ChangePeer change_peer {peer {id: 247777 store_id: 7}} at [term: 187, index: 880] 2019/09/02 16:37:24.372 INFO apply.rs:951: [region 35141] 245552 exec ConfChange “AddNode”, epoch: conf_ver: 1767 version: 145 2019/09/02 16:37:24.372 INFO apply.rs:1002: [region 35141] 245552 add peer id: 247777 store_id: 7 to region id: 35141 start_key: “t200�00�00�00�00�00�01377�25_r200�00�00�00�00377K 361�00�00�00�00�00372” end_key: “t200�00�00�00�00�00�01377�25_r200�00�00�00�00377Y260361�00�00�00�00�00372” region_epoch {conf_ver: 1767 version: 145} peers {id: 239735 store_id: 5} peers {id: 241268 store_id: 1} peers {id: 245552 store_id: 4} peers {id: 247777 store_id: 7 is_learner: true} 2019/09/02 16:37:24.372 INFO peer.rs:1018: [region 35141] 245552 notify pd with change peer region id: 35141 start_key: “t200�00�00�00�00�00�01377�25_r200�00�00�00�00377K 361�00�00�00�00�00372” end_key: “t200�00�00�00�00�00�01377�25_r200�00�00�00�00377Y260361�00�00�00�00�00372” region_epoch {conf_ver: 1768 version: 145} peers {id: 239735 store_id: 5} peers {id: 241268 store_id: 1} peers {id: 245552 store_id: 4} peers {id: 247777 store_id: 7} 2019/09/02 16:37:24.375 INFO pd.rs:575: [region 35141] try to transfer leader from id: 245552 store_id: 4 to id: 239735 store_id: 5 2019/09/02 16:37:24.375 INFO pd.rs:575: [region 35141] try to transfer leader from id: 245552 store_id: 4 to id: 239735 store_id: 5 2019/09/02 16:37:24.375 INFO peer.rs:1539: [region 35141] 245552 transfer leader to id: 239735 store_id: 5 2019/09/02 16:37:24.375 INFO :1277: [region 35141] 245552 [term 187] starts to transfer leadership to 239735 2019/09/02 16:37:24.375 INFO :1287: [region 35141] 245552 sends MsgTimeoutNow to 239735 immediately as 239735 already has up-to-date log 2019/09/02 16:37:24.375 INFO peer.rs:1539: [region 35141] 245552 transfer leader to id: 239735 store_id: 5 2019/09/02 16:37:24.375 INFO :1254: [region 35141] 245552 [term 187] transfer leadership to 239735 is in progress, ignores request to same node 239735 2019/09/02 16:37:24.376 INFO :923: [region 35141] 245552 [term: 187] received a MsgRequestVote message with higher term from 239735 [term: 188] 2019/09/02 16:37:24.376 INFO :722: [region 35141] 245552 became follower at term 188 2019/09/02 16:37:24.376 INFO :1091: [region 35141] 245552 [logterm: 187, index: 880, vote: 0] cast MsgRequestVote for 239735 [logterm: 187, index: 880] at term 188 2019/09/02 16:37:24.380 INFO apply.rs:902: [region 35141] 245552 execute admin command cmd_type: ChangePeer change_peer {change_type: RemoveNode peer {id: 245552 store_id: 4}} at [term: 188, index: 882] 2019/09/02 16:37:24.380 INFO apply.rs:951: [region 35141] 245552 exec ConfChange “RemoveNode”, epoch: conf_ver: 1768 version: 145 2019/09/02 16:37:24.380 INFO apply.rs:1045: [region 35141] 245552 remove 245552 from region:id: 35141 start_key: “t200�00�00�00�00�00�01377�25_r200�00�00�00�00377K 361�00�00�00�00�00372” end_key: “t200�00�00�00�00�00�01377�25_r200�00�00�00�00377Y260361�00�00�00�00�00372” region_epoch {conf_ver: 1768 version: 145} peers {id: 239735 store_id: 5} peers {id: 241268 store_id: 1} peers {id: 245552 store_id: 4} peers {id: 247777 store_id: 7} 2019/09/02 16:37:24.495 INFO peer.rs:911: [region 35141] destroy peer id: 245552 store_id: 4 2019/09/02 16:37:24.495 INFO read.rs:484: [region 35141] 245552 destroy ReadDelegate 2019/09/02 16:37:24.495 INFO peer.rs:513: [region 35141] 245552 begin to destroy 2019/09/02 16:37:24.495 INFO pd.rs:633: [region 35141] remove peer statistic record in pd 2019/09/02 16:37:24.495 INFO peer_storage.rs:1257: [region 35141] clear peer 1 meta key, 1 apply key, 1 raft key and 7 raft logs, takes 93.027µs 2019/09/02 16:37:24.495 INFO peer.rs:550: [region 35141] 245552 destroy itself, takes 316.253µs 2019/09/02 16:37:24.496 INFO region.rs:456: [region 35141] register deleting data in [zt200�00�00�00�00�00�01377�25_r200�00�00�00�00377K 361�00�00�00�00�00372, zt200�00�00�00�00�00�01377�25_r200�00�00�00�00377Y260361�00�00�00�00�00372) 2019/09/02 16:40:08.849 INFO region.rs:425: [region 20447] succeed in deleting data in [zt200�00�00�00�00�00�003779_i200�00�00�00�00377�00�00�02�0175553779da5377a0c37764ea03778837782bea7377f377aae4637377377�00�00�00�00�00�00�00�00377367�00�00�00�00�00�00�00370, zt200�00�00�00�00�00�003779_i200�00�00�00�00377�00�00�02�01c83537780c1377713377e47d0377a6377782c6d3772377fa25158377377�00�00�00�00�00�00�00�00377367�00�00�00�00�00�00�00370) 2019/09/02 16:40:08.850 INFO region.rs:425: [region 23030] succeed in deleting data in [zt200�00�00�00�00�00�01377�03�00�00�00�00�00�00�00370, zt200�00�00�00�00�00�01377�05�00�00�00�00�00�00�00370)

TiDB 操作系统版本要求部署需要在 centos 7.3 以上。

另外,辛苦发下 PD 的配置情况。可以使用 config show all 看下,TiKV 的store 信息也提供一份完整的 json 格式吧。

config show all: all_config.json

all_config.json (2.7 KB)

tikv在释放了两个无region分布的节点的部分存储空间后,region分布在四个节点上均衡了,但是之前有region分布的一个节点的region count变成了1,这个节点的剩余存储空间是五个里面最小的 tikv store info: tikv_store.json

tikv_store.json (3.2 KB)

scheduler show:

» scheduler show

[
  "balance-region-scheduler",
  "balance-leader-scheduler",
  "balance-hot-region-scheduler",
  "label-scheduler"
]

手动调度一下试试,另外看下 PD 面板上 scheduler 有没有在正常进行 region 调度。

释放的两个无 region 分布的节点存储空间是怎么做的?看了 store 信息,store 7 只有一个 leader ,为什么会占用那么多存储空间?其次,TiDB 本身会自动进行数据的 balance ,监控看下 region 调度策略是不是有异常,具体看下 PD 面板 Scheduler 下面 Scheduler is running & Balance leader movement & Balance Region movement & Balance leader scheduler & Balance Region scheduler 这些指标项是不是正常的。

找到 store 7 上的 region_id:

pd-ctl -d -u http://xxxx:2379 region --jq=".regions[] | {id: .id, peer_stores: [.peers[].store_id] | select(any(.==7))}"

根据 region_id 看看这个 region 多大,是不是比其他 region 大,这个 store 的 region 怎么这么大比较奇怪

tikv-ctl --db /path/to/tikv/db size -r 2
{
  "store": {
    "id": 7,
    "address": "server5:20160",
    "version": "2.1.6",
    "state_name": "Up"
  },
  "status": {
    "capacity": "1.8 TiB",
    "available": "637 GiB",
    "leader_count": 1,
    "leader_weight": 1,
    "leader_score": 10,
    "leader_size": 10,
    "region_count": 1,
    "region_weight": 1,
    "region_score": 282520009.65852684,
    "region_size": 10,
    "start_ts": "2019-08-30T17:47:43+08:00",
    "last_heartbeat_ts": "2019-09-03T16:59:37.469121761+08:00",
    "uptime": "95h11m54.469121761s"
  }

监控看下这个 TiKV 实例上面 region 的大小 PD -> Statistics - Balance -> Store Region size

你好,假期里我们集群中的一个节点重启之后,没有启动tidb相关进程,上班回来启动后发现region分布不均

我的配置如下config_show_all.json (3.1 KB)

tikv_store.json (3.2 KB)

scheduler show [ “balance-region-scheduler”, “balance-leader-scheduler”, “balance-hot-region-scheduler”, “label-scheduler” ]

看了下 tikv_store.json 文件中的内容,每个 store 节点所用空间基本相差不大,其他三个节点 region 数量基本相同,只有这个节点 region 数量为 1,并且 “capacity”: “1.8 TiB”,初步判断该节点可能存在大 region。