集群中compaction duration数值较高

🪐 TiDB 技术问题
21

1、每个服务器上都没有其他服务，这是肯定的。
2、我后续会尝试一下，谢谢。如果关闭调度后i/o明显下降 可以认为是调度占用了主要i/o 否则就是业务本身流量较大 或者其他原因

22

不是，目前我怀疑是你的region 太大，导致了一系列问题，关闭 pd 调度，只是暂时验证一下而已（准确来说，我需要 你 tikv 的日志，pd leader 的日志，另外，我还要确认你的表结构，因为我怀疑是你的一行数据太大，导致 region 无法正常分裂导致）（pd leader/tikv 日志，只是验证 region 分裂失败的原因）

23

集群全套日志如下：
链接: https://pan.baidu.com/s/1hfmiA5_vHEXzv3Kw6wefjA 密码: sf3b
虽然是7月16日的，但当时就已经存在该问题了，麻烦您帮忙看一下是否是region的问题，
表结构如下:
mysql> desc tx_info_2;
±----------±------------±-----±-----±--------±------+
| Field | Type | Null | Key | Default | Extra |
±----------±------------±-----±-----±--------±------+
| version | varchar(5) | YES | | NULL | |
| size | int(11) | YES | | NULL | |
| locktime | varchar(14) | YES | | NULL | |
| n_in | int(11) | YES | | NULL | |
| n_out | int(11) | YES | | NULL | |
| vin | json | YES | | NULL | |
| vout | json | YES | | NULL | |
| timestamp | char(10) | YES | MUL | NULL | |
| txhash | char(64) | NO | PRI | NULL | |
±----------±------------±-----±-----±--------±------+
9 rows in set (0.01 sec)

如您所说，一行数据确实很大，因为vin，vout字段为json类型

24

pd -ctl 有个 region 命令，然后 region topsize 命令，有时间方便也提供一下这个（:sweat_smile:忘记要了）（要这个的目的是想看看 region 的真实大小，监控应该是有问题的）

25

执行 region topsize 10 命令输出如下：

{
“count”: 10,
“regions”: [
{
“id”: 304534264,
“start_key”: “74800000000002FEFFD35F728000000000FFA5B4840000000000FA”,
“end_key”: “74800000000002FEFFD600000000000000F8”,
“epoch”: {
“conf_ver”: 107607,
“version”: 73958
},
“peers”: [
{
“id”: 534631049,
“store_id”: 534172210
},
{
“id”: 534639988,
“store_id”: 534414223
},
{
“id”: 537533837,
“store_id”: 534172077
},
{
“id”: 554029230,
“store_id”: 534402022,
“is_learner”: true
}
],
“leader”: {
“id”: 534639988,
“store_id”: 534414223
},
“pending_peers”: [
{
“id”: 534631049,
“store_id”: 534172210
},
{
“id”: 554029230,
“store_id”: 534402022,
“is_learner”: true
}
],
“written_bytes”: 9260334,
“read_bytes”: 3420,
“written_keys”: 608,
“read_keys”: 76,
“approximate_size”: 365070,
“approximate_keys”: 19591852
},
{
“id”: 39230235,
“start_key”: “7480000000000001FF745F728000000022FFE149050000000000FA”,
“end_key”: “7480000000000001FF745F728000000022FFE165730000000000FA”,
“epoch”: {
“conf_ver”: 18464,
“version”: 16861
},
“peers”: [
{
“id”: 534432190,
“store_id”: 534204559
},
{
“id”: 534456021,
“store_id”: 534402022
},
{
“id”: 545089457,
“store_id”: 537425107
}
],
“leader”: {
“id”: 534432190,
“store_id”: 534204559
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 4924,
“approximate_keys”: 0
},
{
“id”: 247750433,
“start_key”: “7480000000000001FF745F728000000024FFFC30390000000000FA”,
“end_key”: “7480000000000001FF745F728000000024FFFC321C0000000000FA”,
“epoch”: {
“conf_ver”: 55247,
“version”: 20822
},
“peers”: [
{
“id”: 537474294,
“store_id”: 534402022
},
{
“id”: 553515332,
“store_id”: 534172077
},
{
“id”: 554553763,
“store_id”: 537425107
}
],
“leader”: {
“id”: 537474294,
“store_id”: 534402022
},
“written_bytes”: 1684431,
“read_bytes”: 0,
“written_keys”: 8,
“read_keys”: 0,
“approximate_size”: 4817,
“approximate_keys”: 40961
},
{
“id”: 248980535,
“start_key”: “7480000000000001FF745F728000000025FF6389DA0000000000FA”,
“end_key”: “7480000000000001FF745F728000000025FF6397B20000000000FA”,
“epoch”: {
“conf_ver”: 69432,
“version”: 20028
},
“peers”: [
{
“id”: 548523811,
“store_id”: 534402022
},
{
“id”: 553601907,
“store_id”: 534204559
},
{
“id”: 554425427,
“store_id”: 534414223
}
],
“leader”: {
“id”: 553601907,
“store_id”: 534204559
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 4763,
“approximate_keys”: 1
},
{
“id”: 39129867,
“start_key”: “7480000000000001FF745F728000000022FFD7A9880000000000FA”,
“end_key”: “7480000000000001FF745F728000000022FFD7BDBC0000000000FA”,
“epoch”: {
“conf_ver”: 17508,
“version”: 16804
},
“peers”: [
{
“id”: 534545598,
“store_id”: 534204559
},
{
“id”: 554223556,
“store_id”: 534414223
},
{
“id”: 554265652,
“store_id”: 537425107
}
],
“leader”: {
“id”: 534545598,
“store_id”: 534204559
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 4761,
“approximate_keys”: 0
},
{
“id”: 247666723,
“start_key”: “7480000000000001FF745F728000000024FFF6C5010000000000FA”,
“end_key”: “7480000000000001FF745F728000000024FFF6C8190000000000FA”,
“epoch”: {
“conf_ver”: 54630,
“version”: 19182
},
“peers”: [
{
“id”: 534238833,
“store_id”: 534172210
},
{
“id”: 553515533,
“store_id”: 537425107
},
{
“id”: 554571316,
“store_id”: 534414223
}
],
“leader”: {
“id”: 534238833,
“store_id”: 534172210
},
“written_bytes”: 3291466,
“read_bytes”: 0,
“written_keys”: 17,
“read_keys”: 0,
“approximate_size”: 4701,
“approximate_keys”: 40963
},
{
“id”: 247118968,
“start_key”: “7480000000000001FF745F728000000024FFC326420000000000FA”,
“end_key”: “7480000000000001FF745F728000000024FFC32D0B0000000000FA”,
“epoch”: {
“conf_ver”: 49355,
“version”: 18803
},
“peers”: [
{
“id”: 534458183,
“store_id”: 534204559
},
{
“id”: 554438459,
“store_id”: 537425107
},
{
“id”: 554528010,
“store_id”: 534172077
}
],
“leader”: {
“id”: 534458183,
“store_id”: 534204559
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 4686,
“approximate_keys”: 9475
},
{
“id”: 262580055,
“start_key”: “7480000000000001FF745F728000000025FF8685F50000000000FA”,
“end_key”: “7480000000000001FF745F728000000025FF868CE40000000000FA”,
“epoch”: {
“conf_ver”: 75546,
“version”: 20343
},
“peers”: [
{
“id”: 534686162,
“store_id”: 534402022
},
{
“id”: 553809012,
“store_id”: 534172210
},
{
“id”: 554452578,
“store_id”: 534204559
}
],
“leader”: {
“id”: 553809012,
“store_id”: 534172210
},
“written_bytes”: 1647204,
“read_bytes”: 45,
“written_keys”: 8,
“read_keys”: 1,
“approximate_size”: 4680,
“approximate_keys”: 1
},
{
“id”: 268598492,
“start_key”: “7480000000000001FF745F728000000025FF9D94A40000000000FA”,
“end_key”: “7480000000000001FF745F728000000025FF9D9FC60000000000FA”,
“epoch”: {
“conf_ver”: 80139,
“version”: 20529
},
“peers”: [
{
“id”: 534945482,
“store_id”: 534204559
},
{
“id”: 534949277,
“store_id”: 534414223
},
{
“id”: 553513842,
“store_id”: 534402022
}
],
“leader”: {
“id”: 534945482,
“store_id”: 534204559
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 4645,
“approximate_keys”: 81921
},
{
“id”: 247067286,
“start_key”: “7480000000000001FF745F728000000024FFB953A30000000000FA”,
“end_key”: “7480000000000001FF745F728000000024FFB969C50000000000FA”,
“epoch”: {
“conf_ver”: 48422,
“version”: 18741
},
“peers”: [
{
“id”: 534457889,
“store_id”: 534402022
},
{
“id”: 553685028,
“store_id”: 534172210
},
{
“id”: 554516420,
“store_id”: 534172077
}
],
“leader”: {
“id”: 553685028,
“store_id”: 534172210
},
“written_bytes”: 3242075,
“read_bytes”: 0,
“written_keys”: 16,
“read_keys”: 0,
“approximate_size”: 4644,
“approximate_keys”: 40962
}
]
}

26

1、可以看一下这个：https://docs.pingcap.com/zh/tidb/v4.0/troubleshoot-high-disk-io#从-log-定位-io-问题
2、建议调整一下 compaction 相关的参数，比如调小 rate-bytes-per-sec 到 300M
3、

image
image1950×490 117 KB

1 个赞
27

感谢您的意见，后续会与同事商量进行调整

28

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。