commit 非常非常慢

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:
【 TiDB 使用环境】
TiKV
Release Version: 2.1.19
Git Commit Hash: b3c27e3a76fd4ca401285375fc7b042775de08d5
Git Commit Branch: HEAD
UTC Build Time: 2019-12-27 11:31:45
Rust Version: rustc 1.29.0-nightly (4f3c7a472 2018-07-17)

TiDB
Release Version: v2.1.19
Git Commit Hash: fcbcc4569fbd7aba9eb92092149f092346b934ec
Git Branch: HEAD
UTC Build Time: 2019-12-27 11:32:43
GoVersion: go version go1.13 linux/amd64
Race Enabled: false
TiKV Min Version: 2.1.0-alpha.1-ff3dd160846b7d1aed9079c389fc188f7f5ea13e
Check Table Before Drop: false

PD
Release Version: v2.1.19
Git Commit Hash: 4a22762aeb7858baae04e8e1d596f577a8cc18a2
Git Branch: HEAD
UTC Build Time: 2019-12-27 11:31:33

【概述】 场景 + 问题概述
— # Time: 2021-08-10T09:58:54.873034017+08:00
— # Txn_start_ts: 426917420337463303
— # User: ptarchiver@10.x.x.x
— # Conn_ID: 5522042
— # Query_time: 15.36248892
— # Parse_time: 0.00000828
— # Compile_time: 0.000015182
— # Prewrite_time: 15.294122957999999 Commit_time: 0.01241276 Get_commit_ts_time: 0.000177951 Write_keys: 18000 Write_size: 1279112 Prewrite_region: 174
— # DB: data_archives
— # Is_internal: false
— # Digest: 9505cacb7c710ed17125fcc6cb3669e8ddca6c8cd8af6a31f6b3cd64604c3098
— # Num_cop_tasks: 0
— # Cop_proc_avg: 0 Cop_proc_p90: 0 Cop_proc_max: 0
— # Cop_wait_avg: 0 Cop_wait_p90: 0 Cop_wait_max: 0
— # Prepared: false
— # Has_more_results: false
— # Succ: true
— # Prev_stmt: REPLACE INTO xxxxxx
— commit;

PD相关设置
“schedule”: {
“max-snapshot-count”: 3,
“max-pending-peer-count”: 16,
“max-merge-region-size”: 0,
“max-merge-region-keys”: 0,
“split-merge-interval”: “1h0m0s”,
“patrol-region-interval”: “100ms”,
“max-store-down-time”: “8h0m0s”,
“leader-schedule-limit”: 16,
“region-schedule-limit”: 8,
“replica-schedule-limit”: 8,
“merge-schedule-limit”: 8,
“hot-region-schedule-limit”: 4,
“hot-region-cache-hits-threshold”: 3,
“tolerant-size-ratio”: 5,
“low-space-ratio”: 0.8,
“high-space-ratio”: 0.6,
“disable-raft-learner”: “false”,
“disable-remove-down-replica”: “false”,
“disable-replace-offline-replica”: “false”,
“disable-make-up-replica”: “false”,
“disable-remove-extra-replica”: “false”,
“disable-location-replacement”: “false”,
“disable-namespace-relocation”: “false”,

机器配置
CPU : Silver 4210 CPU @ 2.20GHz * 2
MEM:256G
Disk:1.92T SSD raid 0 + 14T HDD*10 raid 10,SSD做LVM-Cache

【背景】 做过哪些操作
通过percona工具的pt-archive工具对数据从mysql到tidb的归档操作,把tidb做为归档库。

【现象】 业务和数据库现象
tidb commit阶段很慢

【问题】 当前遇到的问题
tidb commit阶段很慢

pt-archive工具的命令如下:
percona-toolkit/bin/pt-archiver --charset=utf8mb4 --run-time s=76400 --check-columns --replace --why-quit --sleep=1 --retries=3 --limit 2000 --txn-size 2000 --progress 50000

单机2个tikv实例

1 个赞

网络是什么环境,千兆的,还是万兆的?

1 个赞

万兆,mode=4, 总带宽20Gbps

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2+3 (2)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable

Slave Interface: eth0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0

Slave Interface: eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0

1 个赞

那要做很多排查了,可以参考下社区的排查贴
我举个例子:

  • 热点问题

  • 锁冲突

你可以排查一下看看

可以排除CPU或热点问题?

落盘写入延迟太高了,都是 s 级别了。看一下是不是盘的负载被打满了 ?或者盘的有没有性能问题。

1 个赞

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。