TPC-C压测失败，100线程，1400warehouse

magic · 2023 年2 月 16 日 18:37

【 TiDB 使用环境】测试
【 TiDB 版本】v650
【复现路径】tpcc压测，100线程，warehouse 1400，time 15min；tikv配置使用官网压测配置
【遇到的问题：问题现象及影响】100、200线程压测报错如下，50线程压测正常结束；
【资源配置】
tikv 16c128G 5node，其中压测线程在tikv节点a上
tidb 8c16G 5node, 配置了SLB

【附件：截图/日志/监控】
压测开始，线程100…
tiup is checking updates for component bench …
Starting component bench: /root/.tiup/components/bench/v1.12.0/tiup-bench tpcc run -H192.168.120.17 -P4000 -Dtpcc14 -Uroot -pQaz123… --time 900s --warehouses 1400 --interval 90s --threads 100
省略正常输出
[Current] NEW_ORDER - Takes(s): 90.0, Count: 191420, TPM: 127613.2, Sum(ms): 4689326.4, Avg(ms): 24.5, 50th(ms): 24.1, 90th(ms): 33.6, 95th(ms): 37.7, 99th(ms): 46.1, 99.9th(ms): 60.8, Max(ms): 260.0
[Current] ORDER_STATUS - Takes(s): 90.0, Count: 16915, TPM: 11278.0, Sum(ms): 184683.3, Avg(ms): 10.9, 50th(ms): 10.0, 90th(ms): 17.8, 95th(ms): 19.9, 99th(ms): 25.2, 99.9th(ms): 31.5, Max(ms): 218.1
[Current] PAYMENT - Takes(s): 90.0, Count: 182659, TPM: 121773.0, Sum(ms): 2543295.7, Avg(ms): 13.9, 50th(ms): 13.1, 90th(ms): 21.0, 95th(ms): 24.1, 99th(ms): 31.5, 99.9th(ms): 48.2, Max(ms): 226.5
[Current] STOCK_LEVEL - Takes(s): 90.0, Count: 17087, TPM: 11391.4, Sum(ms): 313816.0, Avg(ms): 18.4, 50th(ms): 17.8, 90th(ms): 25.2, 95th(ms): 27.3, 99th(ms): 33.6, 99.9th(ms): 44.0, Max(ms): 302.0
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 packets.go:446: busy buffer
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 packets.go:446: busy buffer
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 packets.go:446: busy buffer
[mysql] 2023/02/16 15:57:34 packets.go:446: busy buffer
[mysql] 2023/02/16 15:57:34 statement.go:96: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:96: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:96: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 packets.go:446: busy buffer
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:96: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:96: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
Finished
[Summary] DELIVERY - Takes(s): 899.9, Count: 171184, TPM: 11413.8, Sum(ms): 11339230.7, Avg(ms): 66.3, 50th(ms): 65.0, 90th(ms): 83.9, 95th(ms): 92.3, 99th(ms): 113.2, 99.9th(ms): 234.9, Max(ms): 1073.7
[Summary] DELIVERY_ERR - Takes(s): 899.9, Count: 12, TPM: 0.8, Sum(ms): 437.1, Avg(ms): 36.2, 50th(ms): 31.5, 90th(ms): 65.0, 95th(ms): 65.0, 99th(ms): 65.0, 99.9th(ms): 65.0, Max(ms): 65.0
[Summary] NEW_ORDER - Takes(s): 899.9, Count: 1930098, TPM: 128683.9, Sum(ms): 46853697.4, Avg(ms): 24.3, 50th(ms): 24.1, 90th(ms): 33.6, 95th(ms): 37.7, 99th(ms): 46.1, 99.9th(ms): 62.9, Max(ms): 1040.2
[Summary] NEW_ORDER_ERR - Takes(s): 899.9, Count: 42, TPM: 2.8, Sum(ms): 391.1, Avg(ms): 9.3, 50th(ms): 7.9, 90th(ms): 17.8, 95th(ms): 18.9, 99th(ms): 22.0, 99.9th(ms): 22.0, Max(ms): 22.0
[Summary] ORDER_STATUS - Takes(s): 899.9, Count: 171617, TPM: 11442.0, Sum(ms): 1855510.8, Avg(ms): 10.8, 50th(ms): 10.0, 90th(ms): 17.8, 95th(ms): 19.9, 99th(ms): 24.1, 99.9th(ms): 32.5, Max(ms): 520.1
[Summary] ORDER_STATUS_ERR - Takes(s): 899.9, Count: 3, TPM: 0.2, Sum(ms): 19.4, Avg(ms): 6.4, 50th(ms): 3.1, 90th(ms): 14.2, 95th(ms): 14.2, 99th(ms): 14.2, 99.9th(ms): 14.2, Max(ms): 14.2
[Summary] PAYMENT - Takes(s): 899.9, Count: 1843839, TPM: 122932.4, Sum(ms): 25536722.0, Avg(ms): 13.9, 50th(ms): 13.1, 90th(ms): 21.0, 95th(ms): 24.1, 99th(ms): 31.5, 99.9th(ms): 50.3, Max(ms): 570.4
[Summary] PAYMENT_ERR - Takes(s): 899.9, Count: 22, TPM: 1.5, Sum(ms): 87.8, Avg(ms): 4.1, 50th(ms): 4.2, 90th(ms): 6.3, 95th(ms): 6.8, 99th(ms): 6.8, 99.9th(ms): 6.8, Max(ms): 6.8
[Summary] STOCK_LEVEL - Takes(s): 899.9, Count: 172072, TPM: 11472.6, Sum(ms): 3047524.0, Avg(ms): 17.7, 50th(ms): 17.8, 90th(ms): 24.1, 95th(ms): 26.2, 99th(ms): 32.5, 99.9th(ms): 44.0, Max(ms): 302.0
[Summary] STOCK_LEVEL_ERR - Takes(s): 899.9, Count: 5, TPM: 0.3, Sum(ms): 47.1, Avg(ms): 9.3, 50th(ms): 4.2, 90th(ms): 21.0, 95th(ms): 21.0, 99th(ms): 21.0, 99.9th(ms): 21.0, Max(ms): 21.0
tpmC: 128683.8, tpmTotal: 285944.6, efficiency: 714.8%

数据小黑 · 2023 年2 月 17 日 00:10

压测应该在集群外、网络内找一台机器做压测机器？找一个tikv，压测软件的流量与tikv的IO相互干扰，能得到有价值的结果么？

托马斯滑板鞋 · 2023 年2 月 17 日 01:05

压测机器是否为独立机器？多少C？SLB是否有最大连接数？

Billdi表弟 · 2023 年2 月 17 日 03:22

有没有检查tidb节点和压测的tikv节点的cpu、内存、磁盘占用。如果都正常，检查tidb日志，如果tidb日志没有error，必然是外部原因，检查tcp的句柄配置，telnet 检查网络，slb的最大连接数，检查。

tidb狂热爱好者 · 2023 年2 月 17 日 04:12

你机器50能压 100线程支撑不住感觉配置有问题。看看是不是本地ssd硬盘 io读取是不是5000m每秒的云ssd不适合tidb

xingzhenxiang · 2023 年2 月 17 日 06:15

tidb 连接数设置多少啊

magic · 2023 年2 月 17 日 10:57

机器性能问题暂时不影响

system · 2023 年4 月 18 日 10:58

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。