使用sysbench 压测haproxy+keepalived负载地址,和分别压测三台独立的tidb,性能差别明显,前者性能下降明显?

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。

  • 【TiDB 版本】:V3.0.12
  • 【问题描述】: 使用sysbench 压测haproxy+keepalived负载地址,和分别压测三台独立的tidb,性能差别明显,前者性能下降明显?

先开始负载地址压测,负载均衡比较好,均匀打到了后端三台tidb: sysbench --config-file=config oltp_point_select --tables=32 --table-size=10000000 --mysql-host=vip --mysql-port=3390 --mysql-user=root --mysql-password=passwd --mysql-db=sbtest --time=600 --report-interval=10 --threads=256 run config: No such file or directory sysbench 1.0.14 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options: Number of threads: 256 Report intermediate results every 10 second(s) Initializing random number generator from current time

Initializing worker threads…

Threads started!

[ 10s ] thds: 256 tps: 117992.41 qps: 117992.41 (r/w/o: 117992.41/0.00/0.00) lat (ms,95%): 2.61 err/s: 0.00 reconn/s: 0.00 [ 20s ] thds: 256 tps: 122899.70 qps: 122899.70 (r/w/o: 122899.70/0.00/0.00) lat (ms,95%): 2.61 err/s: 0.00 reconn/s: 0.00 [ 30s ] thds: 256 tps: 116222.88 qps: 116222.88 (r/w/o: 116222.88/0.00/0.00) lat (ms,95%): 2.86 err/s: 0.00 reconn/s: 0.00 [ 40s ] thds: 256 tps: 124962.99 qps: 124962.99 (r/w/o: 124962.99/0.00/0.00) lat (ms,95%): 2.43 err/s: 0.00 reconn/s: 0.00 [ 50s ] thds: 256 tps: 123986.02 qps: 123986.02 (r/w/o: 123986.02/0.00/0.00) lat (ms,95%): 2.71 err/s: 0.00 reconn/s: 0.00 三台tidb的cpu 消耗也只有10%左右,调大并发也没有变化;

对比独立测试三台tidb地址压测:

tidb1: [ 10s ] thds: 128 tps: 152788.75 qps: 152788.75 (r/w/o: 152788.75/0.00/0.00) lat (ms,95%): 1.50 err/s: 0.00 reconn/s: 0.00 [ 20s ] thds: 128 tps: 129318.44 qps: 129318.44 (r/w/o: 129318.44/0.00/0.00) lat (ms,95%): 1.37 err/s: 0.00 reconn/s: 0.00 [ 30s ] thds: 128 tps: 121320.85 qps: 121320.85 (r/w/o: 121320.85/0.00/0.00) lat (ms,95%): 1.42 err/s: 0.00 reconn/s: 0.00 [ 40s ] thds: 128 tps: 121601.97 qps: 121601.97 (r/w/o: 121601.97/0.00/0.00) lat (ms,95%): 1.39 err/s: 0.00 reconn/s: 0.00 [ 50s ] thds: 128 tps: 87236.73 qps: 87236.73 (r/w/o: 87236.73/0.00/0.00) lat (ms,95%): 1.52 err/s: 0.00 reconn/s: 0.00 [ 60s ] thds: 128 tps: 77346.85 qps: 77346.85 (r/w/o: 77346.85/0.00/0.00) lat (ms,95%): 1.52 err/s: 0.00 reconn/s: 0.00 [ 70s ] thds: 128 tps: 77730.03 qps: 77730.03 (r/w/o: 77730.03/0.00/0.00) lat (ms,95%): 1.55 err/s: 0.00 reconn/s: 0.00 [ 80s ] thds: 128 tps: 76589.63 qps: 76589.63 (r/w/o: 76589.63/0.00/0.00) lat (ms,95%): 1.58 err/s: 0.00 reconn/s: 0.00

tidb2: [ 10s ] thds: 128 tps: 121292.87 qps: 121292.87 (r/w/o: 121292.87/0.00/0.00) lat (ms,95%): 1.34 err/s: 0.00 reconn/s: 0.00 [ 20s ] thds: 128 tps: 120481.85 qps: 120481.85 (r/w/o: 120481.85/0.00/0.00) lat (ms,95%): 1.37 err/s: 0.00 reconn/s: 0.00 [ 30s ] thds: 128 tps: 122076.82 qps: 122076.82 (r/w/o: 122076.82/0.00/0.00) lat (ms,95%): 1.34 err/s: 0.00 reconn/s: 0.00 [ 40s ] thds: 128 tps: 111800.49 qps: 111800.49 (r/w/o: 111800.49/0.00/0.00) lat (ms,95%): 1.44 err/s: 0.00 reconn/s: 0.00 [ 50s ] thds: 128 tps: 111477.84 qps: 111477.84 (r/w/o: 111477.84/0.00/0.00) lat (ms,95%): 1.44 err/s: 0.00 reconn/s: 0.00 [ 60s ] thds: 128 tps: 111361.45 qps: 111361.45 (r/w/o: 111361.45/0.00/0.00) lat (ms,95%): 1.44 err/s: 0.00 reconn/s: 0.00 [ 70s ] thds: 128 tps: 110275.59 qps: 110275.59 (r/w/o: 110275.59/0.00/0.00) lat (ms,95%): 1.44 err/s: 0.00 reconn/s: 0.00

tidb3: [ 10s ] thds: 128 tps: 81051.26 qps: 81051.26 (r/w/o: 81051.26/0.00/0.00) lat (ms,95%): 1.58 err/s: 0.00 reconn/s: 0.00 [ 20s ] thds: 128 tps: 80742.36 qps: 80742.36 (r/w/o: 80742.36/0.00/0.00) lat (ms,95%): 1.58 err/s: 0.00 reconn/s: 0.00 [ 30s ] thds: 128 tps: 79206.43 qps: 79206.43 (r/w/o: 79206.43/0.00/0.00) lat (ms,95%): 1.61 err/s: 0.00 reconn/s: 0.00 [ 40s ] thds: 128 tps: 80642.96 qps: 80642.96 (r/w/o: 80642.96/0.00/0.00) lat (ms,95%): 1.61 err/s: 0.00 reconn/s: 0.00

三台的tidb cpu 消耗都在30-50%之间,合并的qps 要比单独压测haproxy+keepalived要大的多,请问这块有啥参数可以调整的嘛?还是就是软件存在这种限制?

若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出打印结果,请务必全选并复制粘贴上传。

请问,分别测试时,最终连接单台tidb的连接数是否相同?

相同的

问题解决了,调整了参数

请问,具体怎么解决的,调整的哪个参数?

nbproc 40

感谢反馈