mem-quota-query 使用tiup配置后不生效

🪐 TiDB 技术问题 性能调优
1

【TiDB 版本】
V5.0.1

【问题描述】
tiup cluster edit-config tidb 修改配置文件。然后保存没提示错误，重启后，查看参数没生效，还是默认的1G
…
server_configs:
tidb:
log.slow-threshold: 300
mem-quota-query: 34359738368
oom-action: log
…

2

是不是通过 reload 重启的，从哪里看出来没有生效，tidb conf 目录下的 tidb.toml 配置有没有更新

3

是直接 tiup restart tidb 方式重启的。select @@tidb_mem_quota_query;方式查询还是1G。看到tidb conf 目录下的 tidb.toml 配置没有更新的。
所以，我就怀疑TIUP有问题。。。

4

需要用 reload 重启才会生效
https://docs.pingcap.com/zh/tidb/stable/tiup-component-cluster-reload#tiup-cluster-reload

5

谢谢。我试下

6

reload 重启，，也还是一样的效果。

8

tiup cluster reload wuyedb --ignore-config-check时发现错误，，是配置文件里写的不对？

  • Regenerate config tidb -> 192.168.100.9:4000 … Error
  • Regenerate config tidb -> 192.168.100.10:4000 … Error
  • Regenerate config tidb -> 192.168.100.11:4000 … Error

…
server_configs:
tidb:
log.slow-threshold: 300
mem-quota-query: 34359738368
oom-action: log
…

9

看一下 tiup 日志具体什么报错

10

92.168.100.9, path=/root/.tiup/storage/cluster/clusters/wuyedb/config-cache/tidb-4000.service, deploy_dir=/home/tidb-deploy/tidb-4000, data_dir=[], log_dir=/home/tidb-deploy/tidb-4000/log, cache_dir=/root/.tiup/storage/cluster/clusters/wuyedb/config-cache", “error”: “init config failed: 192.168.100.9:4000: failed to scp /root/.tiup/storage/cluster/clusters/wuyedb/config-cache/run_tidb_192.168.100.9_4000.sh to tidb@192.168.100.9:/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh: Process exited with status 1”, “errorVerbose”: “Process exited with status 1\ failed to scp /root/.tiup/storage/cluster/clusters/wuyedb/config-cache/run_tidb_192.168.100.9_4000.sh to tidb@192.168.100.9:/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh\ngithub.com/pingcap/tiup/pkg/cluster/executor.(*EasySSHExecutor).Transfer\ \tgithub.com/pingcap/tiup/pkg/cluster/executor/ssh.go:187\ github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Transfer\ \tgithub.com/pingcap/tiup/pkg/cluster/executor/checkpoint.go:111\ github.com/pingcap/tiup/pkg/cluster/spec.(*TiDBInstance).InitConfig\ \tgithub.com/pingcap/tiup/pkg/cluster/spec/tidb.go:148\ github.com/pingcap/tiup/pkg/cluster/task.(*InitConfig).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/task/init_config.go:51\ github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:85\ github.com/pingcap/tiup/pkg/cluster/task.(*StepDisplay).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/task/step.go:87\ github.com/pingcap/tiup/pkg/cluster/task.(*Parallel).Execute.func1\ \tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:134\ runtime.goexit\ \truntime/asm_amd64.s:1371\ init config failed: 192.168.100.9:4000”}

11

192.168.100.9:4000: failed to scp /root/.tiup/storage/cluster/clusters/wuyedb/config-cache/run_tidb_192.168.100.9_4000.sh to tidb@192.168.100.9:/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh

可能是 ssh 互信有问题，尝试手动执行命令 scp /root/.tiup/storage/cluster/clusters/wuyedb/config-cache/run_tidb_192.168.100.9_4000.sh tidb@192.168.100.9:/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh

12

不行。。不知道tidb用户的密码是多少。手工修改了tidb用户的密码后，还是不行

13

可以配置下 ssh 互信
https://docs.pingcap.com/zh/tidb/stable/check-before-deployment#手动配置-ssh-互信及-sudo-免密码

14

Failed to execute command over SSH for ‘tidb@192.168.100.9:22’ {ssh_stderr: chmod: changing permissions of ‘/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh’: Operation not permitted\ , ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin chmod +x /home/tidb-deploy/tidb-4000/scripts/run_tidb.sh}, cause: Process exited with status 1", “errorVerbose”: “executor.ssh.execute_failed: Failed to execute command over SSH for ‘tidb@192.168.100.9:22’ {ssh_stderr: chmod: changing permissions of ‘/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh’: Operation not permitted\ , ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin chmod +x /home/tidb-deploy/tidb-4000/scripts/run_tidb.sh}, cause: Process exited with status 1\ at github.com/pingcap/tiup/pkg/cluster/executor.(*EasySSHExecutor).Execute()\ \tgithub.com/pingcap/tiup/pkg/cluster/executor/ssh.go:154\ at github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute()\ \tgithub.com/pingcap/tiup/pkg/cluster/executor/checkpoint.go:85\ at github.com/pingcap/tiup/pkg/cluster/spec.(*TiDBInstance).InitConfig()\ \tgithub.com/pingcap/tiup/pkg/cluster/spec/tidb.go:151\ at github.com/pingcap/tiup/pkg/cluster/task.(*InitConfig).Execute()\ \tgithub.com/pingcap/tiup/pkg/cluster/task/init_config.go:51\ at github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute()\ \tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:85\ at github.com/pingcap/tiup/pkg/cluster/task.(*StepDisplay).Execute()\ \tgithub.com/pingcap/tiup/pkg/cluster/task/step.go:87\ at github.com/pingcap/tiup/pkg/cluster/task.(*Parallel).Execute.func1()\ \tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:134\ at runtime.goexit()\ \truntime/asm_amd64.s:1371\ init config failed: 192.168.100.9:4000\ngithub.com/pingcap/tiup/pkg/cluster/task.(*InitConfig).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/task/init_config.go:56\ github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:85\ github.com/pingcap/tiup/pkg/cluster/task.(*StepDisplay).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/task/step.go:87\ github.com/pingcap/tiup/pkg/cluster/task.(*Parallel).Execute.func1\ \tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:134\ runtime.goexit\ \truntime/asm_amd64.s:1371”}

15

Error: init config failed: 192.168.100.9:4000: executor.ssh.execute_failed: Failed to execute command over SSH for ‘tidb@192.168.100.9:22’ {ssh_stderr: chmod: changing permissions of ‘/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh’: Operation not permitted
, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin chmod +x /home/tidb-deploy/tidb-4000/scripts/run_tidb.sh}, cause: Process exited with status 1

16

SSH互信已经配置好了。现在是提示 chmod +x 有问题。。应该是TIUP的BUG吧

17

tidb 用户有没有配置 sudo 权限

18

有的。按手册配置的。

sudo -su root

[root@10.0.1.1 tidb]#

[root@vm16 ~]# su - tidb
上一次登录：二 6月 1 11:38:03 CST 2021pts/0 上
[tidb@vm16 ~]$ ssh 192.168.100.9
Last login: Tue Jun 1 11:22:29 2021 from 192.168.0.16
[tidb@vm04 ~]$ sudo -su root
[root@vm04 tidb]#

19

可以尝试一下 tiup cluster reload 加上 --ssh system 试一下。

20

一样的效果。。不起作用

21

检查下 /home 目录有没有 tidb 用户的读写权限，是所有的节点都会报错，还是只有这个节点报错，可以 reload -N 试试其他节点，之前遇到过扩容后由于系统的问题导致目录权限不正确（https://asktug.com/t/topic/34364/）， 如果权限不对可以先手动改一下目录权限