mem-quota-query 使用tiup配置后不生效

【TiDB 版本】
V5.0.1

【问题描述】
tiup cluster edit-config tidb 修改配置文件。然后保存没提示错误,重启后,查看参数没生效,还是默认的1G

server_configs:
tidb:
log.slow-threshold: 300
mem-quota-query: 34359738368
oom-action: log

是不是通过 reload 重启的,从哪里看出来没有生效,tidb conf 目录下的 tidb.toml 配置有没有更新

是直接 tiup restart tidb 方式重启的。select @@tidb_mem_quota_query;方式查询还是1G。看到tidb conf 目录下的 tidb.toml 配置没有更新的。
所以,我就怀疑TIUP有问题。。。

需要用 reload 重启才会生效
https://docs.pingcap.com/zh/tidb/stable/tiup-component-cluster-reload#tiup-cluster-reload

谢谢。我试下

reload 重启,,也还是一样的效果。

tiup cluster reload wuyedb --ignore-config-check时发现错误,,是配置文件里写的不对?

  • Regenerate config tidb -> 192.168.100.9:4000 … Error
  • Regenerate config tidb -> 192.168.100.10:4000 … Error
  • Regenerate config tidb -> 192.168.100.11:4000 … Error


server_configs:
tidb:
log.slow-threshold: 300
mem-quota-query: 34359738368
oom-action: log

看一下 tiup 日志具体什么报错

92.168.100.9, path=/root/.tiup/storage/cluster/clusters/wuyedb/config-cache/tidb-4000.service, deploy_dir=/home/tidb-deploy/tidb-4000, data_dir=[], log_dir=/home/tidb-deploy/tidb-4000/log, cache_dir=/root/.tiup/storage/cluster/clusters/wuyedb/config-cache", “error”: “init config failed: 192.168.100.9:4000: failed to scp /root/.tiup/storage/cluster/clusters/wuyedb/config-cache/run_tidb_192.168.100.9_4000.sh to tidb@192.168.100.9:/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh: Process exited with status 1”, “errorVerbose”: “Process exited with status 1\nfailed to scp /root/.tiup/storage/cluster/clusters/wuyedb/config-cache/run_tidb_192.168.100.9_4000.sh to tidb@192.168.100.9:/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh\ngithub.com/pingcap/tiup/pkg/cluster/executor.(*EasySSHExecutor).Transfer\n\tgithub.com/pingcap/tiup/pkg/cluster/executor/ssh.go:187\ngithub.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Transfer\n\tgithub.com/pingcap/tiup/pkg/cluster/executor/checkpoint.go:111\ngithub.com/pingcap/tiup/pkg/cluster/spec.(*TiDBInstance).InitConfig\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/tidb.go:148\ngithub.com/pingcap/tiup/pkg/cluster/task.(*InitConfig).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/task/init_config.go:51\ngithub.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:85\ngithub.com/pingcap/tiup/pkg/cluster/task.(*StepDisplay).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/task/step.go:87\ngithub.com/pingcap/tiup/pkg/cluster/task.(*Parallel).Execute.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:134\nruntime.goexit\n\truntime/asm_amd64.s:1371\ninit config failed: 192.168.100.9:4000”}

192.168.100.9:4000: failed to scp /root/.tiup/storage/cluster/clusters/wuyedb/config-cache/run_tidb_192.168.100.9_4000.sh to tidb@192.168.100.9:/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh

可能是 ssh 互信有问题,尝试手动执行命令 scp /root/.tiup/storage/cluster/clusters/wuyedb/config-cache/run_tidb_192.168.100.9_4000.sh tidb@192.168.100.9:/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh

不行。。不知道tidb用户的密码是多少。手工修改了tidb用户的密码后,还是不行

可以配置下 ssh 互信
https://docs.pingcap.com/zh/tidb/stable/check-before-deployment#手动配置-ssh-互信及-sudo-免密码

Failed to execute command over SSH for ‘tidb@192.168.100.9:22’ {ssh_stderr: chmod: changing permissions of ‘/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh’: Operation not permitted\n, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin chmod +x /home/tidb-deploy/tidb-4000/scripts/run_tidb.sh}, cause: Process exited with status 1", “errorVerbose”: “executor.ssh.execute_failed: Failed to execute command over SSH for ‘tidb@192.168.100.9:22’ {ssh_stderr: chmod: changing permissions of ‘/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh’: Operation not permitted\n, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin chmod +x /home/tidb-deploy/tidb-4000/scripts/run_tidb.sh}, cause: Process exited with status 1\n at github.com/pingcap/tiup/pkg/cluster/executor.(*EasySSHExecutor).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/executor/ssh.go:154\n at github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/executor/checkpoint.go:85\n at github.com/pingcap/tiup/pkg/cluster/spec.(*TiDBInstance).InitConfig()\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/tidb.go:151\n at github.com/pingcap/tiup/pkg/cluster/task.(*InitConfig).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/task/init_config.go:51\n at github.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:85\n at github.com/pingcap/tiup/pkg/cluster/task.(*StepDisplay).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/task/step.go:87\n at github.com/pingcap/tiup/pkg/cluster/task.(*Parallel).Execute.func1()\n\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:134\n at runtime.goexit()\n\truntime/asm_amd64.s:1371\ninit config failed: 192.168.100.9:4000\ngithub.com/pingcap/tiup/pkg/cluster/task.(*InitConfig).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/task/init_config.go:56\ngithub.com/pingcap/tiup/pkg/cluster/task.(*Serial).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:85\ngithub.com/pingcap/tiup/pkg/cluster/task.(*StepDisplay).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/task/step.go:87\ngithub.com/pingcap/tiup/pkg/cluster/task.(*Parallel).Execute.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/task/task.go:134\nruntime.goexit\n\truntime/asm_amd64.s:1371”}

Error: init config failed: 192.168.100.9:4000: executor.ssh.execute_failed: Failed to execute command over SSH for ‘tidb@192.168.100.9:22’ {ssh_stderr: chmod: changing permissions of ‘/home/tidb-deploy/tidb-4000/scripts/run_tidb.sh’: Operation not permitted
, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin chmod +x /home/tidb-deploy/tidb-4000/scripts/run_tidb.sh}, cause: Process exited with status 1

SSH互信已经配置好了。现在是提示 chmod +x 有问题。。应该是TIUP的BUG吧

tidb 用户有没有配置 sudo 权限

有的。按手册配置的。

sudo -su root
[root@10.0.1.1 tidb]#

[root@vm16 ~]# su - tidb
上一次登录:二 6月 1 11:38:03 CST 2021pts/0 上
[tidb@vm16 ~]$ ssh 192.168.100.9
Last login: Tue Jun 1 11:22:29 2021 from 192.168.0.16
[tidb@vm04 ~]$ sudo -su root
[root@vm04 tidb]#

可以尝试一下 tiup cluster reload 加上 --ssh system 试一下。

一样的效果。。不起作用

检查下 /home 目录有没有 tidb 用户的读写权限,是所有的节点都会报错,还是只有这个节点报错,可以 reload -N 试试其他节点,之前遇到过扩容后由于系统的问题导致目录权限不正确(https://asktug.com/t/topic/34364/), 如果权限不对可以先手动改一下目录权限