修改集群配置后reload提示失败

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:
【 TiDB 使用环境】
v1.5.6 tiup
集群版本:v5.2.1

【概述】 场景 + 问题概述
通过tiup cluster edit-config xxx 命令在线修改了tidb以下几个参数后reload失败:
server_configs:
tidb:
stmt-count-limit: 10000
tidb_mem_quota_query: 5368709120
txn-total-size-limit: 2097152000
tikv: {}
pd: {}
tiflash: {}
tiflash-learner: {}
pump: {}
drainer: {}
cdc: {}

reload报错信息如下:
[tidb@jtsf-core-pod5-mmp-01-29 conf]$ tiup cluster reload sfjt-tidb -R tidb
Found cluster newer version:

The latest version:         v1.8.2
Local installed version:    v1.5.6
Update current component:   tiup update cluster
Update all components:      tiup update --all

Starting component cluster: /home/tidb/.tiup/components/cluster/v1.5.6/tiup-cluster reload sfjt-tidb -R tidb
Will reload the cluster sfjt-tidb with restart policy is true, nodes: , roles: tidb.
Do you want to continue? [y/N]:(default=N) y

  • [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/sfjt-tidb/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/sfjt-tidb/ssh/id_rsa.pub
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.101
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.101
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.101
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.105
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.103
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.102
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.101
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.103
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.101
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.103
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.102
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.102
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.104
  • [Parallel] - UserSSH: user=tidb, host=10.253.6.101
  • [ Serial ] - UpdateTopology: cluster=sfjt-tidb
    {“level”:“warn”,“ts”:“2022-01-14T11:07:16.663+0800”,“logger”:“etcd-client”,“caller”:“v3@v3.5.0/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“etcd-endpoints://0xc0000e01c0/#initially=[10.253.6.101:2379;10.253.6.102:2379;10.253.6.103:2379]”,“attempt”:0,“error”:“rpc error: code = DeadlineExceeded desc = context deadline exceeded”}

Error: context deadline exceeded

Verbose debug logs has been written to /home/tidb/.tiup/logs/tiup-cluster-debug-2022-01-14-11-07-16.log.
Error: run /home/tidb/.tiup/components/cluster/v1.5.6/tiup-cluster (wd:/home/tidb/.tiup/data/SuSOWIL) failed: exit status 1

/home/tidb/.tiup/logs/tiup-cluster-debug-2022-01-14-11-07-16.log日志如下:

2022-01-14T11:06:58.521+0800 INFO Execute command {“command”: “tiup cluster reload sfjt-tidb -R tidb”}
2022-01-14T11:06:58.521+0800 DEBUG Environment variables {“env”: [“TIUP_HOME=/home/tidb/.tiup”, “TIUP_WORK_DIR=/tidb-deploy/tidb-4000/conf”, “TIUP_USER_INPUT_VERSION=”, “TIUP_VERSION=1.5.6”, “TIUP_INSTANCE_DATA_DIR=/home/tidb/.tiup/data/SuSOWIL”, “TIUP_COMPONENT_DATA_DIR=/home/tidb/.tiup/storage/cluster”, “TIUP_COMPONENT_INSTALL_DIR=/home/tidb/.tiup/components/cluster/v1.5.6”, “TIUP_TELEMETRY_STATUS=enable”, “TIUP_TELEMETRY_UUID=abb25992-12d3-4794-8372-2d7cc261a3ab”, “TIUP_TELEMETRY_SECRET=feb9fbd8e01a7ce2d37dfbe474196e2a”, “TIUP_TAG=SuSOWIL”, “XDG_SESSION_ID=c16”, “HOSTNAME=jtsf-core-pod5-mmp-01-29”, “SHELL=/bin/bash”, “TERM=xterm”, “HISTSIZE=1000”, “USER=tidb”, “LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.axa=01;36:.oga=01;36:.spx=01;36:*.xspf=01;36:”, “MAIL=/var/spool/mail/tidb”, “PATH=/home/tidb/.tiup/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/tidb/.local/bin:/home/tidb/bin”, “PWD=/tidb-deploy/tidb-4000/conf”, “LANG=en_US.UTF-8”, “https_proxy=http://10.252.242.40:38080”,“HISTCONTROL=ignoredups”, “SHLVL=1”, “HOME=/home/tidb”, “LOGNAME=tidb”, “LESSOPEN=||/usr/bin/lesspipe.sh %s”, “_=/home/tidb/.tiup/bin/tiup”, “OLDPWD=/tidb-deploy/tidb-4000”, “TIUP_TELEMETRY_EVENT_UUID=19832539-3ed9-4de9-99e9-bb5fbbc09f0b”, “TIUP_MIRRORS=https://tiup-mirrors.pingcap.com”]}
2022-01-14T11:07:06.651+0800 INFO + [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/sfjt-tidb/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/sfjt-tidb/ssh/id_rsa.pub
2022-01-14T11:07:06.652+0800 DEBUG TaskBegin {“task”: “SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/sfjt-tidb/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/sfjt-tidb/ssh/id_rsa.pub”}
2022-01-14T11:07:06.652+0800 DEBUG TaskFinish {“task”: “SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/sfjt-tidb/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/sfjt-tidb/ssh/id_rsa.pub”}
2022-01-14T11:07:06.652+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.101\nUserSSH: user=tidb, host=10.253.6.102\nUserSSH: user=tidb, host=10.253.6.103\nUserSSH: user=tidb, host=10.253.6.101\nUserSSH: user=tidb, host=10.253.6.102\nUserSSH: user=tidb, host=10.253.6.103\nUserSSH: user=tidb, host=10.253.6.104\nUserSSH: user=tidb, host=10.253.6.105\nUserSSH: user=tidb, host=10.253.6.101\nUserSSH: user=tidb, host=10.253.6.102\nUserSSH: user=tidb, host=10.253.6.103\nUserSSH: user=tidb, host=10.253.6.101\nUserSSH: user=tidb, host=10.253.6.101\nUserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.652+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.101
2022-01-14T11:07:06.652+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.652+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.101
2022-01-14T11:07:06.652+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.652+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.101
2022-01-14T11:07:06.652+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.105
2022-01-14T11:07:06.652+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.653+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.105”}
2022-01-14T11:07:06.652+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.102
2022-01-14T11:07:06.652+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.103
2022-01-14T11:07:06.652+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.101
2022-01-14T11:07:06.652+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.102
2022-01-14T11:07:06.652+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.104
2022-01-14T11:07:06.652+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.103
2022-01-14T11:07:06.653+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.103”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.653+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.102
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.105”}
2022-01-14T11:07:06.653+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.101
2022-01-14T11:07:06.652+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.101
2022-01-14T11:07:06.653+0800 INFO + [Parallel] - UserSSH: user=tidb, host=10.253.6.103
2022-01-14T11:07:06.653+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.103”}
2022-01-14T11:07:06.653+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.102”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.103”}
2022-01-14T11:07:06.653+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.102”}
2022-01-14T11:07:06.653+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.102”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.103”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.102”}
2022-01-14T11:07:06.653+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.102”}
2022-01-14T11:07:06.652+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.653+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.103”}
2022-01-14T11:07:06.653+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.104”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.102”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.103”}
2022-01-14T11:07:06.653+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.104”}
2022-01-14T11:07:06.653+0800 DEBUG TaskBegin {“task”: “UserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.653+0800 DEBUG TaskFinish {“task”: “UserSSH: user=tidb, host=10.253.6.101\nUserSSH: user=tidb, host=10.253.6.102\nUserSSH: user=tidb, host=10.253.6.103\nUserSSH: user=tidb, host=10.253.6.101\nUserSSH: user=tidb, host=10.253.6.102\nUserSSH: user=tidb, host=10.253.6.103\nUserSSH: user=tidb, host=10.253.6.104\nUserSSH: user=tidb, host=10.253.6.105\nUserSSH: user=tidb, host=10.253.6.101\nUserSSH: user=tidb, host=10.253.6.102\nUserSSH: user=tidb, host=10.253.6.103\nUserSSH: user=tidb, host=10.253.6.101\nUserSSH: user=tidb, host=10.253.6.101\nUserSSH: user=tidb, host=10.253.6.101”}
2022-01-14T11:07:06.653+0800 INFO + [ Serial ] - UpdateTopology: cluster=sfjt-tidb
2022-01-14T11:07:06.653+0800 DEBUG TaskBegin {“task”: “UpdateTopology: cluster=sfjt-tidb”}
2022-01-14T11:07:16.663+0800 DEBUG TaskFinish {“task”: “UpdateTopology: cluster=sfjt-tidb”, “error”: “context deadline exceeded”}
2022-01-14T11:07:16.663+0800 INFO Execute command finished {“code”: 1, “error”: “context deadline exceeded”, “errorVerbose”: “context deadline exceeded\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.4/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.4/juju_adaptor.go:15\ngithub.com/pingcap/tiup/pkg/cluster/manager.(*Manager).Reload\n\tgithub.com/pingcap/tiup/pkg/cluster/manager/reload.go:121\ngithub.com/pingcap/tiup/components/cluster/command.newReloadCmd.func1\n\tgithub.com/pingcap/tiup/components/cluster/command/reload.go:40\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.1.3/command.go:852\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.1.3/command.go:960\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.1.3/command.go:897\ngithub.com/pingcap/tiup/components/cluster/command.Execute\n\tgithub.com/pingcap/tiup/components/cluster/command/root.go:264\nmain.main\n\tgithub.com/pingcap/tiup/components/cluster/main.go:23\nruntime.main\n\truntime/proc.go:225\nruntime.goexit\n\truntime/asm_amd64.s:1371”}

1、没在tidb配置文件中看到tidb_mem_quota_query参数
2、txn-total-size-limit配置错误,应该配置为performance.txn-total-size-limit

1、在tidb的配置文件中并没有看到tidb_mem_quota_query参数
2、我把其他的两个参数删除了,只保留tidb_mem_quota_query参数也reload失败。

image 最后的配置参数是这样的。

改成这个试一下呢: mem-quota-query,这个是配置文件的参数

修改成这个参数reload的时候还是失败

image 两个格式不一致

亲,请参照下面修改,已在本地环境reload成功。
server_configs:
tidb:
performance.stmt-count-limit: 10000
mem-quota-query: 5368709120
performance.txn-total-size-limit: 2097152000
image

还是最开始回复的那样,配置一共有两处问题。
1、使用了配置文件中不存在的配置tidb_mem_quota_query,应该配置 mem-quota-query


2、没有注意层次结构,有层级的配置需要在配置前加上层级,比如txn-total-size-limit前面要加上它所属的performance,所以配置应该是performance.txn-total-size-limit。
(PS:配置stmt-count-limit也一样)
image

1赞
mem-quota-query: 65719476736
performance.server-memory-quota: 69719476736
performance.txn-total-size-limit: 10737418240

image

1赞

image
我修改后的配置这样后但是还是reload失败。
具体日志如下:
image
真的不晓得怎么排查了,请各位大佬帮忙看看。到底是不是什么基础环境有问题。

这是我的错,忘记了在系统上设置了环境变量https_proxy,已经解决了,感谢大家的帮助。

:handshake::handshake::handshake: