tiup修改tidb参数后reload报错

[tao7]

tidb版本：v4.0.0
操作步骤：
1、使用tiup cluster edit-config test-cluster修改了tidb的参数：

server_configs:
tidb:
prepared-plan-cache: false

2、使用tiup cluster reload test-cluster -R tidb命令reload，报错日志如下：

• [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xxx.190, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/alertmanager-9093.service, deploy_dir=/data/tidb/deploy, data_dir=[/data/tidb/deploy/data.alertmanager], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
• [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xxx.190, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/prometheus-9090.service, deploy_dir=/data/tidb/deploy, data_dir=[/data/tidb/deploy/prometheus2.0.0.data.metrics], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache
• [ Serial ] - InitConfig: cluster=test-cluster, user=tidb, host=xxx.xxx.xxx.190, path=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache/grafana-3000.service, deploy_dir=/data/tidb/deploy, data_dir=[/home/tidb], log_dir=/data/tidb/deploy/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/test-cluster/config-cache

Error: init config failed: xxx.xxx.xxx.190:2379: fetch /2.root.json from mirror(https://tiup-mirrors.pingcap.com/) failed: download from https://tiup-mirrors.pingcap.com/2.root.json failed: Get https://tiup-mirrors.pingcap.com/2.root.json: Service Unavailable

Verbose debug logs has been written to /home/tidb/logs/tiup-cluster-debug-2020-06-09-15-53-45.log.
Error: run /home/tidb/.tiup/components/cluster/v1.0.3/tiup-cluster (wd:/home/tidb/.tiup/data/S1OnksF) failed: exit status 1

请问这个需要如何处理？

[yilong (yi888long)]

1. 集群是正常状态，只需要修改这个参数吗？ 还是之前有升级过
2. 麻烦截图展示您修改参数的配置格式，多谢。

[tao7]

集群状态在修改前是正常的。该集群是从3.0.9升级到4.0.0的。修改配置的原因是由于dashboard的top sql语句提示error.api.other:invalid connection，查社区资料说需要关闭tidb的prepared-plan-cache参数

[yilong (yi888long)]

1， 参数默认为 false

2. 修改参数是： prepared-plan-cache.enabled: false

[tao7]

谢谢，按prepared-plan-cache.enabled: false修改后，reload成功了。不过我现在有一个新的问题，prepared-plan-cache设置为false后，dashboard报错问题仍然存在，截图如下：

[yilong (yi888long)]

1. 这个参数默认值就是false，所以应该不是这个参数的问题。
2. 可以参考这个试一下 [FAQ] dashboard，qps 和 延迟界面显示异常
3. 请查看是使用 tiup cluster 中显示的 UI 地址登录吗？
4. 查看tiup cluster 上的状态是否都是正常。

[tao7]

已经按照步骤2链接中的方法操作过

tiup cluster 显示如下：

访问地址+页面截图如下：

[yilong (yi888long)]

1. reload 以后， 请问重启 tidb 了吗？
2. 麻烦查看tidb的日志，参数值是多少，多谢。

[tao7]

1、从dashboard上的信息看，执行reload后，tidb组件被重启了

2、tidb日志中，prepared-plan-cache还是true，为什么重启没有生效呢？

[2020/06/09 16:45:24.564 +08:00] [INFO] [printer.go:56] [“loaded config”] [config=“{"host":"0.0.0.0","advertise-address":"xxx.xxx.xxx.188","port":4000,"cors":"","store":"tikv","path":"xxx.xxx.xxx.188:2379,xxx.xxx.xxx.189:2379,xxx.xxx.xxx.190:2379","socket":"","lease":"45s","run-ddl":true,"split-table":true,"token-limit":1000,"oom-use-tmp-storage":true,"tmp-storage-path":"/tmp/1001_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage","oom-action":"log","mem-quota-query":1073741824,"tmp-storage-quota":-1,"enable-streaming":false,"enable-batch-dml":false,"lower-case-table-names":2,"server-version":"","log":{"level":"error","format":"text","disable-timestamp":false,"enable-timestamp":null,"disable-error-stack":null,"enable-error-stack":null,"file":{"filename":"/data/tidb/deploy/log/tidb.log","max-size":300,"max-days":0,"max-backups":0},"enable-slow-log":true,"slow-query-file":"log/tidb_slow_query.log","slow-threshold":300,"expensive-threshold":10000,"query-log-max-len":2048,"record-plan-in-slow-log":1},"security":{"skip-grant-table":false,"ssl-ca":"","ssl-cert":"","ssl-key":"","require-secure-transport":false,"cluster-ssl-ca":"","cluster-ssl-cert":"","cluster-ssl-key":"","cluster-verify-cn":null},"status":{"status-host":"0.0.0.0","metrics-addr":"","status-port":10080,"metrics-interval":15,"report-status":true,"record-db-qps":false},"performance":{"max-procs":0,"max-memory":0,"stats-lease":"3s","stmt-count-limit":5000,"feedback-probability":0.05,"query-feedback-limit":1024,"pseudo-estimate-ratio":0.8,"force-priority":"NO_PRIORITY","bind-info-lease":"3s","txn-total-size-limit":104857600,"tcp-keep-alive":true,"cross-join":true,"run-auto-analyze":true,"agg-push-down-join":false,"committer-concurrency":16,"max-txn-ttl":600000},"prepared-plan-cache":{"enabled":true,"capacity":100,"memory-guard-ratio":0.1},"opentracing":{"enable":false,"rpc-metrics":false,"sampler":{"type":"const","param":1,"sampling-server-url":"","max-operations":0,"sampling-refresh-interval":0},"reporter":{"queue-size":0,"buffer-flush-interval":0,"log-spans":false,"local-agent-host-port":""}},"proxy-protocol":{"networks":"","header-timeout":5},"tikv-client":{"grpc-connection-count":16,"grpc-keepalive-time":10,"grpc-keepalive-timeout":3,"commit-timeout":"41s","max-batch-size":128,"overload-threshold":200,"max-batch-wait-time":0,"batch-wait-size":8,"enable-chunk-rpc":true,"region-cache-ttl":600,"store-limit":0,"store-liveness-timeout":"120s","copr-cache":{"enable":false,"capacity-mb":1000,"admission-max-result-mb":10,"admission-min-process-ms":5}},"binlog":{"enable":false,"ignore-error":false,"write-timeout":"15s","binlog-socket":"","strategy":"range"},"compatible-kill-query":false,"plugin":{"dir":"","load":""},"pessimistic-txn":{"enable":true,"max-retry-count":256},"check-mb4-value-in-utf8":true,"max-index-length":3072,"alter-primary-key":false,"treat-old-version-utf8-as-utf8mb4":true,"enable-table-lock":false,"delay-clean-table-lock":0,"split-region-max-num":1000,"stmt-summary":{"enable":true,"enable-internal-query":false,"max-stmt-count":200,"max-sql-length":4096,"refresh-interval":1800,"history-size":24},"repair-mode":false,"repair-table-list":[],"isolation-read":{"engines":["tikv","tiflash","tidb"]},"max-server-connections":0,"new_collations_enabled_on_first_bootstrap":false,"experimental":{"allow-auto-random":false,"allow-expression-index":false}}”]

[yilong (yi888long)]

麻烦截图看下你的配置文件，怎么写的参数，多谢。

[tao7]

[wish-PingCAP]

Hi，能否方便截取一下 TiDB 实例部署目录的 conf 子目录下的 TiDB 配置文件内容？

[tao7]

详见附件tidb.toml (1.8 KB)

[yilong (yi888long)]

请将配置文件里这个prepare参数删除，service_config 配置就可以了，再重启下

[tao7]

在tidb组件所在的所有节点上vi tidb.toml删除那几行，使用tiup cluster reload test-cluster -R tidb重启后，参数又回来了

[yilong (yi888long)]

抱歉，我没有说清楚，不是每个tidb节点，在部署目录/.tiup/storage/cluster/clusters/xxx/ansible-imported-configs/ 目录下，找一下 tidb 配置 文件，修改这个

[tao7]

按照指导修改目录下tidb的配置后，再reload配置，问题解决。感谢，辛苦了

[yilong (yi888long)]