为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：
【 TiDB 使用环境】
tiup部署 3个PD节点
【概述】场景+问题概述
手动修改 pd 配置文件，增加 tls 加密配置后，pd 异常
【背景】做过哪些操作
生成密钥
【现象】业务和数据库现象
【业务影响】
【TiDB 版本】
5.2.2
【附件】
3个pd都打印如下日志
- 相关日志 和 监控
[2021/11/27 00:40:03.457 +08:00] [WARN] [config_logging.go:279] ["rejected connection"] [remote-addr=172.16.7.147:47918] [server-name=] [error="tls: first record does not look like a TLS handshake"]
- 配置
[replication]
enable-placement-rules = true
[security]
cacert-path = "/home/tidb-deploy/pd-2379/tls/ca.crt"
cert-path = "/home/tidb-deploy/pd-2379/tls/pd.crt"
key-path = "/home/tidb-deploy/pd-2379/tls/pd.pem"
- 配置
启动脚本
#!/bin/bash
set -e
# WARNING: This file was auto-generated. Do not edit!
# All your edit might be overwritten!
DEPLOY_DIR=/home/tidb-deploy/pd-2379
cd "${DEPLOY_DIR}" || exit 1
exec bin/pd-server \
--name="pd-172.16.7.191-2379" \
--client-urls="https://0.0.0.0:2379" \
--advertise-client-urls="https://172.16.7.191:2379" \
--peer-urls="https://0.0.0.0:2380" \
--advertise-peer-urls="https://172.16.7.191:2380" \
--data-dir="/home/tidb-data/pd-2379" \
--join="https://172.16.7.147:2379" \
--config=conf/pd.toml \
--log-file="/home/tidb-deploy/pd-2379/log/pd.log" 2>> "/home/tidb-deploy/pd-2379/log/pd_stderr.log"
情况2
1个pd由未开启tls 设置为开启后，然后新增2个pd 也出现异常
[2021/11/27 02:58:54.547 +08:00] [INFO] [util.go:41] ["Welcome to Placement Driver (PD)"]
[2021/11/27 02:58:54.548 +08:00] [INFO] [util.go:42] [PD] [release-version=v5.2.2]
[2021/11/27 02:58:54.605 +08:00] [INFO] [server.go:221] ["PD Config"] [config="{\"client-urls\":\"https://0.0.0.0:2379\",\"peer-urls\":\"https://0.0.0.0:2380\",\"advertise-client-urls\":\"https://172.16.7.191:2379\",\"advertise-peer-urls\":\"https://172.16.7.191:2380\",\"name\":\"pd-172.16.7.191-2379\",\"data-dir\":\"/home/tidb-data/pd-2379\",\"force-new-cluster\":false,\"enable-grpc-gateway\":true,\"initial-cluster\":\"pd-172.16.7.191-2379=https://172.16.7.191:2380,pd-172.16.7.147-2379=http://172.16.7.147:2380\",\"initial-cluster-state\":\"existing\",\"initial-cluster-token\":\"pd-cluster\",\"join\":\"https://172.16.7.147:2379\",\"lease\":3,\"log\":{\"level\":\"\",\"format\":\"text\",\"disable-timestamp\":false,\"file\":{\"filename\":\"/home/tidb-deploy/pd-2379/log/pd.log\",\"max-size\":300,\"max-days\":0,\"max-backups\":0},\"development\":false,\"disable-caller\":false,\"disable-stacktrace\":false,\"disable-error-verbose\":true,\"sampling\":null},\"tso-save-interval\":\"3s\",\"tso-update-physical-interval\":\"50ms\",\"enable-local-tso\":false,\"metric\":{\"job\":\"pd-172.16.7.191-2379\",\"address\":\"\",\"interval\":\"15s\"},\"schedule\":{\"max-snapshot-count\":3,\"max-pending-peer-count\":16,\"max-merge-region-size\":20,\"max-merge-region-keys\":200000,\"split-merge-interval\":\"1h0m0s\",\"enable-one-way-merge\":\"false\",\"enable-cross-table-merge\":\"true\",\"patrol-region-interval\":\"100ms\",\"max-store-down-time\":\"30m0s\",\"leader-schedule-limit\":4,\"leader-schedule-policy\":\"count\",\"region-schedule-limit\":2048,\"replica-schedule-limit\":64,\"merge-schedule-limit\":8,\"hot-region-schedule-limit\":4,\"hot-region-cache-hits-threshold\":3,\"store-limit\":{},\"tolerant-size-ratio\":0,\"low-space-ratio\":0.8,\"high-space-ratio\":0.7,\"region-score-formula-version\":\"v2\",\"scheduler-max-waiting-operator\":5,\"enable-remove-down-replica\":\"true\",\"enable-replace-offline-replica\":\"true\",\"enable-make-up-replica\":\"true\",\"enable-remove-extra-replica\":\"true\",\"enable-location-replacement\":\"true\",\"enable-debug-metrics\":\"false\",\"enable-joint-consensus\":\"true\",\"schedulers-v2\":[{\"type\":\"balance-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"balance-leader\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"hot-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"}],\"schedulers-payload\":null,\"store-limit-mode\":\"manual\"},\"replication\":{\"max-replicas\":3,\"location-labels\":\"\",\"strictly-match-label\":\"false\",\"enable-placement-rules\":\"true\",\"isolation-level\":\"\"},\"pd-server\":{\"use-region-storage\":\"true\",\"max-gap-reset-ts\":\"24h0m0s\",\"key-type\":\"table\",\"runtime-services\":\"\",\"metric-storage\":\"\",\"dashboard-address\":\"auto\",\"trace-region-flow\":\"true\",\"flow-round-by-digit\":3},\"cluster-version\":\"0.0.0\",\"labels\":{},\"quota-backend-bytes\":\"8GiB\",\"auto-compaction-mode\":\"periodic\",\"auto-compaction-retention-v2\":\"1h\",\"TickInterval\":\"500ms\",\"ElectionInterval\":\"3s\",\"PreVote\":true,\"security\":{\"cacert-path\":\"/home/tidb-deploy/pd-2379/tls/ca.crt\",\"cert-path\":\"/home/tidb-deploy/pd-2379/tls/pd.crt\",\"key-path\":\"/home/tidb-deploy/pd-2379/tls/pd.pem\",\"cert-allowed-cn\":null,\"redact-info-log\":false,\"encryption\":{\"data-encryption-method\":\"plaintext\",\"data-key-rotation-period\":\"168h0m0s\",\"master-key\":{\"type\":\"plaintext\",\"key-id\":\"\",\"region\":\"\",\"endpoint\":\"\",\"path\":\"\"}}},\"label-property\":null,\"WarningMsgs\":null,\"DisableStrictReconfigCheck\":false,\"HeartbeatStreamBindInterval\":\"1m0s\",\"LeaderPriorityCheckInterval\":\"1m0s\",\"dashboard\":{\"tidb-cacert-path\":\"\",\"tidb-cert-path\":\"\",\"tidb-key-path\":\"\",\"public-path-prefix\":\"\",\"internal-proxy\":false,\"enable-telemetry\":true,\"enable-experimental\":false},\"replication-mode\":{\"replication-mode\":\"majority\",\"dr-auto-sync\":{\"label-key\":\"\",\"primary\":\"\",\"dr\":\"\",\"primary-replicas\":0,\"dr-replicas\":0,\"wait-store-timeout\":\"1m0s\",\"wait-sync-timeout\":\"1m0s\",\"wait-async-timeout\":\"2m0s\"}}}"]
[2021/11/27 02:58:54.610 +08:00] [INFO] [server.go:194] ["register REST path"] [path=/pd/api/v1]
[2021/11/27 02:58:54.610 +08:00] [INFO] [server.go:194] ["register REST path"] [path=/swagger/]
[2021/11/27 02:58:54.610 +08:00] [INFO] [server.go:194] ["register REST path"] [path=/autoscaling]
[2021/11/27 02:58:54.613 +08:00] [INFO] [server.go:194] ["register REST path"] [path=/dashboard/api/]
[2021/11/27 02:58:54.613 +08:00] [INFO] [server.go:194] ["register REST path"] [path=/dashboard/]
[2021/11/27 02:58:54.613 +08:00] [INFO] [etcd.go:117] ["configuring peer listeners"] [listen-peer-urls="[https://0.0.0.0:2380]"]
[2021/11/27 02:58:54.614 +08:00] [INFO] [systimemon.go:27] ["start system time monitor"]
[2021/11/27 02:58:54.614 +08:00] [INFO] [etcd.go:465] ["starting with peer TLS"] [tls-info="cert = /home/tidb-deploy/pd-2379/tls/pd.crt, key = /home/tidb-deploy/pd-2379/tls/pd.pem, trusted-ca = /home/tidb-deploy/pd-2379/tls/ca.crt, client-cert-auth = true, crl-file = "] [cipher-suites="[]"]
[2021/11/27 02:58:54.615 +08:00] [INFO] [etcd.go:127] ["configuring client listeners"] [listen-client-urls="[https://0.0.0.0:2379]"]
[2021/11/27 02:58:54.615 +08:00] [INFO] [etcd.go:602] ["pprof is enabled"] [path=/debug/pprof]
[2021/11/27 02:58:54.615 +08:00] [INFO] [etcd.go:299] ["starting an etcd server"] [etcd-version=3.4.3] [git-sha="Not provided (use ./build instead of go build)"] [go-version=go1.16.4] [go-os=linux] [go-arch=amd64] [max-cpu-set=4] [max-cpu-available=4] [member-initialized=false] [name=pd-172.16.7.191-2379] [data-dir=/home/tidb-data/pd-2379] [wal-dir=] [wal-dir-dedicated=] [member-dir=/home/tidb-data/pd-2379/member] [force-new-cluster=false] [heartbeat-interval=500ms] [election-timeout=3s] [initial-election-tick-advance=true] [snapshot-count=100000] [snapshot-catchup-entries=5000] [initial-advertise-peer-urls="[https://172.16.7.191:2380]"] [listen-peer-urls="[https://0.0.0.0:2380]"] [advertise-client-urls="[https://172.16.7.191:2379]"] [listen-client-urls="[https://0.0.0.0:2379]"] [listen-metrics-urls="[]"] [cors="[*]"] [host-whitelist="[*]"] [initial-cluster="pd-172.16.7.147-2379=http://172.16.7.147:2380,pd-172.16.7.191-2379=https://172.16.7.191:2380"] [initial-cluster-state=existing] [initial-cluster-token=pd-cluster] [quota-size-bytes=8589934592] [pre-vote=true] [initial-corrupt-check=false] [corrupt-check-time-interval=0s] [auto-compaction-mode=periodic] [auto-compaction-retention=1h0m0s] [auto-compaction-interval=1h0m0s] [discovery-url=] [discovery-proxy=]
[2021/11/27 02:58:54.618 +08:00] [INFO] [backend.go:79] ["opened backend db"] [path=/home/tidb-data/pd-2379/member/snap/db] [took=1.975156ms]
[2021/11/27 02:58:54.619 +08:00] [WARN] [cluster_util.go:76] ["failed to get cluster response"] [address=http://172.16.7.147:2380/members] [error="Get \"http://172.16.7.147:2380/members\": EOF"]
[2021/11/27 02:58:54.620 +08:00] [INFO] [etcd.go:360] ["closing etcd server"] [name=pd-172.16.7.191-2379] [data-dir=/home/tidb-data/pd-2379] [advertise-peer-urls="[https://172.16.7.191:2380]"] [advertise-client-urls="[https://172.16.7.191:2379]"]
[2021/11/27 02:58:54.620 +08:00] [INFO] [etcd.go:364] ["closed etcd server"] [name=pd-172.16.7.191-2379] [data-dir=/home/tidb-data/pd-2379] [advertise-peer-urls="[https://172.16.7.191:2380]"] [advertise-client-urls="[https://172.16.7.191:2379]"]
[2021/11/27 02:58:54.620 +08:00] [FATAL] [main.go:121] ["run server failed"] [error="[PD:etcd:ErrStartEtcd]cannot fetch cluster info from peer urls: could not retrieve cluster information from the given URLs"] [stack="main.main\
\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:121\
runtime.main\
\t/usr/local/go/src/runtime/proc.go:225"]
#!/bin/bash
set -e
# WARNING: This file was auto-generated. Do not edit!
# All your edit might be overwritten!
DEPLOY_DIR=/home/tidb-deploy/pd-2379
cd "${DEPLOY_DIR}" || exit 1
exec bin/pd-server \
--name="pd-172.16.7.191-2379" \
--client-urls="https://0.0.0.0:2379" \
--advertise-client-urls="https://172.16.7.191:2379" \
--peer-urls="https://0.0.0.0:2380" \
--advertise-peer-urls="https://172.16.7.191:2380" \
--data-dir="/home/tidb-data/pd-2379" \
--join="https://172.16.7.147:2379" \
--config=conf/pd.toml \
--log-file="/home/tidb-deploy/pd-2379/log/pd.log" 2>> "/home/tidb-deploy/pd-2379/log/pd_stderr.log"