重启服务器后PD服务无法启动

重启服务器后PD服务无法启动,报错日志如下

[2022/05/31 06:13:53.606 +00:00] [INFO] [util.go:42] ["Welcome to Placement Driver (PD)"]
[2022/05/31 06:13:53.606 +00:00] [INFO] [util.go:43] [PD] [release-version=v5.4.0]
[2022/05/31 06:13:53.606 +00:00] [INFO] [util.go:44] [PD] [edition=Community]
[2022/05/31 06:13:53.606 +00:00] [INFO] [util.go:45] [PD] [git-hash=e3807695b6fc524f9cb84402937e56e733cabd64]
[2022/05/31 06:13:53.606 +00:00] [INFO] [util.go:46] [PD] [git-branch=heads/refs/tags/v5.4.0]
[2022/05/31 06:13:53.606 +00:00] [INFO] [util.go:47] [PD] [utc-build-time="2022-01-17 02:56:38"]
[2022/05/31 06:13:53.606 +00:00] [INFO] [metricutil.go:82] ["disable Prometheus push client"]
[2022/05/31 06:13:53.606 +00:00] [INFO] [server.go:228] ["PD Config"] [config="{\"client-urls\":\"http://0.0.0.0:22379\",\"peer-urls\":\"http://0.0.0.0:2380\",\"advertise-client-urls\":\"http://192.168.2.45:22379\",\"advertise-peer-urls\":\"http://192.168.2.45:2380\",\"name\":\"pd-192.168.2.45-22379\",\"data-dir\":\"/data/tidb-data/pd-22379\",\"force-new-cluster\":false,\"enable-grpc-gateway\":true,\"initial-cluster\":\"pd-192.168.2.45-22379=http://192.168.2.45:2380\",\"initial-cluster-state\":\"new\",\"initial-cluster-token\":\"pd-cluster\",\"join\":\"\",\"lease\":3,\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":false,\"file\":{\"filename\":\"/data/tidb-deploy/pd-22379/log/pd.log\",\"max-size\":300,\"max-days\":0,\"max-backups\":0},\"development\":false,\"disable-caller\":false,\"disable-stacktrace\":false,\"disable-error-verbose\":true,\"sampling\":null},\"tso-save-interval\":\"3s\",\"tso-update-physical-interval\":\"50ms\",\"enable-local-tso\":false,\"metric\":{\"job\":\"pd-192.168.2.45-22379\",\"address\":\"\",\"interval\":\"15s\"},\"schedule\":{\"max-snapshot-count\":64,\"max-pending-peer-count\":64,\"max-merge-region-size\":20,\"max-merge-region-keys\":200000,\"split-merge-interval\":\"1h0m0s\",\"enable-one-way-merge\":\"false\",\"enable-cross-table-merge\":\"true\",\"patrol-region-interval\":\"10ms\",\"max-store-down-time\":\"30m0s\",\"leader-schedule-limit\":4,\"leader-schedule-policy\":\"count\",\"region-schedule-limit\":2048,\"replica-schedule-limit\":64,\"merge-schedule-limit\":8,\"hot-region-schedule-limit\":4,\"hot-region-cache-hits-threshold\":3,\"store-limit\":{},\"tolerant-size-ratio\":0,\"low-space-ratio\":0.8,\"high-space-ratio\":0.7,\"region-score-formula-version\":\"v2\",\"scheduler-max-waiting-operator\":5,\"enable-remove-down-replica\":\"true\",\"enable-replace-offline-replica\":\"true\",\"enable-make-up-replica\":\"true\",\"enable-remove-extra-replica\":\"true\",\"enable-location-replacement\":\"true\",\"enable-debug-metrics\":\"false\",\"enable-joint-consensus\":\"true\",\"schedulers-v2\":[{\"type\":\"balance-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"balance-leader\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"hot-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"}],\"schedulers-payload\":null,\"store-limit-mode\":\"manual\",\"hot-regions-write-interval\":\"10m0s\",\"hot-regions-reserved-days\":7},\"replication\":{\"max-replicas\":3,\"location-labels\":\"\",\"strictly-match-label\":\"false\",\"enable-placement-rules\":\"true\",\"enable-placement-rules-cache\":\"false\",\"isolation-level\":\"\"},\"pd-server\":{\"use-region-storage\":\"true\",\"max-gap-reset-ts\":\"24h0m0s\",\"key-type\":\"table\",\"runtime-services\":\"\",\"metric-storage\":\"\",\"dashboard-address\":\"auto\",\"trace-region-flow\":\"true\",\"flow-round-by-digit\":3},\"cluster-version\":\"0.0.0\",\"labels\":{},\"quota-backend-bytes\":\"8GiB\",\"auto-compaction-mode\":\"periodic\",\"auto-compaction-retention-v2\":\"1h\",\"TickInterval\":\"500ms\",\"ElectionInterval\":\"3s\",\"PreVote\":true,\"max-request-bytes\":1572864,\"security\":{\"cacert-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null,\"SSLCABytes\":null,\"SSLCertBytes\":null,\"SSLKEYBytes\":null,\"redact-info-log\":false,\"encryption\":{\"data-encryption-method\":\"plaintext\",\"data-key-rotation-period\":\"168h0m0s\",\"master-key\":{\"type\":\"plaintext\",\"key-id\":\"\",\"region\":\"\",\"endpoint\":\"\",\"path\":\"\"}}},\"label-property\":null,\"WarningMsgs\":null,\"DisableStrictReconfigCheck\":false,\"HeartbeatStreamBindInterval\":\"1m0s\",\"LeaderPriorityCheckInterval\":\"1m0s\",\"dashboard\":{\"tidb-cacert-path\":\"\",\"tidb-cert-path\":\"\",\"tidb-key-path\":\"\",\"public-path-prefix\":\"\",\"internal-proxy\":false,\"enable-telemetry\":true,\"enable-experimental\":false},\"replication-mode\":{\"replication-mode\":\"majority\",\"dr-auto-sync\":{\"label-key\":\"\",\"primary\":\"\",\"dr\":\"\",\"primary-replicas\":0,\"dr-replicas\":0,\"wait-store-timeout\":\"1m0s\",\"wait-sync-timeout\":\"1m0s\",\"wait-async-timeout\":\"2m0s\"}}}"]
[2022/05/31 06:13:53.610 +00:00] [INFO] [server.go:201] ["register REST path"] [path=/pd/api/v1]
[2022/05/31 06:13:53.610 +00:00] [INFO] [server.go:201] ["register REST path"] [path=/swagger/]
[2022/05/31 06:13:53.610 +00:00] [INFO] [server.go:201] ["register REST path"] [path=/autoscaling]
[2022/05/31 06:13:53.610 +00:00] [INFO] [distro.go:51] ["Using distribution strings"] [strings={}]
[2022/05/31 06:13:53.612 +00:00] [INFO] [server.go:201] ["register REST path"] [path=/dashboard/api/]
[2022/05/31 06:13:53.612 +00:00] [INFO] [server.go:201] ["register REST path"] [path=/dashboard/]
[2022/05/31 06:13:53.613 +00:00] [INFO] [etcd.go:117] ["configuring peer listeners"] [listen-peer-urls="[http://0.0.0.0:2380]"]
[2022/05/31 06:13:53.613 +00:00] [INFO] [systimemon.go:28] ["start system time monitor"]
[2022/05/31 06:13:53.613 +00:00] [INFO] [etcd.go:127] ["configuring client listeners"] [listen-client-urls="[http://0.0.0.0:22379]"]
[2022/05/31 06:13:53.613 +00:00] [INFO] [etcd.go:602] ["pprof is enabled"] [path=/debug/pprof]
[2022/05/31 06:13:53.613 +00:00] [INFO] [etcd.go:299] ["starting an etcd server"] [etcd-version=3.4.3] [git-sha="Not provided (use ./build instead of go build)"] [go-version=go1.16.4] [go-os=linux] [go-arch=amd64] [max-cpu-set=8] [max-cpu-available=8] [member-initialized=true] [name=pd-192.168.2.45-22379] [data-dir=/data/tidb-data/pd-22379] [wal-dir=] [wal-dir-dedicated=] [member-dir=/data/tidb-data/pd-22379/member] [force-new-cluster=false] [heartbeat-interval=500ms] [election-timeout=3s] [initial-election-tick-advance=true] [snapshot-count=100000] [snapshot-catchup-entries=5000] [initial-advertise-peer-urls="[http://192.168.2.45:2380]"] [listen-peer-urls="[http://0.0.0.0:2380]"] [advertise-client-urls="[http://192.168.2.45:22379]"] [listen-client-urls="[http://0.0.0.0:22379]"] [listen-metrics-urls="[]"] [cors="[*]"] [host-whitelist="[*]"] [initial-cluster=] [initial-cluster-state=new] [initial-cluster-token=] [quota-size-bytes=8589934592] [pre-vote=true] [initial-corrupt-check=false] [corrupt-check-time-interval=0s] [auto-compaction-mode=periodic] [auto-compaction-retention=1h0m0s] [auto-compaction-interval=1h0m0s] [discovery-url=] [discovery-proxy=]
[2022/05/31 06:13:53.615 +00:00] [INFO] [backend.go:79] ["opened backend db"] [path=/data/tidb-data/pd-22379/member/snap/db] [took=2.035907ms]
[2022/05/31 06:13:53.616 +00:00] [INFO] [server.go:443] ["recovered v2 store from snapshot"] [snapshot-index=200002] [snapshot-size="7.9 kB"]
[2022/05/31 06:13:53.621 +00:00] [INFO] [kvstore.go:378] ["restored last compact revision"] [meta-bucket-name=meta] [meta-bucket-name-key=finishedCompactRev] [restored-compact-revision=212151]
[2022/05/31 06:13:53.623 +00:00] [INFO] [server.go:461] ["recovered v3 backend from snapshot"] [backend-size-bytes=2244608] [backend-size="2.2 MB"] [backend-size-in-use-bytes=425984] [backend-size-in-use="426 kB"]
[2022/05/31 06:13:53.836 +00:00] [INFO] [raft.go:506] ["restarting local member"] [cluster-id=fc5d6b56a1685a44] [local-member-id=63c641e203a6aef0] [commit-index=178845]
[2022/05/31 06:13:53.836 +00:00] [INFO] [raft.go:1530] ["63c641e203a6aef0 switched to configuration voters=(7189506294123245296)"]
[2022/05/31 06:13:53.836 +00:00] [PANIC] [raft.go:1579] ["63c641e203a6aef0 state.commit 178845 is out of range [200002, 200002]"]
[2022/05/31 06:13:53.836 +00:00] [FATAL] [log.go:294] [panic] [recover="\"63c641e203a6aef0 state.commit 178845 is out of range [200002, 200002]\""] [stack="github.com/tikv/pd/pkg/logutil.LogPanic\
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/logutil/log.go:294\
runtime.gopanic\
\t/usr/local/go/src/runtime/panic.go:965\
go.uber.org/zap/zapcore.(*CheckedEntry).Write\
\t/nfs/cache/mod/go.uber.org/zap@v1.19.0/zapcore/entry.go:232\
go.uber.org/zap.(*SugaredLogger).log\
\t/nfs/cache/mod/go.uber.org/zap@v1.19.0/sugar.go:227\
go.uber.org/zap.(*SugaredLogger).Panicf\
\t/nfs/cache/mod/go.uber.org/zap@v1.19.0/sugar.go:159\
go.etcd.io/etcd/pkg/logutil.(*zapRaftLogger).Panicf\
\t/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/pkg/logutil/zap_raft.go:101\
go.etcd.io/etcd/raft.(*raft).loadState\
\t/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/raft/raft.go:1579\
go.etcd.io/etcd/raft.newRaft\
\t/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/raft/raft.go:371\
go.etcd.io/etcd/raft.NewRawNode\
\t/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/raft/rawnode.go:48\
go.etcd.io/etcd/raft.RestartNode\
\t/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/raft/node.go:235\
go.etcd.io/etcd/etcdserver.restartNode\
\t/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/etcdserver/raft.go:546\
go.etcd.io/etcd/etcdserver.NewServer\
\t/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/etcdserver/server.go:472\
go.etcd.io/etcd/embed.StartEtcd\
\t/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/embed/etcd.go:211\
github.com/tikv/pd/server.(*Server).startEtcd\
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/server.go:276\
github.com/tikv/pd/server.(*Server).Run\
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/server.go:493\
main.main\
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:121\
runtime.main\
\t/usr/local/go/src/runtime/proc.go:225"]

TiDB的版本为5.4.0

在重启之前,做了些什么操作?可以分享一下么

因为之前内存溢出了,所以想重启一下,别的操作没什么,期间没有任何新增SQL的执行。TiDB部署在多台虚拟机内。。

要急救了,PD 恢复

https://docs.pingcap.com/zh/tidb-in-kubernetes/stable/pd-recover#使用-pd-recover-恢复-pd-集群

是不是在你重启的时候还有sql 呀

是正常的重启集群,还是直接重启的机器。