PD|UI节点状态异常

【 TiDB 使用环境】生产环境
【 TiDB 版本】v5.4.3
【复现路径】强制重启服务器
【遇到的问题:问题现象及影响】
服务器强制重启后,PD|UI节点状态变为down,错误日志如下。集群正常运转,但dashboard站点打不开,这种情况是做PD recover,还是将这个异常节点缩容扩容?

[2023/03/09 18:03:53.326 +08:00] [ERROR] [client.go:171] ["region sync with leader meet error"] [error="[PD:grpc:ErrGRPCRecv]rpc error: code = Unavailable desc = server not started"]
[2023/03/10 00:59:08.601 +08:00] [ERROR] [etcdutil.go:71] ["failed to get cluster from remote"] [error="[PD:etcd:ErrEtcdGetCluster]could not retrieve cluster information from the given URLs"]
[2023/03/10 00:59:18.603 +08:00] [ERROR] [etcdutil.go:71] ["failed to get cluster from remote"] [error="[PD:etcd:ErrEtcdGetCluster]could not retrieve cluster information from the given URLs"]
[2023/03/10 01:12:25.664 +08:00] [ERROR] [etcdutil.go:71] ["failed to get cluster from remote"] [error="[PD:etcd:ErrEtcdGetCluster]could not retrieve cluster information from the given URLs"]
[2023/03/10 01:12:25.684 +08:00] [FATAL] [main.go:122] ["run server failed"] [error="[PD:leveldb:ErrLevelDBOpen]leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000010]"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]
[2023/03/10 01:12:40.945 +08:00] [FATAL] [main.go:122] ["run server failed"] [error="[PD:leveldb:ErrLevelDBOpen]leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000010]"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]
[2023/03/10 01:12:56.447 +08:00] [FATAL] [main.go:122] ["run server failed"] [error="[PD:leveldb:ErrLevelDBOpen]leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000010]"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]
[2023/03/10 01:13:11.950 +08:00] [FATAL] [main.go:122] ["run server failed"] [error="[PD:leveldb:ErrLevelDBOpen]leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000010]"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]
[2023/03/10 01:13:27.372 +08:00] [FATAL] [main.go:122] ["run server failed"] [error="[PD:leveldb:ErrLevelDBOpen]leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000010]"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]
[2023/03/10 01:13:42.876 +08:00] [FATAL] [main.go:122] ["run server failed"] [error="[PD:leveldb:ErrLevelDBOpen]leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000010]"] [stack="main.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]

【资源配置】
【附件:截图/日志/监控】

UI节点就是dashboard节点,L节点就是主节点,如果主节点没问题,只是UI节点down了,直接缩容扩容即可

用tiup把dashboard切走试试

哦吼,还可以直接切么?怎么个操作呀

参考这里,我用测试环境测过可以切走
https://docs.pingcap.com/zh/tidb/stable/dashboard-ops-deploy#切换其他-pd-实例提供-tidb-dashboard-服务

:+1: :+1: :+1:又双叒叕一个小细节~

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。