TIDB的dashboard Top SQL500报错

🪐 TiDB 技术问题 部署&运维管理
1

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】v6.5.1
【复现路径】做过哪些操作出现的问题
服务器停电重启后
【遇到的问题：问题现象及影响】
想要查看Top SQL 排名，页面上直接提示以下内容
错误
API: /topsql/instances
Request failed with status code 500
【资源配置】
【附件：截图/日志/监控】

image
image1920×931 34.6 KB

2

慢sql太多 去删除 清理一下slow log

3

这个数据是在内存中的吧，重启了就没有了吧

4

已清除了部份show log日志 也删除了部门tidb的log日志， 但问题依旧
image

5

看看pd的日志呢，最好改成debug级别的日志

6

我记得 pd的 leader 切换写就行了。您试一下。 有个类似的帖子

7

有链接吗？

8

切换pd leader：member leader transfer pd3 //pd3为转移新的leader

9

[2023/10/09 08:20:51.213 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.215 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.218 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.220 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.221 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.223 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.225 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.226 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.228 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.230 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.232 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.233 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.235 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.237 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.239 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.240 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.242 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.243 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.245 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.247 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.248 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.250 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.252 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.254 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.256 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.258 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.260 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.261 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.263 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.265 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.267 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.269 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.271 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.273 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.275 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.277 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.279 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.281 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.283 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.285 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.287 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.289 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.291 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.293 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.295 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.298 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.301 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.303 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.305 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.307 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.309 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.311 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.313 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]
[2023/10/09 08:20:51.315 +08:00] [ERROR] [middleware.go:99] [“redirect but server is not leader”] [from=pd-10.20.60.132-2379] [server=pd-10.20.60.133-2379] [error=“[PD:apiutil:ErrRedirect]redirect failed”]

这有些报错的日志

10

现在正常了 不 ？

11

可以按上面兄弟说的，切换pd leader

12

切换leader问题依旧。

下面是切换成功的语句
» member leader transfer pd-10.20.60.133-2379
Success!

13

tiup cluster display cluster名字看下pd和dashboard的状态

14

运行是正常的

15

换PD的leader节点 只是不知道逻辑和原理

16

重启一下pd,然后切换到leader

17

500错误一般是服务器问题，使用tiup cluster display 看下集群各节点是否正常，然后把有问题的节点重新启动下，如果还不行就重启下集群服务试试

18

切换一下pd试一下

19

display所有节点都是绿色正常的，没有问题！~

20

切换了PD的leader,问题还是依旧，重启了整个数据库服务，问题还是依旧，这无从下手，不知道怎么处理。