tidb k8s模式启动失败

TiDBer_NEw0xuKK · 2022 年10 月 13 日 04:34

tidb之前按照k8s模式安装的，但是卸载之后重装不起来了，启动pd之后tidb没有启动，也不清楚去哪看报错信息，求解答，感谢。
[root@k8s01 tidb]# kubectl get pods |grep tidb
advanced-tidb-discovery-b5b9d8949-7pw8r 1/1 Running 0 27m
advanced-tidb-pd-0 1/1 Running 0 26m
advanced-tidb-pd-1 1/1 Running 0 26m

songxuecheng · 2022 年10 月 13 日 06:21

describe 一下对应的pod. 看看有什么问题

TiDBer_NEw0xuKK · 2022 年10 月 13 日 07:06

tidb没跑起来，pod都没启动呀，我应该看哪个？

TiDBer_NEw0xuKK · 2022 年10 月 13 日 07:07

tidb-discovery有如下报错信息：
[root@k8s01 I1013 12:07:11.901315 I1013 12:07:11.901404 E1013 12:07:11.919592 I1013 12:07:12.002839 I1013 12:07:12.002875 E1013 12:07:13.484565 E1013 12:07:15.798939 E1013 12:07:21.209706 E1013 12:07:28.624194 E1013 12:07:45.288021 E1013 12:08:31.731011 E1013 12:09:13.163522 E1013 12:09:57.242773 E1013 12:10:36.048883 E1013 12:11:12.618300 E1013 12:11:53.456153 E1013 12:12:49.343276 E1013 12:13:28.268032 E1013 12:14:04.477866 E1013 12:14:55.391926 E1013 12:15:26.253610 E1013 12:16:14.483401 E1013 12:16:46.879133 E1013 12:17:18.665177 E1013 12:17:57.723318 E1013 12:18:43.968591 E1013 12:19:22.336408 E1013 12:20:08.266980 ~]# kubectl logs advanced-tidb-discovery-b5b9d8949-7pw8r
1 version.go:38] Welcome to TiDB Operator.
1 version.go:39] TiDB Operator Version: version.Info{GitVersion:“v1.3.1”, GitCommit:“7b170c3c3e4c9d3de8767fba5246eccdd406371b”, GitTreeState:“clean”, BuildDate:“2022-02-24T00:57:55Z”, GoVersion:“go1.13.8”, Compiler:“gc”, Platform:“linux/amd64”}
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 main.go:109] starting TiDB Discovery server, listening on 0.0.0.0:10261
1 main.go:116] starting TiDB Proxy server, listening on 0.0.0.0:10262
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)
1 reflector.go:127] k8s.io/client-go@v0.19.16/tools/cache/reflector.go:156: Failed to watch *v1.Secret: unknown (get secrets)

songxuecheng · 2022 年10 月 13 日 07:15

那就看下pd的log

TiDBer_NEw0xuKK · 2022 年10 月 13 日 07:21

pd0.log (22.1 KB) pd1.log (23.3 KB)
好的，这是pd0和pd1的两个log，麻烦帮忙看下

songxuecheng · 2022 年10 月 13 日 07:40

k8s 节点个数
看下kubelet日志
pvc pv 截图

TiDBer_NEw0xuKK · 2022 年10 月 13 日 07:51

5个节点，k8s04打了污点

kubelet日志：
kubelet.log (339.7 KB)
pvc：

pv:

songxuecheng · 2022 年10 月 13 日 07:58

pvc pv 没有绑定tidb的资源了

TiDBer_NEw0xuKK · 2022 年10 月 13 日 08:02

您是说这个地方吗？

那我是不是在配置文件里面把需要的磁盘改大点就行了

songxuecheng · 2022 年10 月 13 日 08:13

需要没有绑定的pvc

TiDBer_NEw0xuKK · 2022 年10 月 13 日 08:32

我不是很理解，具体需要怎么做呢，麻烦了

TiDBer_NEw0xuKK · 2022 年10 月 13 日 08:52

是要手动创建一个pvc吗

wuxiangdong · 2022 年10 月 13 日 10:43

describe看下
kubectl describe tidbcluster basic -n tidb-cluster

TiDBer_NEw0xuKK · 2022 年10 月 13 日 12:18

tidb.yaml (8.2 KB)

TiDBer_NEw0xuKK · 2022 年10 月 13 日 12:18

发您了，麻烦看下，谢谢

wuxiangdong · 2022 年10 月 14 日 01:55

local-volume-provisioner-xxxx这个pod的日志看一下

wuxiangdong · 2022 年10 月 14 日 02:29

几个pv是211天前的,把pv清理了，重来一次

TiDBer_NEw0xuKK · 2022 年10 月 14 日 04:16

pv删了数据不是没了吗？
provisioner日志：

另外我昨天成功跑起来了，发现把tiflash注释掉就能正常启动，tiflash是新加的，之前没有运行，是不是因为启动了tiflash导致tikv启动失败？
这是我集群的配置，麻烦帮忙看下
tidb-cluster-local.yaml (36.5 KB)

wuxiangdong · 2022 年10 月 14 日 09:20

kubectl describe tc advanced-tidb
这个看看