k8s tidb-controller-mananger 持续报错

🪐 TiDB 技术问题 K8s
1

controll manager 报错如下，一直在持续刷日志 集群运行正常

I0112 15:52:55.016078       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"default", Name:"tidb-test-cluster", UID:"5b91a246-829d-42fd-9c9f-fde6a97ed87e", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"1534884", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet tidb-test-cluster-pd in  tidb-test-cluster successful
E0112 15:52:55.966398       1 tidbcluster_control.go:90] failed to update TidbCluster: [default/tidb-test-cluster], error: Operation cannot be fulfilled on tidbclusters.pingcap.com "tidb-test-cluster": StorageError: invalid object, Code: 4, Key: /registry/pingcap.com/tidbclusters/default/tidb-test-cluster, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 5b91a246-829d-42fd-9c9f-fde6a97ed87e, UID in object meta: ae8c4cf6-52c7-42c6-bea6-3dcb9412c050
I0112 15:52:55.966427       1 tidb_cluster_controller.go:131] TidbCluster: default/tidb-test-cluster, still need sync: [TidbCluster: [default/tidb-test-cluster], waiting for PD cluster running, Operation cannot be fulfilled on tidbclusters.pingcap.com "tidb-test-cluster": StorageError: invalid object, Code: 4, Key: /registry/pingcap.com/tidbclusters/default/tidb-test-cluster, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 5b91a246-829d-42fd-9c9f-fde6a97ed87e, UID in object meta: ae8c4cf6-52c7-42c6-bea6-3dcb9412c050], requeuing
I0112 15:52:55.970437       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"default", Name:"tidb-test-cluster", UID:"5b91a246-829d-42fd-9c9f-fde6a97ed87e", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"1534884", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Role/tidb-test-cluster-discovery for controller TidbCluster/tidb-test-cluster successfully
I0112 15:52:55.973719       1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"default", Name:"tidb-test-cluster", UID:"5b91a246-829d-42fd-9c9f-fde6a97ed87e", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"1534884", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ServiceAccount/tidb-test-cluster-discovery for controller TidbCluster/tidb-test-cluster successfully

image
image726×199 5.18 KB

2

你这tikv节点一致在重启啊？没问题吗？

3

tikv 是之前重启的，现在是 通过 官网 添加 注解方式 重启tikv 不生效 ，检查control manager有上面的报错 不知道有没有关系

4

是pd的状态一支不对，一直重新等待调度。
kubectl get tc -n xxx -oyaml
看看status里面，pd那一节怎样？
kubectl get tc -n xxx

ready应该是false吧

6

你K8S 版本多少？

7

提供下 operator 版本和完整的日志，以及 get tc -o yaml 的输出

8

看一下 这个 issues
“前提条件失败：前提条件中的 UID”薄片 ·问题 #82130 ·Kubernetes/Kubernetes (github.com)

CM continues requeuing inexisting items · Issue #4437 · cert-manager/cert-manager (github.com)

9

【版本】
OS：4.19.90-17.ky10.aarch64 、 k8s: v1.24.9 、operator: 1.4.0 、tidb 6.1.2
dyrnq/local-volume-provisioner:v2.5.0
集群看起来应该没问题 ，一直在跑read write测试 ， tc配置见附件
tc.yaml (9.4 KB)

image
image985×251 11.2 KB

image
image897×779 92.2 KB

10

提供下 operator 完整日志看下

11

manager.log (262.9 KB)

12

I0113 03:07:40.258419 1 leaderelection.go:243] attempting to acquire leader lease tidb-admin/tidb-controller-manager…
E0113 03:07:47.285894 1 leaderelection.go:361] Failed to update lock: etcdserver: request timed out

需要先解决 k8s 环境的问题

13

日志中的内容 tidb-controller-manager 在做这些动作时 是直接与api-server通信 获取数据和发送命令吗

14

是的，它就是operator，访问 kube-api
Failed to update lock: Put “https://192.168.0.1:443/api/v1/namespaces/tidb-admin/endpoints/tidb-controller-manager”: context deadline exceeded

15

具体原理参考 https://docs.pingcap.com/zh/tidb-in-kubernetes/stable/architecture 和 https://docs.pingcap.com/zh/tidb-in-kubernetes/stable/tidb-operator-rbac

16

具体原因没看出来，重启了k8s master几个组件后目前都正常

17

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。