K8s v1.18部署TiDB 集群失败_(:ι」∠)_

Hacker_sAKN3wef · 2020 年5 月 25 日 01:51

麻烦拿一下 tidb-controller-manager Pod 和 PD Pod 的 log。

如果 Operator 是 1.0，请参考 1.0 文档 https://pingcap.com/docs-cn/tidb-in-kubernetes/v1.0/deploy-on-general-kubernetes/。如果 Operator 是 1.1 版本不建议再使用 Helm 部署集群，参考文档 https://pingcap.com/docs-cn/tidb-in-kubernetes/stable/deploy-on-general-kubernetes/ 直接使用 kubectl apply -f tidb-cluster.yaml。

yunlingfly · 2020 年5 月 25 日 01:54

log 如下

# kubectl describe pod tidb-controller-manager -n tidb-admin
Name:         tidb-controller-manager-7bcff6964b-2c265
Namespace:    tidb-admin
Priority:     0
Node:         acdm1/192.168.128.51
Start Time:   Thu, 21 May 2020 16:15:25 +0800
Labels:       app.kubernetes.io/component=controller-manager
              app.kubernetes.io/instance=tidb-operator
              app.kubernetes.io/name=tidb-operator
              pod-template-hash=7bcff6964b
Annotations:  cni.projectcalico.org/podIP: 10.100.137.197/32
              cni.projectcalico.org/podIPs: 10.100.137.197/32
Status:       Running
IP:           10.100.137.197
IPs:
  IP:           10.100.137.197
Controlled By:  ReplicaSet/tidb-controller-manager-7bcff6964b
Containers:
  tidb-operator:
    Container ID:  docker://40575c970f731241d2dfd8da3b5b7e9bdae4b589df0d0a99ee2c7f0524d7104e
    Image:         pingcap/tidb-operator:v1.0.6
    Image ID:      docker-pullable://pingcap/tidb-operator@sha256:87d88143eda1b2c1f88d063b2ade9e412251320a89702ab7212724b969558c54
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/local/bin/tidb-controller-manager
      -default-storage-class-name=local-storage
      -cluster-scoped=true
      -auto-failover=true
      -pd-failover-period=5m
      -tikv-failover-period=5m
      -tidb-failover-period=5m
      -v=2
    State:          Running
      Started:      Fri, 22 May 2020 14:53:53 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Fri, 22 May 2020 10:42:33 +0800
      Finished:     Fri, 22 May 2020 14:53:51 +0800
    Ready:          True
    Restart Count:  2
    Limits:
      cpu:     250m
      memory:  150Mi
    Requests:
      cpu:     80m
      memory:  50Mi
    Environment:
      NAMESPACE:  tidb-admin (v1:metadata.namespace)
      TZ:         UTC
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from tidb-controller-manager-token-kvbfk (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  tidb-controller-manager-token-kvbfk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  tidb-controller-manager-token-kvbfk
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

Hacker_sAKN3wef · 2020 年5 月 25 日 02:49

kubectl logs -n <ns> <podName>

请使用上面命令拿一下 tidb-controller-manager 和 PD Pod 的 log。

yunlingfly · 2020 年5 月 25 日 02:59

分别如下：

# kubectl logs -n tidb-admin tidb-controller-manager-7bcff6964b-2c265
I0525 02:02:44.524388       1 version.go:38] Welcome to TiDB Operator.
I0525 02:02:44.524415       1 version.go:39] TiDB Operator Version: version.Info{GitVersion:"v1.0.6", G                          itCommit:"982720cd563ece6dbebfc4c579b17fa66a93c550", GitTreeState:"clean", BuildDate:"2019-12-27T16:53:                          41Z", GoVersion:"go1.13", Compiler:"gc", Platform:"linux/amd64"}
E0525 02:03:14.527570       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limi                          t=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:14.527688       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?limit=500&res                          ourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:14.527651       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.PersistentVolume: Get https://10.96.0.1:443/api/v1/persist                          entvolumes?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:14.527585       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1alpha1.TidbCluster: Get https://10.96.0.1:443/apis/pingcap.                          com/v1alpha1/tidbclusters?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:14.528039       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=5                          00&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:14.528042       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.PersistentVolumeClaim: Get https://10.96.0.1:443/api/v1/pe                          rsistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:14.528228       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.StatefulSet: Get https://10.96.0.1:443/apis/apps/v1/statef                          ulsets?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:14.530130       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/pods?limit=500&resou                          rceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:45.528944       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limi                          t=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:45.530625       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?limit=500&res                          ourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:45.532380       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.PersistentVolume: Get https://10.96.0.1:443/api/v1/persist                          entvolumes?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:45.533830       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1alpha1.TidbCluster: Get https://10.96.0.1:443/apis/pingcap.                          com/v1alpha1/tidbclusters?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:45.534866       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=5                          00&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:45.536167       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.PersistentVolumeClaim: Get https://10.96.0.1:443/api/v1/pe                          rsistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:45.537252       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.StatefulSet: Get https://10.96.0.1:443/apis/apps/v1/statef                          ulsets?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:03:45.538483       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/pods?limit=500&resou                          rceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:04:16.529960       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limi                          t=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:04:16.532914       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?limit=500&res                          ourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:04:16.534642       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1alpha1.TidbCluster: Get https://10.96.0.1:443/apis/pingcap.                          com/v1alpha1/tidbclusters?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:04:16.535585       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=5                          00&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:04:16.536785       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.PersistentVolumeClaim: Get https://10.96.0.1:443/api/v1/pe                          rsistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:04:16.537227       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.PersistentVolume: Get https://10.96.0.1:443/api/v1/persist                          entvolumes?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:04:16.537805       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.StatefulSet: Get https://10.96.0.1:443/apis/apps/v1/statef                          ulsets?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:04:16.539146       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/pods?limit=500&resou                          rceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E0525 02:04:18.531638       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limi                          t=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
E0525 02:04:20.535560       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limi                          t=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
E0525 02:04:22.539388       1 reflector.go:134] k8s.io/client-go@v0.0.0-20190115164855-701b91367003/too                          ls/cache/reflector.go:95: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limi                          t=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
I0525 02:04:25.918056       1 main.go:152] cache of informer factories sync successfully
I0525 02:04:25.918690       1 leaderelection.go:205] attempting to acquire leader lease  tidb-admin/tid                          b-controller-manager...
I0525 02:04:26.009549       1 leaderelection.go:214] successfully acquired lease tidb-admin/tidb-contro                          ller-manager
I0525 02:04:26.009689       1 tidb_cluster_controller.go:216] Starting tidbcluster controller
E0525 02:04:31.011177       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:04:31.013282       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:04:36.019250       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:04:36.019985       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:04:41.031124       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:04:41.031899       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:04:46.052898       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:04:46.053611       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:04:51.094565       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:04:51.095000       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:04:56.175894       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:04:56.176221       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:05:01.176820       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:05:01.177594       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:05:06.024606       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: dial tcp: lookup tidb                          -cluster-pd.tidb-cluster on 10.96.0.10:53: read udp 10.100.137.197:49274->10.96.0.10:53: i/o timeout
E0525 02:05:06.025441       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:05:11.026641       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:05:11.027658       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:05:16.028923       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:05:16.029826       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:05:21.030952       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:05:21.031803       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:05:26.032910       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:05:26.033580       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:05:31.034561       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:05:31.035540       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:05:36.036526       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:05:36.037575       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:05:56.516386       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:05:56.517319       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:06:01.518343       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:06:01.519306       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:06:30.021843       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:06:30.022815       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:07:00.021969       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:07:00.022976       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:07:23.438798       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:07:23.439840       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:07:30.022310       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:07:30.023263       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:08:00.022562       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:08:00.023822       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:08:30.022932       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:08:30.023992       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:09:00.023287       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:09:00.024402       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:09:30.023957       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:09:30.025031       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:10:00.023840       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:10:00.024867       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:10:30.024323       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:10:30.025385       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:11:00.023839       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:11:00.024875       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:11:30.024765       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:11:30.025751       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:12:00.025308       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:12:00.038192       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:12:30.025331       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:12:30.026418       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:13:00.025486       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:13:00.026731       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:13:30.026051       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:13:30.026492       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:14:00.026006       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:14:00.027235       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:14:30.026546       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:14:30.027542       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:15:00.026938       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:15:00.027941       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:15:30.026908       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:15:30.028299       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:16:00.027337       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:16:00.027669       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:16:30.027661       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:16:30.028688       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:17:00.027599       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:17:00.030491       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 02:17:30.028324       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:17:30.029355       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
I0525 02:29:35.296605       1 event.go:221] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"def                          ault", Name:"basic", UID:"88cc53ea-3eef-405d-b1fb-9ee883efba22", APIVersion:"pingcap.com/v1alpha1", Res                          ourceVersion:"1319291", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service basic-                          pd in TidbCluster basic successful
I0525 02:29:35.327145       1 event.go:221] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"def                          ault", Name:"basic", UID:"88cc53ea-3eef-405d-b1fb-9ee883efba22", APIVersion:"pingcap.com/v1alpha1", Res                          ourceVersion:"1319291", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service basic-                          pd-peer in TidbCluster basic successful
I0525 02:29:35.370036       1 event.go:221] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"def                          ault", Name:"basic", UID:"88cc53ea-3eef-405d-b1fb-9ee883efba22", APIVersion:"pingcap.com/v1alpha1", Res                          ourceVersion:"1319291", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet ba                          sic-pd in TidbCluster basic successful
I0525 02:29:35.496872       1 tidbcluster_control.go:68] TidbCluster: [default/basic] updated successfu                          lly
I0525 02:29:35.497048       1 tidb_cluster_controller.go:243] TidbCluster: default/basic, still need sy                          nc: TidbCluster: [default/basic], waiting for PD cluster running, requeuing
I0525 02:29:35.497242       1 event.go:221] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"def                          ault", Name:"basic", UID:"88cc53ea-3eef-405d-b1fb-9ee883efba22", APIVersion:"pingcap.com/v1alpha1", Res                          ourceVersion:"1319291", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update TidbCluster ba                          sic successful
E0525 02:29:40.498577       1 pd_member_manager.go:200] failed to sync TidbCluster: [default/basic]'s s                          tatus, error: Get http://basic-pd.default:2379/pd/health: net/http: request canceled while waiting for                           connection (Client.Timeout exceeded while awaiting headers), service default/basic-pd has no endpoints
E0525 02:29:40.536028       1 tidbcluster_control.go:71] failed to update TidbCluster: [default/basic],                           error: Operation cannot be fulfilled on tidbclusters.pingcap.com "basic": the object has been modified                          ; please apply your changes to the latest version and try again
I0525 02:29:40.584375       1 tidbcluster_control.go:68] TidbCluster: [default/basic] updated successfu                          lly
I0525 02:29:40.584444       1 tidb_cluster_controller.go:243] TidbCluster: default/basic, still need sy                          nc: TidbCluster: [default/basic], waiting for PD cluster running, requeuing
I0525 02:29:40.584523       1 event.go:221] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"def                          ault", Name:"basic", UID:"88cc53ea-3eef-405d-b1fb-9ee883efba22", APIVersion:"pingcap.com/v1alpha1", Res                          ourceVersion:"1319308", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update TidbCluster ba                          sic successful
E0525 02:29:45.585153       1 pd_member_manager.go:200] failed to sync TidbCluster: [default/basic]'s s                          tatus, error: Get http://basic-pd.default:2379/pd/health: net/http: request canceled while waiting for                           connection (Client.Timeout exceeded while awaiting headers), service default/basic-pd has no endpoints
E0525 02:29:45.599256       1 tidbcluster_control.go:71] failed to update TidbCluster: [default/basic],                           error: Operation cannot be fulfilled on tidbclusters.pingcap.com "basic": the object has been modified                          ; please apply your changes to the latest version and try again
I0525 02:29:45.621667       1 tidbcluster_control.go:68] TidbCluster: [default/basic] updated successfu                          lly
I0525 02:29:45.621822       1 tidb_cluster_controller.go:243] TidbCluster: default/basic, still need sy                          nc: TidbCluster: [default/basic], waiting for PD cluster running, requeuing
I0525 02:29:45.622047       1 event.go:221] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"def                          ault", Name:"basic", UID:"88cc53ea-3eef-405d-b1fb-9ee883efba22", APIVersion:"pingcap.com/v1alpha1", Res                          ourceVersion:"1319344", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update TidbCluster ba                          sic successful
I0525 02:53:57.883048       1 event.go:221] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"tid                          b-cluster", Name:"tidb-cluster", UID:"5057586a-6904-41d8-92c7-2e79a81869b9", APIVersion:"pingcap.com/v1                          alpha1", ResourceVersion:"1325402", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update Ti                          dbCluster tidb-cluster successful
E0525 02:54:02.884141       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-                          cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request can                          celed while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 02:54:02.884956       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, s                          ync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
[root@acdm2 ~]#

# kubectl logs -n tidb-cluster tidb-cluster-pd-0

nslookup domain tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc failed

Name:      tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc
Address 1: 10.100.30.23 tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc.cluster.local
nslookup domain tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc.svc success
waiting for discovery service to return start args ...
waiting for discovery service to return start args ...
waiting for discovery service to return start args ...

yunlingfly · 2020 年5 月 25 日 05:21

我发现执行下面的命令看不到TiKv?

# kubectl get services --namespace tidb-cluster -l app.kubernetes.io/instance=tidb-cluster
NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
tidb-cluster-discovery          ClusterIP   10.96.105.56    <none>        10261/TCP                        52m
tidb-cluster-grafana            NodePort    10.96.106.182   <none>        3000:31099/TCP                   52m
tidb-cluster-monitor-reloader   NodePort    10.96.76.226    <none>        9089:32380/TCP                   52m
tidb-cluster-pd                 ClusterIP   10.96.213.155   <none>        2379/TCP                         52m
tidb-cluster-pd-peer            ClusterIP   None            <none>        2380/TCP                         52m
tidb-cluster-prometheus         NodePort    10.96.82.67     <none>        9090:31466/TCP                   52m
tidb-cluster-tidb               NodePort    10.96.184.237   <none>        4000:31062/TCP,10080:30658/TCP   52m

Hacker_sAKN3wef · 2020 年5 月 25 日 05:41

麻烦拿一下 discovery Pod 的log
kubectl get tc tidb-cluster -n tidb-cluster -o yaml
kubectl get svc tidb-cluster-pd-peer -n tidb-cluster -o yaml

yunlingfly · 2020 年5 月 25 日 05:48

pvc没有看到tidb-cluster-pd-peer，只有pd-tidb-cluster-pd-0

# kubectl get tc tidb-cluster -n tidb-cluster -o yaml
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  annotations:
    pingcap.com/pd.tidb-cluster-pd.sha: cfa0d77a
    pingcap.com/tidb.tidb-cluster-tidb.sha: a4c4bb14
    pingcap.com/tikv.tidb-cluster-tikv.sha: 3a588779
  creationTimestamp: "2020-05-25T03:09:25Z"
  generation: 5
  labels:
    app.kubernetes.io/component: tidb-cluster
    app.kubernetes.io/instance: tidb-cluster
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: tidb-cluster
    helm.sh/chart: tidb-cluster-v1.0.6
  managedFields:
  - apiVersion: pingcap.com/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:pingcap.com/pd.tidb-cluster-pd.sha: {}
          f:pingcap.com/tidb.tidb-cluster-tidb.sha: {}
          f:pingcap.com/tikv.tidb-cluster-tikv.sha: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/component: {}
          f:app.kubernetes.io/instance: {}
          f:app.kubernetes.io/managed-by: {}
          f:app.kubernetes.io/name: {}
          f:helm.sh/chart: {}
      f:spec:
        .: {}
        f:pd:
          .: {}
          f:affinity: {}
          f:image: {}
          f:imagePullPolicy: {}
          f:limits: {}
          f:replicas: {}
          f:requests:
            .: {}
            f:storage: {}
          f:storageClassName: {}
        f:pvReclaimPolicy: {}
        f:schedulerName: {}
        f:services: {}
        f:tidb:
          .: {}
          f:affinity: {}
          f:image: {}
          f:imagePullPolicy: {}
          f:limits: {}
          f:maxFailoverCount: {}
          f:replicas: {}
          f:requests: {}
          f:separateSlowLog: {}
          f:slowLogTailer:
            .: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:limits:
              .: {}
              f:cpu: {}
              f:memory: {}
            f:requests:
              .: {}
              f:cpu: {}
              f:memory: {}
        f:tikv:
          .: {}
          f:affinity: {}
          f:image: {}
          f:imagePullPolicy: {}
          f:limits: {}
          f:maxFailoverCount: {}
          f:replicas: {}
          f:requests:
            .: {}
            f:storage: {}
          f:storageClassName: {}
        f:timezone: {}
    manager: Go-http-client
    operation: Update
    time: "2020-05-25T03:09:25Z"
  - apiVersion: pingcap.com/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:tikvPromGateway:
          .: {}
          f:image: {}
      f:status:
        .: {}
        f:pd:
          .: {}
          f:leader:
            .: {}
            f:clientURL: {}
            f:health: {}
            f:id: {}
            f:lastTransitionTime: {}
            f:name: {}
          f:phase: {}
          f:statefulSet:
            .: {}
            f:collisionCount: {}
            f:currentReplicas: {}
            f:currentRevision: {}
            f:observedGeneration: {}
            f:readyReplicas: {}
            f:replicas: {}
            f:updateRevision: {}
            f:updatedReplicas: {}
        f:tidb: {}
        f:tikv: {}
    manager: tidb-controller-manager
    operation: Update
    time: "2020-05-25T03:09:41Z"
  name: tidb-cluster
  namespace: tidb-cluster
  resourceVersion: "1329730"
  selfLink: /apis/pingcap.com/v1alpha1/namespaces/tidb-cluster/tidbclusters/tidb-cluster
  uid: 0a726b56-cfb0-4790-90ed-ef5c0b7b3a82
spec:
  pd:
    affinity: {}
    image: pingcap/pd:v3.0.5
    imagePullPolicy: IfNotPresent
    limits: {}
    replicas: 1
    requests:
      storage: 1Gi
    storageClassName: nfs
  pvReclaimPolicy: Retain
  schedulerName: tidb-scheduler
  services:
  - name: pd
    type: ClusterIP
  tidb:
    affinity: {}
    image: pingcap/tidb:v3.0.5
    imagePullPolicy: IfNotPresent
    limits: {}
    maxFailoverCount: 3
    replicas: 2
    requests: {}
    separateSlowLog: true
    slowLogTailer:
      image: busybox:1.26.2
      imagePullPolicy: IfNotPresent
      limits:
        cpu: 100m
        memory: 50Mi
      requests:
        cpu: 20m
        memory: 5Mi
  tikv:
    affinity: {}
    image: pingcap/tikv:v3.0.5
    imagePullPolicy: IfNotPresent
    limits: {}
    maxFailoverCount: 3
    replicas: 1
    requests:
      storage: 10Gi
    storageClassName: nfs
  tikvPromGateway:
    image: ""
  timezone: UTC
status:
  pd:
    leader:
      clientURL: ""
      health: false
      id: ""
      lastTransitionTime: null
      name: ""
    phase: Normal
    statefulSet:
      collisionCount: 0
      currentReplicas: 1
      currentRevision: tidb-cluster-pd-7d645d5d98
      observedGeneration: 1
      readyReplicas: 1
      replicas: 1
      updateRevision: tidb-cluster-pd-7d645d5d98
      updatedReplicas: 1
  tidb: {}
  tikv: {}

# kubectl get pvc pd-tidb-cluster-pd-0 -n tidb-cluster -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    tidb.pingcap.com/pod-scheduling: "2020-05-25T03:09:26Z"
  creationTimestamp: "2020-05-25T03:09:26Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/component: pd
    app.kubernetes.io/instance: tidb-cluster
    app.kubernetes.io/managed-by: tidb-operator
    app.kubernetes.io/name: tidb-cluster
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:tidb.pingcap.com/pod-scheduling: {}
    manager: tidb-scheduler
    operation: Update
    time: "2020-05-25T03:09:26Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:pv.kubernetes.io/bind-completed: {}
          f:pv.kubernetes.io/bound-by-controller: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/component: {}
          f:app.kubernetes.io/instance: {}
          f:app.kubernetes.io/managed-by: {}
          f:app.kubernetes.io/name: {}
      f:spec:
        f:accessModes: {}
        f:resources:
          f:requests:
            .: {}
            f:storage: {}
        f:storageClassName: {}
        f:volumeMode: {}
        f:volumeName: {}
      f:status:
        f:accessModes: {}
        f:capacity:
          .: {}
          f:storage: {}
        f:phase: {}
    manager: kube-controller-manager
    operation: Update
    time: "2020-05-25T03:09:27Z"
  name: pd-tidb-cluster-pd-0
  namespace: tidb-cluster
  resourceVersion: "1329610"
  selfLink: /api/v1/namespaces/tidb-cluster/persistentvolumeclaims/pd-tidb-cluster-pd-0
  uid: cea32df4-57bd-4ed1-8df1-cce5dcf28574
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs
  volumeMode: Filesystem
  volumeName: nfs-pv9
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound

yunlingfly · 2020 年5 月 25 日 05:50

tidb-cluster-pd-peer service如下

# kubectl get service tidb-cluster-pd-peer -n tidb-cluster -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    pingcap.com/last-applied-configuration: '{"ports":[{"name":"peer","protocol":"TCP","port":2380,"targetPort":2380}],"selector":{"app.kubernetes.io/component":"pd","app.kubernetes.io/instance":"tidb-cluster","app.kubernetes.io/managed-by":"tidb-operator","app.kubernetes.io/name":"tidb-cluster"},"clusterIP":"None"}'
  creationTimestamp: "2020-05-25T03:09:25Z"
  labels:
    app.kubernetes.io/component: pd
    app.kubernetes.io/instance: tidb-cluster
    app.kubernetes.io/managed-by: tidb-operator
    app.kubernetes.io/name: tidb-cluster
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:pingcap.com/last-applied-configuration: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/component: {}
          f:app.kubernetes.io/instance: {}
          f:app.kubernetes.io/managed-by: {}
          f:app.kubernetes.io/name: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"0a726b56-cfb0-4790-90ed-ef5c0b7b3a82"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:clusterIP: {}
        f:ports:
          .: {}
          k:{"port":2380,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app.kubernetes.io/component: {}
          f:app.kubernetes.io/instance: {}
          f:app.kubernetes.io/managed-by: {}
          f:app.kubernetes.io/name: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: tidb-controller-manager
    operation: Update
    time: "2020-05-25T03:09:25Z"
  name: tidb-cluster-pd-peer
  namespace: tidb-cluster
  ownerReferences:
  - apiVersion: pingcap.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: TidbCluster
    name: tidb-cluster
    uid: 0a726b56-cfb0-4790-90ed-ef5c0b7b3a82
  resourceVersion: "1329575"
  selfLink: /api/v1/namespaces/tidb-cluster/services/tidb-cluster-pd-peer
  uid: 8233821e-ae88-451a-a598-7e3529e179a3
spec:
  clusterIP: None
  ports:
  - name: peer
    port: 2380
    protocol: TCP
    targetPort: 2380
  selector:
    app.kubernetes.io/component: pd
    app.kubernetes.io/instance: tidb-cluster
    app.kubernetes.io/managed-by: tidb-operator
    app.kubernetes.io/name: tidb-cluster
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Hacker_sAKN3wef · 2020 年5 月 25 日 05:53

麻烦拿一下 tidb-cluster-discovery Pod 的log

yunlingfly · 2020 年5 月 25 日 05:56

# kubectl logs -n tidb-cluster tidb-cluster-discovery-7d4cfc7f4d-6vsjb
I0525 03:09:30.198235       1 version.go:38] Welcome to TiDB Operator.
I0525 03:09:30.198670       1 version.go:39] TiDB Operator Version: version.Info{GitVersion:"v1.0.6", GitCommit:"982720cd563ece6dbebfc4c579b17fa66a93c550", GitTreeState:"clean", BuildDate:"2019-12-27T16:53:46Z", GoVersion:"go1.13", Compiler:"gc", Platform:"linux/amd64"}
I0525 03:09:30.207801       1 mux.go:40] starting TiDB Discovery server, listening on 0.0.0.0:10261
I0525 03:09:32.958181       1 discovery.go:65] advertisePeerUrl is: tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2380
I0525 03:09:33.010968       1 mux.go:65] generated args for tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2380
: --initial-cluster=tidb-cluster-pd-0=http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2380

Hacker_sAKN3wef · 2020 年5 月 25 日 06:09

尝试把 PD Pod 删一下，然后再看下 PD log。

yunlingfly · 2020 年5 月 25 日 06:12

信息如下：

# kubectl delete pod tidb-cluster-pd-0 -n tidb-cluster
pod "tidb-cluster-pd-0" deleted
[root@acdm2 ~]# kubectl get pods -n tidb-cluster
NAME                                      READY   STATUS    RESTARTS   AGE
tidb-cluster-discovery-7d4cfc7f4d-6vsjb   1/1     Running   1          3h1m
tidb-cluster-monitor-588fd6bcd5-7n724     3/3     Running   3          3h1m
tidb-cluster-pd-0                         1/1     Running   0          31s
[root@acdm2 ~]# kubectl logs -n tidb-cluster tidb-cluster-pd-0

nslookup domain tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc failed

Name:      tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc
Address 1: 10.100.30.31 tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc.cluster.local
nslookup domain tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc.svc success
starting pd-server ...
/pd-server --data-dir=/var/lib/pd --name=tidb-cluster-pd-0 --peer-urls=http://0.0.0.0:2380 --advertise-peer-urls=http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2380 --client-urls=http://0.0.0.0:2379 --advertise-client-urls=http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2379 --config=/etc/pd/pd.toml
[2020/05/25 06:10:47.356 +00:00] [INFO] [util.go:59] ["Welcome to Placement Driver (PD)"]
[2020/05/25 06:10:47.356 +00:00] [INFO] [util.go:60] [PD] [release-version=v3.0.5]
[2020/05/25 06:10:47.356 +00:00] [INFO] [util.go:61] [PD] [git-hash=4934a651d2d428411a9610a5b6b9d7156a439355]
[2020/05/25 06:10:47.356 +00:00] [INFO] [util.go:62] [PD] [git-branch=HEAD]
[2020/05/25 06:10:47.356 +00:00] [INFO] [util.go:63] [PD] [utc-build-time="2019-10-25 03:24:57"]
[2020/05/25 06:10:47.356 +00:00] [INFO] [metricutil.go:80] ["disable Prometheus push client"]
[2020/05/25 06:10:47.356 +00:00] [INFO] [server.go:110] ["PD Config"] [config="{\"client-urls\":\"http://0.0.0.0:2379\",\"peer-urls\":\"http://0.0.0.0:2380\",\"advertise-client-urls\":\"http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2379\",\"advertise-peer-urls\":\"http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2380\",\"name\":\"tidb-cluster-pd-0\",\"data-dir\":\"/var/lib/pd\",\"force-new-cluster\":false,\"enable-grpc-gateway\":true,\"initial-cluster\":\"tidb-cluster-pd-0=http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2380\",\"initial-cluster-state\":\"new\",\"join\":\"\",\"lease\":3,\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":false,\"file\":{\"filename\":\"\",\"log-rotate\":true,\"max-size\":0,\"max-days\":0,\"max-backups\":0},\"development\":false,\"disable-caller\":false,\"disable-stacktrace\":false,\"disable-error-verbose\":true,\"sampling\":null},\"log-file\":\"\",\"log-level\":\"\",\"tso-save-interval\":\"3s\",\"metric\":{\"job\":\"tidb-cluster-pd-0\",\"address\":\"\",\"interval\":\"15s\"},\"schedule\":{\"max-snapshot-count\":3,\"max-pending-peer-count\":16,\"max-merge-region-size\":20,\"max-merge-region-keys\":200000,\"split-merge-interval\":\"1h0m0s\",\"enable-one-way-merge\":\"false\",\"patrol-region-interval\":\"100ms\",\"max-store-down-time\":\"30m0s\",\"leader-schedule-limit\":4,\"region-schedule-limit\":64,\"replica-schedule-limit\":64,\"merge-schedule-limit\":8,\"hot-region-schedule-limit\":4,\"hot-region-cache-hits-threshold\":3,\"store-balance-rate\":15,\"tolerant-size-ratio\":0,\"low-space-ratio\":0.8,\"high-space-ratio\":0.6,\"scheduler-max-waiting-operator\":3,\"disable-raft-learner\":\"false\",\"disable-remove-down-replica\":\"false\",\"disable-replace-offline-replica\":\"false\",\"disable-make-up-replica\":\"false\",\"disable-remove-extra-replica\":\"false\",\"disable-location-replacement\":\"false\",\"disable-namespace-relocation\":\"false\",\"schedulers-v2\":[{\"type\":\"balance-region\",\"args\":null,\"disable\":false},{\"type\":\"balance-leader\",\"args\":null,\"disable\":false},{\"type\":\"hot-region\",\"args\":null,\"disable\":false},{\"type\":\"label\",\"args\":null,\"disable\":false}]},\"replication\":{\"max-replicas\":3,\"location-labels\":\"region,zone,rack,host\",\"strictly-match-label\":\"false\"},\"namespace\":{},\"pd-server\":{\"use-region-storage\":\"true\"},\"cluster-version\":\"0.0.0\",\"quota-backend-bytes\":\"0B\",\"auto-compaction-mode\":\"periodic\",\"auto-compaction-retention-v2\":\"1h\",\"TickInterval\":\"500ms\",\"ElectionInterval\":\"3s\",\"PreVote\":true,\"security\":{\"cacert-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\"},\"label-property\":null,\"WarningMsgs\":null,\"namespace-classifier\":\"table\",\"LeaderPriorityCheckInterval\":\"1m0s\"}"]
[2020/05/25 06:10:47.359 +00:00] [INFO] [server.go:145] ["start embed etcd"]
[2020/05/25 06:10:47.359 +00:00] [INFO] [etcd.go:117] ["configuring peer listeners"] [listen-peer-urls="[http://0.0.0.0:2380]"]
[2020/05/25 06:10:47.359 +00:00] [INFO] [systime_mon.go:25] ["start system time monitor"]
[2020/05/25 06:10:47.359 +00:00] [INFO] [etcd.go:127] ["configuring client listeners"] [listen-client-urls="[http://0.0.0.0:2379]"]
[2020/05/25 06:10:47.359 +00:00] [INFO] [etcd.go:600] ["pprof is enabled"] [path=/debug/pprof]
[2020/05/25 06:10:47.360 +00:00] [INFO] [etcd.go:297] ["starting an etcd server"] [etcd-version=3.3.0+git] [git-sha="Not provided (use ./build instead of go build)"] [go-version=go1.12] [go-os=linux] [go-arch=amd64] [max-cpu-set=4] [max-cpu-available=4] [member-initialized=true] [name=tidb-cluster-pd-0] [data-dir=/var/lib/pd] [wal-dir=] [wal-dir-dedicated=] [member-dir=/var/lib/pd/member] [force-new-cluster=false] [heartbeat-interval=500ms] [election-timeout=3s] [initial-election-tick-advance=true] [snapshot-count=100000] [snapshot-catchup-entries=5000] [initial-advertise-peer-urls="[http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2380]"] [listen-peer-urls="[http://0.0.0.0:2380]"] [advertise-client-urls="[http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2379]"] [listen-client-urls="[http://0.0.0.0:2379]"] [listen-metrics-urls="[]"] [cors="[*]"] [host-whitelist="[*]"] [initial-cluster=] [initial-cluster-state=new] [initial-cluster-token=] [quota-size-bytes=2147483648] [pre-vote=true] [initial-corrupt-check=false] [corrupt-check-time-interval=0s] [auto-compaction-mode=periodic] [auto-compaction-retention=1h0m0s] [auto-compaction-interval=1h0m0s] [discovery-url=] [discovery-proxy=]
[2020/05/25 06:10:47.364 +00:00] [INFO] [backend.go:79] ["opened backend db"] [path=/var/lib/pd/member/snap/db] [took=2.80044ms]
[2020/05/25 06:10:47.381 +00:00] [INFO] [raft.go:496] ["restarting local member"] [cluster-id=4efd7a79829dbb9d] [local-member-id=7927c1c88369b5b0] [commit-index=3596]
[2020/05/25 06:10:47.382 +00:00] [INFO] [raft.go:712] ["7927c1c88369b5b0 became follower at term 3"]
[2020/05/25 06:10:47.382 +00:00] [INFO] [raft.go:389] ["newRaft 7927c1c88369b5b0 [peers: [], term: 3, commit: 3596, applied: 0, lastindex: 3596, lastterm: 3]"]
[2020/05/25 06:10:47.383 +00:00] [INFO] [kvstore.go:373] ["restored last compact revision"] [meta-bucket-name=meta] [meta-bucket-name-key=finishedCompactRev] [restored-compact-revision=1083]
[2020/05/25 06:10:47.387 +00:00] [WARN] [store.go:1288] ["simple token is not cryptographically signed"]
[2020/05/25 06:10:47.387 +00:00] [INFO] [quota.go:98] ["enabled backend quota with default value"] [quota-name=v3-applier] [quota-size-bytes=2147483648] [quota-size="2.1 GB"]
[2020/05/25 06:10:47.387 +00:00] [INFO] [server.go:774] ["starting etcd server"] [local-member-id=7927c1c88369b5b0] [local-server-version=3.3.0+git] [cluster-version=to_be_decided]
[2020/05/25 06:10:47.388 +00:00] [INFO] [server.go:662] ["starting initial election tick advance"] [election-ticks=6]
[2020/05/25 06:10:47.388 +00:00] [INFO] [cluster.go:344] ["added member"] [cluster-id=4efd7a79829dbb9d] [local-member-id=7927c1c88369b5b0] [added-peer-id=7927c1c88369b5b0] [added-peer-peer-urls="[http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2380]"]
[2020/05/25 06:10:47.389 +00:00] [INFO] [cluster.go:486] ["set initial cluster version"] [cluster-id=4efd7a79829dbb9d] [local-member-id=7927c1c88369b5b0] [cluster-version=3.3]
[2020/05/25 06:10:47.389 +00:00] [INFO] [capability.go:75] ["enabled capabilities for version"] [cluster-version=3.3]
[2020/05/25 06:10:47.390 +00:00] [INFO] [etcd.go:239] ["now serving peer/client/metrics"] [local-member-id=7927c1c88369b5b0] [initial-advertise-peer-urls="[http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2380]"] [listen-peer-urls="[http://0.0.0.0:2380]"] [advertise-client-urls="[http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2379]"] [listen-client-urls="[http://0.0.0.0:2379]"] [listen-metrics-urls="[]"]
[2020/05/25 06:10:47.390 +00:00] [INFO] [etcd.go:574] ["serving peer traffic"] [address="[::]:2380"]
2020/05/25 06:10:47.449 log.go:86: [warning] etcdserver: [could not get cluster response from http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2380: Get http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2380/members: dial tcp: lookup tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc on 10.96.0.10:53: no such host]
[2020/05/25 06:10:47.449 +00:00] [ERROR] [etcdutil.go:63] ["failed to get cluster from remote"] [error="could not retrieve cluster information from the given URLs"]
[2020/05/25 06:10:52.883 +00:00] [INFO] [raft.go:922] ["7927c1c88369b5b0 is starting a new election at term 3"]
[2020/05/25 06:10:52.883 +00:00] [INFO] [raft.go:741] ["7927c1c88369b5b0 became pre-candidate at term 3"]
[2020/05/25 06:10:52.883 +00:00] [INFO] [raft.go:820] ["7927c1c88369b5b0 received MsgPreVoteResp from 7927c1c88369b5b0 at term 3"]
[2020/05/25 06:10:52.883 +00:00] [INFO] [raft.go:725] ["7927c1c88369b5b0 became candidate at term 4"]
[2020/05/25 06:10:52.883 +00:00] [INFO] [raft.go:820] ["7927c1c88369b5b0 received MsgVoteResp from 7927c1c88369b5b0 at term 4"]
[2020/05/25 06:10:52.883 +00:00] [INFO] [raft.go:777] ["7927c1c88369b5b0 became leader at term 4"]
[2020/05/25 06:10:52.884 +00:00] [INFO] [node.go:330] ["raft.node: 7927c1c88369b5b0 elected leader 7927c1c88369b5b0 at term 4"]
[2020/05/25 06:10:52.884 +00:00] [INFO] [server.go:1824] ["published local member to cluster through raft"] [local-member-id=7927c1c88369b5b0] [local-member-attributes="{Name:tidb-cluster-pd-0 ClientURLs:[http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2379]}"] [request-path=/0/members/7927c1c88369b5b0/attributes] [cluster-id=4efd7a79829dbb9d] [publish-timeout=11s]
[2020/05/25 06:10:52.885 +00:00] [INFO] [server.go:175] ["create etcd v3 client"] [endpoints="[http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2379]"]
[2020/05/25 06:10:52.887 +00:00] [INFO] [serve.go:139] ["serving client traffic insecurely; this is strongly discouraged!"] [address="[::]:2379"]
[2020/05/25 06:10:52.944 +00:00] [WARN] [grpclog.go:60] ["grpc: addrConn.createTransport failed to connect to {http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2379 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing dial tcp: lookup tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc on 10.96.0.10:53: no such host\". Reconnecting..."]
[2020/05/25 06:10:53.894 +00:00] [WARN] [grpclog.go:60] ["grpc: addrConn.createTransport failed to connect to {http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2379 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing dial tcp: lookup tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc on 10.96.0.10:53: no such host\". Reconnecting..."]
[2020/05/25 06:10:55.717 +00:00] [INFO] [server.go:214] ["init cluster id"] [cluster-id=6830613676610887904]
[2020/05/25 06:10:55.738 +00:00] [WARN] [history_buffer.go:137] ["load history index failed"] [error="leveldb: not found"]
[2020/05/25 06:10:55.738 +00:00] [INFO] [history_buffer.go:145] ["start from history index"] [start-index=0]
[2020/05/25 06:10:55.739 +00:00] [INFO] [namespace_classifier.go:459] ["load namespaces information"] [namespace-count=0] [cost=1.164617ms]
[2020/05/25 06:10:55.740 +00:00] [WARN] [leader.go:99] ["the leader has not changed, delete and campaign again"] [old-leader="name:\"tidb-cluster-pd-0\" member_id:8730159469623358896 peer_urls:\"http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2380\" client_urls:\"http://tidb-cluster-pd-0.tidb-cluster-pd-peer.tidb-cluster.svc:2379\" "]
[2020/05/25 06:10:55.742 +00:00] [INFO] [leader.go:215] ["start to campaign leader"] [campaign-leader-name=tidb-cluster-pd-0]
[2020/05/25 06:10:55.750 +00:00] [INFO] [leader.go:257] ["campaign leader ok"] [campaign-leader-name=tidb-cluster-pd-0]
[2020/05/25 06:10:55.752 +00:00] [INFO] [leader.go:415] ["server enable region storage"]
[2020/05/25 06:10:55.754 +00:00] [INFO] [tso.go:105] ["sync and save timestamp"] [last=2020/05/25 06:10:33.936 +00:00] [save=2020/05/25 06:10:58.754 +00:00] [next=2020/05/25 06:10:55.754 +00:00]
[2020/05/25 06:10:55.755 +00:00] [INFO] [util.go:93] ["load cluster version"] [cluster-version=0.0.0]
[2020/05/25 06:10:55.755 +00:00] [INFO] [leader.go:282] ["PD cluster leader is ready to serve"] [leader-name=tidb-cluster-pd-0]

Hacker_sAKN3wef · 2020 年5 月 25 日 06:13

现在 tikv 和 tidb 开始创建了吧？

yunlingfly · 2020 年5 月 25 日 06:13

还是没有tikv

Hacker_sAKN3wef · 2020 年5 月 25 日 06:14

再看下 tidb-controller-manager log

yunlingfly · 2020 年5 月 25 日 06:15

信息如下

# kubectl logs -n tidb-admin tidb-controller-manager-7bcff6964b-2c265                    I0525 06:04:44.523852       1 version.go:38] Welcome to TiDB Operator.
I0525 06:04:44.523887       1 version.go:39] TiDB Operator Version: version.Info{GitVersion:"v1.0.6", GitCommit:"982720cd563ece6dbebfc4c579b17fa66a93c550", GitTreeState:"clean", BuildDate:"2019-12-27T16:53:41Z", GoVersion:"go1.13", Compiler:"gc", Platform:"linux/amd64"}
I0525 06:04:45.920537       1 main.go:152] cache of informer factories sync successfully
I0525 06:04:45.920888       1 leaderelection.go:205] attempting to acquire leader lease  tidb-admin/tidb-controller-manager...
I0525 06:04:46.017528       1 leaderelection.go:214] successfully acquired lease tidb-admin/tidb-controller-manager
I0525 06:04:46.018295       1 tidb_cluster_controller.go:216] Starting tidbcluster controller
E0525 06:04:51.022359       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:04:51.027736       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:04:56.034132       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:04:56.035561       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:05:01.047277       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:05:01.048299       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:05:06.069912       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:05:06.070749       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:05:11.112383       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:05:11.113132       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:05:16.194422       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:05:16.195310       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:05:21.196487       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:05:21.197410       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:05:26.029021       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: dial tcp: lookup tidb-cluster-pd.tidb-cluster on 10.96.0.10:53: read udp 10.100.137.212:40882->10.96.0.10:53: i/o timeout
E0525 06:05:26.029998       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:05:31.031239       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:05:31.032304       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:05:36.033335       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:05:36.034176       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:05:41.035254       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:05:41.036016       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:05:46.037061       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:05:46.037921       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:05:51.039160       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:05:51.039845       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:05:56.041147       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:05:56.041937       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:06:16.521389       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:06:16.521877       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:06:21.522569       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:06:21.523597       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:06:50.124416       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:06:50.125304       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:07:20.124268       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:07:20.126013       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:07:43.443568       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:07:43.444616       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:07:50.125191       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:07:50.126208       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:08:20.125715       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:08:20.126532       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:08:50.125874       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:08:50.126947       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:09:20.130631       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:09:20.132011       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:09:50.126920       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:09:50.128031       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:10:20.126884       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:10:20.128129       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:10:36.884203       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), service tidb-cluster/tidb-cluster-pd has no endpoints
I0525 06:10:36.949265       1 tidbcluster_control.go:68] TidbCluster: [tidb-cluster/tidb-cluster] updated successfully
E0525 06:10:36.949393       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
I0525 06:10:36.949554       1 event.go:221] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"tidb-cluster", Name:"tidb-cluster", UID:"0a726b56-cfb0-4790-90ed-ef5c0b7b3a82", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"1372165", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update TidbCluster tidb-cluster successful
E0525 06:10:41.950275       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), service tidb-cluster/tidb-cluster-pd has no endpoints
I0525 06:10:41.972528       1 tidbcluster_control.go:68] TidbCluster: [tidb-cluster/tidb-cluster] updated successfully
E0525 06:10:41.972704       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
I0525 06:10:41.973751       1 event.go:221] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"tidb-cluster", Name:"tidb-cluster", UID:"0a726b56-cfb0-4790-90ed-ef5c0b7b3a82", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"1373818", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update TidbCluster tidb-cluster successful
E0525 06:10:46.973871       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:10:46.992609       1 tidbcluster_control.go:71] failed to update TidbCluster: [tidb-cluster/tidb-cluster], error: Operation cannot be fulfilled on tidbclusters.pingcap.com "tidb-cluster": the object has been modified; please apply your changes to the latest version and try again
I0525 06:10:47.026425       1 tidbcluster_control.go:68] TidbCluster: [tidb-cluster/tidb-cluster] updated successfully
E0525 06:10:47.026556       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
I0525 06:10:47.026705       1 event.go:221] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"tidb-cluster", Name:"tidb-cluster", UID:"0a726b56-cfb0-4790-90ed-ef5c0b7b3a82", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"1373857", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update TidbCluster tidb-cluster successful
E0525 06:10:52.027916       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0525 06:10:52.048862       1 tidbcluster_control.go:68] TidbCluster: [tidb-cluster/tidb-cluster] updated successfully
E0525 06:10:52.049109       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
I0525 06:10:52.049640       1 event.go:221] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"tidb-cluster", Name:"tidb-cluster", UID:"0a726b56-cfb0-4790-90ed-ef5c0b7b3a82", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"1373881", FieldPath:""}): type: 'Normal' reason: 'SuccessfulUpdate' update TidbCluster tidb-cluster successful
E0525 06:10:57.050885       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:10:57.051671       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:11:20.127218       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:11:20.128006       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:11:50.127825       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:11:50.128988       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:12:20.128226       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:12:20.129540       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:12:50.128184       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:12:50.129391       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:13:20.128413       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:13:20.129476       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:13:50.129303       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:13:50.130399       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:14:20.129481       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:14:20.130580       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing
E0525 06:14:50.129530       1 pd_member_manager.go:200] failed to sync TidbCluster: [tidb-cluster/tidb-cluster]'s status, error: Get http://tidb-cluster-pd.tidb-cluster:2379/pd/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0525 06:14:50.130774       1 tidb_cluster_controller.go:245] TidbCluster: tidb-cluster/tidb-cluster, sync failed TidbCluster: tidb-cluster/tidb-cluster's pd status sync failed, can't failover, requeuing

Hacker_sAKN3wef · 2020 年5 月 25 日 06:22

集群有配置 network policy 吗？看一下 tidb-controller-manager Pod 和 PD Pod 网络通吗？

yunlingfly · 2020 年5 月 25 日 06:23

额具体要怎么测试tidb-controller-manager与PD的网络情况呢？

Hacker_sAKN3wef · 2020 年5 月 25 日 06:29

看下 service 情况 kubectl describe svc tidb-cluster-pd -n tidb-cluster
kubectl get netpol -A
kubectl exec -it -n tidb-admin sh 执行如下命令

curl http://tidb-cluster-pd.tidb-cluster:2379/pd/health
curl http://<pd pod ip>:2379/pd/health

yunlingfly · 2020 年5 月 25 日 06:35

不行，我进入manage后ping都ping不通pd，也没有curl工具

# kubectl describe svc tidb-cluster-pd -n tidb-cluster
Name:              tidb-cluster-pd
Namespace:         tidb-cluster
Labels:            app.kubernetes.io/component=pd
                   app.kubernetes.io/instance=tidb-cluster
                   app.kubernetes.io/managed-by=tidb-operator
                   app.kubernetes.io/name=tidb-cluster
Annotations:       pingcap.com/last-applied-configuration:
                     {"ports":[{"name":"client","protocol":"TCP","port":2379,"targetPort":2379}],"selector":{"app.kubernetes.io/component":"pd","app.kubernetes...
Selector:          app.kubernetes.io/component=pd,app.kubernetes.io/instance=tidb-cluster,app.kubernetes.io/managed-by=tidb-operator,app.kubernetes.io/name=tidb-cluster
Type:              ClusterIP
IP:                10.96.213.155
Port:              client  2379/TCP
TargetPort:        2379/TCP
Endpoints:         10.100.30.31:2379
Session Affinity:  None
Events:            <none>
[root@acdm2 ~]# kubectl get netpol -A
No resources found
[root@acdm2 ~]# kubectl exec -it tidb-controller-manager-7bcff6964b-2c265 -n tidb-admin -- /bin/sh
/ # ping 10.96.213.155
PING 10.96.213.155 (10.96.213.155): 56 data bytes
^C
--- 10.96.213.155 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
/ # curl http://tidb-cluster-pd.tidb-cluster:2379/pd/health
/bin/sh: curl: not found
/ #