集群使用 Hostnetwork 模式部署 TiKV 一直报 PD 节点不通

Meow · 2021 年3 月 19 日 02:03

【TiDB 版本】
TiDB： v4.0.7

TiDB-Operator： v1.2.0-a.1

【问题描述】
集群使用 Hostnetwork 模式部署 TiKV 一直报 PD 节点不通, exec 进入 TiKV Pod，Ping PD 地址是可通的

[root@xxx ~]# kubectl get pods -n uat
NAME                             READY   STATUS    RESTARTS   AGE
tc01-discovery-7f78d4d68-5rgfm   1/1     Running   0          18h
tc01-pd-0                        1/1     Running   0          18h
tc01-tikv-0                      1/1     Running   0          18h
[root@xxx ~]# kubectl logs tc01-tikv-0 -n uat
[2021/03/19 04:05:38.348 +08:00] [INFO] [util.rs:378] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { status: 4-DEADLINE_EXCEEDED, details: Some(\"Deadline Exceeded\") }))"] [endpoints=http://tc01-pd:2379]
[2021/03/19 04:05:38.648 +08:00] [INFO] [util.rs:419] ["connecting to PD endpoint"] [endpoints=http://tc01-pd:2379]
[2021/03/19 04:05:40.649 +08:00] [INFO] [util.rs:378] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { status: 4-DEADLINE_EXCEEDED, details: Some(\"Deadline Exceeded\") }))"] [endpoints=http://tc01-pd:2379]
[2021/03/19 04:05:40.649 +08:00] [WARN] [client.rs:104] ["validate PD endpoints failed"] [err="Other(\"[components/pd_client/src/util.rs:410]: PD cluster failed to respond\")"]
[2021/03/19 04:05:40.949 +08:00] [INFO] [util.rs:419] ["connecting to PD endpoint"] [endpoints=http://tc01-pd:2379]
[2021/03/19 04:05:42.950 +08:00] [INFO] [util.rs:378] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { status: 4-DEADLINE_EXCEEDED, details: Some(\"Deadline Exceeded\") }))"] [endpoints=http://tc01-pd:2379]
[2021/03/19 04:05:43.250 +08:00] [INFO] [util.rs:419] ["connecting to PD endpoint"] [endpoints=http://tc01-pd:2379]
[2021/03/19 04:05:45.251 +08:00] [INFO] [util.rs:378] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { status: 4-DEADLINE_EXCEEDED, details: Some(\"Deadline Exceeded\") }))"] [endpoints=http://tc01-pd:2379]
[2021/03/19 04:05:45.551 +08:00] [INFO] [util.rs:419] ["connecting to PD endpoint"] [endpoints=http://tc01-pd:2379]
[2021/03/19 04:05:47.552 +08:00] [INFO] [util.rs:378] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { status: 4-DEADLINE_EXCEEDED, details: Some(\"Deadline Exceeded\") }))"] [endpoints=http://tc01-pd:2379]
[2021/03/19 04:05:47.852 +08:00] [INFO] [util.rs:419] ["connecting to PD endpoint"] [endpoints=http://tc01-pd:2379]

 cat uat.yaml
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: tc01
  namespace: uat
  labels:
    user: dba

spec:
  version: "v4.0.7"
  timezone: Asia/Shanghai
  configUpdateStrategy: RollingUpdate

  imagePullPolicy: IfNotPresent
  helper:
     image: 172.16.80.11:5000/busybox:1.26.2


  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  pd:
    baseImage: 172.16.80.11:5000/pingcap/pd

    config: {}

    replicas: 1

    requests:
      storage: 1Gi
      memory: 1Gi

    storageClassName: "local-storage"
    hostNetwork: true
    dnsPolicy: ClusterFirstWithHostNet

  tidb:
    service:
      type: NodePort
    baseImage: 172.16.80.11:5000/pingcap/tidb

    replicas: 1

    storageClassName: "fast-storage"
    config: {}
    hostNetwork: true
    dnsPolicy: ClusterFirstWithHostNet

  tikv:
    baseImage: 172.16.80.11:5000/pingcap/tikv

    config: {}

    ulimits:
      nofile:
        soft: 1000000
        hard: 1000000

    replicas: 1

    requests:
      storage: 10Gi

    storageClassName: "fast-storage"
    hostNetwork: true
    dnsPolicy: ClusterFirstWithHostNet

  enablePVReclaim: false

  pvReclaimPolicy: Retain

  tlsCluster: {}

yilong · 2021 年3 月 19 日 03:54

麻烦反馈下当前的tc状态,namespace_name替换为对应的名称：
kubectl get all -n <namespace_name> -o wide
查看下 pd pod 的日志信息

Meow · 2021 年3 月 19 日 06:30

[root@k8sftb11 ~]# kubectl get all -n uat  -o wide
NAME                                 READY   STATUS    RESTARTS   AGE   IP              NODE       NOMINATED NODE   READINESS GATES
pod/tc01-discovery-7f78d4d68-5rgfm   1/1     Running   0          23h   100.73.157.25   k8sftb18   <none>           <none>
pod/tc01-pd-0                        1/1     Running   0          23h   172.16.80.14    k8sftb14   <none>           <none>
pod/tc01-tikv-0                      1/1     Running   0          22h   172.16.80.18    k8sftb18   <none>           <none>

NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)               AGE   SELECTOR
service/tc01-discovery   ClusterIP   10.97.163.69   <none>        10261/TCP,10262/TCP   23h   app.kubernetes.io/component=discovery,app.kubernetes.io/instance=tc01,app.kubernetes.io/managed-by=tidb-operator,app.kubernetes.io/name=tidb-cluster
service/tc01-pd          ClusterIP   10.100.3.4     <none>        2379/TCP              23h   app.kubernetes.io/component=pd,app.kubernetes.io/instance=tc01,app.kubernetes.io/managed-by=tidb-operator,app.kubernetes.io/name=tidb-cluster
service/tc01-pd-peer     ClusterIP   None           <none>        2380/TCP              23h   app.kubernetes.io/component=pd,app.kubernetes.io/instance=tc01,app.kubernetes.io/managed-by=tidb-operator,app.kubernetes.io/name=tidb-cluster
service/tc01-tikv-peer   ClusterIP   None           <none>        20160/TCP             23h   app.kubernetes.io/component=tikv,app.kubernetes.io/instance=tc01,app.kubernetes.io/managed-by=tidb-operator,app.kubernetes.io/name=tidb-cluster

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                   SELECTOR
deployment.apps/tc01-discovery   1/1     1            1           23h   discovery    172.16.80.11:5000/pingcap/tidb-operator:v1.2.0-alpha.1   app.kubernetes.io/component=discovery,app.kubernetes.io/instance=tc01,app.kubernetes.io/managed-by=tidb-operator,app.kubernetes.io/name=tidb-cluster

NAME                                       DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                                                   SELECTOR
replicaset.apps/tc01-discovery-7f78d4d68   1         1         1       23h   discovery    172.16.80.11:5000/pingcap/tidb-operator:v1.2.0-alpha.1   app.kubernetes.io/component=discovery,app.kubernetes.io/instance=tc01,app.kubernetes.io/managed-by=tidb-operator,app.kubernetes.io/name=tidb-cluster,pod-template-hash=7f78d4d68

NAME                         READY   AGE   CONTAINERS   IMAGES
statefulset.apps/tc01-pd     1/1     23h   pd           172.16.80.11:5000/pingcap/pd:v4.0.7
statefulset.apps/tc01-tikv   1/1     23h   tikv         172.16.80.11:5000/pingcap/tikv:v4.0.7
[root@k8sftb11 ~]#
[root@k8sftb11 ~]# kubectl get pods -n uat  -o wide
NAME                             READY   STATUS    RESTARTS   AGE   IP              NODE       NOMINATED NODE   READINESS GATES
tc01-discovery-7f78d4d68-5rgfm   1/1     Running   0          23h   100.73.157.25   k8sftb18   <none>           <none>
tc01-pd-0                        1/1     Running   0          23h   172.16.80.14    k8sftb14   <none>           <none>
tc01-tikv-0                      1/1     Running   0          22h   172.16.80.18    k8sftb18   <none>           <none>
[root@k8sftb11 ~]# kubectl logs tc01-pd-0  -n uat  -o wide
Error: unknown shorthand flag: 'o' in -o
See 'kubectl logs --help' for usage.
[root@k8sftb11 ~]# kubectl logs tc01-pd-0  -n uat

Name:      tc01-pd-0.tc01-pd-peer.uat.svc
Address 1: 172.16.80.14 k8sftb14
nslookup domain tc01-pd-0.tc01-pd-peer.uat.svc.svc success
starting pd-server ...
/pd-server --data-dir=/var/lib/pd --name=tc01-pd-0 --peer-urls=http://0.0.0.0:2380 --advertise-peer-urls=http://tc01-pd-0.tc01-pd-peer.uat.svc:2380 --client-urls=http://0.0.0.0:2379 --advertise-client-urls=http://tc01-pd-0.tc01-pd-peer.uat.svc:2379 --config=/etc/pd/pd.toml --initial-cluster=tc01-pd-0=http://tc01-pd-0.tc01-pd-peer.uat.svc:2380
[2021/03/18 15:06:29.447 +08:00] [INFO] [util.go:42] ["Welcome to Placement Driver (PD)"]
[2021/03/18 15:06:29.447 +08:00] [INFO] [util.go:43] [PD] [release-version=v4.0.7]
[2021/03/18 15:06:29.447 +08:00] [INFO] [util.go:44] [PD] [edition=Community]
[2021/03/18 15:06:29.447 +08:00] [INFO] [util.go:45] [PD] [git-hash=8b0348f545611d5955e32fdcf3c57a3f73657d77]
[2021/03/18 15:06:29.447 +08:00] [INFO] [util.go:46] [PD] [git-branch=heads/refs/tags/v4.0.7]
[2021/03/18 15:06:29.447 +08:00] [INFO] [util.go:47] [PD] [utc-build-time="2020-09-29 06:52:41"]
[2021/03/18 15:06:29.447 +08:00] [INFO] [metricutil.go:81] ["disable Prometheus push client"]
[2021/03/18 15:06:29.447 +08:00] [INFO] [server.go:216] ["PD Config"] [config="{\"client-urls\":\"http://0.0.0.0:2379\",\"peer-urls\":\"http://0.0.0.0:2380\",\"advertise-client-urls\":\"http://tc01-pd-0.tc01-pd-peer.uat.svc:2379\",\"advertise-peer-urls\":\"http://tc01-pd-0.tc01-pd-peer.uat.svc:2380\",\"name\":\"tc01-pd-0\",\"data-dir\":\"/var/lib/pd\",\"force-new-cluster\":false,\"enable-grpc-gateway\":true,\"initial-cluster\":\"tc01-pd-0=http://tc01-pd-0.tc01-pd-peer.uat.svc:2380\",\"initial-cluster-state\":\"new\",\"initial-cluster-token\":\"pd-cluster\",\"join\":\"\",\"lease\":3,\"log\":{\"level\":\"\",\"format\":\"text\",\"disable-timestamp\":false,\"file\":{\"filename\":\"\",\"max-size\":0,\"max-days\":0,\"max-backups\":0},\"development\":false,\"disable-caller\":false,\"disable-stacktrace\":false,\"disable-error-verbose\":true,\"sampling\":null},\"tso-save-interval\":\"3s\",\"metric\":{\"job\":\"tc01-pd-0\",\"address\":\"\",\"interval\":\"15s\"},\"schedule\":{\"max-snapshot-count\":3,\"max-pending-peer-count\":16,\"max-merge-region-size\":20,\"max-merge-region-keys\":200000,\"split-merge-interval\":\"1h0m0s\",\"enable-one-way-merge\":\"false\",\"enable-cross-table-merge\":\"false\",\"patrol-region-interval\":\"100ms\",\"max-store-down-time\":\"30m0s\",\"leader-schedule-limit\":4,\"leader-schedule-policy\":\"count\",\"region-schedule-limit\":2048,\"replica-schedule-limit\":64,\"merge-schedule-limit\":8,\"hot-region-schedule-limit\":4,\"hot-region-cache-hits-threshold\":3,\"store-limit\":null,\"tolerant-size-ratio\":0,\"low-space-ratio\":0.8,\"high-space-ratio\":0.7,\"scheduler-max-waiting-operator\":5,\"enable-remove-down-replica\":\"true\",\"enable-replace-offline-replica\":\"true\",\"enable-make-up-replica\":\"true\",\"enable-remove-extra-replica\":\"true\",\"enable-location-replacement\":\"true\",\"enable-debug-metrics\":\"false\",\"schedulers-v2\":[{\"type\":\"balance-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"balance-leader\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"hot-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"label\",\"args\":null,\"disable\":false,\"args-payload\":\"\"}],\"schedulers-payload\":null,\"store-limit-mode\":\"manual\"},\"replication\":{\"max-replicas\":3,\"location-labels\":\"\",\"strictly-match-label\":\"false\",\"enable-placement-rules\":\"false\"},\"pd-server\":{\"use-region-storage\":\"true\",\"max-gap-reset-ts\":\"24h0m0s\",\"key-type\":\"table\",\"runtime-services\":\"\",\"metric-storage\":\"\",\"dashboard-address\":\"auto\",\"trace-region-flow\":\"true\"},\"cluster-version\":\"0.0.0\",\"quota-backend-bytes\":\"8GiB\",\"auto-compaction-mode\":\"periodic\",\"auto-compaction-retention-v2\":\"1h\",\"TickInterval\":\"500ms\",\"ElectionInterval\":\"3s\",\"PreVote\":true,\"security\":{\"cacert-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null},\"label-property\":null,\"WarningMsgs\":null,\"DisableStrictReconfigCheck\":false,\"HeartbeatStreamBindInterval\":\"1m0s\",\"LeaderPriorityCheckInterval\":\"1m0s\",\"dashboard\":{\"tidb-cacert-path\":\"\",\"tidb-cert-path\":\"\",\"tidb-key-path\":\"\",\"public-path-prefix\":\"\",\"internal-proxy\":false,\"enable-telemetry\":true,\"enable-experimental\":false},\"replication-mode\":{\"replication-mode\":\"majority\",\"dr-auto-sync\":{\"label-key\":\"\",\"primary\":\"\",\"dr\":\"\",\"primary-replicas\":0,\"dr-replicas\":0,\"wait-store-timeout\":\"1m0s\",\"wait-sync-timeout\":\"1m0s\"}}}"]
[2021/03/18 15:06:29.453 +08:00] [INFO] [server.go:189] ["register REST path"] [path=/pd/api/v1]
[2021/03/18 15:06:29.453 +08:00] [INFO] [server.go:189] ["register REST path"] [path=/swagger/]
[2021/03/18 15:06:29.458 +08:00] [INFO] [server.go:189] ["register REST path"] [path=/dashboard/api/]
[2021/03/18 15:06:29.458 +08:00] [INFO] [server.go:189] ["register REST path"] [path=/dashboard/]
[2021/03/18 15:06:29.458 +08:00] [INFO] [etcd.go:117] ["configuring peer listeners"] [listen-peer-urls="[http://0.0.0.0:2380]"]
[2021/03/18 15:06:29.458 +08:00] [INFO] [systime_mon.go:27] ["start system time monitor"]
[2021/03/18 15:06:29.459 +08:00] [INFO] [etcd.go:127] ["configuring client listeners"] [listen-client-urls="[http://0.0.0.0:2379]"]
[2021/03/18 15:06:29.459 +08:00] [INFO] [etcd.go:602] ["pprof is enabled"] [path=/debug/pprof]
[2021/03/18 15:06:29.459 +08:00] [INFO] [etcd.go:299] ["starting an etcd server"] [etcd-version=3.4.3] [git-sha="Not provided (use ./build instead of go build)"] [go-version=go1.13] [go-os=linux] [go-arch=amd64] [max-cpu-set=24] [max-cpu-available=24] [member-initialized=false] [name=tc01-pd-0] [data-dir=/var/lib/pd] [wal-dir=] [wal-dir-dedicated=] [member-dir=/var/lib/pd/member] [force-new-cluster=false] [heartbeat-interval=500ms] [election-timeout=3s] [initial-election-tick-advance=true] [snapshot-count=100000] [snapshot-catchup-entries=5000] [initial-advertise-peer-urls="[http://tc01-pd-0.tc01-pd-peer.uat.svc:2380]"] [listen-peer-urls="[http://0.0.0.0:2380]"] [advertise-client-urls="[http://tc01-pd-0.tc01-pd-peer.uat.svc:2379]"] [listen-client-urls="[http://0.0.0.0:2379]"] [listen-metrics-urls="[]"] [cors="[*]"] [host-whitelist="[*]"] [initial-cluster="tc01-pd-0=http://tc01-pd-0.tc01-pd-peer.uat.svc:2380"] [initial-cluster-state=new] [initial-cluster-token=pd-cluster] [quota-size-bytes=8589934592] [pre-vote=true] [initial-corrupt-check=false] [corrupt-check-time-interval=0s] [auto-compaction-mode=periodic] [auto-compaction-retention=1h0m0s] [auto-compaction-interval=1h0m0s] [discovery-url=] [discovery-proxy=]
[2021/03/18 15:06:29.479 +08:00] [INFO] [backend.go:79] ["opened backend db"] [path=/var/lib/pd/member/snap/db] [took=19.219487ms]
[2021/03/18 15:06:29.483 +08:00] [INFO] [netutil.go:112] ["resolved URL Host"] [url=http://tc01-pd-0.tc01-pd-peer.uat.svc:2380] [host=tc01-pd-0.tc01-pd-peer.uat.svc:2380] [resolved-addr=172.16.80.14:2380]
[2021/03/18 15:06:29.484 +08:00] [INFO] [netutil.go:112] ["resolved URL Host"] [url=http://tc01-pd-0.tc01-pd-peer.uat.svc:2380] [host=tc01-pd-0.tc01-pd-peer.uat.svc:2380] [resolved-addr=172.16.80.14:2380]
[2021/03/18 15:06:29.513 +08:00] [INFO] [raft.go:456] ["starting local member"] [local-member-id=580ada3af5b59fd9] [cluster-id=e8899a61bfec1036]
[2021/03/18 15:06:29.514 +08:00] [INFO] [raft.go:1530] ["580ada3af5b59fd9 switched to configuration voters=()"]
[2021/03/18 15:06:29.514 +08:00] [INFO] [raft.go:700] ["580ada3af5b59fd9 became follower at term 0"]
[2021/03/18 15:06:29.514 +08:00] [INFO] [raft.go:383] ["newRaft 580ada3af5b59fd9 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"]
[2021/03/18 15:06:29.514 +08:00] [INFO] [raft.go:700] ["580ada3af5b59fd9 became follower at term 1"]
[2021/03/18 15:06:29.514 +08:00] [INFO] [raft.go:1530] ["580ada3af5b59fd9 switched to configuration voters=(6344122971870044121)"]
[2021/03/18 15:06:29.553 +08:00] [WARN] [store.go:1317] ["simple token is not cryptographically signed"]
[2021/03/18 15:06:29.564 +08:00] [INFO] [quota.go:126] ["enabled backend quota"] [quota-name=v3-applier] [quota-size-bytes=8589934592] [quota-size="8.6 GB"]
[2021/03/18 15:06:29.576 +08:00] [INFO] [server.go:792] ["starting etcd server"] [local-member-id=580ada3af5b59fd9] [local-server-version=3.4.3] [cluster-version=to_be_decided]
[2021/03/18 15:06:29.578 +08:00] [INFO] [server.go:658] ["started as single-node; fast-forwarding election ticks"] [local-member-id=580ada3af5b59fd9] [forward-ticks=5] [forward-duration=2.5s] [election-ticks=6] [election-timeout=3s]
[2021/03/18 15:06:29.579 +08:00] [INFO] [raft.go:1530] ["580ada3af5b59fd9 switched to configuration voters=(6344122971870044121)"]
[2021/03/18 15:06:29.580 +08:00] [INFO] [cluster.go:392] ["added member"] [cluster-id=e8899a61bfec1036] [local-member-id=580ada3af5b59fd9] [added-peer-id=580ada3af5b59fd9] [added-peer-peer-urls="[http://tc01-pd-0.tc01-pd-peer.uat.svc:2380]"]
[2021/03/18 15:06:29.581 +08:00] [INFO] [etcd.go:576] ["serving peer traffic"] [address="[::]:2380"]
[2021/03/18 15:06:29.581 +08:00] [INFO] [etcd.go:241] ["now serving peer/client/metrics"] [local-member-id=580ada3af5b59fd9] [initial-advertise-peer-urls="[http://tc01-pd-0.tc01-pd-peer.uat.svc:2380]"] [listen-peer-urls="[http://0.0.0.0:2380]"] [advertise-client-urls="[http://tc01-pd-0.tc01-pd-peer.uat.svc:2379]"] [listen-client-urls="[http://0.0.0.0:2379]"] [listen-metrics-urls="[]"]
[2021/03/18 15:06:31.014 +08:00] [INFO] [raft.go:923] ["580ada3af5b59fd9 is starting a new election at term 1"]
[2021/03/18 15:06:31.014 +08:00] [INFO] [raft.go:729] ["580ada3af5b59fd9 became pre-candidate at term 1"]
[2021/03/18 15:06:31.014 +08:00] [INFO] [raft.go:824] ["580ada3af5b59fd9 received MsgPreVoteResp from 580ada3af5b59fd9 at term 1"]
[2021/03/18 15:06:31.014 +08:00] [INFO] [raft.go:713] ["580ada3af5b59fd9 became candidate at term 2"]
[2021/03/18 15:06:31.014 +08:00] [INFO] [raft.go:824] ["580ada3af5b59fd9 received MsgVoteResp from 580ada3af5b59fd9 at term 2"]
[2021/03/18 15:06:31.014 +08:00] [INFO] [raft.go:765] ["580ada3af5b59fd9 became leader at term 2"]
[2021/03/18 15:06:31.014 +08:00] [INFO] [node.go:325] ["raft.node: 580ada3af5b59fd9 elected leader 580ada3af5b59fd9 at term 2"]
[2021/03/18 15:06:31.015 +08:00] [INFO] [server.go:2016] ["published local member to cluster through raft"] [local-member-id=580ada3af5b59fd9] [local-member-attributes="{Name:tc01-pd-0 ClientURLs:[http://tc01-pd-0.tc01-pd-peer.uat.svc:2379]}"] [request-path=/0/members/580ada3af5b59fd9/attributes] [cluster-id=e8899a61bfec1036] [publish-timeout=11s]
[2021/03/18 15:06:31.015 +08:00] [INFO] [server.go:291] ["create etcd v3 client"] [endpoints="[http://tc01-pd-0.tc01-pd-peer.uat.svc:2379]"] [cert="{\"cacert-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null}"]
[2021/03/18 15:06:31.015 +08:00] [INFO] [server.go:2501] ["setting up initial cluster version"] [cluster-version=3.4]
[2021/03/18 15:06:31.017 +08:00] [INFO] [serve.go:139] ["serving client traffic insecurely; this is strongly discouraged!"] [address="[::]:2379"]
[2021/03/18 15:06:31.023 +08:00] [INFO] [cluster.go:558] ["set initial cluster version"] [cluster-id=e8899a61bfec1036] [local-member-id=580ada3af5b59fd9] [cluster-version=3.4]
[2021/03/18 15:06:31.024 +08:00] [INFO] [capability.go:76] ["enabled capabilities for version"] [cluster-version=3.4]
[2021/03/18 15:06:31.024 +08:00] [INFO] [server.go:2533] ["cluster version is updated"] [cluster-version=3.4]
[2021/03/18 15:06:31.030 +08:00] [INFO] [server.go:343] ["init cluster id"] [cluster-id=6940887014484111690]
[2021/03/18 15:06:31.110 +08:00] [WARN] [history_buffer.go:139] ["load history index failed"] [error="leveldb: not found"]
[2021/03/18 15:06:31.110 +08:00] [INFO] [history_buffer.go:147] ["start from history index"] [start-index=0]
[2021/03/18 15:06:31.111 +08:00] [INFO] [server.go:1112] ["start to campaign leader"] [campaign-leader-name=tc01-pd-0]
[2021/03/18 15:06:31.114 +08:00] [INFO] [server.go:1131] ["campaign leader ok"] [campaign-leader-name=tc01-pd-0]
[2021/03/18 15:06:31.120 +08:00] [INFO] [tso.go:158] ["sync and save timestamp"] [last=0001/01/01 00:00:00.000 +00:00] [save=2021/03/18 15:06:34.119 +08:00] [next=2021/03/18 15:06:31.119 +08:00]
[2021/03/18 15:06:31.125 +08:00] [INFO] [server.go:1213] ["server enable region storage"]
[2021/03/18 15:06:31.126 +08:00] [INFO] [util.go:78] ["load cluster version"] [cluster-version=0.0.0]
[2021/03/18 15:06:31.126 +08:00] [INFO] [server.go:1157] ["PD cluster leader is ready to serve"] [leader-name=tc01-pd-0]
[2021/03/18 15:06:32.113 +08:00] [INFO] [server.go:858] ["PD server config is updated"] [new="{\"use-region-storage\":\"true\",\"max-gap-reset-ts\":\"24h0m0s\",\"key-type\":\"table\",\"runtime-services\":\"\",\"metric-storage\":\"\",\"dashboard-address\":\"http://tc01-pd-0.tc01-pd-peer.uat.svc:2379\",\"trace-region-flow\":\"false\"}"] [old="{\"use-region-storage\":\"true\",\"max-gap-reset-ts\":\"24h0m0s\",\"key-type\":\"table\",\"runtime-services\":\"\",\"metric-storage\":\"\",\"dashboard-address\":\"auto\",\"trace-region-flow\":\"true\"}"]
[2021/03/18 15:06:33.114 +08:00] [INFO] [dbstore.go:44] ["Dashboard initializing local storage file"] [path=/var/lib/pd/dashboard.sqlite.db]
[2021/03/18 15:06:33.638 +08:00] [INFO] [version.go:41] ["TiDB Dashboard started"] [internal-version=2020.09.21.1] [standalone=No] [pd-version=v4.0.7] [build-time="2020-09-29 06:52:41"] [build-git-hash=0e148dfc0029]
[2021/03/18 15:06:33.638 +08:00] [INFO] [manager.go:200] ["Dashboard server is started"]
[2021/03/18 15:06:33.638 +08:00] [INFO] [proxy.go:199] ["start serve requests to remotes"] [endpoint=127.0.0.1:34255] [remotes="[]"]
[2021/03/18 15:06:33.638 +08:00] [INFO] [proxy.go:199] ["start serve requests to remotes"] [endpoint=127.0.0.1:38972] [remotes="[]"]
[2021/03/18 15:06:33.639 +08:00] [WARN] [dynamic_config_manager.go:173] ["Dynamic config does not exist in etcd"]
[2021/03/18 15:06:33.640 +08:00] [INFO] [dynamic_config_manager.go:197] ["Save dynamic config to etcd"] [json="{\"keyvisual\":{\"auto_collection_disabled\":false,\"policy\":\"db\",\"policy_kv_separator\":\"\"},\"profiling\":{\"auto_collection_targets\":null,\"auto_collection_duration_secs\":0,\"auto_collection_interval_secs\":0}}"]
[2021/03/18 15:06:33.903 +08:00] [INFO] [manager.go:85] ["Key visual service is started"]
[2021/03/18 15:07:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:08:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:09:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:10:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:11:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:12:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:13:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:14:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:15:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:16:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:17:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:18:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:19:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:20:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:21:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:22:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:23:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:24:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:25:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []
[2021/03/18 15:26:33.643 +08:00] [WARN] [tidb_requests.go:47] ["failed to get tidb schema version"] []

yilong · 2021 年3 月 19 日 08:26

麻烦在检查下网络情况，tikv pod 是否能访问 http://tc01-pd:2379

Meow · 2021 年3 月 22 日 01:44

/ # 
/ # 
/ # cd
~ # 
~ # which nc
/usr/bin/nc
~ # which telnet
~ # which curl
~ # 
~ # 
~ # nc tc01-pd 2379
^Cpunt!

~ # 
~ # 
~ # ping tc01-pd
PING tc01-pd (10.100.3.4): 56 data bytes
64 bytes from 10.100.3.4: seq=0 ttl=64 time=0.095 ms
64 bytes from 10.100.3.4: seq=1 ttl=64 time=0.052 ms
64 bytes from 10.100.3.4: seq=2 ttl=64 time=0.126 ms
64 bytes from 10.100.3.4: seq=3 ttl=64 time=0.072 ms
^C
--- tc01-pd ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.052/0.086/0.126 ms
~ # nc tc01-pd 2379
^A


~ # 
~ # 
~ # nc tc01-pd 2379



~ # 
~ # 
~ # nc tc01-pd 2379

nc 一直没有返回

Meow · 2021 年3 月 22 日 02:34

PD 日志未见异常但是 nc 无返回可能是什么情况？

yilong · 2021 年3 月 24 日 03:22

抱歉，直通的网络问题，可能需要找您的运维人员帮忙查看下。或者可以参考文档使用其他方式部署，多谢。