tidb单机部署集群启停故障

🪐 TiDB 技术问题
1

在单机版部署了TIDB集群，但查看集群状态显示有些状态异常，执行停止集群命令，提示已经停止成功，但status状态却依然显示up.
[tidb@xsky-node3 ~]$ tiup cluster stop tidb-cluster
tiup is checking updates for component cluster …
Starting component cluster: /home/tidb/.tiup/components/cluster/v1.12.3/tiup-cluster stop tidb-cluster
Will stop the cluster tidb-cluster with nodes: , roles: .
Do you want to continue? [y/N]:(default=N) y

  • [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa.pub
  • [Parallel] - UserSSH: user=tidb, host=10.110.7.41
  • [Parallel] - UserSSH: user=tidb, host=10.110.7.41
  • [Parallel] - UserSSH: user=tidb, host=10.110.7.41
  • [Parallel] - UserSSH: user=tidb, host=10.110.7.41
  • [Parallel] - UserSSH: user=tidb, host=10.110.7.41
  • [Parallel] - UserSSH: user=tidb, host=10.110.7.41
  • [ Serial ] - StopCluster
    Stopping component alertmanager
    Stopping instance 10.110.7.41
    Stop alertmanager 10.110.7.41:9093 success
    Stopping component grafana
    Stopping instance 10.110.7.41
    Stop grafana 10.110.7.41:3000 success
    Stopping component prometheus
    Stopping instance 10.110.7.41
    Stop prometheus 10.110.7.41:9090 success
    Stopping component tidb
    Stopping instance 10.110.7.41
    Stop tidb 10.110.7.41:4000 success
    Stopping component tikv
    Stopping instance 10.110.7.41
    Stop tikv 10.110.7.41:20160 success
    Stopping component pd
    Stopping instance 10.110.7.41
    Stop pd 10.110.7.41:2379 success
    Stopping component node_exporter
    Stopping instance 10.110.7.41
    Stop 10.110.7.41 success
    Stopping component blackbox_exporter
    Stopping instance 10.110.7.41
    Stop 10.110.7.41 success
    Stopped cluster tidb-cluster successfully
    [tidb@xsky-node3 ~]$
    [tidb@xsky-node3 ~]$
    [tidb@xsky-node3 ~]$
    [tidb@xsky-node3 ~]$ tiup cluster display tidb-cluster
    tiup is checking updates for component cluster …
    Starting component cluster: /home/tidb/.tiup/components/cluster/v1.12.3/tiup-cluster display tidb-cluster
    Cluster type: tidb
    Cluster name: tidb-cluster
    Cluster version: v7.2.0
    Deploy user: tidb
    SSH type: builtin
    Grafana URL: http://10.110.7.41:3000
    ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

10.110.7.41:9093 alertmanager 10.110.7.41 9093/9094 linux/x86_64 Down /home/tidb/tidb-data/alertmanager-9093 /home/tidb/tidb-deploy/alertmanager-9093
10.110.7.41:3000 grafana 10.110.7.41 3000 linux/x86_64 Down - /home/tidb/tidb-deploy/grafana-3000
10.110.7.41:2379 pd 10.110.7.41 2379/2380 linux/x86_64 Down /home/tidb/tidb-data/pd-2379 /home/tidb/tidb-deploy/pd-2379
10.110.7.41:9090 prometheus 10.110.7.41 9090/12020 linux/x86_64 Down /home/tidb/tidb-data/prometheus-9090 /home/tidb/tidb-deploy/prometheus-9090
10.110.7.41:4000 tidb 10.110.7.41 4000/10080 linux/x86_64 Up - /home/tidb/tidb-deploy/tidb-4000
10.110.7.41:20160 tikv 10.110.7.41 20160/20180 linux/x86_64 N/A /home/tidb/tidb-data/tikv-20160 /home/tidb/tidb-deploy/tikv-20160
Total nodes: 6
[tidb@xsky-node3 ~]$ netstat -ano|grep 4000
tcp 0 0 127.0.0.1:4000 0.0.0.0:* LISTEN off (0.00/0/0)
unix 2 [ ACC ] STREAM LISTENING 4214620330 /tmp/tidb-4000.sock
unix 2 [ ] DGRAM 4210840009
[tidb@xsky-node3 ~]$ lsof -i:4000
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tidb-serv 18762 tidb 20u IPv4 4214620329 0t0 TCP localhost:terabase (LISTEN)
[tidb@xsky-node3 ~]$ kill -9 18762
[tidb@xsky-node3 ~]$ lsof -i:4000
[tidb@xsky-node3 ~]$ netstat -ano|grep 4000
unix 2 [ ] DGRAM 4210840009
[tidb@xsky-node3 ~]$ tiup cluster display tidb-cluster
tiup is checking updates for component cluster …
Starting component cluster: /home/tidb/.tiup/components/cluster/v1.12.3/tiup-cluster display tidb-cluster
Cluster type: tidb
Cluster name: tidb-cluster
Cluster version: v7.2.0
Deploy user: tidb
SSH type: builtin
Grafana URL: http://10.110.7.41:3000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

10.110.7.41:9093 alertmanager 10.110.7.41 9093/9094 linux/x86_64 Down /home/tidb/tidb-data/alertmanager-9093 /home/tidb/tidb-deploy/alertmanager-9093
10.110.7.41:3000 grafana 10.110.7.41 3000 linux/x86_64 Down - /home/tidb/tidb-deploy/grafana-3000
10.110.7.41:2379 pd 10.110.7.41 2379/2380 linux/x86_64 Down /home/tidb/tidb-data/pd-2379 /home/tidb/tidb-deploy/pd-2379
10.110.7.41:9090 prometheus 10.110.7.41 9090/12020 linux/x86_64 Down /home/tidb/tidb-data/prometheus-9090 /home/tidb/tidb-deploy/prometheus-9090
10.110.7.41:4000 tidb 10.110.7.41 4000/10080 linux/x86_64 Down - /home/tidb/tidb-deploy/tidb-4000
10.110.7.41:20160 tikv 10.110.7.41 20160/20180 linux/x86_64 N/A /home/tidb/tidb-data/tikv-20160 /home/tidb/tidb-deploy/tikv-20160
Total nodes: 6
[tidb@xsky-node3 ~]$ netstat -ano|grep 20160
tcp6 0 0 127.0.0.1:20160 :::* LISTEN off (0.00/0/0)
[tidb@xsky-node3 ~]$ lsof -i:20160
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tikv-serv 18474 tidb 132u IPv6 4214605372 0t0 TCP localhost:20160 (LISTEN)
tikv-serv 18474 tidb 134u IPv6 4214605371 0t0 TCP localhost:20160 (LISTEN)
tikv-serv 18474 tidb 136u IPv6 4214605373 0t0 TCP localhost:20160 (LISTEN)
tikv-serv 18474 tidb 137u IPv6 4214605374 0t0 TCP localhost:20160 (LISTEN)
tikv-serv 18474 tidb 138u IPv6 4214605375 0t0 TCP localhost:20160 (LISTEN)
[tidb@xsky-node3 ~]$ kill -9 18474
[tidb@xsky-node3 ~]$ lsof -i:20160
[tidb@xsky-node3 ~]$ netstat -ano|grep 20160
[tidb@xsky-node3 ~]$ tiup cluster display tidb-cluster
tiup is checking updates for component cluster …
Starting component cluster: /home/tidb/.tiup/components/cluster/v1.12.3/tiup-cluster display tidb-cluster
Cluster type: tidb
Cluster name: tidb-cluster
Cluster version: v7.2.0
Deploy user: tidb
SSH type: builtin
Grafana URL: http://10.110.7.41:3000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

10.110.7.41:9093 alertmanager 10.110.7.41 9093/9094 linux/x86_64 Down /home/tidb/tidb-data/alertmanager-9093 /home/tidb/tidb-deploy/alertmanager-9093
10.110.7.41:3000 grafana 10.110.7.41 3000 linux/x86_64 Down - /home/tidb/tidb-deploy/grafana-3000
10.110.7.41:2379 pd 10.110.7.41 2379/2380 linux/x86_64 Down /home/tidb/tidb-data/pd-2379 /home/tidb/tidb-deploy/pd-2379
10.110.7.41:9090 prometheus 10.110.7.41 9090/12020 linux/x86_64 Down /home/tidb/tidb-data/prometheus-9090 /home/tidb/tidb-deploy/prometheus-9090
10.110.7.41:4000 tidb 10.110.7.41 4000/10080 linux/x86_64 Down - /home/tidb/tidb-deploy/tidb-4000
10.110.7.41:20160 tikv 10.110.7.41 20160/20180 linux/x86_64 N/A /home/tidb/tidb-data/tikv-20160 /home/tidb/tidb-deploy/tikv-20160
Total nodes: 6
[tidb@xsky-node3 ~]$ netstat -ano|grep 20180
[tidb@xsky-node3 ~]$

启动集群时也是，提示成功，但普罗米修斯状态显示down
[tidb@xsky-node3 ~]$ tiup cluster start tidb-cluster
tiup is checking updates for component cluster …
Starting component cluster: /home/tidb/.tiup/components/cluster/v1.12.3/tiup-cluster start tidb-cluster
Starting cluster tidb-cluster…

  • [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-cluster/ssh/id_rsa.pub
  • [Parallel] - UserSSH: user=tidb, host=10.110.7.41
  • [Parallel] - UserSSH: user=tidb, host=10.110.7.41
  • [Parallel] - UserSSH: user=tidb, host=10.110.7.41
  • [Parallel] - UserSSH: user=tidb, host=10.110.7.41
  • [Parallel] - UserSSH: user=tidb, host=10.110.7.41
  • [Parallel] - UserSSH: user=tidb, host=10.110.7.41
  • [ Serial ] - StartCluster
    Starting component pd
    Starting instance 10.110.7.41:2379
    Start instance 10.110.7.41:2379 success
    Starting component tikv
    Starting instance 10.110.7.41:20160
    Start instance 10.110.7.41:20160 success
    Starting component tidb
    Starting instance 10.110.7.41:4000
    Start instance 10.110.7.41:4000 success
    Starting component prometheus
    Starting instance 10.110.7.41:9090
    Start instance 10.110.7.41:9090 success
    Starting component grafana
    Starting instance 10.110.7.41:3000
    Start instance 10.110.7.41:3000 success
    Starting component alertmanager
    Starting instance 10.110.7.41:9093
    Start instance 10.110.7.41:9093 success
    Starting component node_exporter
    Starting instance 10.110.7.41
    Start 10.110.7.41 success
    Starting component blackbox_exporter
    Starting instance 10.110.7.41
    Start 10.110.7.41 success
  • [ Serial ] - UpdateTopology: cluster=tidb-cluster
    Started cluster tidb-cluster successfully
    [tidb@xsky-node3 ~]$
    [tidb@xsky-node3 ~]$ tiup cluster display tidb-cluster
    tiup is checking updates for component cluster …
    Starting component cluster: /home/tidb/.tiup/components/cluster/v1.12.3/tiup-cluster display tidb-cluster
    Cluster type: tidb
    Cluster name: tidb-cluster
    Cluster version: v7.2.0
    Deploy user: tidb
    SSH type: builtin
    Dashboard URL: http://10.110.7.41:2379/dashboard
    Grafana URL: http://10.110.7.41:3000
    ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

10.110.7.41:9093 alertmanager 10.110.7.41 9093/9094 linux/x86_64 Up /home/tidb/tidb-data/alertmanager-9093 /home/tidb/tidb-deploy/alertmanager-9093
10.110.7.41:3000 grafana 10.110.7.41 3000 linux/x86_64 Up - /home/tidb/tidb-deploy/grafana-3000
10.110.7.41:2379 pd 10.110.7.41 2379/2380 linux/x86_64 Up|L|UI /home/tidb/tidb-data/pd-2379 /home/tidb/tidb-deploy/pd-2379
10.110.7.41:9090 prometheus 10.110.7.41 9090/12020 linux/x86_64 Down /home/tidb/tidb-data/prometheus-9090 /home/tidb/tidb-deploy/prometheus-9090
10.110.7.41:4000 tidb 10.110.7.41 4000/10080 linux/x86_64 Up - /home/tidb/tidb-deploy/tidb-4000
10.110.7.41:20160 tikv 10.110.7.41 20160/20180 linux/x86_64 Up /home/tidb/tidb-data/tikv-20160 /home/tidb/tidb-deploy/tikv-20160
Total nodes: 6

[tidb@xsky-node3 ~]$ netstat -ano|grep 9090
tcp6 0 0 :::9090 :::* LISTEN off (0.00/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50744 TIME_WAIT timewait (0.54/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50718 TIME_WAIT timewait (0.53/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50712 TIME_WAIT timewait (0.53/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50716 TIME_WAIT timewait (0.53/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50720 TIME_WAIT timewait (0.53/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50736 TIME_WAIT timewait (0.54/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50734 TIME_WAIT timewait (0.54/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50738 TIME_WAIT timewait (0.54/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50728 TIME_WAIT timewait (0.53/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50742 TIME_WAIT timewait (0.54/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50740 TIME_WAIT timewait (0.53/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50726 TIME_WAIT timewait (0.53/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50714 TIME_WAIT timewait (0.53/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50710 TIME_WAIT timewait (0.53/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50724 TIME_WAIT timewait (0.53/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50732 TIME_WAIT timewait (0.53/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50730 TIME_WAIT timewait (0.53/0/0)
tcp6 0 0 10.110.7.41:9090 10.110.7.41:50722 TIME_WAIT timewait (0.53/0/0)

请帮忙分析下原因。

2

我已找到原因。

3

什么原因~解决了吗？

4

解决了，因为其他应用程序占用端口。

1 个赞
5

厉害了

6

可以的 :100:

7

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。