tiflash 8123被占用启动失败

TiDBer_hwEZA4rV · 2022 年2 月 28 日 06:43

[2022/02/28 14:39:59.076 +08:00] [INFO] [] [“Application: Flash grpc server listening on [10.21.206.77:3930]”] [thread_id=1]
[2022/02/28 14:39:59.077 +08:00] [INFO] [] [“Application: Begin to shut down flash grpc server”] [thread_id=1]
[2022/02/28 14:39:59.077 +08:00] [INFO] [] [“Application: Shut down flash grpc server”] [thread_id=1]
[2022/02/28 14:39:59.077 +08:00] [INFO] [] [“Application: Begin to shut down flash service”] [thread_id=1]
[2022/02/28 14:39:59.080 +08:00] [INFO] [] [“Application: Shut down flash service”] [thread_id=1]
[2022/02/28 14:39:59.080 +08:00] [INFO] [] [“Application: Shutting down storages.”] [thread_id=1]
[2022/02/28 14:39:59.080 +08:00] [DEBUG] [] [“Application: Shutted down storages.”] [thread_id=1]
[2022/02/28 14:39:59.083 +08:00] [DEBUG] [] [“PageStorage: RegionPersister gcApply remove 2 invalid snapshots, 1 snapshots left, longest lifetime 0.000 seconds, created from thread_id 0”] [thread_id=4]
[2022/02/28 14:40:01.032 +08:00] [DEBUG] [] [“Application: Destroyed global context.”] [thread_id=1]
[2022/02/28 14:40:01.032 +08:00] [INFO] [] [“Application: let tiflash proxy shutdown”] [thread_id=1]
[2022/02/28 14:40:01.032 +08:00] [INFO] [] [“Application: wait for tiflash proxy thread to join”] [thread_id=1]
[2022/02/28 14:40:01.127 +08:00] [INFO] [] [“Application: tiflash proxy thread is joined”] [thread_id=1]
[2022/02/28 14:40:01.127 +08:00] [ERROR] [] [“Application: Net Exception: Address already in use: 0.0.0.0:8123”] [thread_id=1]
[2022/02/28 14:40:01.127 +08:00] [INFO] [] [“Application: shutting down”] [thread_id=1]
[2022/02/28 14:40:01.127 +08:00] [DEBUG] [] [“Application: Uninitializing subsystem: Logging Subsystem”] [thread_id=1]
[2022/02/28 14:40:01.127 +08:00] [INFO] [] [“BaseDaemon: Stop SignalListener thread”] [thread_id=5]

tiup启动成功，但实际服务错误现实端口被占用，服务没起来，实际却没有被占用

啦啦啦啦啦 · 2022 年2 月 28 日 06:47

lsof -i:8123看看端口占用情况

TiDBer_hwEZA4rV · 2022 年2 月 28 日 07:01

通过再次确认，这个端口和clickhouse冲突了，8123是对应服务端口可以指定么

啦啦啦啦啦 · 2022 年2 月 28 日 07:05

扩容的话可以在scale-out.yaml里直接指定

TiDBer_hwEZA4rV · 2022 年2 月 28 日 07:14

tiup cluster edit-config tidb-test2
Starting component cluster: /home/admin/.tiup/components/cluster/v1.5.3/tiup-cluster edit-config tidb-test2
New topology could not be saved: immutable field changed: TiFlashServers.0.HTTPPort changed from ‘8123’ to ‘8124’
Do you want to continue editing? [Y/n]: (default=Y) y
New topology could not be saved: immutable field changed: TiFlashServers.0.HTTPPort changed from ‘8123’ to ‘8124’
Do you want to continue editing? [Y/n]: (default=Y) Y
New topology could not be saved: immutable field changed: TiFlashServers.0.HTTPPort changed from ‘8123’ to ‘8124’
Do you want to continue editing? [Y/n]: (default=Y) Y
New topology could not be saved: immutable field changed: TiFlashServers.0.HTTPPort changed from ‘8123’ to ‘8124’
Do you want to continue editing? [Y/n]: (default=Y) n
已部署的可否修改

啦啦啦啦啦 · 2022 年2 月 28 日 07:29

反正tiflah也没启动成功，不如直接缩容在指定端口扩容

TiDBer_hwEZA4rV · 2022 年2 月 28 日 07:34

The component tiflash will become tombstone, maybe exists in several minutes or hours, after that you can use the prune command to clean it

缩容后tiup display还存在，怎么彻底删除

啦啦啦啦啦 · 2022 年2 月 28 日 07:38

先用tiup试试

tiup cluster scale-in <cluster-name> --node 10.0.1.4:9000

tiup操作失败的话可以手动缩容试试
https://docs.pingcap.com/zh/tidb/stable/scale-tidb-using-tiup#%E6%96%B9%E6%A1%88%E4%BA%8C%E6%89%8B%E5%8A%A8%E7%BC%A9%E5%AE%B9-tiflash-%E8%8A%82%E7%82%B9

TiDBer_hwEZA4rV · 2022 年2 月 28 日 07:49

tiup操作成功
但tiup dispaly还在
X:9000 tiflash X 9000/8123/3930/20170/20292/8234 linux/x86_64 N/A /data17/tiflash,/data18/tiflash,/data19/tiflash,/data20/tiflash /tidb-deploy/tiflash

TiDBer_hwEZA4rV · 2022 年2 月 28 日 07:55

tiup cluster edit-config tidb-test2
Starting component cluster: /home/admin/.tiup/components/cluster/v1.5.3/tiup-cluster edit-config tidb-test2
New topology could not be saved: immutable field changed: removed TiFlashServers.0.Host with value ‘10.21.206.77’, removed TiFlashServers.0.ssh_port with value ‘22’, removed TiFlashServers.0.TCPPort with value ‘9000’, removed TiFlashServers.0.HTTPPort with value ‘8123’, removed TiFlashServers.0.FlashServicePort with value ‘3930’, removed TiFlashServers.0.FlashProxyPort with value ‘20170’, removed TiFlashServers.0.FlashProxyStatusPort with value ‘20292’, removed TiFlashServers.0.StatusPort with value ‘8234’, removed TiFlashServers.0.DeployDir with value ‘/tidb-deploy/tiflash’, removed TiFlashServers.0.data_dir with value ‘/data17/tiflash,/data18/tiflash,/data19/tiflash,/data20/tiflash’, removed TiFlashServers.0.LogDir with value ‘/tidb-deploy/tiflash/log’, removed TiFlashServers.0.Offline with value ‘true’, removed TiFlashServers.0.Arch with value ‘amd64’, removed TiFlashServers.0.OS with value ‘linux’
Do you want to continue editing? [Y/n]: (default=Y) Y
New topology could not be saved: immutable field changed: removed TiFlashServers.0.Host with value ‘10.21.206.77’, removed TiFlashServers.0.ssh_port with value ‘22’, removed TiFlashServers.0.TCPPort with value ‘9000’, removed TiFlashServers.0.HTTPPort with value ‘8123’, removed TiFlashServers.0.FlashServicePort with value ‘3930’, removed TiFlashServers.0.FlashProxyPort with value ‘20170’, removed TiFlashServers.0.FlashProxyStatusPort with value ‘20292’, removed TiFlashServers.0.StatusPort with value ‘8234’, removed TiFlashServers.0.DeployDir with value ‘/tidb-deploy/tiflash’, removed TiFlashServers.0.data_dir with value ‘/data17/tiflash,/data18/tiflash,/data19/tiflash,/data20/tiflash’, removed TiFlashServers.0.LogDir with value ‘/tidb-deploy/tiflash/log’, removed TiFlashServers.0.Offline with value ‘true’, removed TiFlashServers.0.Arch with value ‘amd64’, removed TiFlashServers.0.OS with value ‘linux’
Do you want to continue editing? [Y/n]: (default=Y) Y
New topology could not be saved: immutable field changed: removed TiFlashServers.0.Host with value ‘10.21.206.77’, removed TiFlashServers.0.ssh_port with value ‘22’, removed TiFlashServers.0.TCPPort with value ‘9000’, removed TiFlashServers.0.HTTPPort with value ‘8123’, removed TiFlashServers.0.FlashServicePort with value ‘3930’, removed TiFlashServers.0.FlashProxyPort with value ‘20170’, removed TiFlashServers.0.FlashProxyStatusPort with value ‘20292’, removed TiFlashServers.0.StatusPort with value ‘8234’, removed TiFlashServers.0.DeployDir with value ‘/tidb-deploy/tiflash’, removed TiFlashServers.0.data_dir with value ‘/data17/tiflash,/data18/tiflash,/data19/tiflash,/data20/tiflash’, removed TiFlashServers.0.LogDir with value ‘/tidb-deploy/tiflash/log’, removed TiFlashServers.0.Offline with value ‘true’, removed TiFlashServers.0.Arch with value ‘amd64’, removed TiFlashServers.0.OS with value ‘linux’
Do you want to continue editing? [Y/n]: (default=Y) G
New topology could not be saved: immutable field changed: removed TiFlashServers.0.Host with value ‘10.21.206.77’, removed TiFlashServers.0.ssh_port with value ‘22’, removed TiFlashServers.0.TCPPort with value ‘9000’, removed TiFlashServers.0.HTTPPort with value ‘8123’, removed TiFlashServers.0.FlashServicePort with value ‘3930’, removed TiFlashServers.0.FlashProxyPort with value ‘20170’, removed TiFlashServers.0.FlashProxyStatusPort with value ‘20292’, removed TiFlashServers.0.StatusPort with value ‘8234’, removed TiFlashServers.0.DeployDir with value ‘/tidb-deploy/tiflash’, removed TiFlashServers.0.data_dir with value ‘/data17/tiflash,/data18/tiflash,/data19/tiflash,/data20/tiflash’, removed TiFlashServers.0.LogDir with value ‘/tidb-deploy/tiflash/log’, removed TiFlashServers.0.Offline with value ‘true’, removed TiFlashServers.0.Arch with value ‘amd64’, removed TiFlashServers.0.OS with value ‘linux’
Do you want to continue editing? [Y/n]: (default=Y) Y
New topology could not be saved: immutable field changed: removed TiFlashServers.0.Host with value ‘10.21.206.77’, removed TiFlashServers.0.ssh_port with value ‘22’, removed TiFlashServers.0.TCPPort with value ‘9000’, removed TiFlashServers.0.HTTPPort with value ‘8123’, removed TiFlashServers.0.FlashServicePort with value ‘3930’, removed TiFlashServers.0.FlashProxyPort with value ‘20170’, removed TiFlashServers.0.FlashProxyStatusPort with value ‘20292’, removed TiFlashServers.0.StatusPort with value ‘8234’, removed TiFlashServers.0.DeployDir with value ‘/tidb-deploy/tiflash’, removed TiFlashServers.0.data_dir with value ‘/data17/tiflash,/data18/tiflash,/data19/tiflash,/data20/tiflash’, removed TiFlashServers.0.LogDir with value ‘/tidb-deploy/tiflash/log’, removed TiFlashServers.0.Offline with value ‘true’, removed TiFlashServers.0.Arch with value ‘amd64’, removed TiFlashServers.0.OS with value ‘linux’
Do you want to continue editing? [Y/n]: (default=Y)

手动删除最后一步配置删除tiflash无法更新，点Y就重新进入编辑页面

啦啦啦啦啦 · 2022 年2 月 28 日 08:02

tiup cluster scale-in <cluster-name> --node 10.0.1.4:9000 --force
加–force强制下线试试,要是生产环境的话谨慎点，别下线错了

TiDBer_hwEZA4rV · 2022 年2 月 28 日 08:13

tiup cluster scale-in tidb-test2 --node 10.21.206.77:9000 --force

Starting component cluster: /home/admin/.tiup/components/cluster/v1.5.3/tiup-cluster display tidb-test2
Cluster type: tidb
Cluster name: tidb-test2
Cluster version: v5.1.1
Deploy user: admin
SSH type: builtin
Dashboard URL: http://10.21.84.24:2379/dashboard
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

10.21.84.24:2379 pd 10.21.84.24 2379/2380 linux/x86_64 Up|L|UI /data17/pd-2379 /home/admin/tidb-deploy/pd-2379
10.21.84.24:4000 tidb 10.21.84.24 4000/10080 linux/x86_64 Up - /home/admin/tidb-deploy/tidb-4000
10.21.206.80:20160 tikv 10.21.206.80 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160
10.21.83.130:20160 tikv 10.21.83.130 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160
10.21.84.151:20160 tikv 10.21.84.151 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160

tikv节点变成offline了

啦啦啦啦啦 · 2022 年2 月 28 日 08:17

刚才是只指定下线tifash节点吗，方便的话发下tiup cluster display 看看集群拓扑吧，敏感信息可以打码

TiDBer_hwEZA4rV · 2022 年2 月 28 日 08:19

[admin@tidb-p-192168065035 ~]$ tiup cluster display tidb-test2
Starting component cluster: /home/admin/.tiup/components/cluster/v1.5.3/tiup-cluster display tidb-test2
Cluster type: tidb
Cluster name: tidb-test2
Cluster version: v5.1.1
Deploy user: admin
SSH type: builtin
Dashboard URL: http://10.21.84.24:2379/dashboard
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

10.21.84.24:2379 pd 10.21.84.24 2379/2380 linux/x86_64 Up|L|UI /data17/pd-2379 /home/admin/tidb-deploy/pd-2379
10.21.84.24:4000 tidb 10.21.84.24 4000/10080 linux/x86_64 Up - /home/admin/tidb-deploy/tidb-4000
10.21.206.80:20160 tikv 10.21.206.80 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160
10.21.83.130:20160 tikv 10.21.83.130 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160
10.21.84.151:20160 tikv 10.21.84.151 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160

TiDBer_hwEZA4rV · 2022 年2 月 28 日 08:20

是的，
[admin@tidb-p-192168065035 ~]$ tiup cluster scale-in tidb-test2 --node 10.21.206.77:9000 --force
Starting component cluster: /home/admin/.tiup/components/cluster/v1.5.3/tiup-cluster scale-in tidb-test2 --node 10.21.206.77:9000 --force
Forcing scale in is unsafe and may result in data loss for stateful components.
The process is irreversible and could NOT be cancelled.
Only use --force when some of the servers are already permanently offline.
Are you sure to continue? [y/N]:(default=N) y
This operation will delete the 10.21.206.77:9000 nodes in tidb-test2 and all their data.
Do you want to continue? [y/N]:(default=N) y
Scale-in nodes…

[ Serial ] - SSHKeySet: privateKey=/home/admin/.tiup/storage/cluster/clusters/tidb-test2/ssh/id_rsa, publicKey=/home/admin/.tiup/storage/cluster/clusters/tidb-test2/ssh/id_rsa.pub
[Parallel] - UserSSH: user=admin, host=10.21.206.77
[Parallel] - UserSSH: user=admin, host=10.21.84.24
[Parallel] - UserSSH: user=admin, host=10.21.83.130
[Parallel] - UserSSH: user=admin, host=10.21.84.24
[Parallel] - UserSSH: user=admin, host=10.21.84.151
[Parallel] - UserSSH: user=admin, host=10.21.206.80
[ Serial ] - ClusterOperate: operation=ScaleInOperation, options={Roles:[] Nodes:[10.21.206.77:9000] Force:true SSHTimeout:5 OptTimeout:120 APITimeout:300 IgnoreConfigCheck:false NativeSSH:false SSHType: CleanupData:false CleanupLog:false RetainDataRoles:[] RetainDataNodes:[] ShowUptime:false JSON:false Operation:StartOperation}
Stopping component tiflash
Stopping instance 10.21.206.77
Stop tiflash 10.21.206.77:9000 success
Destroying component tiflash
Destroying instance 10.21.206.77
Destroy 10.21.206.77 success

Destroy tiflash paths: [/data20/tiflash /tidb-deploy/tiflash/log /tidb-deploy/tiflash /etc/systemd/system/tiflash-9000.service /data17/tiflash /data18/tiflash /data19/tiflash]
Stopping component node_exporter
Stopping instance 10.21.206.77
Stop 10.21.206.77 success
Stopping component blackbox_exporter
Stopping instance 10.21.206.77
Stop 10.21.206.77 success
Destroying monitored 10.21.206.77
Destroying instance 10.21.206.77
Destroy monitored on 10.21.206.77 success
Delete public key 10.21.206.77
Delete public key 10.21.206.77 success

[ Serial ] - UpdateMeta: cluster=tidb-test2, deleted='10.21.206.77:9000'
[ Serial ] - UpdateTopology: cluster=tidb-test2
Refresh instance configs
- Regenerate config pd -> 10.21.84.24:2379 … Done
- Regenerate config tikv -> 10.21.84.151:20160 … Done
- Regenerate config tikv -> 10.21.83.130:20160 … Done
- Regenerate config tikv -> 10.21.206.80:20160 … Done
- Regenerate config tidb -> 10.21.84.24:4000 … Done
  Scaled cluster tidb-test2 in successfully
  [admin@tidb-p-192168065035 ~]$ tiup cluster display tidb-test2
  Starting component cluster: /home/admin/.tiup/components/cluster/v1.5.3/tiup-cluster display tidb-test2
  Cluster type: tidb
  Cluster name: tidb-test2
  Cluster version: v5.1.1
  Deploy user: admin
  SSH type: builtin
  Dashboard URL: http://10.21.84.24:2379/dashboard
  ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

10.21.84.24:2379 pd 10.21.84.24 2379/2380 linux/x86_64 Up|L|UI /data17/pd-2379 /home/admin/tidb-deploy/pd-2379
10.21.84.24:4000 tidb 10.21.84.24 4000/10080 linux/x86_64 Up - /home/admin/tidb-deploy/tidb-4000
10.21.206.80:20160 tikv 10.21.206.80 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160
10.21.83.130:20160 tikv 10.21.83.130 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160
10.21.84.151:20160 tikv 10.21.84.151 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160
Total nodes: 5

[admin@tidb-p-192168065035 ~]$
[admin@tidb-p-192168065035 ~]$ tiup cluster display tidb-test2
Starting component cluster: /home/admin/.tiup/components/cluster/v1.5.3/tiup-cluster display tidb-test2
Cluster type: tidb
Cluster name: tidb-test2
Cluster version: v5.1.1
Deploy user: admin
SSH type: builtin
Dashboard URL: http://10.21.84.24:2379/dashboard
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

10.21.84.24:2379 pd 10.21.84.24 2379/2380 linux/x86_64 Up|L|UI /data17/pd-2379 /home/admin/tidb-deploy/pd-2379
10.21.84.24:4000 tidb 10.21.84.24 4000/10080 linux/x86_64 Up - /home/admin/tidb-deploy/tidb-4000
10.21.206.80:20160 tikv 10.21.206.80 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160
10.21.83.130:20160 tikv 10.21.83.130 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160
10.21.84.151:20160 tikv 10.21.84.151 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160
Total nodes: 5
[admin@tidb-p-192168065035 ~]$ tiup cluster display tidb-test2
Starting component cluster: /home/admin/.tiup/components/cluster/v1.5.3/tiup-cluster display tidb-test2
Cluster type: tidb
Cluster name: tidb-test2
Cluster version: v5.1.1
Deploy user: admin
SSH type: builtin
Dashboard URL: http://10.21.84.24:2379/dashboard
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

10.21.84.24:2379 pd 10.21.84.24 2379/2380 linux/x86_64 Up|L|UI /data17/pd-2379 /home/admin/tidb-deploy/pd-2379
10.21.84.24:4000 tidb 10.21.84.24 4000/10080 linux/x86_64 Up - /home/admin/tidb-deploy/tidb-4000
10.21.206.80:20160 tikv 10.21.206.80 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160
10.21.83.130:20160 tikv 10.21.83.130 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160
10.21.84.151:20160 tikv 10.21.84.151 20160/20180 linux/x86_64 Offline /data17/tikv-20160 /home/admin/tidb-deploy/tikv-20160

啦啦啦啦啦 · 2022 年2 月 28 日 08:30

我按你的操作试了下，我这测试没问题，看你返回的日志也是只下线tiflash,tikv咋offline了。。。
看看tikv的日志为啥offline了

TiDBer_hwEZA4rV · 2022 年2 月 28 日 09:01

我销毁再重新创建了，另外咨询下，tikv可以指定副本数么，不是tiflash

啦啦啦啦啦 · 2022 年2 月 28 日 09:08

还是重建快啊
可以，pd的max-replicas这个参数控制

Orange_sword · 2022 年2 月 28 日 09:18

重搞，粗暴简单

system · 2022 年10 月 31 日 19:18

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。