dm master扩容启动失败

🪐 TiDB 技术问题 备份&数据迁移
1

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】
【复现路径】做过哪些操作出现的问题
【遇到的问题：问题现象及影响】
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【附件：截图/日志/监控】

image
image1920×406 77.5 KB

master日志

dm-master.log (5.8 MB)

2

你这日志对吗？2进制文件啊？

3

二进制文件怎么找？日志应该是对的

4

我打开是这个样的啊，你打开试下。。。。

image
image749×560 4.95 KB

5

[2024/01/17 22:33:01.857 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:33:01.857 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:33:01.863 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]
[2024/01/17 22:33:17.170 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:33:17.170 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:33:17.174 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]
[2024/01/17 22:33:32.453 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:33:32.453 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:33:32.458 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]
[2024/01/17 22:33:47.696 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:33:47.696 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:33:47.704 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]
[2024/01/17 22:34:02.926 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:34:02.926 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:34:02.930 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]
[2024/01/17 22:34:18.196 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:34:18.196 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:34:18.201 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]
[2024/01/17 22:34:33.423 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:34:33.423 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:34:33.428 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]
[2024/01/17 22:34:48.560 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:34:48.561 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:34:48.565 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]

6

看不出啥。

7

消息：未能加入嵌入etcd:有一个成员未成功加入，请继续加入或删除它，解决方法：请检查主配置文件中的配置

8

https://docs.pingcap.com/zh/tidb-data-migration/v5.3/dm-error-handling
貌似没有38040的故障代码

9

fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”

10

扩容的配置文件有文件吗？ 同样的配置文件，我打开worker ，注释调master就没有问题。