在单机上模拟部署生产环境集群,部署成功后启动不了

【 TiDB 使用环境】测试
【 TiDB 版本】v5.4.2
【复现路径】tiup部署新的测试集群
【遇到的问题:问题现象及影响】
部署成功后,集群启动不了。参考链接:在单机上模拟部署生产环境集群
【资源配置】
【附件:截图/日志/监控】
在单机上模拟部署生产环境集群,部署成功后启动不了。1pd,1kv,1tidb
使用配置文件:tiup cluster template > topology.yaml

Cluster test-cluster deployed successfully, you can start it with command: tiup cluster start test-cluster --init
[test-deploy@localhost ~]$ tiup cluster start test-cluster --init

执行命令tiup cluster start test-cluster --init后报错:
Error: failed to start tidb: failed to start: 10.0.0.56 tidb-4000.service, please check the instance’s log(/data/tidb/deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s

详细tidb_stderr.log日志:
{“level”:“warn”,“ts”:“2023-04-25T17:52:05.278+0800”,“caller”:“clientv3/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“endpoint://client-0c043b83-12c8-450e-91e8-546ff1efdc93/10.0.0.56:2379”,“attempt”:0,“error”:“rpc error: code = DeadlineExceeded desc = context deadline exceeded”}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x28148d7]

goroutine 1 [running]:
github.com/pingcap/tidb/ddl.(*ddl).close(0xc000bff260)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ddl/ddl.go:399 +0x77
github.com/pingcap/tidb/ddl.(*ddl).Stop(0xc000bff260, 0x0, 0x0)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ddl/ddl.go:327 +0x8a
github.com/pingcap/tidb/domain.(*Domain).Close(0xc0007fbe00)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/domain/domain.go:695 +0x377
github.com/pingcap/tidb/session.(*domainMap).Get.func1(0xc0013bf1a0, 0xc00133f8d8, 0x1369afc)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/tidb.go:86 +0x69e
github.com/pingcap/tidb/util.RunWithRetry(0x1e, 0x1f4, 0xc00189f938, 0x18, 0x6468280)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/misc.go:65 +0x7f
github.com/pingcap/tidb/session.(*domainMap).Get(0x642b450, 0x4538850, 0xc0006fbbd0, 0xc0007fbe00, 0x0, 0x0)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/tidb.go:71 +0x1f0
github.com/pingcap/tidb/session.createSessionWithOpt(0x4538850, 0xc0006fbbd0, 0x0, 0x0, 0xc10a068d909177ba, 0x34ef6bc)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2767 +0x59
github.com/pingcap/tidb/session.createSession(...)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2763
github.com/pingcap/tidb/session.runInBootstrapSession(0x4538850, 0xc0006fbbd0, 0x4097f48)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2746 +0x59
github.com/pingcap/tidb/session.BootstrapSession(0x4538850, 0xc0006fbbd0, 0x0, 0x0, 0x0)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2593 +0xfef
main.createStoreAndDomain(0x64312a0, 0x3ff6a97, 0x2c)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:296 +0x189
main.main()
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:202 +0x29e

什么系统、什么资源配置?

有参考官方要求配置吗?https://docs.pingcap.com/zh/tidb/stable/hardware-and-software-requirements

44核,128G内存。

有钱就是好

看这这么像内存分配地址冲突?

5.4.3提到个bug修复的,不知道有没有关系?换个版本试试?
https://docs.pingcap.com/zh/tidb/stable/release-5.4.3

  • 修复执行 SHOW WARNINGS 时可能会报 invalid memory address or nil pointer dereference 的问题 #31569

只有1台单机部署啊。 :joy: 为了测试tiflash而买的。没钱,只买了两台二手的,1台混合pd,kv,tidb,另1台tiflash

我们生产环境就是这个版, :sob:

invalid memory address or nil pointer dereference这种一般是触发go的报错了

用check命令看看环境有没有哪些配置不满足要求的

另外的集群平常在运维中也时不时出现go错误,不明所以然。

必须是通过的。

楼主,最后咋解决的?什么原因?

不知道什么错。多试几次就好了。我搭的是测试环境 :sweat_smile:

搞不清楚重装系统重新搭建最快