Tidb-ansiable启动tidb-server

乌云先生 · 2021 年1 月 27 日 07:08

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：

【TiDB 版本】
tidb_version = v3.0.1

【问题描述】
ansible-playbook rolling_update_monitor.yml --tags=prometheus
更新Prometheus配置重启时failed，报错如下:
[xxx]: Ansible FAILED! => playbook: rolling_update_monitor.yml; TASK: prometheus : create configuration file; message: {“changed”: false, “msg”: “AnsibleUndefinedVariable: ‘dict object’ has no attribute ‘alertmanager_servers’”}

若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出的打印结果，请务必全选并复制粘贴上传。

来了老弟 · 2021 年1 月 27 日 07:10

ansible-playbook rolling_update_monitor.yml 试下

乌云先生 · 2021 年1 月 27 日 07:56

先看下另个问题吧，扩容一个tidb-server之后，新节点tidb-serber已启动
报错如下：
[2021/01/27 10:45:35.003 +08:00] [ERROR] [kv.go:255] [“fail to load safepoint from pd”] [error=“context deadline exceeded”] [errorVerbose=“context deadline exceeded\ngithub.com/pingcap/errors.AddStack\ \t/home/
jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/errors.go:174\ngithub.com/pingcap/errors.Trace\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/error
s@v0.11.4/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/tikv.(*EtcdSafePointKV).Get\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/safepoint.go:108\ngithub.com/pingc
ap/tidb/store/tikv.loadSafePoint\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/safepoint.go:127\ngithub.com/pingcap/tidb/store/tikv.(*tikvStore).runSafePointChecker\ \t
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/kv.go:248\ runtime.goexit\ \t/usr/local/go/src/runtime/asm_amd64.s:1357”] [stack=“github.com/pingcap/tidb/store/tikv.(*tikvSt
ore).runSafePointChecker\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/kv.go:255”]
2021/01/27 10:45:58.494 terror.go:357: [error] EOF
github.com/pingcap/errors.AddStack
/home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/errors.go:174
github.com/pingcap/errors.Trace
/home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/juju_adaptor.go:15
github.com/pingcap/tidb/server.(*packetIO).readOnePacket
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/packetio.go:80
github.com/pingcap/tidb/server.(*packetIO).readPacket
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/packetio.go:105
github.com/pingcap/tidb/server.(*clientConn).readPacket
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/conn.go:265
github.com/pingcap/tidb/server.(*clientConn).readOptionalSSLRequestAndHandshakeResponse
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/conn.go:471
github.com/pingcap/tidb/server.(*clientConn).handshake
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/conn.go:172
github.com/pingcap/tidb/server.(*Server).onConn
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/server/server.go:345
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357

乌云先生 · 2021 年1 月 27 日 07:57

这个也不行，有点问题，多谢您哈，一会我再看下这个

来了老弟 · 2021 年1 月 27 日 09:33

这个参数在 tidb ansible 是必须配置的, 你看看 inventory 文件中,有木有把

乌云先生 · 2021 年1 月 27 日 09:37

有这个。没有alertmanager_servers

来了老弟 · 2021 年1 月 27 日 09:47

[alertmanager_servers]
貌似才是正确的标签名称,

高亮部分展示下

乌云先生 · 2021 年1 月 27 日 09:49

来了老弟 · 2021 年1 月 27 日 09:54

[alertmanager]
变成
[alertmanager_servers]

保存后,在进行其他操作, 如果 tidb-ansible 版本与 tidb 版本不一致请及更换, inventory 文件建议一点一点 vi, cp inventory 文件我们并不建议, 版本的变更, 里面参数会有变化. 请知晓

乌云先生 · 2021 年1 月 27 日 09:58

恩这个已经好了。多谢您。
另外有时间的话帮忙看下启动的tidb-server 日志报错。

[2021/01/27 17:14:24.205 +08:00] [ERROR] [kv.go:255] [“fail to load safepoint from pd”] [error=“context deadline exceeded”] [errorVerbose=“context deadline exceeded
github.com/pingcap/errors.AddStack
\t/home/
jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/errors.go:174
github.com/pingcap/errors.Trace
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/error
s@v0.11.4/juju_adaptor.go:15
github.com/pingcap/tidb/store/tikv.(*EtcdSafePointKV).Get
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/safepoint.go:108
github.com/pingc
ap/tidb/store/tikv.loadSafePoint
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/safepoint.go:127
github.com/pingcap/tidb/store/tikv.(*tikvStore).runSafePointChecker
\t
/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/kv.go:248
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”] [stack=“github.com/pingcap/tidb/store/tikv.(*tikvSt
ore).runSafePointChecker
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/kv.go:255”]
[2021/01/27 17:14:44.146 +08:00] [ERROR] [tidb.go:83] [“[ddl] init domain failed”] [error=“rpc error: code = Unknown desc = context deadline exceeded”] [errorVerbose=“rpc error: code = Unknown desc = context de
adline exceeded
github.com/pingcap/errors.AddStack
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/errors.go:174
github.com/pingcap/errors.Trace
\t/home/jenkins/age
nt/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/juju_adaptor.go:15
github.com/pingcap/tidb/ddl/util.(*schemaVersionSyncer).Init
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/githu
b.com/pingcap/tidb/ddl/util/syncer.go:169
github.com/pingcap/tidb/domain.(*Domain).Init
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/domain/domain.go:633
github.com/pingcap/tid
b/session.(*domainMap).Get.func1
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/session/tidb.go:79
github.com/pingcap/tidb/util.RunWithRetry
\t/home/jenkins/agent/workspace/tidb_
v3.0.12/go/src/github.com/pingcap/tidb/util/misc.go:54
github.com/pingcap/tidb/session.(*domainMap).Get
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/session/tidb.go:71
github.c
om/pingcap/tidb/session.createSession
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/session/session.go:1594
github.com/pingcap/tidb/session.BootstrapSession
\t/home/jenkins/agen
t/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/session/session.go:1503
main.createStoreAndDomain
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/tidb-server/main.go:210
m
ain.main
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/tidb-server/main.go:172
runtime.main
\t/usr/local/go/src/runtime/proc.go:203
runtime.goexit
\t/usr/local/go/src/runtime/
asm_amd64.s:1357”] [stack=“github.com/pingcap/tidb/session.(*domainMap).Get.func1
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/session/tidb.go:83
github.com/pingcap/tidb/util.Ru
nWithRetry
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/util/misc.go:54
github.com/pingcap/tidb/session.(*domainMap).Get
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/git
hub.com/pingcap/tidb/session/tidb.go:71
github.com/pingcap/tidb/session.createSession
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/session/session.go:1594
github.com/pingcap/ti
db/session.BootstrapSession
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/session/session.go:1503
main.createStoreAndDomain
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/g
ithub.com/pingcap/tidb/tidb-server/main.go:210
main.main
\t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/tidb-server/main.go:172
runtime.main
\t/usr/local/go/src/runtime/proc.go:
203”]

来了老弟 · 2021 年1 月 27 日 10:00

可否开新帖提供下完整的信息, 包括 tidb.log .

乌云先生 · 2021 年1 月 27 日 10:01

恩好的。你们是按这个算任务量吗哈哈

来了老弟 · 2021 年1 月 27 日 10:06

技术导向吧, 也不是专职回复. 主要是为了区分帖子内容, 让一个回复变得简单些.

乌云先生 · 2021 年1 月 27 日 10:10

好的。辛苦您。

来了老弟 · 2021 年1 月 27 日 10:11

客气了,

system · 2022 年10 月 31 日 19:05

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。