node_exporter-9100.service服务启动失败

【 TiDB 使用环境】生产环境
【 TiDB 版本】v6.5.3
【复现路径】集群依次做过机房迁移、PD重建、扩容缩容。
【遇到的问题：问题现象及影响】
node_exporter-9100 reload报错
报错内容如下：

|2023-06-29T05:46:24.958+0800|DEBUG|retry error|{error: operation timed out after 2m0s}|
|---|---|---|---|
|2023-06-29T05:46:24.958+0800|DEBUG|TaskFinish|{task: StartCluster, error: failed to start: 10.11.209.128 node_exporter-9100.service, please check the instance's log() for more detail.: timed out waiting for port 9100 to be started after 2m0s, errorVerbose: timed out waiting for port 9100 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(*WaitFor).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ngithub.com/pingcap/tiup/pkg/cluster/spec.PortStarted\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:121\ngithub.com/pingcap/tiup/pkg/cluster/operation.systemctlMonitor.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:336\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594\nfailed to start: 10.11.209.128 node_exporter-9100.service, please check the instance's log() for more detail.}|
|2023-06-29T05:46:24.958+0800|INFO|Execute command finished|{code: 1, error: failed to start: 10.11.209.128 node_exporter-9100.service, please check the instance's log() for more detail.: timed out waiting for port 9100 to be started after 2m0s, errorVerbose: timed out waiting for port 9100 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(*WaitFor).Execute\n\tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ngithub.com/pingcap/tiup/pkg/cluster/spec.PortStarted\n\tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:121\ngithub.com/pingcap/tiup/pkg/cluster/operation.systemctlMonitor.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:336\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594\nfailed to start: 10.11.209.128 node_exporter-9100.service, please check the instance's log() for more detail.}|

检查对应节点下的node_exporter.log日志，没有任何记录。

检查下node_exporter进程在吗 如果不在 手动执行下tidb-deploy/monitored-9100/scripts/run_node_exporter.sh 这个启动脚本看看会不会出日志

应该是正常的吧

image1100×148 9.1 KB

中午手工关闭试试

我一般这么搞的
cd /etc/systemd/system/

systemctl start node_exporter-9100.service

systemctl start blackbox_exporter-9115.service

中午我试一下

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。