TiCDC 扩容启动失败,

【 TiDB 使用环境】测试环境
【 TiDB 版本】v5.1.3
【遇到的问题】cdc无法启动,timed out waiting for port 8300 to be started after 2m0s
【问题现象及影响】 无法正常启动cdc,日志中init log

cdc-scale-out.yml

cdc_servers:
  - host: x.x.x.x
    port: 8300
    gc-ttl: 86400
    deploy_dir: /home/tidb/tidb50/tidb-deploy/cdc-8300
    data_dir: /home/tidb/tidb50/tidb-data/cdc-8300
    log_dir: /home/tidb/tidb50/tidb-deploy/cdc-8300/log

启动日志

## cdc_stderr.log 日志是空的
$ tailf -1000 cdc.log

[2022/04/07 10:42:04.074 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:42:19.324 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:42:34.574 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:42:49.824 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:43:05.074 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:43:20.324 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:43:35.574 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:43:50.825 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:44:06.074 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:44:21.324 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:44:36.574 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:44:51.824 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:45:07.073 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:45:22.324 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]
[2022/04/07 10:45:37.573 +08:00] [INFO] [util.go:100] ["init log"] [file=/home/tidb/tidb50/tidb-deploy/cdc-8300/log/cdc.log] [level=info]

集群中 ticdc 的状态是正常的么?

在集群中是 down

tiup cluster scale-out <cluster-name> scale-out.yaml

这个命令执行的结果是啥?

不行就缩掉,再来一遍

缩掉后,要删除目录信息…
避免有垃圾数据的干扰

1 个赞

重新扩容了一遍,在scale-in之后,cdc组件不在集群,重新scale-out,最终会卡住Starting component cdc,直到timeout

有没有 TiDB 的大佬提供一下排查思路

登陆到 cdc 服务器,在cdc 部署目录下的scripts 目录有一个 run_cdc.sh 的脚本,执行 sh -v run_cdc.sh 看看输出内容是什么

排查下来是时区timezone问题

      Local time: Thu 2022-04-07 15:21:18 CST
  Universal time: Thu 2022-04-07 07:21:18 UTC
        RTC time: Thu 2022-04-07 07:21:16
       Time zone: n/a (CST, +0800)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: n/a
1 个赞

该主题在最后一个回复创建后60天后自动关闭。不再允许新的回复。