线上一套tidb集群部署了ticdc后,将任务创建好,总共18个任务,17个任务都能正常将binlog同步到下游kafka,但是有一个任务在kafka没有找到对应的binlog。
排查发现cdc在不断的重启。
1、通过display查看集群状态,cdc的状态一直在down跟up之间切换。
2、分析cdc日志发现,cdc.log一直在刷日志,但是都是info类跟warn类日志,cdc_stderr.log日志一直在刷下面的日志段。
goroutine 4789 [running]:
github.com/pingcap/ticdc/cdc/sink/codec.rowEventToMaxwellMessage(0xc007058080, 0x2fa3e80, 0xc009b5fec0)
github.com/pingcap/ticdc@/cdc/sink/codec/maxwell.go:105 +0xfae
github.com/pingcap/ticdc/cdc/sink/codec.(*MaxwellEventBatchEncoder).AppendRowChangedEvent(0xc009b5fec0, 0xc007058080, 0x3, 0x2, 0x1)
github.com/pingcap/ticdc@/cdc/sink/codec/maxwell.go:160 +0x2f
github.com/pingcap/ticdc/cdc/sink.(*mqSink).runWorker(0xc00a3da630, 0x2f70da0, 0xc00903be80, 0x3, 0x0, 0x0)
github.com/pingcap/ticdc@/cdc/sink/mq.go:351 +0x3c8
github.com/pingcap/ticdc/cdc/sink.(*mqSink).run.func1(0xc0342c2f68, 0xc02d6c0700)
github.com/pingcap/ticdc@/cdc/sink/mq.go:281 +0x46
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc026329530, 0xc009b5fe80)
golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:54 +0x66
正常的任务的点位都是正常在变的
"status": {
"resolved-ts": 444249339662434305,
"checkpoint-ts": 444249339518255105,
"admin-job-type": 0
},
不正常的任务的点位异常
"resolved-ts": 正常变化,
"checkpoint-ts": 固定值,不变化,