cdc输出为canal-json格式时,ts=0的问题。

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:

【TiDB 版本】4.0.10

【问题描述】这里面的ts字段为0,这个是不对的

{“id”:0,“database”:“sold_product”,“table”:“soldItem”,“pkNames”:[“itemId”],“isDdl”:false,“type”:“UPDATE”,“es”:1612503773357,“ts”:0,“sql”:"",“sqlType”:{“addTime”:-5,“approach”:5,“author”:12,“bearShipping”:1,“beginSaleTime”:-5,“bgImgUrl”:12,“bizType”:5,“booklibId”:-5,“catId”:3,“certifyStatus”:1,“discount”:5,“endFlashSaleTime”:-5,“endSaleTime”:-5,“imgShowType”:5,“imgUrl”:12,“isBuildIndex”:5,“isDelete”:5,“isFlashSale”:5,“isNewBook”:5,“isOnSale”:5,“isPreSale”:5,“isRelatedISBN”:5,“isSoldOut”:5,“isSyncISBN”:5,“isUseMould”:5,“isbn”:12,“itemId”:3,“itemName”:12,“itemSn”:12,“limitBuyNum”:5,“mouldId”:-5,“myCatId”:-5,“number”:4,“oriPrice”:3,“preSaleNum”:4,“press”:12,“price”:3,“productArea”:3,“pubDate”:12,“quality”:5,“reCertifyStatus”:5,“soldTime”:-5,“startFlashSaleTime”:-5,“updateTime”:93,“userId”:-5,“weight”:7,“weightPiece”:7,“years”:3,“yearsGroup”:5},“mysqlType”:{“addTime”:“int”,“approach”:“tinyint”,“author”:“varchar”,“bearShipping”:“enum”,“beginSaleTime”:“int”,“bgImgUrl”:“varchar”,“bizType”:“tinyint”,“booklibId”:“int”,“catId”:“bigint”,“certifyStatus”:“enum”,“discount”:“tinyint”,“endFlashSaleTime”:“int”,“endSaleTime”:“int”,“imgShowType”:“tinyint”,“imgUrl”:“varchar”,“isBuildIndex”:“tinyint”,“isDelete”:“tinyint”,“isFlashSale”:“tinyint”,“isNewBook”:“tinyint”,“isOnSale”:“tinyint”,“isPreSale”:“tinyint”,“isRelatedISBN”:“tinyint”,“isSoldOut”:“tinyint”,“isSyncISBN”:“tinyint”,“isUseMould”:“tinyint”,“isbn”:“varchar”,“itemId”:“bigint”,“itemName”:“varchar”,“itemSn”:“varchar”,“limitBuyNum”:“tinyint”,“mouldId”:“int”,“myCatId”:“int”,“number”:“smallint”,“oriPrice”:“decimal”,“preSaleNum”:“smallint”,“press”:“varchar”,“price”:“decimal”,“productArea”:“bigint”,“pubDate”:“varchar”,“quality”:“tinyint”,“reCertifyStatus”:“tinyint”,“soldTime”:“int”,“startFlashSaleTime”:“int”,“updateTime”:“timestamp”,“userId”:“int”,“weight”:“float”,“weightPiece”:“float”,“years”:“bigint”,“yearsGroup”:“tinyint”},“data”:[{“addTime”:“1115875179”,“approach”:“0”,“author”:“胡源改编 聂昌硕 马永进 海晨绘画”,“bearShipping”:“1”,“beginSaleTime”:“1”,“bgImgUrl”:"",“bizType”:“1”,“booklibId”:“0”,“catId”:“7000000000000000”,“certifyStatus”:“2”,“discount”:“100”,“endFlashSaleTime”:“0”,“endSaleTime”:“1352967356”,“imgShowType”:“0”,“imgUrl”:"",“isBuildIndex”:“1”,“isDelete”:“0”,“isFlashSale”:“0”,“isNewBook”:“0”,“isOnSale”:“0”,“isPreSale”:“0”,“isRelatedISBN”:“0”,“isSoldOut”:“1”,“isSyncISBN”:“0”,“isUseMould”:“0”,“isbn”:"",“itemId”:“1710035”,“itemName”:"《神跤甄三》之六:苦尽甜来",“itemSn”:"",“limitBuyNum”:“0”,“mouldId”:“0”,“myCatId”:“0”,“number”:“0”,“oriPrice”:“0.34”,“preSaleNum”:“0”,“press”:“内蒙古人民”,“price”:“2.01”,“productArea”:“6003001000”,“pubDate”:“1985-01-01”,“quality”:“90”,“reCertifyStatus”:“1”,“soldTime”:“1169008509”,“startFlashSaleTime”:“0”,“updateTime”:“2021-02-05 13:42:53”,“userId”:“32165”,“weight”:“0”,“weightPiece”:“0”,“years”:“1004000000”,“yearsGroup”:“0”}],“old”:[{“addTime”:“1115875179”,“approach”:“0”,“author”:“胡源改编 聂昌硕 马永进 海晨绘画”,“bearShipping”:“1”,“beginSaleTime”:“1”,“bgImgUrl”:"",“bizType”:“1”,“booklibId”:“0”,“catId”:“7000000000000000”,“certifyStatus”:“2”,“discount”:“100”,“endFlashSaleTime”:“0”,“endSaleTime”:“1352967356”,“imgShowType”:“0”,“imgUrl”:"",“isBuildIndex”:“1”,“isDelete”:“0”,“isFlashSale”:“0”,“isNewBook”:“0”,“isOnSale”:“0”,“isPreSale”:“0”,“isRelatedISBN”:“0”,“isSoldOut”:“1”,“isSyncISBN”:“0”,“isUseMould”:“0”,“isbn”:"",“itemId”:“1710035”,“itemName”:"《神跤甄三》之六:苦尽甜来",“itemSn”:"",“limitBuyNum”:“0”,“mouldId”:“0”,“myCatId”:“0”,“number”:“0”,“oriPrice”:“0.34”,“preSaleNum”:“0”,“press”:“内蒙古人民”,“price”:“2.00”,“productArea”:“6003001000”,“pubDate”:“1985-01-01”,“quality”:“90”,“reCertifyStatus”:“1”,“soldTime”:“1169008509”,“startFlashSaleTime”:“0”,“updateTime”:“2020-12-24 12:21:18”,“userId”:“32165”,“weight”:“0”,“weightPiece”:“0”,“years”:“1004000000”,“yearsGroup”:“0”}]}


若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

感谢您的反馈。

根据 Canal 协议的定义,ts 字段是 build time。但实际上我们调研时参考的两个 Canal Consumer 的实现 (Alibaba 与 Flink) 在处理消息的时候均没有处理 ts 字段。考虑到 TiCDC 为分布式架构,多节点写入时有可能出现时序错乱的问题,这个 ts 字段我们特意留空了。

如果您需要 TiDB 侧的写入时间,可以使用 es 字段。如果您遇到其他障碍或有其他需求可以继续在这里反馈,谢谢~