ticdc 异常

表渣本渣 · 2024 年6 月 28 日 08:17

【 TiDB 使用环境】生产环境
【 TiDB 版本】v5.4.1
【复现路径】
昨天晚上建表，对表进行加字段，写入数据，新建视等操作；然后 cdc 就频繁异常；

建表：（安全起见，更改了字段，表名，comment ,其他未更改）

CREATE TABLE dim.`1111_vest` (
  `id` int(11) NOT NULL COMMENT '主键',
  `1111_id` int(11) DEFAULT NULL COMMENT 'id',
  `1111_name` varchar(48) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '',
  `1111_name` varchar(100) DEFAULT NULL COMMENT '）',
  `qy_wx_id` varchar(48) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '',
  `1111_phone` varchar(24) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '',
  `1111_dep` varchar(100) DEFAULT NULL COMMENT '）',
  `owner_name` varchar(100) DEFAULT NULL COMMENT '',
  `owner_job_id` varchar(100) DEFAULT NULL COMMENT '',
  `owner_dep` varchar(100) DEFAULT NULL COMMENT '',
  `consult_id` int(11) DEFAULT NULL COMMENT '',
  `owner_222_id` int(11) DEFAULT NULL COMMENT '',
  `owner_333_id` int(11) DEFAULT NULL COMMENT 'id',
  `owner_333_name` varchar(50) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '',
  `owner_333_222_id` int(11) DEFAULT NULL COMMENT 'id，
  `owner_333_222_name` varchar(50) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '',
  `owner_333_center_id` int(11) DEFAULT NULL COMMENT 'id，',
  `owner_333_center_name` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '',
  `dddd_type` int(11) NOT NULL DEFAULT '0' COMMENT ' ',
  PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;

增加字段：

alter table dim.dim_2222_user add column `555_333_center_name` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '' after consult_id;
alter table dim.dim_2222_user add column `555_333_center_id` int(11) DEFAULT NULL COMMENT '，id' after consult_id;
alter table dim.dim_2222_user add column `555_333_111_name` varchar(50) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '' after consult_id;
alter table dim.dim_2222_user add column `555_333_111_id` int(11) DEFAULT NULL COMMENT '，id' after consult_id;
alter table dim.dim_2222_user add column `555_333_name` varchar(50) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '' after consult_id;
alter table dim.dim_2222_user add column `555_333_id` int(11) DEFAULT NULL COMMENT 'id，' after consult_id;
alter table dim.dim_2222_user add column `555_111_name` varchar(45) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '' after consult_id;
alter table dim.dim_2222_user add column `555_job_number` varchar(32) COLLATE utf8mb4_general_ci DEFAULT '' COMMENT '' after consult_id;
alter table dim.dim_2222_user add column `555_name` varchar(45) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '' after consult_id;
alter table dim.dim_2222_user add column `555_111_id` int(11) DEFAULT NULL COMMENT ' id' after consult_id;
alter table dim.dim_2222_user add column `555_consult_id` int(11) DEFAULT NULL COMMENT 'id' after consult_id;
alter table dim.dim_2222_user add column `000_333_center_name` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '' after consult_id;
alter table dim.dim_2222_user add column `000_333_center_id` int(11) DEFAULT NULL COMMENT 'id，id' after consult_id;
alter table dim.dim_2222_user add column `000_333_111_name` varchar(50) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '' after consult_id;
alter table dim.dim_2222_user add column `000_333_111_id` int(11) DEFAULT NULL COMMENT 'id，' after consult_id;
alter table dim.dim_2222_user add column `000_333_name` varchar(50) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '' after consult_id;
alter table dim.dim_2222_user add column `000_333_id` int(11) DEFAULT NULL COMMENT 'id，' after consult_id;
alter table dim.dim_2222_user add column `000_111_id` int(11) DEFAULT NULL COMMENT ' 对应的id' after consult_id;

然后cdc 停止同步，3个cdc 组件轮番挂，同步停止；
于是停掉昨天建表和加字段表的数据写入，
定位到就是dim.* 的同步异常，于是，删除，重建cdc 同步任务，

但是只要启动，就会报错；

持续到下午，启动，好了

在这期间，我对cdc 同步的任务多次停止，删除，重建；

丈二的和尚

【附件：截图/日志/监控】

日志找到异常：

[2024/06/27 23:43:27.459 +08:00] [ERROR] [client.go:752] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]
[2024/06/27 23:43:27.459 +08:00] [INFO] [client.go:666] ["[pd] exit tso dispatcher"] [dc-location=global]
[2024/06/27 23:43:27.459 +08:00] [INFO] [capture.go:323] ["the processor routine has exited"] [error="[CDC:ErrPDEtcdAPIError]etcd api call error: context canceled"] [errorVerbose="[CDC:ErrPDEtcdAPIError]etcd api call error: context canceled\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:30\ngithub.com/pingcap/tiflow/cdc/capture.(*Capture).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:476\ngithub.com/pingcap/tiflow/cdc/capture.(*Capture).run.func3\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:322\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2024/06/27 23:43:27.459 +08:00] [INFO] [acquirer.go:72] ["TimeAcquirer exit"]
[2024/06/27 23:43:27.459 +08:00] [INFO] [client.go:234] ["WatchWithChan exited"] [role=processor]
[2024/06/27 23:43:27.459 +08:00] [INFO] [capture.go:299] ["the owner routine has exited"] []
[2024/06/27 23:43:27.459 +08:00] [ERROR] [http_status.go:74] ["http server error"] [error="[CDC:ErrServeHTTP]serve http error: mux: server closed"] [errorVerbose="[CDC:ErrServeHTTP]serve http error: mux: server closed\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:30\ngithub.com/pingcap/tiflow/cdc.(*Server).startStatusHTTP.func1\n\tgithub.com/pingcap/tiflow/cdc/http_status.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2024/06/27 23:43:27.460 +08:00] [INFO] [capture.go:257] ["capture recovered"] [capture-id=99bbe3c0-fe7d-47f4-8619-684e223a4d7a]
[2024/06/27 23:43:27.460 +08:00] [INFO] [capture.go:234] ["the capture routine has exited"]
[2024/06/27 23:43:27.460 +08:00] [ERROR] [client.go:752] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]

表渣本渣 · 2024 年6 月 28 日 08:19

说明一点：我们这个cdc 同步的时候是 dim库下所有表

Billmay表妹 · 2024 年6 月 28 日 08:24

https://github.com/pingcap/tidb/issues/31335

6.x 上解决了，请升级到 6.x 以上的版本

表渣本渣 · 2024 年7 月 2 日 07:53

关闭这个sql

表渣本渣 · 2024 年7 月 2 日 07:55

cdc.log (3.1 MB)
测试环境日志

system · 2024 年8 月 31 日 07:56

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。