tikv结合ticdc导入T级数量时发现,所有tikv总是重启

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:

【TiDB 版本】
5.0.1

【问题描述】
tikv结合ticdc导入T级数量时发现,tikv总是重启

[2021/05/15 19:43:26.232 +08:00] [WARN] [store.rs:691] [“[store 3] handle 350 pending peers include 349 ready, 21 entries, 349 messages and 0 snapshots”] [takes=2778]
[2021/05/15 19:43:26.239 +08:00] [INFO] [raft.rs:1307] [“received a message with higher term from 2172045”] [“msg type”=MsgRequestVote] [message_term=31] [term=30] [from=2172045] [raft_id=2172044] [region_id=2172043]
[2021/05/15 19:43:26.245 +08:00] [INFO] [raft.rs:1064] [“became follower at term 31”] [term=31] [raft_id=2172044] [region_id=2172043]
[2021/05/15 19:43:26.251 +08:00] [INFO] [raft.rs:1503] [“[logterm: 30, index: 1483, vote: 0] cast vote for 2172045 [logterm: 30, index: 1483] at term 31”] [“msg type”=MsgRequestVote] [term=31] [msg_index=1483] [msg_term=30] [from=2172045] [vote=0] [log_index=1483] [log_term=30] [raft_id=2172044] [region_id=2172043]
[2021/05/15 19:43:26.392 +08:00] [INFO] [] [“ipv4:10.1.13.9:39844: Keepalive watchdog fired. Closing transport.”]
[2021/05/15 19:43:26.434 +08:00] [INFO] [service.rs:366] [“cdc receive closed”] [conn_id=ConnID(520)] [downstream=ipv4:10.1.13.9:39844]
[2021/05/15 19:43:26.440 +08:00] [WARN] [service.rs:390] [“cdc send failed”] [conn_id=ConnID(520)] [downstream=ipv4:10.1.13.9:39844] [error=RemoteStopped]
[2021/05/15 19:43:26.502 +08:00] [ERROR] [util.rs:416] [“request failed, retry”] [err_code=KV:PD:gRPC] [err=“Grpc(RpcFailure(RpcStatus { status: 4-DEADLINE_EXCEEDED, details: Some("Deadline Exceeded") }))”]
[2021/05/15 19:43:55.353 +08:00] [INFO] [lib.rs:89] [“Welcome to TiKV”]
[2021/05/15 19:43:55.353 +08:00] [INFO] [lib.rs:94] [“Release Version: 5.0.1”]
[2021/05/15 19:43:55.353 +08:00] [INFO] [lib.rs:94] [“Edition: Community”]
[2021/05/15 19:43:55.353 +08:00] [INFO] [lib.rs:94] [“Git Commit Hash: e26389a278116b2f61addfa9f15ca25ecf38bc80”]
[2021/05/15 19:43:55.353 +08:00] [INFO] [lib.rs:94] [“Git Commit Branch: heads/refs/tags/v5.0.1”]
[2021/05/15 19:43:55.353 +08:00] [INFO] [lib.rs:94] [“UTC Build Time: 2021-04-23 08:24:13”]
[2021/05/15 19:43:55.353 +08:00] [INFO] [lib.rs:94] [“Rust Version: rustc 1.51.0-nightly (bc39d4d9c 2021-01-15)”]
[2021/05/15 19:43:55.353 +08:00] [INFO] [lib.rs:94] [“Enable Features: jemalloc mem-profiling portable sse protobuf-codec test-engines-rocksdb cloud-aws cloud-gcp”]


若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

麻烦描述一下上下游关系,现在 TiKV 结合 TiCDC 导入到哪里 ?下游是什么? TiKV 重启有没有发生大量的oom 问题。

从mysql使用dm全量+增量导入到tidb,再通过ticdc把全量和增量数据,从tidb数据同步到kafka

TiKV 重启不一定是 TiCDC 同步引起的,现在 DM 完成全量同步了嘛 ?需要看一下 DM 的状态是 load/sync 哪个阶段。如果 load 阶段,建议调整一下降低并发,并且观察一下 TiKV 的负载情况

同样的数据导入过很多次了,都没出过问题,只是这种我把set GLOBAl tidb_enable_clustered_index = off,参数给关掉了

你的意思说把 cluster index 开启,就会出现导入 TiKV 重启的现象么 ?帮忙确认一下是 load 阶段 ?还是 sync 阶段?

是Load 阶段,我关掉这个参数正在重新拉取同样的数据,目前还未发现重启问题

感谢反馈,如果关闭参数之后可以导入成功的话,也麻烦再反馈一下

,已确认应该跟这个参数有关系

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。