tidb集群有一个节点故障，数据同步变慢

搬砖er · 2024 年12 月 2 日 04:38

【 TiDB 使用环境】生产环境
【 TiDB 版本】4.0.2
【复现路径】
1.线上TiDB集群20+节点，其中一个节点故障，手工删除节点，节点已经是offlien状态。此时集群工作正常，该集群称为A。
2.线上部署了两套tidb集群，使用pump/drainer方式同步。此时另外一套集群B同步数据到集群A报错，时延非常长。
【遇到的问题：问题现象及影响】
两套tidb集团AB同步数据，方向是B->A。
drainer通过A集群的tidb写入数据，tidb有大量慢查：
Time: 2024-12-02T12:28:26.53089392+08:00
Txn_start_ts: 454325348827661289
User: drainer@10.1.0.230
Conn_ID: 95
Query_time: 40.437295963
Parse_time: 0.000007968
Compile_time: 0.000018548
Prewrite_time: 40.437156516 Wait_prewrite_binlog_time: 0.000000225 Commit_backoff_time: 40.01 Backoff_types: [regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMissregionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss regionMiss] Write_keys: 16 Write_size: 1073 Prewrite_region: 94
Is_internal: false
Digest: 9505cacb7c710ed17125fcc6cb3669e8ddca6c8cd8af6a31f6b3cd64604c3098
Num_cop_tasks: 0
Prepared: false
Plan_from_cache: false
Has_more_results: false
Succ: false
Prev_stmt: REPLACE INTO d_msgcenter.t_relat_op_gz(addtime,app,msgid,status,storage_time,tag,tuid,uid) VALUES(1733106661,‘kgpure’,7269176203168906330,1,1733109424416,‘mchat:520021309’,2039700267,520021309)
COMMIT;
【资源配置】
【附件：截图/日志/监控】