pump突然几乎不写日志

kuweilong666 · 2022 年2 月 14 日 10:34

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：
【 TiDB 使用环境】
4.0

【概述】场景 + 问题概述
tidbbinlog
pump集群（3台机器）
drainer（5个进程，pump的其中一台）

【备份和数据迁移策略逻辑】

【背景】做过哪些操作

【现象】业务和数据库现象
突然不往下游同步数据，监控中drainer更新时间正常走，pump也正常，pump和drainer日志中没有报错

【问题】当前遇到的问题

【业务影响】
下游好几个库数据丢失

【TiDB 版本】

【附件】

相关日志、配置文件、Grafana 监控（https://metricstool.pingcap.com/)
TiUP Cluster Display 信息
TiUP CLuster Edit config 信息
TiDB-Overview 监控
对应模块的 Grafana 监控（如有 BR、TiDB-binlog、TiCDC 等）
对应模块日志（包含问题前后 1 小时日志）

若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出的打印结果，请务必全选并复制粘贴上传。

TiDBer_NtDbUm7p · 2022 年2 月 14 日 10:37

提供一下日志呗

kuweilong666 · 2022 年2 月 14 日 10:39

[2022/02/14 15:47:50.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431180970220584961]
[2022/02/14 15:48:00.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1152668,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:48:00.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431180973366312961]
[2022/02/14 15:48:10.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1153245,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:48:10.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431180975725608962]
[2022/02/14 15:48:20.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1153737,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:48:20.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431180978098012161]
[2022/02/14 15:48:30.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1154308,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:48:30.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431180981243740161]
[2022/02/14 15:48:40.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1154885,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:48:40.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431180983603036161]
[2022/02/14 15:48:50.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1155377,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:48:50.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431180985975439361]
[2022/02/14 15:49:00.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1155955,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:49:00.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431180989121167361]
[2022/02/14 15:49:10.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1156532,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:49:10.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431180991480463362]
[2022/02/14 15:49:20.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1157024,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:49:20.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431180993852866562]
[2022/02/14 15:49:30.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1157595,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:49:30.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431180996998594565]
[2022/02/14 15:49:40.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1158172,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:49:40.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431180999357890577]
[2022/02/14 15:49:50.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1158664,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:49:50.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181001730293761]
[2022/02/14 15:50:00.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1159235,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:50:00.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181004876021761]
[2022/02/14 15:50:10.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1159812,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:50:10.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181007235317761]
[2022/02/14 15:50:20.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1160304,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:50:20.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181009607720961]
[2022/02/14 15:50:30.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1160875,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:50:30.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181012753448961]
[2022/02/14 15:50:40.675 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1161452,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:50:40.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181015112744961]
[2022/02/14 15:50:40.676 +08:00] [INFO] [server.go:544] [“use gc ts to detect drainer checkpoint”] [“gc ts”=431021684340031488]
[2022/02/14 15:50:50.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1161944,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:50:50.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181017485148161]
[2022/02/14 15:51:00.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1162515,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:51:00.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181020630876161]
[2022/02/14 15:51:10.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1163092,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:51:10.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181022990172161]
[2022/02/14 15:51:20.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1163584,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:51:20.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181025349468161]
[2022/02/14 15:51:30.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1164155,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:51:30.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181028508303361]
[2022/02/14 15:51:40.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1164732,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:51:40.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181030867599361]
[2022/02/14 15:51:50.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1165224,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:51:50.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181033226895369]
[2022/02/14 15:52:00.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1165795,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:52:00.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181036385730561]
[2022/02/14 15:52:10.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1166372,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:52:10.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181038745026561]
[2022/02/14 15:52:20.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1166864,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:52:20.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181041104322561]
[2022/02/14 15:52:30.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1167435,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:52:30.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181044263157761]
[2022/02/14 15:52:40.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1168012,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:52:40.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181046622453761]
[2022/02/14 15:52:50.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1168504,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:52:50.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181048981749761]
[2022/02/14 15:53:00.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1169075,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:53:00.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181052140584961]
[2022/02/14 15:53:10.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1169652,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:53:10.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181054499880962]
[2022/02/14 15:53:20.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1170144,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:53:20.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181056859176965]
[2022/02/14 15:53:30.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1170715,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:53:30.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=5] [MaxCommitTS=431181060018012162]
[2022/02/14 15:53:40.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1171292,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:53:40.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=4] [MaxCommitTS=431181062377308167]
[2022/02/14 15:53:50.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1171784,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:53:50.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=3] [MaxCommitTS=431181064736604162]
[2022/02/14 15:54:00.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1172276,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:54:00.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=3] [MaxCommitTS=431181067095900165]
[2022/02/14 15:54:10.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1172847,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:54:10.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=2] [MaxCommitTS=431181070254735361]
[2022/02/14 15:54:20.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1173424,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:54:20.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=1] [MaxCommitTS=431181072614031361]
[2022/02/14 15:54:30.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1173916,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:54:30.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=0] [MaxCommitTS=431181074973327362]
[2022/02/14 15:54:40.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1174487,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:54:40.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=0] [MaxCommitTS=431181078132162567]
[2022/02/14 15:54:50.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1175064,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:54:50.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=0] [MaxCommitTS=431181080491458561]
[2022/02/14 15:55:00.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1175556,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:55:00.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=0] [MaxCommitTS=431181082863861761]
[2022/02/14 15:55:10.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1176127,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:55:10.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=0] [MaxCommitTS=431181086009589761]
[2022/02/14 15:55:20.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1176704,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:55:20.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=0] [MaxCommitTS=431181088368885761]
[2022/02/14 15:55:30.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1177196,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:55:30.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=0] [MaxCommitTS=431181090728181766]
[2022/02/14 15:55:40.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1177767,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:55:40.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=0] [MaxCommitTS=431181093887016966]
[2022/02/14 15:55:50.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1178344,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:55:50.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=0] [MaxCommitTS=431181096246312961]
[2022/02/14 15:56:00.674 +08:00] [INFO] [storage.go:384] [DBStats] [DBStats="{“WriteDelayCount”:0,“WriteDelayDuration”:0,“WritePaused”:false,“AliveSnapshots”:0,“AliveIterators”:0,“IOWrite”:1178836,“IORead”:4228388,“BlockCacheSize”:7093849,“OpenedTablesCount”:4,“LevelSizes”:[24379060,17113210],“LevelTablesCounts”:[3,1],“LevelRead”:[0,0],“LevelWrite”:[0,0],“LevelDurations”:[0,0]}"]
[2022/02/14 15:56:00.676 +08:00] [INFO] [server.go:563] [“server info tick”] [writeBinlogCount=0] [alivePullerCount=0] [MaxCommitTS=431181098605608961]

kuweilong666 · 2022 年2 月 14 日 10:42

kuweilong666 · 2022 年2 月 14 日 10:43

binlog增长缓慢

kuweilong666 · 2022 年2 月 15 日 02:24

tidb重启之后，pump正常写binlog了，怀疑是PD有问题

pepezzzz · 2022 年2 月 16 日 07:49

SQL>show config where type=‘tidb’ and name like ‘%ignore%’;
binlog.ignore-error是否为 true。
tidb 的日志内是否有 write binlog fail but error ignored 关键字，可能是消息过限导致的。

如果以上符合，参考 https://github.com/pingcap/tidb/blob/master/docs/tidb_http_api.md 的 “Resume the binlog writing when Pump is recovered.” 的一节描述。

kuweilong666 · 2022 年2 月 21 日 04:59

binlog.ignore-error= true，没有找到write binlog fail but error ignored

pepezzzz · 2022 年2 月 21 日 06:55

看一下 {clustername}-tidb → query detail → Panic And Critial Error 监控期间的曲线，是否如下图有 critical 的上升？

kuweilong666 · 2022 年2 月 21 日 10:34

该用什么方法处理让pump一直能写日志？

system · 2022 年10 月 31 日 19:06

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。