为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。
- 【TiDB 版本】:TiDB v2.1.15, DM v1.0.0-alpha-83-gf6f0566
- 【问题描述】:
这边观察到DM 里有个任务有点慢,基本和线上MySQL的数据 有1小时的延迟。
在dmctl.sh 中,使用query-status xx 观察到"syncerBinlog" 这里 数值变化较慢。
比如:
master:
"masterBinlog": "(mysql-bin.010537, 1047990566)",
syncer:
"syncerBinlog": "(mysql-bin|000003.010534, 524853955)"
线上除了该任务外,没有任务在重放binlog file : mysql-bin.010534。通过观察dm-worker.log日志[grep ‘, syncer-binlog=(mysql-bin|000003.010534,’ dm-worker.log]发现:
该任务tps 比较低,200 - 300 之间,此时syncer worker-count 30 ,batch 100
调整worker-count, tps 逐步降低1.3w → 440 ,此时 syncer worker-count 128, batch 100
# syncer worker-count 30 ,batch 100
2020/06/08 10:31:39.906 syncer.go:1828: [info] [syncer]total events = 397983, total tps = 301, recent tps = 25, master-binlog = (mysql-bin.010537, 872751874), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825635047, syncer-binlog=(mysql-bin|000003.010534, 419877638)(flushed (mysql-bin|000003.010534, 419520059))
2020/06/08 10:32:09.906 syncer.go:1828: [info] [syncer]total events = 398979, total tps = 295, recent tps = 33, master-binlog = (mysql-bin.010537, 873697784), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825635368, syncer-binlog=(mysql-bin|000003.010534, 427482530)(flushed (mysql-bin|000003.010534, 426964031))
2020/06/08 10:32:39.907 syncer.go:1828: [info] [syncer]total events = 400162, total tps = 289, recent tps = 39, master-binlog = (mysql-bin.010537, 875183855), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825636617, syncer-binlog=(mysql-bin|000003.010534, 436009174)(flushed (mysql-bin|000003.010534, 435527049))
2020/06/08 10:33:09.906 syncer.go:1828: [info] [syncer]total events = 401389, total tps = 284, recent tps = 40, master-binlog = (mysql-bin.010537, 876946547), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825637519, syncer-binlog=(mysql-bin|000003.010534, 445064503)(flushed (mysql-bin|000003.010534, 444818104))
2020/06/08 10:33:39.906 syncer.go:1828: [info] [syncer]total events = 402422, total tps = 279, recent tps = 34, master-binlog = (mysql-bin.010537, 884054032), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825640958, syncer-binlog=(mysql-bin|000003.010534, 452229409)(flushed (mysql-bin|000003.010534, 451903463))
2020/06/08 10:34:09.906 syncer.go:1828: [info] [syncer]total events = 404153, total tps = 274, recent tps = 57, master-binlog = (mysql-bin.010537, 885631492), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825641620, syncer-binlog=(mysql-bin|000003.010534, 464895200)(flushed (mysql-bin|000003.010534, 464843664))
2020/06/08 10:34:39.906 syncer.go:1828: [info] [syncer]total events = 406123, total tps = 270, recent tps = 65, master-binlog = (mysql-bin.010537, 887040699), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825642057, syncer-binlog=(mysql-bin|000003.010534, 479118218)(flushed (mysql-bin|000003.010534, 478893215))
2020/06/08 10:35:09.906 syncer.go:1828: [info] [syncer]total events = 407748, total tps = 266, recent tps = 54, master-binlog = (mysql-bin.010537, 908582176), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825642837, syncer-binlog=(mysql-bin|000003.010534, 491445926)(flushed (mysql-bin|000003.010534, 491163532))
2020/06/08 10:35:39.906 syncer.go:1828: [info] [syncer]total events = 409636, total tps = 262, recent tps = 62, master-binlog = (mysql-bin.010537, 909834511), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825643179, syncer-binlog=(mysql-bin|000003.010534, 500983977)(flushed (mysql-bin|000003.010534, 500798150))
# syncer worker-count 128 ,batch 100
2020/06/08 10:37:18.891 syncer.go:1828: [info] [syncer]total events = 412580, total tps = 13752, recent tps = 98, master-binlog = (mysql-bin.010537, 913791726), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825644383, syncer-binlog=(mysql-bin|000003.010534, 515439631)(flushed (mysql-bin|000003.010534, 515422674))
2020/06/08 10:37:48.891 syncer.go:1828: [info] [syncer]total events = 414175, total tps = 6902, recent tps = 53, master-binlog = (mysql-bin.010537, 914844280), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825644753, syncer-binlog=(mysql-bin|000003.010534, 521594432)(flushed (mysql-bin|000003.010534, 521512913))
2020/06/08 10:38:18.891 syncer.go:1828: [info] [syncer]total events = 416826, total tps = 4631, recent tps = 88, master-binlog = (mysql-bin.010537, 915954918), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825645068, syncer-binlog=(mysql-bin|000003.010534, 532305300)(flushed (mysql-bin|000003.010534, 532027852))
2020/06/08 10:38:48.891 syncer.go:1828: [info] [syncer]total events = 419212, total tps = 3493, recent tps = 79, master-binlog = (mysql-bin.010537, 916806985), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825645293, syncer-binlog=(mysql-bin|000003.010534, 542167332)(flushed (mysql-bin|000003.010534, 542054104))
2020/06/08 10:39:18.891 syncer.go:1828: [info] [syncer]total events = 421677, total tps = 2811, recent tps = 82, master-binlog = (mysql-bin.010537, 917918096), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825645699, syncer-binlog=(mysql-bin|000003.010534, 552412361)(flushed (mysql-bin|000003.010534, 552394580))
2020/06/08 10:39:48.891 syncer.go:1828: [info] [syncer]total events = 424314, total tps = 2357, recent tps = 87, master-binlog = (mysql-bin.010537, 919466235), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825646118, syncer-binlog=(mysql-bin|000003.010534, 563120650)(flushed (mysql-bin|000003.010534, 562648885))
2020/06/08 10:40:18.891 syncer.go:1828: [info] [syncer]total events = 426941, total tps = 2033, recent tps = 87, master-binlog = (mysql-bin.010537, 927025729), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825646642, syncer-binlog=(mysql-bin|000003.010534, 574482594)(flushed (mysql-bin|000003.010534, 574440358))
2020/06/08 10:40:48.891 syncer.go:1828: [info] [syncer]total events = 429358, total tps = 1788, recent tps = 80, master-binlog = (mysql-bin.010537, 929787247), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825647699, syncer-binlog=(mysql-bin|000003.010534, 584562877)(flushed (mysql-bin|000003.010534, 584550291))
2020/06/08 10:41:18.891 syncer.go:1828: [info] [syncer]total events = 431824, total tps = 1599, recent tps = 82, master-binlog = (mysql-bin.010537, 930758106), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825648030, syncer-binlog=(mysql-bin|000003.010534, 594972512)(flushed (mysql-bin|000003.010534, 594504145))
2020/06/08 10:41:48.891 syncer.go:1828: [info] [syncer]total events = 434580, total tps = 1448, recent tps = 91, master-binlog = (mysql-bin.010537, 932315926), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825648417, syncer-binlog=(mysql-bin|000003.010534, 605905786)(flushed (mysql-bin|000003.010534, 605718289))
2020/06/08 10:42:18.891 syncer.go:1828: [info] [syncer]total events = 436537, total tps = 1322, recent tps = 65, master-binlog = (mysql-bin.010537, 937658288), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825650284, syncer-binlog=(mysql-bin|000003.010534, 640862138)(flushed (mysql-bin|000003.010534, 639364184))
2020/06/08 10:42:48.891 syncer.go:1828: [info] [syncer]total events = 439254, total tps = 1220, recent tps = 90, master-binlog = (mysql-bin.010537, 938994274), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825650689, syncer-binlog=(mysql-bin|000003.010534, 690250476)(flushed (mysql-bin|000003.010534, 689306959))
2020/06/08 10:43:18.891 syncer.go:1828: [info] [syncer]total events = 441787, total tps = 1132, recent tps = 84, master-binlog = (mysql-bin.010537, 940077518), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825651046, syncer-binlog=(mysql-bin|000003.010534, 708967088)(flushed (mysql-bin|000003.010534, 708050374))
2020/06/08 10:43:48.891 syncer.go:1828: [info] [syncer]total events = 444575, total tps = 1058, recent tps = 92, master-binlog = (mysql-bin.010537, 941902636), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825651568, syncer-binlog=(mysql-bin|000003.010534, 729349656)(flushed (mysql-bin|000003.010534, 729042615))
2020/06/08 10:44:18.891 syncer.go:1828: [info] [syncer]total events = 446609, total tps = 992, recent tps = 67, master-binlog = (mysql-bin.010537, 943044155), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825651957, syncer-binlog=(mysql-bin|000003.010534, 747521668)(flushed (mysql-bin|000003.010534, 747395114))
2020/06/08 10:44:48.891 syncer.go:1828: [info] [syncer]total events = 448482, total tps = 934, recent tps = 62, master-binlog = (mysql-bin.010537, 944241512), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825652335, syncer-binlog=(mysql-bin|000003.010534, 762839812)(flushed (mysql-bin|000003.010534, 762638802))
2020/06/08 10:45:18.891 syncer.go:1828: [info] [syncer]total events = 450752, total tps = 883, recent tps = 75, master-binlog = (mysql-bin.010537, 945400977), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825652751, syncer-binlog=(mysql-bin|000003.010534, 778993464)(flushed (mysql-bin|000003.010534, 778772941))
2020/06/08 10:45:48.891 syncer.go:1828: [info] [syncer]total events = 454540, total tps = 841, recent tps = 126, master-binlog = (mysql-bin.010537, 946777135), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825653118, syncer-binlog=(mysql-bin|000003.010534, 806677642)(flushed (mysql-bin|000003.010534, 806218686))
2020/06/08 10:46:18.891 syncer.go:1828: [info] [syncer]total events = 458307, total tps = 804, recent tps = 125, master-binlog = (mysql-bin.010537, 947860968), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825653473, syncer-binlog=(mysql-bin|000003.010534, 832003335)(flushed (mysql-bin|000003.010534, 831927720))
2020/06/08 10:46:48.891 syncer.go:1828: [info] [syncer]total events = 460881, total tps = 768, recent tps = 85, master-binlog = (mysql-bin.010537, 948975976), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825653906, syncer-binlog=(mysql-bin|000003.010534, 850881863)(flushed (mysql-bin|000003.010534, 850486131))
2020/06/08 10:47:18.892 syncer.go:1828: [info] [syncer]total events = 463562, total tps = 735, recent tps = 89, master-binlog = (mysql-bin.010537, 950369267), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825654272, syncer-binlog=(mysql-bin|000003.010534, 870426024)(flushed (mysql-bin|000003.010534, 870307616))
2020/06/08 10:47:48.891 syncer.go:1828: [info] [syncer]total events = 465376, total tps = 705, recent tps = 60, master-binlog = (mysql-bin.010537, 951269610), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825654573, syncer-binlog=(mysql-bin|000003.010534, 884472695)(flushed (mysql-bin|000003.010534, 884421439))
2020/06/08 10:48:18.891 syncer.go:1828: [info] [syncer]total events = 466841, total tps = 676, recent tps = 48, master-binlog = (mysql-bin.010537, 952301231), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825654926, syncer-binlog=(mysql-bin|000003.010534, 896480193)(flushed (mysql-bin|000003.010534, 896401493))
2020/06/08 10:48:48.891 syncer.go:1828: [info] [syncer]total events = 468318, total tps = 650, recent tps = 49, master-binlog = (mysql-bin.010537, 953513783), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825655343, syncer-binlog=(mysql-bin|000003.010534, 907965769)(flushed (mysql-bin|000003.010534, 907736841))
2020/06/08 10:49:18.891 syncer.go:1828: [info] [syncer]total events = 469654, total tps = 626, recent tps = 44, master-binlog = (mysql-bin.010537, 954970094), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825655774, syncer-binlog=(mysql-bin|000003.010534, 917693855)(flushed (mysql-bin|000003.010534, 917638814))
2020/06/08 10:49:48.891 syncer.go:1828: [info] [syncer]total events = 471203, total tps = 604, recent tps = 51, master-binlog = (mysql-bin.010537, 956019232), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825656181, syncer-binlog=(mysql-bin|000003.010534, 930126843)(flushed (mysql-bin|000003.010534, 929474870))
2020/06/08 10:50:18.891 syncer.go:1828: [info] [syncer]total events = 472583, total tps = 583, recent tps = 46, master-binlog = (mysql-bin.010537, 962586204), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825656804, syncer-binlog=(mysql-bin|000003.010534, 942803458)(flushed (mysql-bin|000003.010534, 942641754))
2020/06/08 10:50:48.891 syncer.go:1828: [info] [syncer]total events = 474432, total tps = 564, recent tps = 61, master-binlog = (mysql-bin.010537, 963317064), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825657071, syncer-binlog=(mysql-bin|000003.010534, 956967695)(flushed (mysql-bin|000003.010534, 956936442))
2020/06/08 10:51:18.891 syncer.go:1828: [info] [syncer]total events = 476154, total tps = 547, recent tps = 57, master-binlog = (mysql-bin.010537, 966310160), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825658209, syncer-binlog=(mysql-bin|000003.010534, 969266921)(flushed (mysql-bin|000003.010534, 969188437))
2020/06/08 10:51:48.891 syncer.go:1828: [info] [syncer]total events = 477842, total tps = 530, recent tps = 56, master-binlog = (mysql-bin.010537, 967373710), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825658530, syncer-binlog=(mysql-bin|000003.010534, 981617207)(flushed (mysql-bin|000003.010534, 981602811))
2020/06/08 10:52:18.891 syncer.go:1828: [info] [syncer]total events = 478806, total tps = 514, recent tps = 32, master-binlog = (mysql-bin.010537, 968230294), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825658880, syncer-binlog=(mysql-bin|000003.010534, 987965895)(flushed (mysql-bin|000003.010534, 987914250))
2020/06/08 10:52:48.891 syncer.go:1828: [info] [syncer]total events = 480171, total tps = 500, recent tps = 45, master-binlog = (mysql-bin.010537, 969728217), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825659375, syncer-binlog=(mysql-bin|000003.010534, 997104675)(flushed (mysql-bin|000003.010534, 996797402))
2020/06/08 10:53:18.891 syncer.go:1828: [info] [syncer]total events = 481466, total tps = 486, recent tps = 43, master-binlog = (mysql-bin.010537, 971588546), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825659814, syncer-binlog=(mysql-bin|000003.010534, 1005857359)(flushed (mysql-bin|000003.010534, 1005692157))
2020/06/08 10:53:48.891 syncer.go:1828: [info] [syncer]total events = 482958, total tps = 473, recent tps = 49, master-binlog = (mysql-bin.010537, 972509015), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825660177, syncer-binlog=(mysql-bin|000003.010534, 1016489254)(flushed (mysql-bin|000003.010534, 1016371801))
2020/06/08 10:54:18.891 syncer.go:1828: [info] [syncer]total events = 485168, total tps = 462, recent tps = 73, master-binlog = (mysql-bin.010537, 974251196), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825660700, syncer-binlog=(mysql-bin|000003.010534, 1031802261)(flushed (mysql-bin|000003.010534, 1031619190))
2020/06/08 10:54:48.891 syncer.go:1828: [info] [syncer]total events = 487362, total tps = 451, recent tps = 73, master-binlog = (mysql-bin.010537, 975602577), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825661130, syncer-binlog=(mysql-bin|000003.010534, 1046729977)(flushed (mysql-bin|000003.010534, 1046635462))
2020/06/08 10:55:18.891 syncer.go:1828: [info] [syncer]total events = 489699, total tps = 441, recent tps = 77, master-binlog = (mysql-bin.010537, 977280445), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825661630, syncer-binlog=(mysql-bin|000003.010534, 1063245806)(flushed (mysql-bin|000003.010534, 1063146869))
```