DM 做binlog重放时,通过worker-count 控制同步速度,不符合预期

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。

  • 【TiDB 版本】:TiDB v2.1.15, DM v1.0.0-alpha-83-gf6f0566
  • 【问题描述】:

这边观察到DM 里有个任务有点慢,基本和线上MySQL的数据 有1小时的延迟。

在dmctl.sh 中,使用query-status xx 观察到"syncerBinlog" 这里 数值变化较慢。

比如:

master:
"masterBinlog": "(mysql-bin.010537, 1047990566)",

syncer:
"syncerBinlog": "(mysql-bin|000003.010534, 524853955)"

线上除了该任务外,没有任务在重放binlog file : mysql-bin.010534。通过观察dm-worker.log日志[grep ‘, syncer-binlog=(mysql-bin|000003.010534,’ dm-worker.log]发现:

该任务tps 比较低,200 - 300 之间,此时syncer worker-count 30 ,batch 100

调整worker-count, tps 逐步降低1.3w → 440 ,此时 syncer worker-count 128, batch 100

# syncer worker-count 30 ,batch 100

2020/06/08 10:31:39.906 syncer.go:1828: [info] [syncer]total events = 397983, total tps = 301, recent tps = 25, master-binlog = (mysql-bin.010537, 872751874), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825635047, syncer-binlog=(mysql-bin|000003.010534, 419877638)(flushed (mysql-bin|000003.010534, 419520059))
2020/06/08 10:32:09.906 syncer.go:1828: [info] [syncer]total events = 398979, total tps = 295, recent tps = 33, master-binlog = (mysql-bin.010537, 873697784), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825635368, syncer-binlog=(mysql-bin|000003.010534, 427482530)(flushed (mysql-bin|000003.010534, 426964031))
2020/06/08 10:32:39.907 syncer.go:1828: [info] [syncer]total events = 400162, total tps = 289, recent tps = 39, master-binlog = (mysql-bin.010537, 875183855), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825636617, syncer-binlog=(mysql-bin|000003.010534, 436009174)(flushed (mysql-bin|000003.010534, 435527049))
2020/06/08 10:33:09.906 syncer.go:1828: [info] [syncer]total events = 401389, total tps = 284, recent tps = 40, master-binlog = (mysql-bin.010537, 876946547), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825637519, syncer-binlog=(mysql-bin|000003.010534, 445064503)(flushed (mysql-bin|000003.010534, 444818104))
2020/06/08 10:33:39.906 syncer.go:1828: [info] [syncer]total events = 402422, total tps = 279, recent tps = 34, master-binlog = (mysql-bin.010537, 884054032), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825640958, syncer-binlog=(mysql-bin|000003.010534, 452229409)(flushed (mysql-bin|000003.010534, 451903463))
2020/06/08 10:34:09.906 syncer.go:1828: [info] [syncer]total events = 404153, total tps = 274, recent tps = 57, master-binlog = (mysql-bin.010537, 885631492), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825641620, syncer-binlog=(mysql-bin|000003.010534, 464895200)(flushed (mysql-bin|000003.010534, 464843664))
2020/06/08 10:34:39.906 syncer.go:1828: [info] [syncer]total events = 406123, total tps = 270, recent tps = 65, master-binlog = (mysql-bin.010537, 887040699), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825642057, syncer-binlog=(mysql-bin|000003.010534, 479118218)(flushed (mysql-bin|000003.010534, 478893215))
2020/06/08 10:35:09.906 syncer.go:1828: [info] [syncer]total events = 407748, total tps = 266, recent tps = 54, master-binlog = (mysql-bin.010537, 908582176), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825642837, syncer-binlog=(mysql-bin|000003.010534, 491445926)(flushed (mysql-bin|000003.010534, 491163532))
2020/06/08 10:35:39.906 syncer.go:1828: [info] [syncer]total events = 409636, total tps = 262, recent tps = 62, master-binlog = (mysql-bin.010537, 909834511), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825643179, syncer-binlog=(mysql-bin|000003.010534, 500983977)(flushed (mysql-bin|000003.010534, 500798150))


# syncer worker-count 128 ,batch 100
2020/06/08 10:37:18.891 syncer.go:1828: [info] [syncer]total events = 412580, total tps = 13752, recent tps = 98, master-binlog = (mysql-bin.010537, 913791726), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825644383, syncer-binlog=(mysql-bin|000003.010534, 515439631)(flushed (mysql-bin|000003.010534, 515422674))
2020/06/08 10:37:48.891 syncer.go:1828: [info] [syncer]total events = 414175, total tps = 6902, recent tps = 53, master-binlog = (mysql-bin.010537, 914844280), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825644753, syncer-binlog=(mysql-bin|000003.010534, 521594432)(flushed (mysql-bin|000003.010534, 521512913))
2020/06/08 10:38:18.891 syncer.go:1828: [info] [syncer]total events = 416826, total tps = 4631, recent tps = 88, master-binlog = (mysql-bin.010537, 915954918), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825645068, syncer-binlog=(mysql-bin|000003.010534, 532305300)(flushed (mysql-bin|000003.010534, 532027852))
2020/06/08 10:38:48.891 syncer.go:1828: [info] [syncer]total events = 419212, total tps = 3493, recent tps = 79, master-binlog = (mysql-bin.010537, 916806985), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825645293, syncer-binlog=(mysql-bin|000003.010534, 542167332)(flushed (mysql-bin|000003.010534, 542054104))
2020/06/08 10:39:18.891 syncer.go:1828: [info] [syncer]total events = 421677, total tps = 2811, recent tps = 82, master-binlog = (mysql-bin.010537, 917918096), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825645699, syncer-binlog=(mysql-bin|000003.010534, 552412361)(flushed (mysql-bin|000003.010534, 552394580))
2020/06/08 10:39:48.891 syncer.go:1828: [info] [syncer]total events = 424314, total tps = 2357, recent tps = 87, master-binlog = (mysql-bin.010537, 919466235), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825646118, syncer-binlog=(mysql-bin|000003.010534, 563120650)(flushed (mysql-bin|000003.010534, 562648885))
2020/06/08 10:40:18.891 syncer.go:1828: [info] [syncer]total events = 426941, total tps = 2033, recent tps = 87, master-binlog = (mysql-bin.010537, 927025729), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825646642, syncer-binlog=(mysql-bin|000003.010534, 574482594)(flushed (mysql-bin|000003.010534, 574440358))
2020/06/08 10:40:48.891 syncer.go:1828: [info] [syncer]total events = 429358, total tps = 1788, recent tps = 80, master-binlog = (mysql-bin.010537, 929787247), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825647699, syncer-binlog=(mysql-bin|000003.010534, 584562877)(flushed (mysql-bin|000003.010534, 584550291))
2020/06/08 10:41:18.891 syncer.go:1828: [info] [syncer]total events = 431824, total tps = 1599, recent tps = 82, master-binlog = (mysql-bin.010537, 930758106), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825648030, syncer-binlog=(mysql-bin|000003.010534, 594972512)(flushed (mysql-bin|000003.010534, 594504145))
2020/06/08 10:41:48.891 syncer.go:1828: [info] [syncer]total events = 434580, total tps = 1448, recent tps = 91, master-binlog = (mysql-bin.010537, 932315926), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825648417, syncer-binlog=(mysql-bin|000003.010534, 605905786)(flushed (mysql-bin|000003.010534, 605718289))
2020/06/08 10:42:18.891 syncer.go:1828: [info] [syncer]total events = 436537, total tps = 1322, recent tps = 65, master-binlog = (mysql-bin.010537, 937658288), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825650284, syncer-binlog=(mysql-bin|000003.010534, 640862138)(flushed (mysql-bin|000003.010534, 639364184))
2020/06/08 10:42:48.891 syncer.go:1828: [info] [syncer]total events = 439254, total tps = 1220, recent tps = 90, master-binlog = (mysql-bin.010537, 938994274), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825650689, syncer-binlog=(mysql-bin|000003.010534, 690250476)(flushed (mysql-bin|000003.010534, 689306959))
2020/06/08 10:43:18.891 syncer.go:1828: [info] [syncer]total events = 441787, total tps = 1132, recent tps = 84, master-binlog = (mysql-bin.010537, 940077518), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825651046, syncer-binlog=(mysql-bin|000003.010534, 708967088)(flushed (mysql-bin|000003.010534, 708050374))
2020/06/08 10:43:48.891 syncer.go:1828: [info] [syncer]total events = 444575, total tps = 1058, recent tps = 92, master-binlog = (mysql-bin.010537, 941902636), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825651568, syncer-binlog=(mysql-bin|000003.010534, 729349656)(flushed (mysql-bin|000003.010534, 729042615))
2020/06/08 10:44:18.891 syncer.go:1828: [info] [syncer]total events = 446609, total tps = 992, recent tps = 67, master-binlog = (mysql-bin.010537, 943044155), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825651957, syncer-binlog=(mysql-bin|000003.010534, 747521668)(flushed (mysql-bin|000003.010534, 747395114))
2020/06/08 10:44:48.891 syncer.go:1828: [info] [syncer]total events = 448482, total tps = 934, recent tps = 62, master-binlog = (mysql-bin.010537, 944241512), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825652335, syncer-binlog=(mysql-bin|000003.010534, 762839812)(flushed (mysql-bin|000003.010534, 762638802))
2020/06/08 10:45:18.891 syncer.go:1828: [info] [syncer]total events = 450752, total tps = 883, recent tps = 75, master-binlog = (mysql-bin.010537, 945400977), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825652751, syncer-binlog=(mysql-bin|000003.010534, 778993464)(flushed (mysql-bin|000003.010534, 778772941))
2020/06/08 10:45:48.891 syncer.go:1828: [info] [syncer]total events = 454540, total tps = 841, recent tps = 126, master-binlog = (mysql-bin.010537, 946777135), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825653118, syncer-binlog=(mysql-bin|000003.010534, 806677642)(flushed (mysql-bin|000003.010534, 806218686))
2020/06/08 10:46:18.891 syncer.go:1828: [info] [syncer]total events = 458307, total tps = 804, recent tps = 125, master-binlog = (mysql-bin.010537, 947860968), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825653473, syncer-binlog=(mysql-bin|000003.010534, 832003335)(flushed (mysql-bin|000003.010534, 831927720))
2020/06/08 10:46:48.891 syncer.go:1828: [info] [syncer]total events = 460881, total tps = 768, recent tps = 85, master-binlog = (mysql-bin.010537, 948975976), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825653906, syncer-binlog=(mysql-bin|000003.010534, 850881863)(flushed (mysql-bin|000003.010534, 850486131))
2020/06/08 10:47:18.892 syncer.go:1828: [info] [syncer]total events = 463562, total tps = 735, recent tps = 89, master-binlog = (mysql-bin.010537, 950369267), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825654272, syncer-binlog=(mysql-bin|000003.010534, 870426024)(flushed (mysql-bin|000003.010534, 870307616))
2020/06/08 10:47:48.891 syncer.go:1828: [info] [syncer]total events = 465376, total tps = 705, recent tps = 60, master-binlog = (mysql-bin.010537, 951269610), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825654573, syncer-binlog=(mysql-bin|000003.010534, 884472695)(flushed (mysql-bin|000003.010534, 884421439))
2020/06/08 10:48:18.891 syncer.go:1828: [info] [syncer]total events = 466841, total tps = 676, recent tps = 48, master-binlog = (mysql-bin.010537, 952301231), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825654926, syncer-binlog=(mysql-bin|000003.010534, 896480193)(flushed (mysql-bin|000003.010534, 896401493))
2020/06/08 10:48:48.891 syncer.go:1828: [info] [syncer]total events = 468318, total tps = 650, recent tps = 49, master-binlog = (mysql-bin.010537, 953513783), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825655343, syncer-binlog=(mysql-bin|000003.010534, 907965769)(flushed (mysql-bin|000003.010534, 907736841))
2020/06/08 10:49:18.891 syncer.go:1828: [info] [syncer]total events = 469654, total tps = 626, recent tps = 44, master-binlog = (mysql-bin.010537, 954970094), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825655774, syncer-binlog=(mysql-bin|000003.010534, 917693855)(flushed (mysql-bin|000003.010534, 917638814))
2020/06/08 10:49:48.891 syncer.go:1828: [info] [syncer]total events = 471203, total tps = 604, recent tps = 51, master-binlog = (mysql-bin.010537, 956019232), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825656181, syncer-binlog=(mysql-bin|000003.010534, 930126843)(flushed (mysql-bin|000003.010534, 929474870))
2020/06/08 10:50:18.891 syncer.go:1828: [info] [syncer]total events = 472583, total tps = 583, recent tps = 46, master-binlog = (mysql-bin.010537, 962586204), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825656804, syncer-binlog=(mysql-bin|000003.010534, 942803458)(flushed (mysql-bin|000003.010534, 942641754))
2020/06/08 10:50:48.891 syncer.go:1828: [info] [syncer]total events = 474432, total tps = 564, recent tps = 61, master-binlog = (mysql-bin.010537, 963317064), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825657071, syncer-binlog=(mysql-bin|000003.010534, 956967695)(flushed (mysql-bin|000003.010534, 956936442))
2020/06/08 10:51:18.891 syncer.go:1828: [info] [syncer]total events = 476154, total tps = 547, recent tps = 57, master-binlog = (mysql-bin.010537, 966310160), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825658209, syncer-binlog=(mysql-bin|000003.010534, 969266921)(flushed (mysql-bin|000003.010534, 969188437))
2020/06/08 10:51:48.891 syncer.go:1828: [info] [syncer]total events = 477842, total tps = 530, recent tps = 56, master-binlog = (mysql-bin.010537, 967373710), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825658530, syncer-binlog=(mysql-bin|000003.010534, 981617207)(flushed (mysql-bin|000003.010534, 981602811))
2020/06/08 10:52:18.891 syncer.go:1828: [info] [syncer]total events = 478806, total tps = 514, recent tps = 32, master-binlog = (mysql-bin.010537, 968230294), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825658880, syncer-binlog=(mysql-bin|000003.010534, 987965895)(flushed (mysql-bin|000003.010534, 987914250))
2020/06/08 10:52:48.891 syncer.go:1828: [info] [syncer]total events = 480171, total tps = 500, recent tps = 45, master-binlog = (mysql-bin.010537, 969728217), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825659375, syncer-binlog=(mysql-bin|000003.010534, 997104675)(flushed (mysql-bin|000003.010534, 996797402))
2020/06/08 10:53:18.891 syncer.go:1828: [info] [syncer]total events = 481466, total tps = 486, recent tps = 43, master-binlog = (mysql-bin.010537, 971588546), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825659814, syncer-binlog=(mysql-bin|000003.010534, 1005857359)(flushed (mysql-bin|000003.010534, 1005692157))
2020/06/08 10:53:48.891 syncer.go:1828: [info] [syncer]total events = 482958, total tps = 473, recent tps = 49, master-binlog = (mysql-bin.010537, 972509015), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825660177, syncer-binlog=(mysql-bin|000003.010534, 1016489254)(flushed (mysql-bin|000003.010534, 1016371801))
2020/06/08 10:54:18.891 syncer.go:1828: [info] [syncer]total events = 485168, total tps = 462, recent tps = 73, master-binlog = (mysql-bin.010537, 974251196), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825660700, syncer-binlog=(mysql-bin|000003.010534, 1031802261)(flushed (mysql-bin|000003.010534, 1031619190))
2020/06/08 10:54:48.891 syncer.go:1828: [info] [syncer]total events = 487362, total tps = 451, recent tps = 73, master-binlog = (mysql-bin.010537, 975602577), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825661130, syncer-binlog=(mysql-bin|000003.010534, 1046729977)(flushed (mysql-bin|000003.010534, 1046635462))
2020/06/08 10:55:18.891 syncer.go:1828: [info] [syncer]total events = 489699, total tps = 441, recent tps = 77, master-binlog = (mysql-bin.010537, 977280445), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825661630, syncer-binlog=(mysql-bin|000003.010534, 1063245806)(flushed (mysql-bin|000003.010534, 1063146869))

```

整过同步过程,top 发现 CPU 使用一直不高,低于 5%。

查看了该任务的所有表结构 ,都有 主键。

  1. 查看的是syncer配置的服务器吗?
  2. 在下游 tidb 执行 show processlist ,看下同时有几个进程在同步。
  3. 上传配置,文件调整的哪里
  1. 是的

刚上来,用show processlist, 发现 一共有397个DM syncer 连接

397 rows in set (0.00 sec)

我把该任务stop 之后 ,再show 一下

335 rows in set (0.00 sec)

重新start 该任务之后,show 结果

464 rows in set (0.01 sec)

感觉 找到问题了,因为,DM 是单独指向了一台独立的tidb server节点,无其他服务在用。

需要stop 在start 任务,才可以使用worker-count这个参数。

但是,速度 还是有点慢。

目前 只有该任务 在重放 mysql-bin.010537 ,TPS 还是比较低

2020/06/08 11:30:54.198 syncer.go:1828: [info] [syncer]total events = 15056, total tps = 167, recent tps = 151, master-binlog = (mysql-bin.010538, 48078716), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825697250, syncer-binlog=(mysql-bin|000003.010537, 79817217)(flushed (mysql-bin|000003.010537, 79072728))
2020/06/08 11:31:24.198 syncer.go:1828: [info] [syncer]total events = 19856, total tps = 165, recent tps = 160, master-binlog = (mysql-bin.010538, 49059423), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825697592, syncer-binlog=(mysql-bin|000003.010537, 99742055)(flushed (mysql-bin|000003.010537, 99699504))
2020/06/08 11:31:54.198 syncer.go:1828: [info] [syncer]total events = 23723, total tps = 158, recent tps = 128, master-binlog = (mysql-bin.010538, 268226946), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825698320, syncer-binlog=(mysql-bin|000003.010537, 114256013)(flushed (mysql-bin|000003.010537, 114174154))
2020/06/08 11:32:24.198 syncer.go:1828: [info] [syncer]total events = 27704, total tps = 153, recent tps = 132, master-binlog = (mysql-bin.010538, 297365913), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825698751, syncer-binlog=(mysql-bin|000003.010537, 129983714)(flushed (mysql-bin|000003.010537, 129740340))
2020/06/08 11:32:54.221 syncer.go:1828: [info] [syncer]total events = 30862, total tps = 146, recent tps = 105, master-binlog = (mysql-bin.010538, 302849152), master-binlog-gtid=a06ce942-d0ec-11e7-8959-00163e0c3232:1-1825699377, syncer-binlog=(mysql-bin|000003.010537, 144456661)(flushed (mysql-bin|000003.010537, 144305559))

3 整体的配置:

name: "ment"
# 全量+增量 (all) 同步模式。
task-mode: "incremental"

ignore-checking-items: ["dump_privilege"]
# 下游 TiDB 配置信息。
target-database:
  host: "xx.net"
  port: 4000
  user: "xx"
  password: "xx"

# 当前数据同步任务需要的全部上游 MySQL 实例配置。
mysql-instances:
-
  # 上游实例或者复制组 ID,参考 `inventory.ini` 的 `source_id` 或者 `dm-master.toml` 的 `source-id 配置`。
  source-id: "mysql-replica-01"
  meta:
    binlog-name: mysql-bin.006726
    binlog-pos: 300508273

  # 需要同步的库名或表名的黑白名单的配置项名称,用于引用全局的黑白名单配置,全局配置见下面的 `black-white-list` 的配置。
  black-white-list: "db-rule"
  filter-rules: ["filter-partition-rule"]
  syncer-config-name: "global"

# 黑白名单全局配置,各实例通过配置项名引用。
black-white-list:
  db-rule:
    do-dbs: ["ment"]

filters:
  filter-partition-rule:
    schema-pattern: "*"
    sql-pattern: []
    action: Ignore

syncers:
  global:
    worker-count: 128
    batch: 100

syncer的配置,主要改的这里:

syncers:
  global:
    worker-count: 128
    batch: 100

改完后,用start-task or update-task 的时候,dm-worker.log 里 会打印出该任务信息。检查过,worker-count的确是128

  1. show processlist 里你可以看到有很多进程同步,都是运行状态还是有很多事 sleep 之类的状态?
  2. 麻烦上传 over-view 和 tidb 监控,以及 DM 监控,多谢。
  1. 有很多sleep,要占90%

好的,麻烦反馈下监控信息吧,多谢。