drainer扩容后不往kafka写消息

qufudcj · 2023 年4 月 6 日 09:17

【 TiDB 使用环境】测试
【 TiDB 版本】v4.0.4
【复现路径】做过哪些操作出现的问题
在一个drainer内存不足的情况下，新扩容了一个drainer
【遇到的问题：问题现象及影响】
两个drainer目前启动正常，但是并不往kafka里面写消息，kafka的队列是存在的，并且其他队列收发消息正常，也并没有发现kafka报错消息过大
【资源配置】
【附件：截图/日志/监控】
新drainer启动日志

[2023/04/06 16:53:13.774 +08:00] [INFO] [version.go:50] ["Welcome to Drainer"] ["Release Version"=v4.0.3-dirty] ["Git Commit Hash"=fb0cc0d5290202bb6d1b6f189549f629cfd810ef] ["Build TS"="2021-10-21 02:40:08"] ["Go Version"=go1.15.6] ["Go OS/Arch"=linux/amd64]
[2023/04/06 16:53:13.774 +08:00] [INFO] [main.go:46] ["start drainer..."] [config="{\"log-level\":\"info\",\"node-id\":\"172.16.12.169:8249\",\"addr\":\"http://172.16.12.169:8249\",\"advertise-addr\":\"http://172.16.12.169:8249\",\"data-dir\":\"/alidata/tidb/deploy/data\",\"detect-interval\":5,\"pd-urls\":\"http://172.16.12.73:2379,http://172.16.12.75:2379,http://172.16.12.76:2379\",\"log-file\":\"/alidata/tidb/deploy/log/drainer.log\",\"initial-commit-ts\":0,\"sycner\":{\"sql-mode\":null,\"ignore-txn-commit-ts\":null,\"ignore-schemas\":\"INFORMATION_SCHEMA,METRICS_SCHEMA,PERFORMANCE_SCHEMA,aaa,aliprod119,bak_marketing_0925,dusto_job,goinception,mysql,stress_testing,test,test33,test55,thisaliprod,thisisuat,tidb_loader,txmanager,uat_databank,uat_goods,uat_grid,uat_info,uat_integration,uat_job,uat_marketing,uat_member,uat_message,uat_nacos,uat_order,uat_pay,uat_product,uat_report,uat_saga,uat_security,uat_stock,uat_support,uat_zipkin,pre_databank,pre_info,pre_job,pre_message,pre_nacos,pre_nacos113,pre_product,pre_report,pre_security,pre_support,dev_member,dev_databank,dev_message,dev_info,dev_order,dev_pay,dev_marketing,dev_report,dev_stock,dev_grid,dev_support,dev_security,dev_goods,dev_dusto_dmp,uat_dusto_dmp,pre_dusto_dmp,nacos_dusto_dmp,uat_plan\",\"ignore-table\":[{\"db-name\":\"test\",\"tbl-name\":\"test\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_comp_region_rel_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_category_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_color_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_price_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_size_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_sku_sap\"},{\"db-name\":\"pre_grid\",\"tbl-name\":\"buffer_clerk_sap\"},{\"db-name\":\"pre_grid\",\"tbl-name\":\"buffer_data_err\"},{\"db-name\":\"pre_grid\",\"tbl-name\":\"buffer_store_sap\"},{\"db-name\":\"pre_grid\",\"tbl-name\":\"bonus_rate_rule\"},{\"db-name\":\"pre_order\",\"tbl-name\":\"resend_sap_intf\"},{\"db-name\":\"pre_order\",\"tbl-name\":\"send_sap_intf\"},{\"db-name\":\"pre_order\",\"tbl-name\":\"order_pay_mode_bs\"},{\"db-name\":\"pre_order\",\"tbl-name\":\"order_sub_bs\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_inventory_order_detail_provisional\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_inventory_range\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_allocation\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_sap_inv_sync_detail\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_sap_inv_sync\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_shelf\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_stock_his\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_stock_history\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_warn_history\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_transfer_order\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_transfer_order_detail\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"qrtz_fired_triggers\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"sales_orders\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"stock_temp\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"storeinfo_temp\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"stores\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"t_post_id\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_inventory_order_detail_history\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_inventory_order_history\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"stock_store\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_0611\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_0618\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_2019017\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_bak_0918\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_temp20190515\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_temp20190521\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_transaction_detail_2019017\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_transaction_detail_bak_0918\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp201905_01\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp201905_old\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190521\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190522\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190528\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190529\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190529_01\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_201906011\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190603\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190618\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190624\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190625\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190627\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190701\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190708\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190722\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190729\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190805\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190812\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190820\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190826\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190827\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190912\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190925\"}],\"txn-batch\":20,\"loopback-control\":false,\"sync-ddl\":true,\"channel-id\":0,\"worker-count\":1,\"to\":{\"host\":\"\",\"user\":\"\",\"password\":\"\",\"security\":{\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cert-allowed-cn\":null},\"encrypted_password\":\"\",\"sync-mode\":0,\"port\":0,\"checkpoint\":{\"type\":\"\",\"schema\":\"\",\"host\":\"\",\"user\":\"\",\"password\":\"\",\"encrypted_password\":\"\",\"port\":0,\"security\":{\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cert-allowed-cn\":null}},\"dir\":\"\",\"retention-time\":0,\"merge\":false,\"zookeeper-addrs\":\"\",\"kafka-addrs\":\"172.16.16.221:9092,172.16.17.65:9092,172.16.18.27:9092\",\"kafka-version\":\"1.0.2\",\"kafka-max-messages\":1536,\"kafka-client-id\":\"\",\"kafka-max-message-size\":1610612736,\"topic-name\":\"tidb-binlog-pre\"},\"replicate-do-table\":null,\"replicate-do-db\":null,\"db-type\":\"kafka\",\"relay\":{\"log-dir\":\"\",\"max-file-size\":10485760},\"disable-dispatch-flag\":null,\"enable-dispatch-flag\":null,\"disable-dispatch\":null,\"enable-dispatch\":null,\"safe-mode\":false,\"disable-detect-flag\":null,\"enable-detect-flag\":null,\"disable-detect\":null,\"enable-detect\":null},\"security\":{\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cert-allowed-cn\":null},\"synced-check-time\":5,\"compressor\":\"\",\"EtcdTimeout\":5000000000,\"MetricsAddr\":\"\",\"MetricsInterval\":15}"]
[2023/04/06 16:53:13.775 +08:00] [INFO] [client.go:135] ["[pd] create pd client with endpoints"] [pd-address="[http://172.16.12.73:2379,http://172.16.12.75:2379,http://172.16.12.76:2379]"]
[2023/04/06 16:53:13.778 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://172.16.12.76:2379] [old-leader=]
[2023/04/06 16:53:13.778 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6667489703447796057]
[2023/04/06 16:53:13.778 +08:00] [INFO] [server.go:120] ["get cluster id from pd"] [id=6667489703447796057]
[2023/04/06 16:53:13.780 +08:00] [INFO] [checkpoint.go:64] ["initialize checkpoint"] [type=file] [checkpoint=440603977927622705] [cfg="{\"CheckpointType\":\"file\",\"Db\":null,\"Schema\":\"\",\"Table\":\"\",\"ClusterID\":6667489703447796057,\"InitialCommitTS\":0,\"dir\":\"/alidata/tidb/deploy/data/savepoint\"}"]
[2023/04/06 16:53:13.780 +08:00] [INFO] [store.go:68] ["new store"] [path="tikv://172.16.12.73:2379,172.16.12.75:2379,172.16.12.76:2379?disableGC=true"]
[2023/04/06 16:53:13.780 +08:00] [INFO] [client.go:135] ["[pd] create pd client with endpoints"] [pd-address="[172.16.12.73:2379,172.16.12.75:2379,172.16.12.76:2379]"]
[2023/04/06 16:53:13.782 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://172.16.12.76:2379] [old-leader=]
[2023/04/06 16:53:13.782 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6667489703447796057]
[2023/04/06 16:53:13.783 +08:00] [INFO] [store.go:74] ["new store with retry success"]
[2023/04/06 16:53:26.021 +08:00] [INFO] [client.go:127] ["[sarama] Initializing new client"]
[2023/04/06 16:53:26.021 +08:00] [INFO] [client.go:174] ["[sarama] Successfully initialized new client"]
[2023/04/06 16:53:26.021 +08:00] [INFO] [store.go:68] ["new store"] [path="tikv://172.16.12.73:2379,172.16.12.75:2379,172.16.12.76:2379?disableGC=true"]
[2023/04/06 16:53:26.021 +08:00] [INFO] [client.go:135] ["[pd] create pd client with endpoints"] [pd-address="[172.16.12.73:2379,172.16.12.75:2379,172.16.12.76:2379]"]
[2023/04/06 16:53:26.024 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://172.16.12.76:2379] [old-leader=]
[2023/04/06 16:53:26.024 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6667489703447796057]
[2023/04/06 16:53:26.025 +08:00] [INFO] [store.go:74] ["new store with retry success"]
[2023/04/06 16:53:26.028 +08:00] [INFO] [server.go:265] ["register success"] ["drainer node id"=172.16.12.169:8249]
[2023/04/06 16:53:26.029 +08:00] [INFO] [server.go:326] ["start to server request"] [addr=http://172.16.12.169:8249]
[2023/04/06 16:53:26.030 +08:00] [INFO] [merge.go:222] ["merger add source"] ["source id"=172.16.16.221:8250]
[2023/04/06 16:53:26.030 +08:00] [INFO] [merge.go:222] ["merger add source"] ["source id"=172.16.17.65:8250]
[2023/04/06 16:53:26.030 +08:00] [INFO] [merge.go:222] ["merger add source"] ["source id"=172.16.18.27:8250]
[2023/04/06 16:53:26.030 +08:00] [INFO] [pump.go:138] ["pump create pull binlogs client"] [id=172.16.16.221:8250]
[2023/04/06 16:53:26.030 +08:00] [INFO] [pump.go:138] ["pump create pull binlogs client"] [id=172.16.18.27:8250]
[2023/04/06 16:53:26.030 +08:00] [INFO] [pump.go:138] ["pump create pull binlogs client"] [id=172.16.17.65:8250]
[2023/04/06 16:53:29.544 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604087608672371]
[2023/04/06 16:53:32.547 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604088395104378]
[2023/04/06 16:53:35.650 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604089181536471]
[2023/04/06 17:03:26.021 +08:00] [INFO] [client.go:716] ["[sarama] Client background metadata update:kafka: no specific topics to update metadata"]
[2023/04/06 16:53:38.653 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604089967968464]

旧drainer启动日志

[2023/04/06 16:53:13.786 +08:00] [INFO] [version.go:50] ["Welcome to Drainer"] ["Release Version"=v4.0.3-dirty] ["Git Commit Hash"=fb0cc0d5290202bb6d1b6f189549f629cfd810ef] ["Build TS"="2021-10-21 02:40:08"] ["Go Version"=go1.15.6] ["Go OS/Arch"=linux/amd64]
[2023/04/06 16:53:13.786 +08:00] [INFO] [main.go:46] ["start drainer..."] [config="{\"log-level\":\"info\",\"node-id\":\"172.16.18.27:8249\",\"addr\":\"http://172.16.18.27:8249\",\"advertise-addr\":\"http://172.16.18.27:8249\",\"data-dir\":\"/alidata/tidb/deploy/drainer-8249/data\",\"detect-interval\":5,\"pd-urls\":\"http://172.16.12.73:2379,http://172.16.12.75:2379,http://172.16.12.76:2379\",\"log-file\":\"/alidata/tidb/deploy/drainer-8249/log/drainer.log\",\"initial-commit-ts\":-1,\"sycner\":{\"sql-mode\":null,\"ignore-txn-commit-ts\":null,\"ignore-schemas\":\"INFORMATION_SCHEMA,METRICS_SCHEMA,PERFORMANCE_SCHEMA,aaa,aliprod119,bak_marketing_0925,dusto_job,goinception,mysql,stress_testing,test,test33,test55,thisaliprod,thisisuat,tidb_loader,txmanager,uat_databank,uat_goods,uat_grid,uat_info,uat_integration,uat_job,uat_marketing,uat_member,uat_message,uat_nacos,uat_order,uat_pay,uat_product,uat_report,uat_saga,uat_security,uat_stock,uat_support,uat_zipkin,pre_databank,pre_info,pre_job,pre_message,pre_nacos,pre_nacos113,pre_product,pre_report,pre_security,pre_support,dev_member,dev_databank,dev_message,dev_info,dev_order,dev_pay,dev_marketing,dev_report,dev_stock,dev_grid,dev_support,dev_security,dev_goods,dev_dusto_dmp,uat_dusto_dmp,pre_dusto_dmp,nacos_dusto_dmp,uat_plan\",\"ignore-table\":[{\"db-name\":\"test\",\"tbl-name\":\"test\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_comp_region_rel_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_category_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_color_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_price_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_size_sap\"},{\"db-name\":\"pre_goods\",\"tbl-name\":\"buffer_product_sku_sap\"},{\"db-name\":\"pre_grid\",\"tbl-name\":\"buffer_clerk_sap\"},{\"db-name\":\"pre_grid\",\"tbl-name\":\"buffer_data_err\"},{\"db-name\":\"pre_grid\",\"tbl-name\":\"buffer_store_sap\"},{\"db-name\":\"pre_grid\",\"tbl-name\":\"bonus_rate_rule\"},{\"db-name\":\"pre_order\",\"tbl-name\":\"resend_sap_intf\"},{\"db-name\":\"pre_order\",\"tbl-name\":\"send_sap_intf\"},{\"db-name\":\"pre_order\",\"tbl-name\":\"order_pay_mode_bs\"},{\"db-name\":\"pre_order\",\"tbl-name\":\"order_sub_bs\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_inventory_order_detail_provisional\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_inventory_range\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_allocation\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_sap_inv_sync_detail\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_sap_inv_sync\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_shelf\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_stock_his\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_stock_history\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_sku_warn_history\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_transfer_order\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_transfer_order_detail\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"qrtz_fired_triggers\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"sales_orders\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"stock_temp\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"storeinfo_temp\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"stores\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"t_post_id\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_inventory_order_detail_history\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"kc_inventory_order_history\"},{\"db-name\":\"pre_stock\",\"tbl-name\":\"stock_store\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_0611\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_0618\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_2019017\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_bak_0918\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_temp20190515\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_common_account_temp20190521\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_transaction_detail_2019017\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"mb_points_transaction_detail_bak_0918\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp201905_01\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp201905_old\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190521\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190522\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190528\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190529\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190529_01\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_201906011\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190603\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190618\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190624\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190625\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190627\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190701\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190708\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190722\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190729\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190805\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190812\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190820\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190826\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190827\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190912\"},{\"db-name\":\"pre_member\",\"tbl-name\":\"member_info_temp_20190925\"}],\"txn-batch\":20,\"loopback-control\":false,\"sync-ddl\":true,\"channel-id\":0,\"worker-count\":1,\"to\":{\"host\":\"\",\"user\":\"\",\"password\":\"\",\"security\":{\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cert-allowed-cn\":null},\"encrypted_password\":\"\",\"sync-mode\":0,\"port\":0,\"checkpoint\":{\"type\":\"\",\"schema\":\"\",\"host\":\"\",\"user\":\"\",\"password\":\"\",\"encrypted_password\":\"\",\"port\":0,\"security\":{\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cert-allowed-cn\":null}},\"dir\":\"\",\"retention-time\":0,\"merge\":false,\"zookeeper-addrs\":\"\",\"kafka-addrs\":\"172.16.16.221:9092,172.16.17.65:9092,172.16.18.27:9092\",\"kafka-version\":\"1.0.2\",\"kafka-max-messages\":1536,\"kafka-client-id\":\"\",\"kafka-max-message-size\":1610612736,\"topic-name\":\"tidb-binlog-pre\"},\"replicate-do-table\":null,\"replicate-do-db\":null,\"db-type\":\"kafka\",\"relay\":{\"log-dir\":\"\",\"max-file-size\":10485760},\"disable-dispatch-flag\":null,\"enable-dispatch-flag\":null,\"disable-dispatch\":null,\"enable-dispatch\":null,\"safe-mode\":false,\"disable-detect-flag\":null,\"enable-detect-flag\":null,\"disable-detect\":null,\"enable-detect\":null},\"security\":{\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cert-allowed-cn\":null},\"synced-check-time\":5,\"compressor\":\"\",\"EtcdTimeout\":5000000000,\"MetricsAddr\":\"\",\"MetricsInterval\":15}"]
[2023/04/06 16:53:13.786 +08:00] [INFO] [client.go:135] ["[pd] create pd client with endpoints"] [pd-address="[http://172.16.12.73:2379,http://172.16.12.75:2379,http://172.16.12.76:2379]"]
[2023/04/06 16:53:13.789 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://172.16.12.76:2379] [old-leader=]
[2023/04/06 16:53:13.789 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6667489703447796057]
[2023/04/06 16:53:13.789 +08:00] [INFO] [server.go:120] ["get cluster id from pd"] [id=6667489703447796057]
[2023/04/06 16:53:13.790 +08:00] [INFO] [server.go:129] ["set InitialCommitTS"] [ts=440604083820691565]
[2023/04/06 16:53:13.790 +08:00] [INFO] [checkpoint.go:64] ["initialize checkpoint"] [type=file] [checkpoint=440602031873327109] [cfg="{\"CheckpointType\":\"file\",\"Db\":null,\"Schema\":\"\",\"Table\":\"\",\"ClusterID\":6667489703447796057,\"InitialCommitTS\":440604083820691565,\"dir\":\"/alidata/tidb/deploy/drainer-8249/data/savepoint\"}"]
[2023/04/06 16:53:13.790 +08:00] [INFO] [store.go:68] ["new store"] [path="tikv://172.16.12.73:2379,172.16.12.75:2379,172.16.12.76:2379?disableGC=true"]
[2023/04/06 16:53:13.791 +08:00] [INFO] [client.go:135] ["[pd] create pd client with endpoints"] [pd-address="[172.16.12.73:2379,172.16.12.75:2379,172.16.12.76:2379]"]
[2023/04/06 16:53:13.792 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://172.16.12.76:2379] [old-leader=]
[2023/04/06 16:53:13.792 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6667489703447796057]
[2023/04/06 16:53:13.794 +08:00] [INFO] [store.go:74] ["new store with retry success"]
[2023/04/06 16:53:26.501 +08:00] [INFO] [client.go:127] ["[sarama] Initializing new client"]
[2023/04/06 16:53:26.501 +08:00] [INFO] [client.go:174] ["[sarama] Successfully initialized new client"]
[2023/04/06 16:53:26.502 +08:00] [INFO] [store.go:68] ["new store"] [path="tikv://172.16.12.73:2379,172.16.12.75:2379,172.16.12.76:2379?disableGC=true"]
[2023/04/06 16:53:26.502 +08:00] [INFO] [client.go:135] ["[pd] create pd client with endpoints"] [pd-address="[172.16.12.73:2379,172.16.12.75:2379,172.16.12.76:2379]"]
[2023/04/06 16:53:26.504 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://172.16.12.76:2379] [old-leader=]
[2023/04/06 16:53:26.504 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6667489703447796057]
[2023/04/06 16:53:26.506 +08:00] [INFO] [store.go:74] ["new store with retry success"]
[2023/04/06 16:53:26.510 +08:00] [INFO] [server.go:265] ["register success"] ["drainer node id"=172.16.18.27:8249]
[2023/04/06 16:53:26.510 +08:00] [INFO] [server.go:326] ["start to server request"] [addr=http://172.16.18.27:8249]
[2023/04/06 16:53:26.511 +08:00] [INFO] [merge.go:222] ["merger add source"] ["source id"=172.16.17.65:8250]
[2023/04/06 16:53:26.511 +08:00] [INFO] [merge.go:222] ["merger add source"] ["source id"=172.16.18.27:8250]
[2023/04/06 16:53:26.511 +08:00] [INFO] [merge.go:222] ["merger add source"] ["source id"=172.16.16.221:8250]
[2023/04/06 16:53:26.511 +08:00] [INFO] [pump.go:138] ["pump create pull binlogs client"] [id=172.16.16.221:8250]
[2023/04/06 16:53:26.511 +08:00] [INFO] [pump.go:138] ["pump create pull binlogs client"] [id=172.16.17.65:8250]
[2023/04/06 16:53:26.511 +08:00] [INFO] [pump.go:138] ["pump create pull binlogs client"] [id=172.16.18.27:8250]
[2023/04/06 16:53:29.538 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604087608672371]
[2023/04/06 16:53:32.541 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604088395104378]
[2023/04/06 16:53:35.645 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604089181536471]
[2023/04/06 16:53:38.649 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604089967968464]
[2023/04/06 17:03:26.502 +08:00] [INFO] [client.go:716] ["[sarama] Client background metadata update:kafka: no specific topics to update metadata"]
[2023/04/06 17:03:28.223 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604244462534658]
[2023/04/06 17:03:31.227 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604245248966660]
[2023/04/06 17:03:34.231 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604246035398661]
[2023/04/06 17:09:26.523 +08:00] [WARN] [base_client.go:184] ["[pd] cannot update leader"] [address=http://172.16.12.73:2379] [error="error:rpc error: code = Unavailable desc = transport is closing target:172.16.12.73:2379 status:CONNECTING"] [errorVerbose="error:rpc error: code = Unavailable desc = transport is closing target:172.16.12.73:2379 status:CONNECTING\ngithub.com/pingcap/pd/v4/client.(*baseClient).getMembers\n\t/Users/weizheng/go/pkg/mod/github.com/pingcap/pd/v4@v4.0.0-beta.1.0.20200305072537-61d9f9cc35d3/client/base_client.go:208\ngithub.com/pingcap/pd/v4/client.(*baseClient).updateLeader\n\t/Users/weizheng/go/pkg/mod/github.com/pingcap/pd/v4@v4.0.0-beta.1.0.20200305072537-61d9f9cc35d3/client/base_client.go:182\ngithub.com/pingcap/pd/v4/client.(*baseClient).leaderLoop\n\t/Users/weizheng/go/pkg/mod/github.com/pingcap/pd/v4@v4.0.0-beta.1.0.20200305072537-61d9f9cc35d3/client/base_client.go:129\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374\ngithub.com/pingcap/pd/v4/client.(*baseClient).getMembers\n\t/Users/weizheng/go/pkg/mod/github.com/pingcap/pd/v4@v4.0.0-beta.1.0.20200305072537-61d9f9cc35d3/client/base_client.go:209\ngithub.com/pingcap/pd/v4/client.(*baseClient).updateLeader\n\t/Users/weizheng/go/pkg/mod/github.com/pingcap/pd/v4@v4.0.0-beta.1.0.20200305072537-61d9f9cc35d3/client/base_client.go:182\ngithub.com/pingcap/pd/v4/client.(*baseClient).leaderLoop\n\t/Users/weizheng/go/pkg/mod/github.com/pingcap/pd/v4@v4.0.0-beta.1.0.20200305072537-61d9f9cc35d3/client/base_client.go:129\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"]
[2023/04/06 17:09:29.522 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604339083411493]
[2023/04/06 17:09:32.526 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440604339869843531]

qufudcj · 2023 年4 月 6 日 09:29

不知道这个
[“[sarama] Client background metadata update:kafka: no specific topics to update metadata”]
是不是有关系

qufudcj · 2023 年4 月 6 日 09:34

并且用github上的java消费者尝试消费，是读不出消息的，用kafka的命令行消费也会卡住，不显示任何返回

qufudcj · 2023 年4 月 6 日 09:51

这个是不是找不到topic的意思？需要我删掉重建吗

qufudcj · 2023 年4 月 6 日 10:07

我换了一个新的topic也还是没有消息进来…

qufudcj · 2023 年4 月 6 日 10:24

drainer的配置是：

- host: 172.16.12.169
  ssh_port: 40022
  port: 8249
  deploy_dir: /alidata/tidb/deploy
  data_dir: data
  config:
    syncer.db-type: kafka
    syncer.to.kafka-addrs: 172.16.16.221:9092,172.16.17.65:9092,172.16.18.27:9092
    syncer.to.kafka-max-message-size: 1610612736
    syncer.to.kafka-max-messages: 1536
    syncer.to.kafka-version: 1.0.2
    syncer.to.topic-name: dcj-test
  arch: amd64
  os: linux

tidb狂热爱好者 · 2023 年4 月 6 日 10:28

你这个很老的tidb了吧

qufudcj · 2023 年4 月 6 日 10:29

是的，4.0.4，一直没有升级，因为生产环境还牵扯到binlog同步，所以一直没有升级计划

db_user · 2023 年4 月 6 日 10:49

看下这个，检查下状态
https://docs.pingcap.com/zh/tidb/stable/binlog-control

qufudcj · 2023 年4 月 6 日 12:09

我目前看俩都是在线状态，上面一行是新加的drainer，下面的是原来的

qufudcj · 2023 年4 月 6 日 12:21

我也曾尝试过把那个老的标记为下线状态，消息队列中仍然没有新消息

db_user · 2023 年4 月 7 日 01:23

pumper的状态也正常么

qufudcj · 2023 年4 月 7 日 01:29

都是在线

目前display看都是正常的

“kafka: no specific topics to update metadata”这个报错我翻了下之前的老drainer日志，是没有这个报错德尔

db_user · 2023 年4 月 7 日 01:42

这个应该是个bug,但是好像没有解决方案：
remove confused sarama log kafka: no specific topics to update metadata · Issue #969 · pingcap/tidb-binlog (github.com)

qufudcj · 2023 年4 月 7 日 02:02

我刚尝试又扩了一个drainer，发现有消息送入了，但是没一会儿就又有报错自己挂掉了，这个是不是消息太大，kafka在30s没反馈，于是超时了呢？

[2023/04/07 09:45:16.393 +08:00] [INFO] [broker.go:212] ["[sarama] Connected to broker at 172.16.18.27:9092 (registered as #3)\n"]
[2023/04/07 09:45:19.622 +08:00] [INFO] [pump.go:166] ["receive big size binlog"] [size="592 MB"]
[2023/04/07 09:46:57.271 +08:00] [INFO] [syncer.go:257] ["write save point"] [ts=440465904976789506]
[2023/04/07 09:46:57.271 +08:00] [ERROR] [syncer.go:457] ["Failed to close syncer"] [error="fail to push msg to kafka after 30s, check if kafka is up and working"] [errorVerbose="fail to push msg to kafka after 30s, check if kafka is up and working\ngithub.com/pingcap/tidb-binlog/drainer/sync.(*KafkaSyncer).run\n\t/home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/drainer/sync/kafka.go:236\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
[2023/04/07 09:46:58.418 +08:00] [INFO] [async_producer.go:971] ["[sarama] producer/broker/3 state change to [closing] because write tcp 172.16.12.165:47234->172.16.18.27:9092: i/o timeout\n"]
[2023/04/07 09:46:58.418 +08:00] [INFO] [broker.go:253] ["[sarama] Closed connection to broker 172.16.18.27:9092\n"]
[2023/04/07 09:46:58.418 +08:00] [INFO] [async_producer.go:578] ["[sarama] producer/leader/tidb-binlog-pre/0 state change to [retrying-1]\n"]
[2023/04/07 09:46:58.418 +08:00] [INFO] [async_producer.go:588] ["[sarama] producer/leader/tidb-binlog-pre/0 abandoning broker 3\n"]
[2023/04/07 09:46:58.418 +08:00] [INFO] [async_producer.go:717] ["[sarama] producer/broker/3 input chan closed\n"]
[2023/04/07 09:46:58.419 +08:00] [INFO] [async_producer.go:801] ["[sarama] producer/broker/3 shut down\n"]
[2023/04/07 09:46:58.919 +08:00] [INFO] [client.go:772] ["[sarama] client/metadata fetching metadata for [tidb-binlog-pre] from broker 172.16.17.65:9092\n"]
[2023/04/07 09:46:58.922 +08:00] [INFO] [async_producer.go:711] ["[sarama] producer/broker/3 starting up\n"]
[2023/04/07 09:46:58.922 +08:00] [INFO] [async_producer.go:727] ["[sarama] producer/broker/3 state change to [open] on tidb-binlog-pre/0\n"]
[2023/04/07 09:46:58.922 +08:00] [INFO] [async_producer.go:570] ["[sarama] producer/leader/tidb-binlog-pre/0 selected broker 3\n"]
[2023/04/07 09:46:58.922 +08:00] [INFO] [async_producer.go:594] ["[sarama] producer/leader/tidb-binlog-pre/0 state change to [flushing-1]\n"]
[2023/04/07 09:46:58.922 +08:00] [INFO] [async_producer.go:616] ["[sarama] producer/leader/tidb-binlog-pre/0 state change to [normal]\n"]
[2023/04/07 09:46:58.922 +08:00] [INFO] [broker.go:212] ["[sarama] Connected to broker at 172.16.18.27:9092 (registered as #3)\n"]
[2023/04/07 09:47:07.272 +08:00] [INFO] [server.go:451] ["begin to close drainer server"]
[2023/04/07 09:47:07.272 +08:00] [ERROR] [util.go:65] ["Recovered from panic"] [err="\"Waiting too long for `Syncer.run` to quit.\""] ["real stack"="github.com/pingcap/tidb-binlog/drainer.(*taskGroup).start.func1.1\n\t/home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/drainer/util.go:67\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:679\ngithub.com/pingcap/tidb-binlog/drainer.(*Syncer).run\n\t/home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:463\ngithub.com/pingcap/tidb-binlog/drainer.(*Syncer).Start\n\t/home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:132\ngithub.com/pingcap/tidb-binlog/drainer.(*Server).Start.func4\n\t/home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/drainer/server.go:288\ngithub.com/pingcap/tidb-binlog/drainer.(*taskGroup).start.func1\n\t/home/jenkins/agent/workspace/build_binlog_multi_branch_v4.0.4/go/src/github.com/pingcap/tidb-binlog/drainer/util.go:75"] [name=syncer]
[2023/04/07 09:47:07.272 +08:00] [INFO] [util.go:72] [Exit] [name=syncer]
[2023/04/07 09:47:07.275 +08:00] [INFO] [server.go:416] ["has already update status"] [id=172.16.12.165:8249]

db_user · 2023 年4 月 7 日 02:10

看下kafka的状态和报错日志，有可能是message-bytes的配置关系

qufudcj · 2023 年4 月 7 日 02:27

这个值已经改的挺大了，而且kafka也没有“ Message was too large”的报错

drainer是否可以主动限制消息大小呢

db_user · 2023 年4 月 7 日 02:32

https://docs.pingcap.com/zh/tidb/stable/tidb-binlog-configuration-file#drainer
https://docs.pingcap.com/zh/tidb/stable/deploy-tidb-binlog
可以根据这两篇来调整下配置

qufudcj · 2023 年4 月 7 日 02:38

4.0的版本中，似乎是没有这个选项的

qufudcj · 2023 年4 月 7 日 08:55

我尝试了各种操作，比如升高kafka版本，搞了个新队列等等，还是不大行

我想尝试单独使用高版本的drainer替换二进制文件，请问会有什么灾难性后果吗