drainer修改配置后无法进行重启变更,重启操作无效果

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。

  • 【TiDB 版本】: V3.0.14
  • 【问题描述】:修改了中控机tidb部署目录/config/drainer配置文件,然后使用rolling_update.yml进行重启,结果发现,drainer并未进行重启,与没有重启一样。

当前 inventory.ini 中关于drainer配置

[drainer_servers]
pt2bigdata ansible_host=192.168.17.3 drainer_port=8449 deploy_dir=/data/tidb/4200/drainer initial_commit_ts="417122067683475461"

变更操作步骤:

1> 在 ~/tidb-ansible-v3.0.14/conf/pt2bigdata_drainer.toml 配置文件中增加 过滤表规则

[[syncer.replicate-do-table]]
db-name ="pt_activity"
tbl-name = "activity"

2> 通过 ansible-playbook rolling_update.yml 命令,尝试使用3种方法重启drainer节点

第一种方法:  ansible-playbook rolling_update.yml -t drainer -l pt2bigdata

第二种方法:  ansible-playbook rolling_update.yml -t drainer -l 192.168.17.3

第三种方法:  ansible-playbook rolling_update.yml -t drainer

3> 3种方法分别进行操作后,均发现drainer部署节点192.168.17.3上drainer进程号没有改变,drainer.log没有出现任何关于重启的日志,配置文件中也未出现新增的表过滤规则

以下是发起变更的日志 ,帮确认下drainer节点无法进行重启操作的原因,谢谢

[tidb@bj-mt-tidb-olap-master-prod-01 tidb-ansible-v3.0.14]$ ansible-playbook rolling_update.yml -t drainer -l pt2bigdata

PLAY [check config locally] ******************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [check system environment] **************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [gather all facts, and check dest] ******************************************************************************************************************************************************************************************************

TASK [check_config_dynamic : Set enable_binlog variable] *************************************************************************************************************************************************************************************

TASK [check_config_dynamic : Set deploy_dir if not set] **************************************************************************************************************************************************************************************

▽

TASK [check_config_dynamic : environment check (deploy dir)] *********************************************************************************************************************************************************************************
ok: [pt2bigdata]

TASK [check_config_dynamic : Preflight check - Does deploy dir have appropriate permission] **************************************************************************************************************************************************

TASK [check_config_dynamic : environment check (supervise)] **********************************************************************************************************************************************************************************

TASK [check_config_dynamic : config skip variables (default)] ********************************************************************************************************************************************************************************
ok: [pt2bigdata]

TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************
ok: [pt2bigdata]

TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************


PLAY [Pre-check PD configuration] ************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Pre-check TiKV configuration] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Pre-check TiDB configuration] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [pd_servers[0]] *************************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [pd_servers] ****************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
 [WARNING]: Could not match supplied host pattern, ignoring: pd_servers_followers

 [WARNING]: Could not match supplied host pattern, ignoring: pd_servers_leader


PLAY [rolling update PD cluster] *************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [rolling update TiKV cluster] ***********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [rolling update pump cluster] ***********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [rolling update TiDB cluster] ***********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
pt2bigdata                 : ok=3    changed=0    unreachable=0    failed=0

Congrats! All goes well. :-)



*tidb@bj-mt-tidb-olap-master-prod-01 tidb-ansible-v3.0.14]$ ansible-playbook rolling_update.yml -t drainer -l 192.168.17.3

PLAY [check config locally] ******************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [check system environment] **************************************************************************************************************************************************************************************************************

TASK [check_system_dynamic : Disk space check - Fail task when disk is full] *****************************************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_system_dynamic : get facts] ******************************************************************************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_system_dynamic : Preflight check - Get hostnames of all nodes in cluster] ********************************************************************************************************************************************************

TASK [check_system_dynamic : Preflight check - Does every node in cluster have different hostname] *******************************************************************************************************************************************

TASK [check_system_dynamic : Preflight check - Get NTP service status] ***********************************************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_system_dynamic : Preflight check - NTP service] **********************************************************************************************************************************************************************************

TASK [check_system_dynamic : Preflight check - Get umask] ************************************************************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_system_dynamic : Preflight check - Does the system have a standard umask] ********************************************************************************************************************************************************

TASK [check_system_dynamic : Preflight check - Get maximum number of open file descriptors limit] ********************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_system_dynamic : Preflight check - ulimit -n] ************************************************************************************************************************************************************************************

TASK [check_system_dynamic : Preflight check - Check swap] ***********************************************************************************************************************************************************************************

PLAY [gather all facts, and check dest] ******************************************************************************************************************************************************************************************************

TASK [check_config_dynamic : Set enable_binlog variable] *************************************************************************************************************************************************************************************

TASK [check_config_dynamic : Set deploy_dir if not set] **************************************************************************************************************************************************************************************

TASK [check_config_dynamic : environment check (deploy dir)] *********************************************************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_config_dynamic : Preflight check - Does deploy dir have appropriate permission] **************************************************************************************************************************************************

TASK [check_config_dynamic : environment check (supervise)] **********************************************************************************************************************************************************************************

TASK [check_config_dynamic : config skip variables (default)] ********************************************************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************

PLAY [Pre-check for rolling update] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Pre-check PD configuration] ************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Pre-check TiKV configuration] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Pre-check TiDB configuration] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [pd_servers[0]] *************************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [pd_servers] ****************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
 [WARNING]: Could not match supplied host pattern, ignoring: pd_servers_followers

 [WARNING]: Could not match supplied host pattern, ignoring: pd_servers_leader


PLAY [rolling update PD cluster] *************************************************************************************************************************************************************************************************************
skipping: no hosts matched



TASK [check_system_dynamic : Preflight check - NTP service] **********************************************************************************************************************************************************************************

TASK [check_system_dynamic : Preflight check - Get umask] ************************************************************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_system_dynamic : Preflight check - Does the system have a standard umask] ********************************************************************************************************************************************************

TASK [check_system_dynamic : Preflight check - Get maximum number of open file descriptors limit] ********************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_system_dynamic : Preflight check - ulimit -n] ************************************************************************************************************************************************************************************

TASK [check_system_dynamic : Preflight check - Check swap] ***********************************************************************************************************************************************************************************

PLAY [gather all facts, and check dest] ******************************************************************************************************************************************************************************************************

TASK [check_config_dynamic : Set enable_binlog variable] *************************************************************************************************************************************************************************************

TASK [check_config_dynamic : Set deploy_dir if not set] **************************************************************************************************************************************************************************************

TASK [check_config_dynamic : environment check (deploy dir)] *********************************************************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_config_dynamic : Preflight check - Does deploy dir have appropriate permission] **************************************************************************************************************************************************

TASK [check_config_dynamic : environment check (supervise)] **********************************************************************************************************************************************************************************

TASK [check_config_dynamic : config skip variables (default)] ********************************************************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************
ok: [192.168.17.3]

TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************

PLAY [Pre-check for rolling update] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Pre-check PD configuration] ************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Pre-check TiKV configuration] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Pre-check TiDB configuration] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [pd_servers[0]] *************************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [pd_servers] ****************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
 [WARNING]: Could not match supplied host pattern, ignoring: pd_servers_followers

 [WARNING]: Could not match supplied host pattern, ignoring: pd_servers_leader


PLAY [rolling update PD cluster] *************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [rolling update TiKV cluster] ***********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [rolling update pump cluster] ***********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [rolling update TiDB cluster] ***********************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
192.168.17.3               : ok=8    changed=0    unreachable=0    failed=0

Congrats! All goes well. :-)
[tidb@bj-mt-tidb-olap-master-prod-01 tidb-ansible-v3.0.14]$ ansible-playbook rolling_update.yml -t drainer

PLAY [check config locally] ******************************************************************************************************************************************************************************************************************

TASK [check_config_static : Ensure monitoring_servers exists] ********************************************************************************************************************************************************************************

TASK [check_config_static : Ensure monitored_servers exists] *********************************************************************************************************************************************************************************

TASK [check_config_static : Ensure TiDB host exists] *****************************************************************************************************************************************************************************************

TASK [check_config_static : Ensure PD host exists] *******************************************************************************************************************************************************************************************

TASK [check_config_static : Ensure TiKV host exists] *****************************************************************************************************************************************************************************************

TASK [check_config_static : Check ansible_user variable] *************************************************************************************************************************************************************************************

TASK [check_config_static : Ensure timezone variable is set] *********************************************************************************************************************************************************************************

TASK [check_config_static : Close old SSH control master processes] **************************************************************************************************************************************************************************
ok: [localhost]

TASK [check_config_static : Check ansible version] *******************************************************************************************************************************************************************************************

TASK [check_config_static : Check if jmespath is installed] **********************************************************************************************************************************************************************************
changed: [localhost]

TASK [check_config_static : Check if jinja2 is installed] ************************************************************************************************************************************************************************************
changed: [localhost]

TASK [check_config_static : Preflight check - Fail when jmespath or jinja2 doesn't exist] ****************************************************************************************************************************************************

TASK [check_config_static : Get jmespath info] ***********************************************************************************************************************************************************************************************
changed: [localhost]

TASK [check_config_static : Get jmespath version] ********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [check_config_static : Get jinja2 info] *************************************************************************************************************************************************************************************************
changed: [localhost]

TASK [check_config_static : Get jinja2 version] **********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [check_config_static : Preflight check - Fail when the versions of jmespath and jinja2 doesn't meet the requirements] *******************************************************************************************************************

PLAY [check system environment] **************************************************************************************************************************************************************************************************************

TASK [check_system_dynamic : Disk space check - Fail task when disk is full] *****************************************************************************************************************************************************************
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.155]
ok: [192.168.17.154]
ok: [192.168.17.167]
ok: [192.168.17.158]
ok: [192.168.16.72]
ok: [192.168.17.169]
ok: [192.168.17.168]
ok: [192.168.17.251]
ok: [192.168.17.5]
ok: [192.168.17.252]
ok: [192.168.17.3]
ok: [192.168.17.4]

TASK [check_system_dynamic : get facts] ******************************************************************************************************************************************************************************************************
ok: [192.168.17.155]
ok: [192.168.17.167]
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.154]
ok: [192.168.16.72]
ok: [192.168.17.158]
ok: [192.168.17.168]
ok: [192.168.17.169]
ok: [192.168.17.251]
ok: [192.168.17.5]
ok: [192.168.17.3]
ok: [192.168.17.252]
ok: [192.168.17.4]

TASK [check_system_dynamic : Preflight check - Get hostnames of all nodes in cluster] ********************************************************************************************************************************************************
ok: [192.168.17.152]

TASK [check_system_dynamic : Preflight check - Does every node in cluster have different hostname] *******************************************************************************************************************************************

TASK [check_system_dynamic : Preflight check - Get NTP service status] ***********************************************************************************************************************************************************************
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.155]
ok: [192.168.17.167]
ok: [192.168.17.154]
ok: [192.168.17.158]
ok: [192.168.16.72]
ok: [192.168.17.169]
ok: [192.168.17.168]
ok: [192.168.17.251]
ok: [192.168.17.5]
ok: [192.168.17.3]
ok: [192.168.17.252]
ok: [192.168.17.4]

TASK [check_system_dynamic : Preflight check - NTP service] **********************************************************************************************************************************************************************************

TASK [check_system_dynamic : Preflight check - Get umask] ************************************************************************************************************************************************************************************
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.155]
ok: [192.168.17.167]
ok: [192.168.17.154]
ok: [192.168.17.158]
ok: [192.168.16.72]
ok: [192.168.17.168]
ok: [192.168.17.169]
ok: [192.168.17.251]
ok: [192.168.17.5]
ok: [192.168.17.3]
ok: [192.168.17.4]
ok: [192.168.17.252]

TASK [check_system_dynamic : Preflight check - Does the system have a standard umask] ********************************************************************************************************************************************************

TASK [check_system_dynamic : Preflight check - Get maximum number of open file descriptors limit] ********************************************************************************************************************************************
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.155]
ok: [192.168.17.167]
ok: [192.168.17.154]
ok: [192.168.17.158]
ok: [192.168.16.72]
ok: [192.168.17.169]
ok: [192.168.17.168]
ok: [192.168.17.251]
ok: [192.168.17.5]
ok: [192.168.17.252]
ok: [192.168.17.3]
ok: [192.168.17.4]

TASK [check_system_dynamic : Preflight check - ulimit -n] ************************************************************************************************************************************************************************************

TASK [check_system_dynamic : Preflight check - Check swap] ***********************************************************************************************************************************************************************************

PLAY [gather all facts, and check dest] ******************************************************************************************************************************************************************************************************

TASK [check_config_dynamic : Set enable_binlog variable] *************************************************************************************************************************************************************************************

TASK [check_config_dynamic : Set deploy_dir if not set] **************************************************************************************************************************************************************************************

TASK [check_config_dynamic : environment check (deploy dir)] *********************************************************************************************************************************************************************************
ok: [192.168.17.158]
ok: [192.168.17.155]
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.154]
ok: [192.168.17.167]
ok: [192.168.16.72]
ok: [192.168.17.168]
ok: [192.168.17.169]
ok: [192.168.17.251]
ok: [192.168.17.5]
ok: [192.168.17.3]
ok: [192.168.17.4]
ok: [192.168.17.252]
ok: [pump1]
ok: [pump2]
ok: [pump3]
ok: [pt2bigdata]
ok: [TiKV1-2]
ok: [TiKV2-2]
ok: [TiKV3-2]
ok: [TiKV4-2]
ok: [TiKV5-2]

TASK [check_config_dynamic : Preflight check - Does deploy dir have appropriate permission] **************************************************************************************************************************************************

TASK [check_config_dynamic : environment check (supervise)] **********************************************************************************************************************************************************************************

TASK [check_config_dynamic : config skip variables (default)] ********************************************************************************************************************************************************************************
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.154]
ok: [192.168.17.158]
ok: [192.168.17.155]
ok: [192.168.17.167]
ok: [192.168.17.168]
ok: [192.168.17.169]
ok: [192.168.16.72]
ok: [192.168.17.251]
ok: [192.168.17.252]
ok: [192.168.17.5]
ok: [192.168.17.3]
ok: [192.168.17.4]
ok: [pump2]
ok: [pump1]
ok: [pump3]
ok: [TiKV1-2]
ok: [pt2bigdata]
ok: [TiKV2-2]
ok: [TiKV3-2]
ok: [TiKV4-2]
ok: [TiKV5-2]

TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.154]
ok: [192.168.17.158]
ok: [192.168.17.155]
ok: [192.168.17.167]
ok: [192.168.17.168]
ok: [192.168.17.169]
ok: [192.168.16.72]
ok: [192.168.17.251]
ok: [192.168.17.252]
ok: [192.168.17.5]
ok: [192.168.17.3]
ok: [pump1]
ok: [192.168.17.4]
ok: [pump2]
ok: [pump3]
ok: [pt2bigdata]
ok: [TiKV1-2]
ok: [TiKV2-2]
ok: [TiKV3-2]
ok: [TiKV4-2]
ok: [TiKV5-2]

TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************

PLAY [Pre-check for rolling update] **********************************************************************************************************************************************************************************************************

TASK [shell] *********************************************************************************************************************************************************************************************************************************
changed: [192.168.17.152]
changed: [192.168.17.153]
changed: [192.168.17.154]

TASK [Check whether can perform rolling update] **********************************************************************************************************************************************************************************************

PLAY [Pre-check PD configuration] ************************************************************************************************************************************************************************************************************

PLAY [Pre-check TiKV configuration] **********************************************************************************************************************************************************************************************************

PLAY [Pre-check TiDB configuration] **********************************************************************************************************************************************************************************************************

PLAY [pd_servers[0]] *************************************************************************************************************************************************************************************************************************

PLAY [pd_servers] ****************************************************************************************************************************************************************************************************************************

.......。。。。。。。。。。

PLAY [rolling update TiKV cluster] ***********************************************************************************************************************************************************************************************************

PLAY [rolling update TiKV cluster] ***********************************************************************************************************************************************************************************************************

▽

PLAY [rolling update TiKV cluster] ***********************************************************************************************************************************************************************************************************

PLAY [rolling update pump cluster] ***********************************************************************************************************************************************************************************************************


▽
PLAY [rolling update pump cluster] ***********************************************************************************************************************************************************************************************************

PLAY [rolling update pump cluster] ***********************************************************************************************************************************************************************************************************

PLAY [rolling update TiDB cluster] ***********************************************************************************************************************************************************************************************************

PLAY [rolling update TiDB cluster] ***********************************************************************************************************************************************************************************************************

PLAY [rolling update TiDB cluster] ***********************************************************************************************************************************************************************************************************

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
192.168.16.72              : ok=8    changed=0    unreachable=0    failed=0
192.168.17.152             : ok=10   changed=1    unreachable=0    failed=0
192.168.17.153             : ok=9    changed=1    unreachable=0    failed=0
192.168.17.154             : ok=9    changed=1    unreachable=0    failed=0
192.168.17.155             : ok=8    changed=0    unreachable=0    failed=0
192.168.17.158             : ok=8    changed=0    unreachable=0    failed=0
192.168.17.167             : ok=8    changed=0    unreachable=0    failed=0
192.168.17.168             : ok=8    changed=0    unreachable=0    failed=0
192.168.17.169             : ok=8    changed=0    unreachable=0    failed=0
192.168.17.251             : ok=8    changed=0    unreachable=0    failed=0
192.168.17.252             : ok=8    changed=0    unreachable=0    failed=0
192.168.17.3               : ok=8    changed=0    unreachable=0    failed=0
192.168.17.4               : ok=8    changed=0    unreachable=0    failed=0
192.168.17.5               : ok=8    changed=0    unreachable=0    failed=0
TiKV1-2                    : ok=3    changed=0    unreachable=0    failed=0
TiKV2-2                    : ok=3    changed=0    unreachable=0    failed=0
TiKV3-2                    : ok=3    changed=0    unreachable=0    failed=0
TiKV4-2                    : ok=3    changed=0    unreachable=0    failed=0
TiKV5-2                    : ok=3    changed=0    unreachable=0    failed=0
localhost                  : ok=7    changed=4    unreachable=0    failed=0
pt2bigdata                 : ok=3    changed=0    unreachable=0    failed=0
pump1                      : ok=3    changed=0    unreachable=0    failed=0
pump2                      : ok=3    changed=0    unreachable=0    failed=0
pump3                      : ok=3    changed=0    unreachable=0    failed=0

Congrats! All goes well. :-)



你好,

修改 inventory 文件中的配置需要执行 deploy -t drainer 进行配置刷新。

对于 drainer 可以通过 stop_drainer.yml 和 start_drainer.yml 进行重启

大佬,我修改的不是inventory文件,是config目录下的某个drainer的配置文件,也是需要使用 deploy -t drainer 进行配置刷新 吗 ?

对于 drainer 可以通过 stop_drainer.yml 和 start_drainer.yml 进行重启 。如果只是修改了多个drainer节点中的一个节点,那么 stop_drainer.yml 和 start_drainer.yml 是重启所有drainer节点吧 ,有没有只重启单个节点的语法呢 ,帮确认下

多谢

不好意思,看错了。

也需要 deploy 进行更新。

stop start drainer 功能类似 rolling_update ,所以可以通过 -l 参数可以使用,可以 --help 查询下。

1赞

好的 ,大佬帮check一下流程是否准确

1: ansible-playbook stop_drainer.yml -l 某个drainer节点名称

2: 修改该drainer节点的配置文件

3: 刷新配置文件: ansible-playbook deploy_drainer.yml -l 某个drainer节点名称

4: ansible-playbook start_drainer.yml -l 某个drainer节点名称

感谢

可以的,deploy 之后可以增加一个验证步骤,cat drainer_deploy-dir/conf/drainer.toml 是否刷新了更改的配置

好的 :smiling_face_with_three_hearts::smiling_face_with_three_hearts::smiling_face_with_three_hearts:

好的,测试通过可以将结果反馈到帖子中,如果有问题欢迎继续跟帖

嗯 ,已完美实现需求 ,完整流程如下

1> ansible-playbook stop_drainer.yml -l pt2bigdata
2> 修改pt2bigdata节点的配置文件
3> 刷新配置文件: ansible-playbook deploy_drainer.yml -l pt2bigdata
4> 在该drainer部署节点上确认配置文件修改成功
5> ansible-playbook start_drainer.yml -l pt2bigdata

感谢大佬指导 ~~

:call_me_hand: