为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。
- 【TiDB 版本】: V3.0.14
- 【问题描述】:修改了中控机tidb部署目录/config/drainer配置文件,然后使用rolling_update.yml进行重启,结果发现,drainer并未进行重启,与没有重启一样。
当前 inventory.ini 中关于drainer配置
[drainer_servers]
pt2bigdata ansible_host=192.168.17.3 drainer_port=8449 deploy_dir=/data/tidb/4200/drainer initial_commit_ts="417122067683475461"
变更操作步骤:
1> 在 ~/tidb-ansible-v3.0.14/conf/pt2bigdata_drainer.toml 配置文件中增加 过滤表规则
[[syncer.replicate-do-table]]
db-name ="pt_activity"
tbl-name = "activity"
2> 通过 ansible-playbook rolling_update.yml 命令,尝试使用3种方法重启drainer节点
第一种方法: ansible-playbook rolling_update.yml -t drainer -l pt2bigdata
第二种方法: ansible-playbook rolling_update.yml -t drainer -l 192.168.17.3
第三种方法: ansible-playbook rolling_update.yml -t drainer
3> 3种方法分别进行操作后,均发现drainer部署节点192.168.17.3上drainer进程号没有改变,drainer.log没有出现任何关于重启的日志,配置文件中也未出现新增的表过滤规则
以下是发起变更的日志 ,帮确认下drainer节点无法进行重启操作的原因,谢谢
[tidb@bj-mt-tidb-olap-master-prod-01 tidb-ansible-v3.0.14]$ ansible-playbook rolling_update.yml -t drainer -l pt2bigdata
PLAY [check config locally] ******************************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [check system environment] **************************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [gather all facts, and check dest] ******************************************************************************************************************************************************************************************************
TASK [check_config_dynamic : Set enable_binlog variable] *************************************************************************************************************************************************************************************
TASK [check_config_dynamic : Set deploy_dir if not set] **************************************************************************************************************************************************************************************
▽
TASK [check_config_dynamic : environment check (deploy dir)] *********************************************************************************************************************************************************************************
ok: [pt2bigdata]
TASK [check_config_dynamic : Preflight check - Does deploy dir have appropriate permission] **************************************************************************************************************************************************
TASK [check_config_dynamic : environment check (supervise)] **********************************************************************************************************************************************************************************
TASK [check_config_dynamic : config skip variables (default)] ********************************************************************************************************************************************************************************
ok: [pt2bigdata]
TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************
ok: [pt2bigdata]
TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************
PLAY [Pre-check PD configuration] ************************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [Pre-check TiKV configuration] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [Pre-check TiDB configuration] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [pd_servers[0]] *************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [pd_servers] ****************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: pd_servers_followers
[WARNING]: Could not match supplied host pattern, ignoring: pd_servers_leader
PLAY [rolling update PD cluster] *************************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [rolling update TiKV cluster] ***********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [rolling update pump cluster] ***********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [rolling update TiDB cluster] ***********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
pt2bigdata : ok=3 changed=0 unreachable=0 failed=0
Congrats! All goes well. :-)
*tidb@bj-mt-tidb-olap-master-prod-01 tidb-ansible-v3.0.14]$ ansible-playbook rolling_update.yml -t drainer -l 192.168.17.3
PLAY [check config locally] ******************************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [check system environment] **************************************************************************************************************************************************************************************************************
TASK [check_system_dynamic : Disk space check - Fail task when disk is full] *****************************************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_system_dynamic : get facts] ******************************************************************************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_system_dynamic : Preflight check - Get hostnames of all nodes in cluster] ********************************************************************************************************************************************************
TASK [check_system_dynamic : Preflight check - Does every node in cluster have different hostname] *******************************************************************************************************************************************
TASK [check_system_dynamic : Preflight check - Get NTP service status] ***********************************************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_system_dynamic : Preflight check - NTP service] **********************************************************************************************************************************************************************************
TASK [check_system_dynamic : Preflight check - Get umask] ************************************************************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_system_dynamic : Preflight check - Does the system have a standard umask] ********************************************************************************************************************************************************
TASK [check_system_dynamic : Preflight check - Get maximum number of open file descriptors limit] ********************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_system_dynamic : Preflight check - ulimit -n] ************************************************************************************************************************************************************************************
TASK [check_system_dynamic : Preflight check - Check swap] ***********************************************************************************************************************************************************************************
PLAY [gather all facts, and check dest] ******************************************************************************************************************************************************************************************************
TASK [check_config_dynamic : Set enable_binlog variable] *************************************************************************************************************************************************************************************
TASK [check_config_dynamic : Set deploy_dir if not set] **************************************************************************************************************************************************************************************
TASK [check_config_dynamic : environment check (deploy dir)] *********************************************************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_config_dynamic : Preflight check - Does deploy dir have appropriate permission] **************************************************************************************************************************************************
TASK [check_config_dynamic : environment check (supervise)] **********************************************************************************************************************************************************************************
TASK [check_config_dynamic : config skip variables (default)] ********************************************************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************
PLAY [Pre-check for rolling update] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [Pre-check PD configuration] ************************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [Pre-check TiKV configuration] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [Pre-check TiDB configuration] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [pd_servers[0]] *************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [pd_servers] ****************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: pd_servers_followers
[WARNING]: Could not match supplied host pattern, ignoring: pd_servers_leader
PLAY [rolling update PD cluster] *************************************************************************************************************************************************************************************************************
skipping: no hosts matched
TASK [check_system_dynamic : Preflight check - NTP service] **********************************************************************************************************************************************************************************
TASK [check_system_dynamic : Preflight check - Get umask] ************************************************************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_system_dynamic : Preflight check - Does the system have a standard umask] ********************************************************************************************************************************************************
TASK [check_system_dynamic : Preflight check - Get maximum number of open file descriptors limit] ********************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_system_dynamic : Preflight check - ulimit -n] ************************************************************************************************************************************************************************************
TASK [check_system_dynamic : Preflight check - Check swap] ***********************************************************************************************************************************************************************************
PLAY [gather all facts, and check dest] ******************************************************************************************************************************************************************************************************
TASK [check_config_dynamic : Set enable_binlog variable] *************************************************************************************************************************************************************************************
TASK [check_config_dynamic : Set deploy_dir if not set] **************************************************************************************************************************************************************************************
TASK [check_config_dynamic : environment check (deploy dir)] *********************************************************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_config_dynamic : Preflight check - Does deploy dir have appropriate permission] **************************************************************************************************************************************************
TASK [check_config_dynamic : environment check (supervise)] **********************************************************************************************************************************************************************************
TASK [check_config_dynamic : config skip variables (default)] ********************************************************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************
ok: [192.168.17.3]
TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************
PLAY [Pre-check for rolling update] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [Pre-check PD configuration] ************************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [Pre-check TiKV configuration] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [Pre-check TiDB configuration] **********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [pd_servers[0]] *************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [pd_servers] ****************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: pd_servers_followers
[WARNING]: Could not match supplied host pattern, ignoring: pd_servers_leader
PLAY [rolling update PD cluster] *************************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [rolling update TiKV cluster] ***********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [rolling update pump cluster] ***********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY [rolling update TiDB cluster] ***********************************************************************************************************************************************************************************************************
skipping: no hosts matched
PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
192.168.17.3 : ok=8 changed=0 unreachable=0 failed=0
Congrats! All goes well. :-)
[tidb@bj-mt-tidb-olap-master-prod-01 tidb-ansible-v3.0.14]$ ansible-playbook rolling_update.yml -t drainer
PLAY [check config locally] ******************************************************************************************************************************************************************************************************************
TASK [check_config_static : Ensure monitoring_servers exists] ********************************************************************************************************************************************************************************
TASK [check_config_static : Ensure monitored_servers exists] *********************************************************************************************************************************************************************************
TASK [check_config_static : Ensure TiDB host exists] *****************************************************************************************************************************************************************************************
TASK [check_config_static : Ensure PD host exists] *******************************************************************************************************************************************************************************************
TASK [check_config_static : Ensure TiKV host exists] *****************************************************************************************************************************************************************************************
TASK [check_config_static : Check ansible_user variable] *************************************************************************************************************************************************************************************
TASK [check_config_static : Ensure timezone variable is set] *********************************************************************************************************************************************************************************
TASK [check_config_static : Close old SSH control master processes] **************************************************************************************************************************************************************************
ok: [localhost]
TASK [check_config_static : Check ansible version] *******************************************************************************************************************************************************************************************
TASK [check_config_static : Check if jmespath is installed] **********************************************************************************************************************************************************************************
changed: [localhost]
TASK [check_config_static : Check if jinja2 is installed] ************************************************************************************************************************************************************************************
changed: [localhost]
TASK [check_config_static : Preflight check - Fail when jmespath or jinja2 doesn't exist] ****************************************************************************************************************************************************
TASK [check_config_static : Get jmespath info] ***********************************************************************************************************************************************************************************************
changed: [localhost]
TASK [check_config_static : Get jmespath version] ********************************************************************************************************************************************************************************************
ok: [localhost]
TASK [check_config_static : Get jinja2 info] *************************************************************************************************************************************************************************************************
changed: [localhost]
TASK [check_config_static : Get jinja2 version] **********************************************************************************************************************************************************************************************
ok: [localhost]
TASK [check_config_static : Preflight check - Fail when the versions of jmespath and jinja2 doesn't meet the requirements] *******************************************************************************************************************
PLAY [check system environment] **************************************************************************************************************************************************************************************************************
TASK [check_system_dynamic : Disk space check - Fail task when disk is full] *****************************************************************************************************************************************************************
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.155]
ok: [192.168.17.154]
ok: [192.168.17.167]
ok: [192.168.17.158]
ok: [192.168.16.72]
ok: [192.168.17.169]
ok: [192.168.17.168]
ok: [192.168.17.251]
ok: [192.168.17.5]
ok: [192.168.17.252]
ok: [192.168.17.3]
ok: [192.168.17.4]
TASK [check_system_dynamic : get facts] ******************************************************************************************************************************************************************************************************
ok: [192.168.17.155]
ok: [192.168.17.167]
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.154]
ok: [192.168.16.72]
ok: [192.168.17.158]
ok: [192.168.17.168]
ok: [192.168.17.169]
ok: [192.168.17.251]
ok: [192.168.17.5]
ok: [192.168.17.3]
ok: [192.168.17.252]
ok: [192.168.17.4]
TASK [check_system_dynamic : Preflight check - Get hostnames of all nodes in cluster] ********************************************************************************************************************************************************
ok: [192.168.17.152]
TASK [check_system_dynamic : Preflight check - Does every node in cluster have different hostname] *******************************************************************************************************************************************
TASK [check_system_dynamic : Preflight check - Get NTP service status] ***********************************************************************************************************************************************************************
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.155]
ok: [192.168.17.167]
ok: [192.168.17.154]
ok: [192.168.17.158]
ok: [192.168.16.72]
ok: [192.168.17.169]
ok: [192.168.17.168]
ok: [192.168.17.251]
ok: [192.168.17.5]
ok: [192.168.17.3]
ok: [192.168.17.252]
ok: [192.168.17.4]
TASK [check_system_dynamic : Preflight check - NTP service] **********************************************************************************************************************************************************************************
TASK [check_system_dynamic : Preflight check - Get umask] ************************************************************************************************************************************************************************************
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.155]
ok: [192.168.17.167]
ok: [192.168.17.154]
ok: [192.168.17.158]
ok: [192.168.16.72]
ok: [192.168.17.168]
ok: [192.168.17.169]
ok: [192.168.17.251]
ok: [192.168.17.5]
ok: [192.168.17.3]
ok: [192.168.17.4]
ok: [192.168.17.252]
TASK [check_system_dynamic : Preflight check - Does the system have a standard umask] ********************************************************************************************************************************************************
TASK [check_system_dynamic : Preflight check - Get maximum number of open file descriptors limit] ********************************************************************************************************************************************
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.155]
ok: [192.168.17.167]
ok: [192.168.17.154]
ok: [192.168.17.158]
ok: [192.168.16.72]
ok: [192.168.17.169]
ok: [192.168.17.168]
ok: [192.168.17.251]
ok: [192.168.17.5]
ok: [192.168.17.252]
ok: [192.168.17.3]
ok: [192.168.17.4]
TASK [check_system_dynamic : Preflight check - ulimit -n] ************************************************************************************************************************************************************************************
TASK [check_system_dynamic : Preflight check - Check swap] ***********************************************************************************************************************************************************************************
PLAY [gather all facts, and check dest] ******************************************************************************************************************************************************************************************************
TASK [check_config_dynamic : Set enable_binlog variable] *************************************************************************************************************************************************************************************
TASK [check_config_dynamic : Set deploy_dir if not set] **************************************************************************************************************************************************************************************
TASK [check_config_dynamic : environment check (deploy dir)] *********************************************************************************************************************************************************************************
ok: [192.168.17.158]
ok: [192.168.17.155]
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.154]
ok: [192.168.17.167]
ok: [192.168.16.72]
ok: [192.168.17.168]
ok: [192.168.17.169]
ok: [192.168.17.251]
ok: [192.168.17.5]
ok: [192.168.17.3]
ok: [192.168.17.4]
ok: [192.168.17.252]
ok: [pump1]
ok: [pump2]
ok: [pump3]
ok: [pt2bigdata]
ok: [TiKV1-2]
ok: [TiKV2-2]
ok: [TiKV3-2]
ok: [TiKV4-2]
ok: [TiKV5-2]
TASK [check_config_dynamic : Preflight check - Does deploy dir have appropriate permission] **************************************************************************************************************************************************
TASK [check_config_dynamic : environment check (supervise)] **********************************************************************************************************************************************************************************
TASK [check_config_dynamic : config skip variables (default)] ********************************************************************************************************************************************************************************
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.154]
ok: [192.168.17.158]
ok: [192.168.17.155]
ok: [192.168.17.167]
ok: [192.168.17.168]
ok: [192.168.17.169]
ok: [192.168.16.72]
ok: [192.168.17.251]
ok: [192.168.17.252]
ok: [192.168.17.5]
ok: [192.168.17.3]
ok: [192.168.17.4]
ok: [pump2]
ok: [pump1]
ok: [pump3]
ok: [TiKV1-2]
ok: [pt2bigdata]
ok: [TiKV2-2]
ok: [TiKV3-2]
ok: [TiKV4-2]
ok: [TiKV5-2]
TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************
ok: [192.168.17.152]
ok: [192.168.17.153]
ok: [192.168.17.154]
ok: [192.168.17.158]
ok: [192.168.17.155]
ok: [192.168.17.167]
ok: [192.168.17.168]
ok: [192.168.17.169]
ok: [192.168.16.72]
ok: [192.168.17.251]
ok: [192.168.17.252]
ok: [192.168.17.5]
ok: [192.168.17.3]
ok: [pump1]
ok: [192.168.17.4]
ok: [pump2]
ok: [pump3]
ok: [pt2bigdata]
ok: [TiKV1-2]
ok: [TiKV2-2]
ok: [TiKV3-2]
ok: [TiKV4-2]
ok: [TiKV5-2]
TASK [check_config_dynamic : config skip variables] ******************************************************************************************************************************************************************************************
PLAY [Pre-check for rolling update] **********************************************************************************************************************************************************************************************************
TASK [shell] *********************************************************************************************************************************************************************************************************************************
changed: [192.168.17.152]
changed: [192.168.17.153]
changed: [192.168.17.154]
TASK [Check whether can perform rolling update] **********************************************************************************************************************************************************************************************
PLAY [Pre-check PD configuration] ************************************************************************************************************************************************************************************************************
PLAY [Pre-check TiKV configuration] **********************************************************************************************************************************************************************************************************
PLAY [Pre-check TiDB configuration] **********************************************************************************************************************************************************************************************************
PLAY [pd_servers[0]] *************************************************************************************************************************************************************************************************************************
PLAY [pd_servers] ****************************************************************************************************************************************************************************************************************************
.......。。。。。。。。。。
PLAY [rolling update TiKV cluster] ***********************************************************************************************************************************************************************************************************
PLAY [rolling update TiKV cluster] ***********************************************************************************************************************************************************************************************************
▽
PLAY [rolling update TiKV cluster] ***********************************************************************************************************************************************************************************************************
PLAY [rolling update pump cluster] ***********************************************************************************************************************************************************************************************************
▽
PLAY [rolling update pump cluster] ***********************************************************************************************************************************************************************************************************
PLAY [rolling update pump cluster] ***********************************************************************************************************************************************************************************************************
PLAY [rolling update TiDB cluster] ***********************************************************************************************************************************************************************************************************
PLAY [rolling update TiDB cluster] ***********************************************************************************************************************************************************************************************************
PLAY [rolling update TiDB cluster] ***********************************************************************************************************************************************************************************************************
PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
192.168.16.72 : ok=8 changed=0 unreachable=0 failed=0
192.168.17.152 : ok=10 changed=1 unreachable=0 failed=0
192.168.17.153 : ok=9 changed=1 unreachable=0 failed=0
192.168.17.154 : ok=9 changed=1 unreachable=0 failed=0
192.168.17.155 : ok=8 changed=0 unreachable=0 failed=0
192.168.17.158 : ok=8 changed=0 unreachable=0 failed=0
192.168.17.167 : ok=8 changed=0 unreachable=0 failed=0
192.168.17.168 : ok=8 changed=0 unreachable=0 failed=0
192.168.17.169 : ok=8 changed=0 unreachable=0 failed=0
192.168.17.251 : ok=8 changed=0 unreachable=0 failed=0
192.168.17.252 : ok=8 changed=0 unreachable=0 failed=0
192.168.17.3 : ok=8 changed=0 unreachable=0 failed=0
192.168.17.4 : ok=8 changed=0 unreachable=0 failed=0
192.168.17.5 : ok=8 changed=0 unreachable=0 failed=0
TiKV1-2 : ok=3 changed=0 unreachable=0 failed=0
TiKV2-2 : ok=3 changed=0 unreachable=0 failed=0
TiKV3-2 : ok=3 changed=0 unreachable=0 failed=0
TiKV4-2 : ok=3 changed=0 unreachable=0 failed=0
TiKV5-2 : ok=3 changed=0 unreachable=0 failed=0
localhost : ok=7 changed=4 unreachable=0 failed=0
pt2bigdata : ok=3 changed=0 unreachable=0 failed=0
pump1 : ok=3 changed=0 unreachable=0 failed=0
pump2 : ok=3 changed=0 unreachable=0 failed=0
pump3 : ok=3 changed=0 unreachable=0 failed=0
Congrats! All goes well. :-)