Ansible 滚动更新至 4.0.0-beta.1 ,Check pd cluster status 失败

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。

  • 【TiDB 版本】:v4.0.0-alpha-161-g2488cb97
  • 【问题描述】:Ansible 滚动更新至 4.0.0-beta.1 ,Check pd cluster status 失败

检测的URL报404:http://192.168.13.26:2379/pd/health

使用curl测试http://192.168.13.26:2379/health是正常的,请问是不是版本不兼容?

curl http://192.168.13.26:2379/health
{"health":"true"}

image

若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

你好.请问是从什么版本升级到 4.0.0-beta.1 呢?

你好,ansible部署的时候tidb_version使用的是latest,pd-server -V 显示的版本是v4.0.0-alpha-161-g2488cb97

麻烦检查下 {tidb_ansible_path}/inventory.ini 文件的 enable_tls 是否设置成 true 了。

你好,inventory.ini 文件只添加了对应服务器的IP,其他都用默认值. enable_tls=False

## Global variables
[all:vars]
deploy_dir = /data/deploy

## Connection
# ssh via normal user
ansible_user = tidb

cluster_name = test-cluster

#tidb_version = latest
#tidb_version = v3.1.0-beta.1
tidb_version = v4.0.0-beta.1

# process supervision, [systemd, supervise]
process_supervision = systemd

timezone = Asia/Shanghai

enable_firewalld = False
# check NTP service
enable_ntpd = True
set_hostname = False

## binlog trigger
enable_binlog = False

# kafka cluster address for monitoring, example:
# kafka_addrs = "192.168.0.11:9092,192.168.0.12:9092,192.168.0.13:9092"
kafka_addrs = ""

# zookeeper address of kafka cluster for monitoring, example:
# zookeeper_addrs = "192.168.0.11:2181,192.168.0.12:2181,192.168.0.13:2181"
zookeeper_addrs = ""

# enable TLS authentication in the TiDB cluster
enable_tls = False

# KV mode
deploy_without_tidb = False

# wait for region replication complete before start tidb-server.
wait_replication = True

# Optional: Set if you already have a alertmanager server.
# Format: alertmanager_host:alertmanager_port
alertmanager_target = ""

grafana_admin_user = "xxx"
grafana_admin_password = "xxx"


### Collect diagnosis
collect_log_recent_hours = 2

enable_bandwidth_limit = True
# default: 10Mb/s, unit: Kbit/s
collect_bandwidth_limit = 10000

麻烦到问题的机器看下 PD server 的日志是否正常。正常情况下应该 curl http://{{ ansible_host }}:{{ pd_client_port }}/pd/health 也是有输出的。

今天只有几条WARN级别的错误

[2020/02/29 00:13:00.139 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414949226318397440]
[2020/02/29 00:23:00.082 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414949383604797440]
[2020/02/29 00:33:00.084 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414949540891197440]
[2020/02/29 00:39:30.093 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3677347] [compact-period=1h0m0s]
[2020/02/29 00:39:30.100 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3677347]
[2020/02/29 00:39:30.100 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3677347] [compact-period=1h0m0s] [took=1h0m0.007405175s]
[2020/02/29 00:39:30.101 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3677347] [took=1.220431ms]
[2020/02/29 00:43:00.108 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414949698177597440]
[2020/02/29 00:53:00.068 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414949855463997440]
[2020/02/29 01:03:00.106 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414950012750397440]
[2020/02/29 01:13:00.140 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414950170036797440]
[2020/02/29 01:23:00.096 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414950327323197440]
[2020/02/29 01:33:00.125 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414950484609597440]
[2020/02/29 01:39:30.101 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3678799] [compact-period=1h0m0s]
[2020/02/29 01:39:30.104 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3678799]
[2020/02/29 01:39:30.104 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3678799] [compact-period=1h0m0s] [took=1h0m0.004629575s]
[2020/02/29 01:39:30.106 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3678799] [took=1.439111ms]
[2020/02/29 01:43:00.094 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414950641895997440]
[2020/02/29 01:53:00.096 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414950799182397440]
[2020/02/29 02:03:00.081 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414950956468797440]
[2020/02/29 02:13:00.071 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414951113755197440]
[2020/02/29 02:23:00.427 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414951271054704640]
[2020/02/29 02:33:00.095 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414951428327997440]
[2020/02/29 02:39:30.105 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3680251] [compact-period=1h0m0s]
[2020/02/29 02:39:30.109 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3680251]
[2020/02/29 02:39:30.109 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3680251] [compact-period=1h0m0s] [took=1h0m0.004527473s]
[2020/02/29 02:39:30.110 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3680251] [took=1.458284ms]
[2020/02/29 02:43:00.097 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414951585614397440]
[2020/02/29 02:53:00.074 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414951742900797440]
[2020/02/29 03:03:00.080 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414951900187197440]
[2020/02/29 03:13:00.101 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414952057473597440]
[2020/02/29 03:23:00.068 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414952214759997440]
[2020/02/29 03:33:00.064 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414952372046397440]
[2020/02/29 03:39:30.110 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3681703] [compact-period=1h0m0s]
[2020/02/29 03:39:30.114 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3681703]
[2020/02/29 03:39:30.114 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3681703] [compact-period=1h0m0s] [took=1h0m0.005120381s]
[2020/02/29 03:39:30.115 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3681703] [took=1.391564ms]
[2020/02/29 03:43:00.098 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414952529332797440]
[2020/02/29 03:53:00.068 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414952686619197440]
[2020/02/29 04:03:00.083 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414952843905597440]
[2020/02/29 04:13:00.065 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414953001191997440]
[2020/02/29 04:23:00.079 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414953158478397440]
[2020/02/29 04:33:00.106 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414953315764797440]
[2020/02/29 04:39:30.115 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3683155] [compact-period=1h0m0s]
[2020/02/29 04:39:30.118 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3683155]
[2020/02/29 04:39:30.119 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3683155] [compact-period=1h0m0s] [took=1h0m0.00456289s]
[2020/02/29 04:39:30.120 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3683155] [took=1.372119ms]
[2020/02/29 04:43:00.078 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414953473051197440]
[2020/02/29 04:53:00.057 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414953630337597440]
[2020/02/29 05:03:00.071 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414953787623997440]
[2020/02/29 05:13:00.080 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414953944910397440]
[2020/02/29 05:23:00.150 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414954102196797440]
[2020/02/29 05:33:00.079 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414954259483197440]
[2020/02/29 05:39:30.120 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3684607] [compact-period=1h0m0s]
[2020/02/29 05:39:30.141 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3684607]
[2020/02/29 05:39:30.141 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3684607] [compact-period=1h0m0s] [took=1h0m0.022240937s]
[2020/02/29 05:39:30.142 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3684607] [took=1.213407ms]
[2020/02/29 05:43:00.175 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414954416769597440]
[2020/02/29 05:53:00.069 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414954574055997440]
[2020/02/29 06:00:01.656 +08:00] [INFO] [cluster.go:369] ["region Version changed"] [region-id=2] [detail="StartKey Changed:{7480000000000027FF4100000000000000F8} -> {7480000000000027FF4400000000000000F8}, EndKey:{}"] [old-version=3399] [new-version=3400]
[2020/02/29 06:00:01.656 +08:00] [INFO] [cluster_worker.go:208] ["region batch split, generate new regions"] [region-id=2] [origin="id:484461 start_key:\"7480000000000027FF4100000000000000F8\" end_key:\"7480000000000027FF4400000000000000F8\" region_epoch:<conf_ver:182 version:3400 > peers:<id:484462 store_id:5 > peers:<id:484463 store_id:1 > peers:<id:484464 store_id:4 >"] [total=1]
[2020/02/29 06:00:05.502 +08:00] [INFO] [cluster.go:369] ["region Version changed"] [region-id=2] [detail="StartKey Changed:{7480000000000027FF4400000000000000F8} -> {7480000000000027FF4700000000000000F8}, EndKey:{}"] [old-version=3400] [new-version=3401]
[2020/02/29 06:00:05.502 +08:00] [INFO] [cluster_worker.go:208] ["region batch split, generate new regions"] [region-id=2] [origin="id:484465 start_key:\"7480000000000027FF4400000000000000F8\" end_key:\"7480000000000027FF4700000000000000F8\" region_epoch:<conf_ver:182 version:3401 > peers:<id:484466 store_id:5 > peers:<id:484467 store_id:1 > peers:<id:484468 store_id:4 >"] [total=1]
[2020/02/29 06:00:09.764 +08:00] [INFO] [operator_controller.go:330] ["add operator"] [region-id=19129] [operator="\"balance-leader {transfer leader: store 1 to 5} (kind:leader,balance, region:19129(785,122), createAt:2020-02-29 06:00:09.763994741 +0800 CST m=+930041.910665252, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, steps:[transfer leader from store 1 to store 5])\""]
[2020/02/29 06:00:09.764 +08:00] [INFO] [operator_controller.go:428] ["send schedule command"] [region-id=19129] [step="transfer leader from store 1 to store 5"] [source=create]
[2020/02/29 06:00:09.772 +08:00] [INFO] [cluster.go:390] ["leader changed"] [region-id=19129] [from=1] [to=5]
[2020/02/29 06:00:09.772 +08:00] [INFO] [operator_controller.go:140] ["operator finish"] [region-id=19129] [takes=8.25472ms] [operator="\"balance-leader {transfer leader: store 1 to 5} (kind:leader,balance, region:19129(785,122), createAt:2020-02-29 06:00:09.763994741 +0800 CST m=+930041.910665252, startAt:2020-02-29 06:00:09.76409583 +0800 CST m=+930041.910766366, currentStep:1, steps:[transfer leader from store 1 to store 5]) finished\""]
[2020/02/29 06:03:00.102 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414954731342397440]
[2020/02/29 06:13:00.207 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414954888628797440]
[2020/02/29 06:23:00.102 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414955045915197440]
[2020/02/29 06:33:00.073 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414955203201597440]
[2020/02/29 06:39:30.142 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3686059] [compact-period=1h0m0s]
[2020/02/29 06:39:30.145 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3686059]
[2020/02/29 06:39:30.145 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3686059] [compact-period=1h0m0s] [took=1h0m0.0043959s]
[2020/02/29 06:39:30.147 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3686059] [took=1.240856ms]
[2020/02/29 06:43:00.092 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414955360487997440]
[2020/02/29 06:53:00.104 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414955517774397440]
[2020/02/29 07:03:00.095 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414955675060797440]
[2020/02/29 07:13:00.092 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414955832347197440]
[2020/02/29 07:23:00.089 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414955989633597440]
[2020/02/29 07:33:00.069 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414956146919997440]
[2020/02/29 07:39:30.146 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3687534] [compact-period=1h0m0s]
[2020/02/29 07:39:30.150 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3687534]
[2020/02/29 07:39:30.150 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3687534] [compact-period=1h0m0s] [took=1h0m0.004409611s]
[2020/02/29 07:39:30.151 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3687534] [took=1.229976ms]
[2020/02/29 07:43:00.090 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414956304206397440]
[2020/02/29 07:53:00.087 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414956461492797440]
[2020/02/29 08:03:00.080 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414956618779197440]
[2020/02/29 08:13:00.105 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414956776065597440]
[2020/02/29 08:23:00.094 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414956933351997440]
[2020/02/29 08:33:00.080 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414957090638397440]
[2020/02/29 08:39:30.151 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3688986] [compact-period=1h0m0s]
[2020/02/29 08:39:30.154 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3688986]
[2020/02/29 08:39:30.154 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3688986] [compact-period=1h0m0s] [took=1h0m0.004531956s]
[2020/02/29 08:39:30.156 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3688986] [took=1.656744ms]
[2020/02/29 08:43:00.081 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414957247924797440]
[2020/02/29 08:53:00.089 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414957405211197440]
[2020/02/29 08:54:13.366 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=155.513094ms] [prev-physical=2020/02/29 08:54:13.210 +08:00] [now=2020/02/29 08:54:13.366 +08:00]
[2020/02/29 08:57:28.547 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=336.738525ms] [prev-physical=2020/02/29 08:57:28.210 +08:00] [now=2020/02/29 08:57:28.547 +08:00]
[2020/02/29 09:03:00.079 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414957562497597440]
[2020/02/29 09:13:00.088 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414957719783997440]
[2020/02/29 09:19:01.420 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=159.806412ms] [prev-physical=2020/02/29 09:19:01.260 +08:00] [now=2020/02/29 09:19:01.420 +08:00]
[2020/02/29 09:23:00.099 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414957877070397440]
[2020/02/29 09:27:55.663 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=152.936674ms] [prev-physical=2020/02/29 09:27:55.510 +08:00] [now=2020/02/29 09:27:55.663 +08:00]
[2020/02/29 09:33:00.182 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414958034356797440]
[2020/02/29 09:39:30.156 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3690438] [compact-period=1h0m0s]
[2020/02/29 09:39:30.166 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3690438]
[2020/02/29 09:39:30.166 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3690438] [compact-period=1h0m0s] [took=1h0m0.011939859s]
[2020/02/29 09:39:30.168 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3690438] [took=1.362012ms]
[2020/02/29 09:43:00.247 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414958191643197440]
[2020/02/29 09:53:00.272 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414958348929597440]
[2020/02/29 09:55:04.908 +08:00] [WARN] [util.go:144] ["apply request took too long"] [took=199.375734ms] [expected-duration=100ms] [prefix="read-only range "] [request="key:\"/tidb/store/gcworker/saved_safe_point\" "] [response="range_response_count:1 size:81"] []
[2020/02/29 09:56:43.884 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=174.038203ms] [prev-physical=2020/02/29 09:56:43.710 +08:00] [now=2020/02/29 09:56:43.884 +08:00]
[2020/02/29 10:01:44.052 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=191.627917ms] [prev-physical=2020/02/29 10:01:43.860 +08:00] [now=2020/02/29 10:01:44.052 +08:00]
[2020/02/29 10:03:00.115 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414958506215997440]
[2020/02/29 10:13:00.085 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414958663502397440]
[2020/02/29 10:23:00.094 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414958820788797440]
[2020/02/29 10:33:00.098 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414958978075197440]
[2020/02/29 10:39:30.167 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3691890] [compact-period=1h0m0s]
[2020/02/29 10:39:30.172 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3691890]
[2020/02/29 10:39:30.172 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3691890] [compact-period=1h0m0s] [took=1h0m0.005899269s]
[2020/02/29 10:39:30.173 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3691890] [took=1.222979ms]
[2020/02/29 10:40:41.211 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=250.481772ms] [prev-physical=2020/02/29 10:40:40.960 +08:00] [now=2020/02/29 10:40:41.211 +08:00]
[2020/02/29 10:43:00.095 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414959135361597440]
[2020/02/29 10:45:56.280 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=269.832643ms] [prev-physical=2020/02/29 10:45:56.010 +08:00] [now=2020/02/29 10:45:56.280 +08:00]
[2020/02/29 10:53:00.108 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414959292647997440]
[2020/02/29 10:55:56.398 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=337.827939ms] [prev-physical=2020/02/29 10:55:56.060 +08:00] [now=2020/02/29 10:55:56.398 +08:00]
[2020/02/29 11:03:00.111 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414959449934397440]
[2020/02/29 11:13:00.102 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414959607220797440]
[2020/02/29 11:23:00.100 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414959764507197440]
[2020/02/29 11:28:23.417 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=256.489765ms] [prev-physical=2020/02/29 11:28:23.160 +08:00] [now=2020/02/29 11:28:23.417 +08:00]
[2020/02/29 11:33:00.114 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414959921806704640]
[2020/02/29 11:39:30.173 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3693342] [compact-period=1h0m0s]
[2020/02/29 11:39:30.177 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3693342]
[2020/02/29 11:39:30.177 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3693342] [compact-period=1h0m0s] [took=1h0m0.005112002s]
[2020/02/29 11:39:30.179 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3693342] [took=1.329144ms]
[2020/02/29 11:43:00.093 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414960079079997440]
[2020/02/29 11:53:00.092 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414960236366397440]
[2020/02/29 12:03:00.077 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414960393652797440]
[2020/02/29 12:08:29.565 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=255.013201ms] [prev-physical=2020/02/29 12:08:29.310 +08:00] [now=2020/02/29 12:08:29.565 +08:00]
[2020/02/29 12:13:00.092 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414960550939197440]
[2020/02/29 12:23:00.090 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414960708225597440]
[2020/02/29 12:33:00.080 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414960865511997440]
[2020/02/29 12:39:05.635 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=224.658986ms] [prev-physical=2020/02/29 12:39:05.410 +08:00] [now=2020/02/29 12:39:05.635 +08:00]
[2020/02/29 12:39:30.179 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3694794] [compact-period=1h0m0s]
[2020/02/29 12:39:30.183 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3694794]
[2020/02/29 12:39:30.183 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3694794] [compact-period=1h0m0s] [took=1h0m0.006008201s]
[2020/02/29 12:39:30.185 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3694794] [took=1.503485ms]
[2020/02/29 12:43:00.073 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414961022798397440]
[2020/02/29 12:53:00.101 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414961180084797440]
[2020/02/29 12:53:23.614 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=153.435545ms] [prev-physical=2020/02/29 12:53:23.460 +08:00] [now=2020/02/29 12:53:23.614 +08:00]
[2020/02/29 13:03:00.106 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414961337371197440]
[2020/02/29 13:07:26.823 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=212.320703ms] [prev-physical=2020/02/29 13:07:26.610 +08:00] [now=2020/02/29 13:07:26.823 +08:00]
[2020/02/29 13:13:00.095 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414961494657597440]
[2020/02/29 13:23:00.114 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414961651943997440]
[2020/02/29 13:28:53.886 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=176.033716ms] [prev-physical=2020/02/29 13:28:53.710 +08:00] [now=2020/02/29 13:28:53.886 +08:00]
[2020/02/29 13:33:00.100 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414961809230397440]
[2020/02/29 13:39:30.185 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3696246] [compact-period=1h0m0s]
[2020/02/29 13:39:30.188 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3696246]
[2020/02/29 13:39:30.188 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3696246] [compact-period=1h0m0s] [took=1h0m0.004608928s]
[2020/02/29 13:39:30.189 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3696246] [took=1.282271ms]
[2020/02/29 13:41:59.911 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=150.698789ms] [prev-physical=2020/02/29 13:41:59.760 +08:00] [now=2020/02/29 13:41:59.911 +08:00]
[2020/02/29 13:43:00.088 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414961966516797440]
[2020/02/29 13:53:00.094 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414962123803197440]
[2020/02/29 14:03:00.111 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414962281089597440]
[2020/02/29 14:13:00.093 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414962438375997440]
[2020/02/29 14:19:03.051 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=190.555313ms] [prev-physical=2020/02/29 14:19:02.860 +08:00] [now=2020/02/29 14:19:03.051 +08:00]
[2020/02/29 14:21:00.104 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=243.156159ms] [prev-physical=2020/02/29 14:20:59.860 +08:00] [now=2020/02/29 14:21:00.104 +08:00]
[2020/02/29 14:23:00.076 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414962595662397440]
[2020/02/29 14:28:48.150 +08:00] [WARN] [util.go:144] ["apply request took too long"] [took=139.323278ms] [expected-duration=100ms] [prefix="read-only range "] [request="key:\"/tidb/store/gcworker/saved_safe_point\" "] [response="range_response_count:1 size:81"] []
[2020/02/29 14:31:00.012 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=151.762529ms] [prev-physical=2020/02/29 14:30:59.860 +08:00] [now=2020/02/29 14:31:00.012 +08:00]
[2020/02/29 14:33:00.082 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414962752948797440]
[2020/02/29 14:39:30.189 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3697698] [compact-period=1h0m0s]
[2020/02/29 14:39:30.193 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3697698]
[2020/02/29 14:39:30.193 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3697698] [compact-period=1h0m0s] [took=1h0m0.004959928s]
[2020/02/29 14:39:30.195 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3697698] [took=1.975383ms]
[2020/02/29 14:43:00.087 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414962910235197440]
[2020/02/29 14:53:00.091 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414963067521597440]
[2020/02/29 15:03:00.095 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414963224807997440]
[2020/02/29 15:13:00.080 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414963382094397440]
[2020/02/29 15:23:00.076 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414963539380797440]
[2020/02/29 15:33:00.095 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414963696667197440]
[2020/02/29 15:39:30.194 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3699150] [compact-period=1h0m0s]
[2020/02/29 15:39:30.197 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3699150]
[2020/02/29 15:39:30.197 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3699150] [compact-period=1h0m0s] [took=1h0m0.004392704s]
[2020/02/29 15:39:30.199 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3699150] [took=1.454106ms]
[2020/02/29 15:43:00.117 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414963853953597440]
[2020/02/29 15:53:00.116 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414964011239997440]
[2020/02/29 15:53:57.280 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=219.82589ms] [prev-physical=2020/02/29 15:53:57.060 +08:00] [now=2020/02/29 15:53:57.280 +08:00]
[2020/02/29 16:03:00.098 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414964168526397440]
[2020/02/29 16:13:00.130 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414964325812797440]
[2020/02/29 16:23:00.119 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414964483099197440]
[2020/02/29 16:24:03.361 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=200.43748ms] [prev-physical=2020/02/29 16:24:03.160 +08:00] [now=2020/02/29 16:24:03.361 +08:00]
[2020/02/29 16:33:00.095 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414964640385597440]
[2020/02/29 16:37:03.569 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=408.968773ms] [prev-physical=2020/02/29 16:37:03.160 +08:00] [now=2020/02/29 16:37:03.569 +08:00]
[2020/02/29 16:39:30.216 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3700602] [compact-period=1h0m0s]
[2020/02/29 16:39:30.219 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3700602]
[2020/02/29 16:39:30.219 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3700602] [compact-period=1h0m0s] [took=1h0m0.021902406s]
[2020/02/29 16:39:30.221 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3700602] [took=1.474848ms]
[2020/02/29 16:43:00.113 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414964797671997440]
[2020/02/29 16:53:00.145 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414964954958397440]
[2020/02/29 17:03:00.096 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414965112244797440]
[2020/02/29 17:13:00.239 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414965269570519040]
[2020/02/29 17:19:57.432 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=171.442481ms] [prev-physical=2020/02/29 17:19:57.260 +08:00] [now=2020/02/29 17:19:57.432 +08:00]
[2020/02/29 17:23:00.102 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414965426817597440]
[2020/02/29 17:23:39.554 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=293.840552ms] [prev-physical=2020/02/29 17:23:39.260 +08:00] [now=2020/02/29 17:23:39.554 +08:00]
[2020/02/29 17:33:00.083 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414965584103997440]
[2020/02/29 17:39:30.220 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3702054] [compact-period=1h0m0s]
[2020/02/29 17:39:30.224 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3702054]
[2020/02/29 17:39:30.224 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3702054] [compact-period=1h0m0s] [took=1h0m0.00451303s]
[2020/02/29 17:39:30.225 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3702054] [took=1.487995ms]
[2020/02/29 17:43:00.083 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414965741390397440]
[2020/02/29 17:53:00.089 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414965898676797440]
[2020/02/29 18:00:01.870 +08:00] [INFO] [cluster.go:369] ["region Version changed"] [region-id=2] [detail="StartKey Changed:{7480000000000027FF4700000000000000F8} -> {7480000000000027FF4A00000000000000F8}, EndKey:{}"] [old-version=3401] [new-version=3402]
[2020/02/29 18:00:01.870 +08:00] [INFO] [cluster_worker.go:208] ["region batch split, generate new regions"] [region-id=2] [origin="id:484469 start_key:\"7480000000000027FF4700000000000000F8\" end_key:\"7480000000000027FF4A00000000000000F8\" region_epoch:<conf_ver:182 version:3402 > peers:<id:484470 store_id:5 > peers:<id:484471 store_id:1 > peers:<id:484472 store_id:4 >"] [total=1]
[2020/02/29 18:00:03.798 +08:00] [INFO] [cluster.go:369] ["region Version changed"] [region-id=2] [detail="StartKey Changed:{7480000000000027FF4A00000000000000F8} -> {7480000000000027FF4D00000000000000F8}, EndKey:{}"] [old-version=3402] [new-version=3403]
[2020/02/29 18:00:03.798 +08:00] [INFO] [cluster_worker.go:208] ["region batch split, generate new regions"] [region-id=2] [origin="id:484473 start_key:\"7480000000000027FF4A00000000000000F8\" end_key:\"7480000000000027FF4D00000000000000F8\" region_epoch:<conf_ver:182 version:3403 > peers:<id:484474 store_id:5 > peers:<id:484475 store_id:1 > peers:<id:484476 store_id:4 >"] [total=1]
[2020/02/29 18:00:03.968 +08:00] [INFO] [operator_controller.go:330] ["add operator"] [region-id=36205] [operator="\"balance-leader {transfer leader: store 1 to 5} (kind:leader,balance, region:36205(1306,116), createAt:2020-02-29 18:00:03.968300384 +0800 CST m=+973236.114970896, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, steps:[transfer leader from store 1 to store 5])\""]
[2020/02/29 18:00:03.968 +08:00] [INFO] [operator_controller.go:428] ["send schedule command"] [region-id=36205] [step="transfer leader from store 1 to store 5"] [source=create]
[2020/02/29 18:00:03.975 +08:00] [INFO] [cluster.go:390] ["leader changed"] [region-id=36205] [from=1] [to=5]
[2020/02/29 18:00:03.975 +08:00] [INFO] [operator_controller.go:140] ["operator finish"] [region-id=36205] [takes=7.279654ms] [operator="\"balance-leader {transfer leader: store 1 to 5} (kind:leader,balance, region:36205(1306,116), createAt:2020-02-29 18:00:03.968300384 +0800 CST m=+973236.114970896, startAt:2020-02-29 18:00:03.968393671 +0800 CST m=+973236.115064201, currentStep:1, steps:[transfer leader from store 1 to store 5]) finished\""]
[2020/02/29 18:03:00.100 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414966055963197440]
[2020/02/29 18:13:00.254 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414966213249597440]
[2020/02/29 18:23:00.467 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414966370535997440]
[2020/02/29 18:33:00.104 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414966527822397440]
[2020/02/29 18:39:30.225 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3703505] [compact-period=1h0m0s]
[2020/02/29 18:39:30.229 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3703505]
[2020/02/29 18:39:30.229 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3703505] [compact-period=1h0m0s] [took=1h0m0.005289094s]
[2020/02/29 18:39:30.230 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3703505] [took=1.259172ms]
[2020/02/29 18:42:40.075 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=214.85496ms] [prev-physical=2020/02/29 18:42:39.860 +08:00] [now=2020/02/29 18:42:40.075 +08:00]
[2020/02/29 18:43:00.292 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414966685108797440]
[2020/02/29 18:53:00.101 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414966842395197440]
[2020/02/29 19:03:00.123 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414966999681597440]
[2020/02/29 19:13:00.095 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414967156967997440]
[2020/02/29 19:23:00.096 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414967314254397440]
[2020/02/29 19:33:00.093 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414967471540797440]
[2020/02/29 19:39:30.230 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3704981] [compact-period=1h0m0s]
[2020/02/29 19:39:30.235 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3704981]
[2020/02/29 19:39:30.235 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3704981] [compact-period=1h0m0s] [took=1h0m0.005698586s]
[2020/02/29 19:39:30.236 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3704981] [took=1.482815ms]
[2020/02/29 19:43:00.094 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414967628827197440]
[2020/02/29 19:53:00.098 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414967786113597440]
[2020/02/29 20:03:00.210 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414967943426211840]
[2020/02/29 20:13:00.102 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414968100686397440]
[2020/02/29 20:23:00.098 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414968257972797440]
[2020/02/29 20:23:40.263 +08:00] [WARN] [tso.go:205] ["clock offset"] [jet-lag=152.840606ms] [prev-physical=2020/02/29 20:23:40.110 +08:00] [now=2020/02/29 20:23:40.263 +08:00]
[2020/02/29 20:33:00.116 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414968415259197440]
[2020/02/29 20:39:30.236 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3706433] [compact-period=1h0m0s]
[2020/02/29 20:39:30.239 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3706433]
[2020/02/29 20:39:30.239 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3706433] [compact-period=1h0m0s] [took=1h0m0.004426064s]
[2020/02/29 20:39:30.241 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3706433] [took=1.256238ms]
[2020/02/29 20:43:00.097 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414968572545597440]
[2020/02/29 20:53:00.092 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414968729831997440]
[2020/02/29 21:03:00.094 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414968887118397440]
[2020/02/29 21:13:00.140 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414969044404797440]
[2020/02/29 21:23:00.152 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414969201691197440]
[2020/02/29 21:33:00.128 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414969358977597440]
[2020/02/29 21:39:30.241 +08:00] [INFO] [periodic.go:135] ["starting auto periodic compaction"] [revision=3707885] [compact-period=1h0m0s]
[2020/02/29 21:39:30.244 +08:00] [INFO] [index.go:190] ["compact tree index"] [revision=3707885]
[2020/02/29 21:39:30.244 +08:00] [INFO] [periodic.go:146] ["completed auto periodic compaction"] [revision=3707885] [compact-period=1h0m0s] [took=1h0m0.004643721s]
[2020/02/29 21:39:30.245 +08:00] [INFO] [kvstore_compaction.go:57] ["finished scheduled compaction"] [compact-revision=3707885] [took=1.391225ms]
[2020/02/29 21:43:00.112 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414969516263997440]
[2020/02/29 21:53:00.085 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414969673550397440]
[2020/02/29 22:03:00.091 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414969830836797440]
[2020/02/29 22:13:00.108 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414969988123197440]
[2020/02/29 22:23:00.108 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414970145409597440]
[2020/02/29 22:33:00.091 +08:00] [INFO] [grpc_service.go:707] ["updated gc safe point"] [safe-point=414970302695997440]

请问问题在吗?

对应机器的PD server日志看不出有什么问题。其他几台存活的PD server请求这个接口也是404

curl 192.168.13.27:2379/pd
<a href="/pd/">Moved Permanently</a>.

curl 192.168.13.27:2379/pd/health
404 page not found

curl 192.168.13.27:2379/health
{"health":"true"}

麻烦发一下升级前的 pd-server 的版本,可以使用 ./pd-server -V 查看

你好,之前ansible部署的时候tidb_version使用的是latest, pd-server -V 显示的版本是v4.0.0-alpha-161-g2488cb97

./pd-server -V
Release Version: v4.0.0-alpha-161-g2488cb97
Git Commit Hash: 2488cb9786444100f163bd5ce53faf835c1fc42f
Git Branch: master
UTC Build Time:  2019-11-14 01:00:19

确认了一下,这个是一个已知的已经修复的问题。这个情况升级,如果可以停集群的话,可以使用如下方式:

  1. ansible-playbook stop.yml 停掉旧集群
  2. ansible-playbook deploy.yml 部署新版本
  3. ansible-playbook start.yml 启动集群

注:不需要清掉集群,不影响数据。

感谢帮助,已按照你提供的信息解决问题

ansible-playbook stop.yml --tags=pd
ansible-playbook deploy.yml --tags=pd
ansible-playbook start.yml --tags=pd
ansible-playbook rolling_update.yml

:+1: