tikv扩容缩容后集群信息错误，仍然访问已下线的tikv节点

ojsl · 2021 年11 月 12 日 03:44

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：
【 TiDB 使用环境】
【概述】场景+问题概述
【背景】做过哪些操作
Tidb对Tikv扩容缩容后，在检索数据时仍访问已下线的tikv节点
【现象】业务和数据库现象
【业务影响】
【TiDB 版本】
【附件】

相关日志和监控

TiUP Cluster Display 信息
TiUP Cluster Edit Config 信息
TiDB- Overview 监控

对应模块日志（包含问题前后1小时日志）
[raft_client.rs:407] [“connection aborted”] [addr=tidb-8.dtsta.local:20160] [receiver_err=“Some(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: “DNS resolution failed for service: tidb-8.dtsta.local:20160”, details: [] }))”] [sink_error=Some(RemoteStopped)] [store_id=754634]

xfworld · 2021 年11 月 12 日 06:02

麻烦提供一下版本，以及集群状态等信息

ojsl · 2021 年11 月 12 日 06:17

Cluster type:       tidb
Cluster name:       dtsta_tidb
Cluster version:    v5.2.1
Deploy user:        app_tidb
SSH type:           builtin
Dashboard URL:      http://tidb-1.dtsta.local:2379/dashboard
ID                              Role        Host                       Ports        OS/Arch        Status  Data Dir                    Deploy Dir
--                              ----        ----                       -----        -------        ------  --------                    ----------
172.32.0.180:8249               drainer     172.32.0.180               8249         linux/x86_64   Up      /tidb/data/drainer-8249     /tidb/deploy/drainer-8249
tidb-binlog-1.dtsta.local:8249  drainer     tidb-binlog-1.dtsta.local  8249         linux/x86_64   Up      /tidb/data/drainer-8249     /tidb/deploy/drainer-8249
tidb-1.dtsta.local:3000         grafana     tidb-1.dtsta.local         3000         linux/x86_64   Up      -                           /tidb/deploy/grafana-3000
tidb-1.dtsta.local:2379         pd          tidb-1.dtsta.local         2379/2380    linux/x86_64   Up|UI   /tidb/data/pd-2379          /tidb/deploy/pd-2379
tidb-2.dtsta.local:2379         pd          tidb-2.dtsta.local         2379/2380    linux/x86_64   Up      /tidb/data/pd-2379          /tidb/deploy/pd-2379
tidb-3.dtsta.local:2379         pd          tidb-3.dtsta.local         2379/2380    linux/x86_64   Up|L    /tidb/data/pd-2379          /tidb/deploy/pd-2379
tidb-1.dtsta.local:9090         prometheus  tidb-1.dtsta.local         9090         linux/x86_64   Up      /tidb/data/prometheus-9090  /tidb/deploy/prometheus-9090
tidb-binlog-1.dtsta.local:8250  pump        tidb-binlog-1.dtsta.local  8250         linux/x86_64   Up      /tidb/data/pump-8250        /tidb/deploy/pump-8250
tidb-binlog-2.dtsta.local:8250  pump        tidb-binlog-2.dtsta.local  8250         linux/x86_64   Up      /tidb/data/pump-8250        /tidb/deploy/pump-8250
tidb-binlog-3.dtsta.local:8250  pump        tidb-binlog-3.dtsta.local  8250         linux/x86_64   Up      /tidb/data/pump-8250        /tidb/deploy/pump-8250
tidb-1.dtsta.local:4000         tidb        tidb-1.dtsta.local         4000/10080   linux/x86_64   Up      -                           /tidb/deploy/tidb-4000
tidb-2.dtsta.local:4000         tidb        tidb-2.dtsta.local         4000/10080   linux/x86_64   Up      -                           /tidb/deploy/tidb-4000
tidb-3.dtsta.local:4000         tidb        tidb-3.dtsta.local         4000/10080   linux/x86_64   Up      -                           /tidb/deploy/tidb-4000
tidb-4.dtsta.local:4000         tidb        tidb-4.dtsta.local         4000/10080   linux/aarch64  Up      -                           /tidb/deploy/tidb-4000
tidb-1.dtsta.local:20160        tikv        tidb-1.dtsta.local         20160/20180  linux/x86_64   Up      /tidb/data/tikv-20160       /tidb/deploy/tikv-20160
tidb-2.dtsta.local:20160        tikv        tidb-2.dtsta.local         20160/20180  linux/x86_64   Up      /tidb/data/tikv-20160       /tidb/deploy/tikv-20160
tidb-3.dtsta.local:20160        tikv        tidb-3.dtsta.local         20160/20180  linux/x86_64   Up      /tidb/data/tikv-20160       /tidb/deploy/tikv-20160

xfworld · 2021 年11 月 12 日 06:50

状态全是 UP 呢，哪个节点下线了？

或者说，你的操作过程是什么样的？结果是否达到这个操作过程的预期了？

Meditator · 2021 年11 月 12 日 07:09

1) select * from information_schema.TIKV_STORE_STATUS；
2) curl "http://{pd_ip}:${pd_port}/pd/api/v1/stores"
3) pd-ctl  store

麻烦把这三个信息发下，谢谢。

Meditator · 2021 年11 月 12 日 07:10

下线了的这个

Meditator · 2021 年11 月 12 日 07:10

另外，你的下线步骤是怎样的，可以简单表述下吗

ojsl · 2021 年11 月 12 日 08:41

pd-ctl 里下线的节点状态都是offline，我尝试重启集群现在碰到另外一个问题
tikv启动报错：
[ERROR] [server.rs:1030] [“failed to init io snooper”] [err_code=KV:Unknown] [err="“IO snooper is not started due to not compiling with BCC”"]
[“failed to create kv engine: Storage Engine Corruption: Can’t access /1253103.sst: IO error: No such file or directorywhile stat a file for size: /tidb/data/tikv-20160/db/1253103.sst: No such file or directory\ ”]

ojsl · 2021 年11 月 12 日 08:45

都是用tiup工具操作，先进行下线任务提交，等待监控中任务完毕后，再操作tiup提交下线

Meditator · 2021 年11 月 12 日 09:15

这个明显是底层的rocksdb的数据文件已经丢失

Meditator · 2021 年11 月 12 日 09:17

在描述问题的时候，tug有个模板，请按模板填写，你只给出报错内容，不给出之前的上下文信息（操作了什么，入如何操作？），有时很难定位到具体信息，或者需要很久才能定位到。

ojsl · 2021 年11 月 12 日 09:27

执行了一次重启集群，就报这个错了，没有操作过任何文件。

Meditator · 2021 年11 月 12 日 09:32

tiup cluster audit

列出你的执行tiup命令的历史看看

ojsl · 2021 年11 月 12 日 09:34

fHRVMfmcpRP 2021-09-10T07:01:24Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check
fHS6CqkZwPB 2021-09-10T09:43:37Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check /tidb/opt/topology.yaml
fHS6WKVsfFn 2021-09-10T09:48:24Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check /tidb/opt/topology.yaml
fHS6ZfFz3rC 2021-09-10T09:49:09Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check /tidb/opt/topology.yaml
fHS72gC0fg1 2021-09-10T09:50:02Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check /tidb/opt/topology.yaml
fHS7crgprSL 2021-09-10T09:52:43Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check /tidb/opt/topology.yaml
fHS7pvtv3w0 2021-09-10T09:55:22Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check /tidb/opt/topology.yaml
fHS8tRKZS7j 2021-09-10T10:11:37Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check /tidb/opt/topology.yaml
fHS8CSJqV2k 2021-09-10T10:13:40Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster deploy dtsta_tidb v5.2.1 /tidb/opt/topology.yaml
fHS8ZyTQ4rN 2021-09-10T10:19:09Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster deploy dtsta_tidb v5.2.1 /tidb/opt/topology.yaml
fHS9dKQFYzw 2021-09-10T10:23:01Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check /tidb/opt/topology.yaml
fHS9xqbwx0Y 2021-09-10T10:27:19Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check /tidb/opt/topology.yaml
fHS9FLf678j 2021-09-10T10:29:10Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check /tidb/opt/topology.yaml
fHS9M3nth3K 2021-09-10T10:30:44Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster deploy dtsta_tidb v5.2.1 /tidb/opt/topology.yaml
fHS9QSgjRht 2021-09-10T10:31:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster start dtsta_tidb
fJcQrjKJVrR 2021-09-20T07:29:47Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check /tidb/opt/topology.yaml
fJd3Qc2Cv8S 2021-09-20T10:35:06Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check /tidb/opt/topology.yaml
fJd3YH9dHbS 2021-09-20T10:37:17Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check
fJd3Z5x6BxT 2021-09-20T10:37:25Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check dtsta_tidb
fJd412CvwZx 2021-09-20T10:37:59Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster list
fJd4dVJcv70 2021-09-20T10:41:28Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fJd4fW80Gx3 2021-09-20T10:41:45Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster template
fJd4gGZZ5nS 2021-09-20T10:41:59Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster template dtsta_tidb
fJd4sKWDQ2f 2021-09-20T10:44:38Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fJd4B9PFGN3 2021-09-20T10:46:32Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fJd4N41YVHR 2021-09-20T10:49:26Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fJd4RM7ZwTp 2021-09-20T10:50:31Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display
fJd4SkCnw75 2021-09-20T10:50:41Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fJd51DH9rc0 2021-09-20T10:53:07Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fJd5dtQWFrp 2021-09-20T10:56:18Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fJd5gNkdB34 2021-09-20T10:56:58Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fJNjRm3DvRz 2021-10-05T09:48:15Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fK4b0t9tTGS 2021-10-13T06:47:55Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fK4b1h99fFL 2021-10-13T06:48:09Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb
fK4b5V1VCpG 2021-10-13T06:49:30Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/topology.yaml
fK4bf4BSbC7 2021-10-13T06:51:37Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster list
fK4bgM9Mj0d 2021-10-13T06:52:06Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-scale-out.yaml.yaml
fK4bjsQvZ2p 2021-10-13T06:52:36Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-scale-out.yaml
fK4bp6ycWWz 2021-10-13T06:53:41Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-scale-out.yaml
fK4bpNcMv8P 2021-10-13T06:53:52Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-scale-out-tikv.yaml
fK4bxGHnb9p 2021-10-13T06:55:54Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-scale-out-tikv.yaml
fK4bGkqnLwG 2021-10-13T06:57:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-scale-out-tikv.yaml
fK4bMnwYn6R 2021-10-13T06:59:20Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-scale-out-tikv.yaml
fK4cTzZr8Tr 2021-10-13T07:16:06Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-scale-out-tikv.yaml
fK4d3z2nKHc 2021-10-13T07:18:44Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-scale-out-tikv.yaml
fK4d5gRZX07 2021-10-13T07:19:14Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check dtsta_local
fK4d5KMMv62 2021-10-13T07:19:22Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check dtsta_tidb
fK4d6gR0J0w 2021-10-13T07:19:32Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check
fK4d6Lv4kWY 2021-10-13T07:19:40Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster list
fK4d77CpXcr 2021-10-13T07:19:47Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check dtsta_tidb
fK4d96Vry4f 2021-10-13T07:20:22Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check dtsta_tidb --cluster
fK4dsNcH1D3 2021-10-13T07:24:40Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check dtsta_tidb --cluster
fK4ndMSVdjm 2021-10-13T09:06:08Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-scale-out-tikv.yaml
fK4nhPCqZc7 2021-10-13T09:07:02Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check dtsta_tidb --cluster
fK4pXYMwjfc 2021-10-13T09:31:42Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check dtsta_tidb --cluster
fK4rSXnJ1RZ 2021-10-13T10:00:26Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster check dtsta_tidb --cluster
fK4zcBPfdH0 2021-10-13T11:35:22Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-scale-out-tidb-pd.yaml
fK4zrm3r39q 2021-10-13T11:38:31Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-scale-out-tidb-pd.yaml
fK4zw417Yjx 2021-10-13T11:39:36Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fK4zHJttMRS 2021-10-13T11:42:26Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in
fK4zZXdm0V8 2021-10-13T11:46:54Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fK4BfRGNLxh 2021-10-13T11:50:59Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fK4BgVrw8Ch 2021-10-13T11:51:17Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fK4Bj7z1PDT 2021-10-13T11:51:39Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fK4BmVtccjW 2021-10-13T11:52:28Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fK4BB5bxq26 2021-10-13T11:55:44Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fK4BSypgJ3t 2021-10-13T11:59:59Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fK4BWKbk2G6 2021-10-13T12:00:55Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fK4C2k61cQF 2021-10-13T12:02:33Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fK4C2sGtScc 2021-10-13T12:02:36Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fK4CjWc7r6B 2021-10-13T12:06:50Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb -N tidb-1.dtsta.local:2379
fK4CkdjK7wq 2021-10-13T12:06:56Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fK4Cn7ywBVg 2021-10-13T12:07:29Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb -N tidb-2.dtsta.local:2379
fK4CqB4fzVX 2021-10-13T12:08:12Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb -N tidb-3.dtsta.local:2379
fK4Cr0XFMHW 2021-10-13T12:08:20Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fK4CsWk76ZC 2021-10-13T12:08:53Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb -N tidb-3.dtsta.local:4000
fK4CB9N7Gv0 2021-10-13T12:10:43Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb -N tidb-2.dtsta.local:4000
fK4CJJjZhmH 2021-10-13T12:12:38Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb -N tidb-1.dtsta.local:4000
fK4CM9prgcP 2021-10-13T12:13:22Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKyHxpvDSDY 2021-10-26T06:16:55Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKBMGbTZQ1H 2021-10-27T08:44:20Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKBMKTTJgbG 2021-10-27T08:45:25Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node tidb-4.dtsta.local:20160 tidb-5.dtsta.local:20160 tidb-6.dtsta.local:20160 tidb-7.dtsta.local:20160 tidb-8.dtsta.local:20160 tidb-9.dtsta.local:20160
fKBMNDdRNYN 2021-10-27T08:46:13Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node tidb-4.dtsta.local:20160 --node tidb-5.dtsta.local:20160 --node tidb-6.dtsta.local:20160 --node tidb-7.dtsta.local:20160 --node tidb-8.dtsta.local:20160 --node tidb-9.dtsta.local:20160
fKBMPjsqcJf 2021-10-27T08:46:25Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKBNyrc7hqR 2021-10-27T08:57:36Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/performance-test-revert-scale-out-tidb-pd.yaml
fKBNBwvgBYG 2021-10-27T08:58:12Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKBP7pYND68 2021-10-27T09:06:23Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node tidb-14.dtsta.local:2379 --node tidb-15.dtsta.local:2379 --node tidb-16.dtsta.local:2379 --node tidb-17.dtsta.local:2379 --node tidb-18.dtsta.local:2379 --node tidb-10.dtsta.local:4000 --node tidb-11.dtsta.local:4000 --node tidb-12.dtsta.local:4000 --node tidb-13.dtsta.local:4000 --node tidb-14.dtsta.local:4000 --node tidb-15.dtsta.local:4000 --node tidb-16.dtsta.local:4000 --node tidb-17.dtsta.local:4000
fKBPfz2bcrp 2021-10-27T09:08:12Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKBPgrZkjL7 2021-10-27T09:08:27Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKBPgwsr5z0 2021-10-27T09:08:28Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKBPN4CrH2y 2021-10-27T09:15:58Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKDjd5cGcz1 2021-10-28T04:19:26Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKG4QgR3xcd 2021-10-29T02:54:25Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKG7mgnfyVz 2021-10-29T03:32:14Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKN8LPQPvC5 2021-11-01T08:10:01Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKPZxj3F02h 2021-11-02T07:17:38Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKPZXPnDZ07 2021-11-02T07:23:56Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node tidb-4.dtsta.local:20160 tidb-5.dtsta.local:20160 tidb-6.dtsta.local:20160 tidb-7.dtsta.local:20160 tidb-8.dtsta.local:20160 tidb-9.dtsta.local:20160
fKPZZvtpwYM 2021-11-02T07:24:26Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node tidb-4.dtsta.local:20160 --node tidb-5.dtsta.local:20160 --node tidb-6.dtsta.local:20160 --node tidb-7.dtsta.local:20160 --node tidb-8.dtsta.local:20160 --node tidb-9.dtsta.local:20160
fKPZZMrrsHJ 2021-11-02T07:24:30Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKQ00jwsqLQ 2021-11-02T07:24:40Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKQ04N838gn 2021-11-02T07:25:59Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRvFBBVP22 2021-11-03T02:31:09Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster list
fKRvGPXsfHK 2021-11-03T02:31:30Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster list
fKRxzt1cgtH 2021-11-03T02:59:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRxWHFMw3T 2021-11-03T03:05:12Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node tidb-4.dtsta.local:20160 --node tidb-5.dtsta.local:20160 --node tidb-6.dtsta.local:20160 --node tidb-7.dtsta.local:20160 --node tidb-8.dtsta.local:20160 --node tidb-9.dtsta.local:20160
fKRxWRRXBNb 2021-11-03T03:05:15Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRyyXvCXGw 2021-11-03T03:14:39Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRCby1MWZr 2021-11-03T03:54:08Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb ./binlog-scale-out.yaml
fKRCjp3THJQ 2021-11-03T03:55:50Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb ./binlog-scale-out.yaml
fKRCmzFmk7N 2021-11-03T03:56:29Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRCpBRsrqg 2021-11-03T03:57:05Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config
fKRCy80cC4r 2021-11-03T03:59:17Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRCzBgnjzL 2021-11-03T03:59:43Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRCBM9xJQ5 2021-11-03T04:00:03Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRCGnzMYs7 2021-11-03T04:01:07Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N tidb-1.dtsta.local:4000 -N tidb-2.dtsta.local:4000 -N tidb-3.dtsta.local:4000
fKRCGFwxQSK 2021-11-03T04:01:12Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRCH663Jtp 2021-11-03T04:01:20Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRCP1HBP9J 2021-11-03T04:03:04Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRCRSJ5rRY 2021-11-03T04:03:54Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRCSF4fb0m 2021-11-03T04:04:08Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRCTCxN6Fp 2021-11-03T04:04:25Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRCZrySZ2r 2021-11-03T04:05:50Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRD0vjPp7g 2021-11-03T04:06:08Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRDvfj1XX5 2021-11-03T04:13:24Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta-tidb
fKRDx7D6G00 2021-11-03T04:13:57Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster list
fKRDyLk8t8f 2021-11-03T04:14:25Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRDBnV7PDP 2021-11-03T04:14:54Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRDDnNt2b2 2021-11-03T04:15:29Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRDFw7Rh4c 2021-11-03T04:15:49Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster prune
fKRDKVwvv7d 2021-11-03T04:17:07Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster prune dtsta_tidb
fKRF54qKmDP 2021-11-03T04:22:26Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node tidb-4.dtsta.local:20160 --node tidb-5.dtsta.local:20160 --node tidb-6.dtsta.local:20160 --node tidb-7.dtsta.local:20160 --node tidb-8.dtsta.local:20160 --node tidb-9.dtsta.local:20160
fKRF5t3NdmQ 2021-11-03T04:22:33Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRF929BF9d 2021-11-03T04:23:36Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRF9yQYbXC 2021-11-03T04:23:45Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRFmzxvdqd 2021-11-03T04:26:24Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRFnjRFjmt 2021-11-03T04:26:37Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRFxWdD1wp 2021-11-03T04:29:08Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRFz9qBrvR 2021-11-03T04:29:30Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node tidb-4.dtsta.local:20160 --node tidb-5.dtsta.local:20160 --node tidb-6.dtsta.local:20160 --node tidb-7.dtsta.local:20160 --node tidb-8.dtsta.local:20160 --node tidb-9.dtsta.local:20160
fKRFCJPqWxX 2021-11-03T04:30:15Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster prune dtsta_tidb
fKRFCY94NM7 2021-11-03T04:30:19Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRFFmnmX9R 2021-11-03T04:30:44Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node tidb-4.dtsta.local:20160 --node tidb-5.dtsta.local:20160 --node tidb-6.dtsta.local:20160 --node tidb-7.dtsta.local:20160 --node tidb-8.dtsta.local:20160 --node tidb-9.dtsta.local:20160
fKRFLMN02Bm 2021-11-03T04:32:19Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster prune dtsta_tidb
fKRFZL0W7qY 2021-11-03T04:35:50Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRGhR1FfG4 2021-11-03T04:40:33Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRGkgDn8j7 2021-11-03T04:40:58Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRGLV1J7Vj 2021-11-03T04:47:19Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRGMdqNZvf 2021-11-03T04:47:25Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRGNtTywcJ 2021-11-03T04:47:47Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRGVRhBXXj 2021-11-03T04:49:39Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRGYKDvXT6 2021-11-03T04:50:29Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRH1PpvzXQ 2021-11-03T04:51:24Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRH3ZzPFCh 2021-11-03T04:52:02Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRHfKPC8br 2021-11-03T04:54:53Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRHkpBHTnk 2021-11-03T04:55:58Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRHvDw7wn5 2021-11-03T04:58:23Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRK0d1j3qv 2021-11-03T05:20:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRR6bHCLcQ 2021-11-03T06:52:20Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRRtKYG3jn 2021-11-03T06:57:46Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRRTXXDGW1 2021-11-03T07:04:00Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRSJ0Bz3Kv 2021-11-03T07:16:02Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fKRSP3sjkSB 2021-11-03T07:17:31Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRTdbH7SH5 2021-11-03T07:24:01Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRTfJtDWPJ 2021-11-03T07:24:27Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKRTmg3VgCr 2021-11-03T07:25:47Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKRTG71d1gQ 2021-11-03T07:30:26Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N tidb-binlog-1.dtsta.local:8249 -N tidb-1.dtsta.local:9093 -N tidb-binlog-1.dtsta.local:8250 -N tidb-binlog-2.dtsta.local:8250 -N tidb-binlog-3.dtsta.local:8250 -N tidb-1.dtsta.local:4000 -N tidb-2.dtsta.local:4000 -N tidb-3.dtsta.local:4000
fKRTJcgGNxT 2021-11-03T07:31:03Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKSc6fTcC1g 2021-11-03T11:36:32Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta-tidb
fKSccLP2Nn1 2021-11-03T11:38:09Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKSdNsGnRSy 2021-11-03T12:01:31Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta-tidb
fKTQqP2Y8cN 2021-11-04T08:07:33Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display
fKTQswgG7Px 2021-11-04T08:08:03Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKWHcRL6D9h 2021-11-05T07:45:32Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb binlog-cross-scale-out-20211105.yaml
fKWKPQbbpZj 2021-11-05T08:24:14Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb binlog-cross-scale-out-20211105.yaml
fKWKR3v2d1y 2021-11-05T08:24:36Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb binlog-cross-scale-out-20211105.yaml
fKWLjttgv7p 2021-11-05T08:31:46Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb binlog-cross-scale-out-20211105.yaml
fKWMcnbsjKN 2021-11-05T08:45:13Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb binlog-cross-scale-out-20211105.yaml
fKWR9JncmD5 2021-11-05T09:44:34Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb binlog-cross-scale-out-20211105.yaml
fKWRbHxTTTS 2021-11-05T09:44:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKWRnRM6XfN 2021-11-05T09:47:32Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N 172.32.0.180:8249
fKWRFyJdQhh 2021-11-05T09:51:33Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N 172.32.0.180:8249
fKWRZ1vR0JP 2021-11-05T09:56:23Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fKWS7vW4bZ6 2021-11-05T09:58:52Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N 172.32.0.180:8249
fKWSmTn6H6k 2021-11-05T10:02:13Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fKWSXRKSj0Z 2021-11-05T10:11:00Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N 172.32.0.180:8249
fL1TVQsLvzz 2021-11-08T01:59:21Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL1TX1qCP51 2021-11-08T01:59:42Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N 172.32.0.180:8249
fL1WtQYwMzJ 2021-11-08T02:22:49Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N 172.32.0.180:8249
fL1YDBtxjxk 2021-11-08T02:55:01Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL20Fy21mDK 2021-11-08T03:25:12Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL20G7yPY32 2021-11-08T03:25:23Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL21kPhmcgW 2021-11-08T03:35:33Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N 172.32.0.180:8249
fL2g1Q9pMtz 2021-11-08T06:45:18Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL2qwLZb04F 2021-11-08T08:37:18Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb -N 172.32.0.180:8249
fL2qyhyRv4G 2021-11-08T08:37:45Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL2qCDNv8R9 2021-11-08T08:38:44Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node 172.32.0.180:8249
fL2qJxNkpS0 2021-11-08T08:40:10Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node 172.32.0.180:8249
fL2rdPbK1f4 2021-11-08T08:48:10Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL2rz20vjpx 2021-11-08T08:52:56Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -R tidb
fL2rMTT8Tkq 2021-11-08T08:56:07Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL2rNpxL21N 2021-11-08T08:56:16Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL2s0JN4xbG 2021-11-08T08:59:35Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster start dtsta_tidb -N tidb -N tidb-3.dtsta.local
fL2s1SFtFHP 2021-11-08T08:59:55Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster start dtsta_tidb -R tidb -N tidb-3.dtsta.local
fL2s3wfynRd 2021-11-08T09:00:24Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL2sf6Mqn8y 2021-11-08T09:03:13Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster start dtsta_tidb -R tidb -N tidb-3.dtsta.local:4000
fL2syTjxYz3 2021-11-08T09:07:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL2sCK3pfxb 2021-11-08T09:08:40Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster start dtsta_tidb -R tidb -N tidb-3.dtsta.local:4000
fL2sNyW8gHb 2021-11-08T09:11:16Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N tidb-3.dtsta.local:4000
fL2sYjX4mXN 2021-11-08T09:13:50Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N tidb-3.dtsta.local:4000
fL2sZK7PjPX 2021-11-08T09:14:15Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL2v3kQjf3P 2021-11-08T09:30:16Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL2v3DttbP1 2021-11-08T09:30:21Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N tidb-3.dtsta.local:4000
fL2vdR3WbzV 2021-11-08T09:33:03Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N 172.32.0.180:8249
fL2vZVjK2Nh 2021-11-08T09:44:13Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N 172.32.0.180:8249
fL2xyPpVtG9 2021-11-08T10:07:39Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -N 172.32.0.180:8249
fL2z1V40qSm 2021-11-08T10:29:40Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL3MzwChWMp 2021-11-09T01:50:08Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL4d8NLxBmR 2021-11-09T07:43:01Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster list
fL4dc1BPn5d 2021-11-09T07:43:41Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out binlog-scale-out.yaml
fL4ddggJrwm 2021-11-09T07:44:03Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb binlog-scale-out.yaml
fL4dmSYxB3T 2021-11-09T07:45:59Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config
fL4dnx5ZHvB 2021-11-09T07:46:10Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta-tidb
fL4fcLg02Cr 2021-11-09T07:58:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL4hWSBQSn2 2021-11-09T08:39:21Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL4hZNxXFJx 2021-11-09T08:40:13Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL4j2rSm4yq 2021-11-09T08:40:59Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node 172.32.0.180:8249
fL4j3sK2xmt 2021-11-09T08:41:17Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node 172.32.0.180:8249
fL4j4XvZ3FF 2021-11-09T08:41:43Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node [172.32.0.180:8249]
fL4jczFZLdm 2021-11-09T08:43:40Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node 172.32.0.180:8249
fL4jkFPGXt3 2021-11-09T08:45:27Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tiup binlog-scale-out-2.yaml
fL4jmWjhf3J 2021-11-09T08:45:49Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb binlog-scale-out-2.yaml
fL4jGgr548Z 2021-11-09T08:50:19Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb binlog-scale-out-2.yaml
fL5FWwqhd5W 2021-11-10T01:51:16Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL5GvBBrwj6 2021-11-10T01:59:48Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL67XMfHBV3 2021-11-10T08:05:34Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL683XTW2GF 2021-11-10T08:07:22Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb -N tidb-1.dtsta.local:9093
fL684pdb8Zk 2021-11-10T08:07:30Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL685zVPkGn 2021-11-10T08:07:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb -N 172.32.0.180:8249
fL6d35VN3Xr 2021-11-10T09:06:57Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node 172.32.0.180:8249
fL6d63ZLYdm 2021-11-10T09:07:49Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node 172.32.0.180:8249
fL6d6HG2wT5 2021-11-10T09:08:00Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node [172.32.0.180:8249]
fL6dnK9mWtr 2021-11-10T09:11:50Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL6dqcqG24J 2021-11-10T09:12:16Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node [172.32.0.180:8249]
fL6dsbYM8Y7 2021-11-10T09:12:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node 172.32.0.180:8249
fL6f16vpGQp 2021-11-10T09:21:20Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta-tidb
fL6f1MRxfX7 2021-11-10T09:21:31Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL6fq2trBh0 2021-11-10T09:27:10Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node 172.32.0.180
fL6fGw2Xyh3 2021-11-10T09:31:07Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL6fHRY9ztL 2021-11-10T09:31:31Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-in dtsta_tidb --node 172.32.0.180:8249
fL6fNqdpLb4 2021-11-10T09:32:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL6fSkx73wd 2021-11-10T09:34:00Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL6fTRVHcpg 2021-11-10T09:34:27Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL6gmY1HKsy 2021-11-10T09:41:31Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb binlog-scale-out-2.yaml
fL6gssq7RV1 2021-11-10T09:42:50Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL6hg5s5cvg 2021-11-10T09:55:03Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fL6hxsZfMNn 2021-11-10T09:58:58Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL6kvR9nPGK 2021-11-10T10:28:24Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb --node 172.32.0.180:8249
fL6kzsbnVDK 2021-11-10T10:29:28Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL6kWR8RrkS 2021-11-10T10:34:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL6m88qBpy6 2021-11-10T10:38:11Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb --node 172.32.0.180:8249
fL6nFh0DkG5 2021-11-10T11:00:30Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL6nR7KVgwy 2021-11-10T11:03:24Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb --node 172.32.0.180:8249
fL6pb0d6vJb 2021-11-10T11:08:38Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL6pfqzkcF6 2021-11-10T11:09:38Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb --node 172.32.0.180:8249 --wait-timeout 15
fL6q1d9wR1X 2021-11-10T11:21:01Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL6q3zFngJY 2021-11-10T11:21:42Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL6qtRN4LQb 2021-11-10T11:27:57Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL6qH7XZGxH 2021-11-10T11:30:58Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL6ryFBY5gr 2021-11-10T11:44:01Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb --node 172.32.0.180:8249 --wait-timeout 750
fL6shkF2HDh 2021-11-10T11:55:04Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL6w1XNPvnx 2021-11-10T12:36:01Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_dm_tidb
fL6yx16vPgp 2021-11-10T13:13:16Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster list
fL6yxn9T2Nv 2021-11-10T13:13:23Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster list dtsta_tidb
fL6yxLswyDk 2021-11-10T13:13:29Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL6BWFhXBW4 2021-11-10T13:49:14Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL82bxj8pF2 2021-11-11T08:05:10Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster list
fL82c6mgNS9 2021-11-11T08:05:20Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster list dtsta_tidb
fL82dfPsYdc 2021-11-11T08:05:40Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL8nBHs65hW 2021-11-11T12:25:20Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/tidb-arm-scale-out.yaml
fL8pcF1h5gn 2021-11-11T12:34:43Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/tidb-arm-scale-out.yaml
fL8pvWCXKSd 2021-11-11T12:38:54Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/tidb-arm-scale-out.yaml
fL8pD9vWgc2 2021-11-11T12:40:44Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster scale-out dtsta_tidb /tidb/opt/tidb-arm-scale-out.yaml
fL8pDNMPh6b 2021-11-11T12:40:55Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL9yKQKgTX3 2021-11-12T03:24:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL9Gxxp2dXm 2021-11-12T04:51:34Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL9QF5vWpRp 2021-11-12T06:52:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL9QSgNz9Qf 2021-11-12T06:56:08Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster restart dtsta_tidb
fL9QXv5BykT 2021-11-12T06:57:22Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL9RsvM1bHz 2021-11-12T07:05:00Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL9RxtstC7s 2021-11-12T07:06:10Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fL9RG8Lh33x 2021-11-12T07:08:07Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fL9RTbFw7Md 2021-11-12T07:11:22Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster help
fL9RTYW66bc 2021-11-12T07:11:35Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster start dtsta_tidb
fL9RWgZCTGK 2021-11-12T07:11:58Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster restart
fL9RWtjZp6j 2021-11-12T07:12:01Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster restart dista_tidb
fL9S5D2d31c 2021-11-12T07:14:42Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster restart dtsta_tidb
fL9Sy90cgKV 2021-11-12T07:21:19Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL9Tfzp2cjv 2021-11-12T07:32:00Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster start dtsta_tidb
fL9Ty0nWzDk 2021-11-12T07:36:14Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster start dtsta_tidb
fL9V6TTMGCP 2021-11-12T07:45:00Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster display dtsta_tidb
fL9VhsGQ9hp 2021-11-12T07:47:30Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster start dtsta_tidb
fL9VM4RBZBt 2021-11-12T07:54:26Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster start dtsta_tidb
fL9WbmTdHZB 2021-11-12T08:00:58Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL9WwXxzxVB 2021-11-12T08:05:50Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL9WyG81JXj 2021-11-12T08:06:20Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fL9WCnBXSRm 2021-11-12T08:07:08Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL9WFTs2m3f 2021-11-12T08:07:52Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fL9WRWJkFtG 2021-11-12T08:10:49Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster start dtsta_tidb
fL9XfvYrDn2 2021-11-12T08:16:51Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL9XgWQG1VT 2021-11-12T08:17:16Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster edit-config dtsta_tidb
fL9Xkk4Q9NC 2021-11-12T08:17:58Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb -R tikv
fL9XmXSxHMw 2021-11-12T08:18:27Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster reload dtsta_tidb
fL9XxqQkt0z 2021-11-12T08:20:56Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster start dtsta_tidb

ojsl · 2021 年11 月 12 日 09:38

fL9QSgNz9Qf 2021-11-12T06:56:08Z /tidb/opt/components/cluster/v1.5.6/tiup-cluster restart dtsta_tidb
这个是今天第一次尝试的重启

ojsl · 2021 年11 月 12 日 09:45

另外这个丢失文件的错误在3个tikv实例上都有，只不过是不同的文件名。

Meditator · 2021 年11 月 12 日 09:49

tiup cluster audit fL9QSgNz9Qf

看看有什么输出

目前看是不规范的操作的引起的，至于怎么引起的还需要分析下.

Meditator · 2021 年11 月 12 日 09:55

正常的操作

tiup  cluster scale-in  xxx

触发region的leader和peer调度
待state变成 Tombstone后，执行

tiup  cluster prune  xxx

来手工清理集群组件中的元数据

ojsl · 2021 年11 月 12 日 09:59

2021-11-12T06:56:08.425Z	DEBUG	TaskFinish	{“task”: “RestartCluster”, “error”: “failed to start: failed to start tikv: failed to start: tidb-2.dtsta.local tikv-20160.service, please check the instance’s log(/tidb/deploy/tikv-20160/log) for more detail.: timed out waiting for port 20160 to be started after 2m0s”, “errorVerbose”: “timed out waiting for port 20160 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(WaitFor).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ github.com/pingcap/tiup/pkg/cluster/spec.PortStarted\ \tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:115\ github.com/pingcap/tiup/pkg/cluster/spec.(BaseInstance).Ready\ \tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:147\ github.com/pingcap/tiup/pkg/cluster/operation.startInstance\ \tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:351\ github.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1\ \tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:454\ golang.org/x/sync/errgroup.(*Group).Go.func1\ \tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\ runtime.goexit\ \truntime/asm_amd64.s:1371\ failed to start: tidb-2.dtsta.local tikv-20160.service, please check the instance’s log(/tidb/deploy/tikv-20160/log) for more detail.\ failed to start tikv\ failed to start”}
2021-11-12T06:56:08.425Z	INFO	Execute command finished	{“code”: 1, “error”: “failed to start: failed to start tikv: failed to start: tidb-2.dtsta.local tikv-20160.service, please check the instance’s log(/tidb/deploy/tikv-20160/log) for more detail.: timed out waiting for port 20160 to be started after 2m0s”, “errorVerbose”: “timed out waiting for port 20160 to be started after 2m0s\ngithub.com/pingcap/tiup/pkg/cluster/module.(WaitFor).Execute\ \tgithub.com/pingcap/tiup/pkg/cluster/module/wait_for.go:91\ github.com/pingcap/tiup/pkg/cluster/spec.PortStarted\ \tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:115\ github.com/pingcap/tiup/pkg/cluster/spec.(BaseInstance).Ready\ \tgithub.com/pingcap/tiup/pkg/cluster/spec/instance.go:147\ github.com/pingcap/tiup/pkg/cluster/operation.startInstance\ \tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:351\ github.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1\ \tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:454\ golang.org/x/sync/errgroup.(*Group).Go.func1\ \tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\ runtime.goexit\ \truntime/asm_amd64.s:1371\ failed to start: tidb-2.dtsta.local tikv-20160.service, please check the instance’s log(/tidb/deploy/tikv-20160/log) for more detail.\ failed to start tikv\ failed to start”}

只有结束的时候有这个日志，其他看起来是正常的

ojsl · 2021 年11 月 12 日 10:00

2021-11-12T06:53:40.640Z INFO Stopping component tikv

2021-11-12T06:53:40.640Z INFO Stopping instance tidb-1.dtsta.local

2021-11-12T06:53:40.640Z INFO Stopping instance tidb-3.dtsta.local

2021-11-12T06:53:40.640Z INFO Stopping instance tidb-2.dtsta.local

2021-11-12T06:53:42.680Z INFO SSHCommand {“host”: “tidb-1.dtsta.local”, “port”: “22”, “cmd”: “export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H bash -c “systemctl daemon-reload && systemctl stop tikv-20160.service””, “stdout”: “”, “stderr”: “”}

2021-11-12T06:53:42.680Z INFO CheckPoint {“host”: “tidb-1.dtsta.local”, “port”: 22, “user”: “app_tidb”, “sudo”: true, “cmd”: “systemctl daemon-reload && systemctl stop tikv-20160.service”, “stdout”: “”, “stderr”: “”, “hash”: “1d1478dbec0d06ef5e909d3323331914538d4cb0”, “func”: “github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute”, “hit”: false}

2021-11-12T06:53:42.680Z INFO Stop tikv tidb-1.dtsta.local:20160 success

2021-11-12T06:53:42.989Z INFO SSHCommand {“host”: “tidb-3.dtsta.local”, “port”: “22”, “cmd”: “export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H bash -c “systemctl daemon-reload && systemctl stop tikv-20160.service””, “stdout”: “”, “stderr”: “”}

2021-11-12T06:53:42.989Z INFO CheckPoint {“host”: “tidb-3.dtsta.local”, “port”: 22, “user”: “app_tidb”, “sudo”: true, “cmd”: “systemctl daemon-reload && systemctl stop tikv-20160.service”, “stdout”: “”, “stderr”: “”, “hash”: “1d1478dbec0d06ef5e909d3323331914538d4cb0”, “func”: “github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute”, “hit”: false}

2021-11-12T06:53:42.989Z INFO Stop tikv tidb-3.dtsta.local:20160 success

2021-11-12T06:53:43.272Z INFO SSHCommand {“host”: “tidb-2.dtsta.local”, “port”: “22”, “cmd”: “export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H bash -c “systemctl daemon-reload && systemctl stop tikv-20160.service””, “stdout”: “”, “stderr”: “”}

2021-11-12T06:53:43.272Z INFO CheckPoint {“host”: “tidb-2.dtsta.local”, “port”: 22, “user”: “app_tidb”, “sudo”: true, “cmd”: “systemctl daemon-reload && systemctl stop tikv-20160.service”, “stdout”: “”, “stderr”: “”, “hash”: “1d1478dbec0d06ef5e909d3323331914538d4cb0”, “func”: “github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute”, “hit”: false}

2021-11-12T06:53:43.272Z INFO Stop tikv tidb-2.dtsta.local:20160 success

这是stop tikv部分的操作