tikv down,重启后恢复,看起来是一个bug

tikv down,日志说连不上pd,我上去那个down的集群,telnet一下pd,是可以连上的。

重启一下tikv,tikv恢复了。 看起来,这是tikv的一个bug呀。

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:
【 TiDB 使用环境】
【概述】场景+问题概述
正常使用,但是收到tikv down的报警。
【背景】做过哪些操作
没有
【现象】业务和数据库现象
tikv down
【业务影响】tikv down
【TiDB 版本】v5.2.1
【附件】

  1. TiUP Cluster Display 信息
    Starting component cluster: /home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster display tidb_prod
    Cluster type: tidb
    Cluster name: tidb_prod
    Cluster version: v5.2.1
    Deploy user: tidb
    SSH type: builtin
    Dashboard URL: http://192.168.0.66:2379/dashboard
    ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

192.168.0.44:9093 alertmanager 192.168.0.44 9093/9094 linux/x86_64 Up /home/tidb/tidb-data/alertmanager-9093 /home/tidb/tidb-deploy/alertmanager-9093
116.57.100.238:3000 grafana 116.57.100.238 3000 linux/x86_64 Up - /home/tidb/tidb-deploy/grafana-3000
192.168.0.55:3000 grafana 192.168.0.55 3000 linux/x86_64 Up - /home/tidb/tidb-deploy/grafana-3000
192.168.0.11:2379 pd 192.168.0.11 2379/2380 linux/x86_64 Up|L /home/tidb/tidb-data/pd-2379 /home/tidb/tidb-deploy/pd-2379
192.168.0.22:2379 pd 192.168.0.22 2379/2380 linux/x86_64 Up /home/tidb/tidb-data/pd-2379 /home/tidb/tidb-deploy/pd-2379
192.168.0.33:2379 pd 192.168.0.33 2379/2380 linux/x86_64 Up /home/tidb/tidb-data/pd-2379 /home/tidb/tidb-deploy/pd-2379
192.168.0.44:2379 pd 192.168.0.44 2379/2380 linux/x86_64 Up /home/tidb/tidb-data/pd-2379 /home/tidb/tidb-deploy/pd-2379
192.168.0.55:2379 pd 192.168.0.55 2379/2380 linux/x86_64 Up /home/tidb/tidb-data/pd-2379 /home/tidb/tidb-deploy/pd-2379
192.168.0.66:2379 pd 192.168.0.66 2379/2380 linux/x86_64 Up|UI /home/tidb/tidb-data/pd-2379 /home/tidb/tidb-deploy/pd-2379
192.168.0.44:9090 prometheus 192.168.0.44 9090 linux/x86_64 Up /home/tidb/tidb-data/prometheus-9090 /home/tidb/tidb-deploy/prometheus-9090
192.168.0.11:4000 tidb 192.168.0.11 4000/10080 linux/x86_64 Up - /home/tidb/tidb-deploy/tidb-4000
192.168.0.22:4000 tidb 192.168.0.22 4000/10080 linux/x86_64 Up - /home/tidb/tidb-deploy/tidb-4000
192.168.0.33:4000 tidb 192.168.0.33 4000/10080 linux/x86_64 Up - /home/tidb/tidb-deploy/tidb-4000
192.168.0.44:4000 tidb 192.168.0.44 4000/10080 linux/x86_64 Up - /home/tidb/tidb-deploy/tidb-4000
192.168.0.55:4000 tidb 192.168.0.55 4000/10080 linux/x86_64 Up - /home/tidb/tidb-deploy/tidb-4000
192.168.0.66:4000 tidb 192.168.0.66 4000/10080 linux/x86_64 Up - /home/tidb/tidb-deploy/tidb-4000
192.168.0.11:20160 tikv 192.168.0.11 20160/20180 linux/x86_64 Up /tidb_ssd_data/tikv-20160 /home/tidb/tidb-deploy/tikv-20160
192.168.0.22:20160 tikv 192.168.0.22 20160/20180 linux/x86_64 Up /tidb_ssd_data/tikv-20160 /home/tidb/tidb-deploy/tikv-20160
192.168.0.33:20160 tikv 192.168.0.33 20160/20180 linux/x86_64 Up /tidb_ssd_data/tikv-20160 /home/tidb/tidb-deploy/tikv-20160
192.168.0.44:20160 tikv 192.168.0.44 20160/20180 linux/x86_64 Down /tidb_ssd_data/tikv-20160 /home/tidb/tidb-deploy/tikv-20160
192.168.0.55:20160 tikv 192.168.0.55 20160/20180 linux/x86_64 Up /tidb_ssd_data/tikv-20160 /home/tidb/tidb-deploy/tikv-20160
192.168.0.66:20160 tikv 192.168.0.66 20160/20180 linux/x86_64 Up /tidb_ssd_data/tikv-20160 /home/tidb/tidb-deploy/tikv-20160

  1. TiUP Cluster Edit Config 信息

  2. TiDB- Overview 监控

  • 对应模块日志(包含问题前后1小时日志)
    这是重启前的日志:

[2021/12/18 15:04:04.166 +08:00] [INFO] [tracker.rs:256] [slow-query] [perf_stats.internal_delete_skipped_count=0] [perf_stats.internal_key_skipped_count=482657] [perf_stats.block_read_byte=17179509] [perf_stats.block_read_count=1488] [perf_stats.block_cache_hit_count=10] [scan.range.first=“Some(start: 74800000000000006D5F728000000001B7C670 end: 74800000000000006D5F728000000001BB7521)”] [scan.ranges=1] [scan.total=241330] [scan.processed_size=87390171] [scan.processed=241329] [scan.is_desc=None] [tag=analyze_index] [table_id=109] [txn_start_ts=429866509339721730] [total_suspend_time=0ns] [total_process_time=2.712s] [handler_build_time=0ns] [wait_time.snapshot=1ms] [wait_time.schedule=0ns] [wait_time=1ms] [total_lifetime=2.713s] [remote_host=ipv4:192.168.0.44:46922] [region_id=53892240]
[2021/12/18 15:04:51.404 +08:00] [INFO] [tracker.rs:256] [slow-query] [perf_stats.internal_delete_skipped_count=0] [perf_stats.internal_key_skipped_count=482161] [perf_stats.block_read_byte=17249262] [perf_stats.block_read_count=1489] [perf_stats.block_cache_hit_count=10] [scan.range.first=“Some(start: 74800000000000006D5F728000000001E7E641 end: 74800000000000006D5F728000000001EB93FA)”] [scan.ranges=1] [scan.total=241082] [scan.processed_size=87403619] [scan.processed=241081] [scan.is_desc=None] [tag=analyze_index] [table_id=109] [txn_start_ts=429866509339721730] [total_suspend_time=0ns] [total_process_time=2.723s] [handler_build_time=0ns] [wait_time.snapshot=0ns] [wait_time.schedule=1ms] [wait_time=1ms] [total_lifetime=2.724s] [remote_host=ipv4:192.168.0.44:46926] [region_id=53988918]
[2021/12/18 15:04:54.183 +08:00] [INFO] [tracker.rs:256] [slow-query] [perf_stats.internal_delete_skipped_count=0] [perf_stats.internal_key_skipped_count=481137] [perf_stats.block_read_byte=17397836] [perf_stats.block_read_count=1489] [perf_stats.block_cache_hit_count=10] [scan.range.first=“Some(start: 74800000000000006D5F728000000001EB93FA end: 74800000000000006D5F728000000001EF3FB3)”] [scan.ranges=1] [scan.total=240570] [scan.processed_size=87429083] [scan.processed=240569] [scan.is_desc=None] [tag=analyze_index] [table_id=109] [txn_start_ts=429866509339721730] [total_suspend_time=0ns] [total_process_time=2.739s] [handler_build_time=0ns] [wait_time.snapshot=1ms] [wait_time.schedule=0ns] [wait_time=1ms] [total_lifetime=2.74s] [remote_host=ipv4:192.168.0.44:46928] [region_id=53979482]
@ @
/ERROR
[2021/12/19 01:15:28.478 +08:00] [WARN] [scheduler.rs:655] ["[region 54441815] scheduler handle command: commit, ts: 429876241881366561"] [takes=2920]
[2021/12/19 01:15:28.545 +08:00] [ERROR] [util.rs:419] [“request failed, retry”] [err_code=KV:PD:Unknown] [err=“Other(”[components/pd_client/src/client.rs:834]: get timestamp timeout")"]
[2021/12/19 01:15:28.689 +08:00] [ERROR] [util.rs:419] [“request failed, retry”] [err_code=KV:PD:Unknown] [err=“Other(”[components/pd_client/src/client.rs:834]: get timestamp timeout")"]
[2021/12/19 01:15:28.689 +08:00] [WARN] [apply.rs:641] ["[store 435001] handle ready 1 committed entries"] [takes=3593]
[2021/12/19 01:20:18.649 +08:00] [WARN] [store.rs:603] ["[store 435001] handle timeout RaftEnginePurge"] [takes=3233]
[2021/12/19 01:20:18.649 +08:00] [INFO] [] [“ipv4:192.168.0.66:36814: Keepalive watchdog fired. Closing transport.”]
[2021/12/19 01:20:18.649 +08:00] [INFO] [] [“ipv4:192.168.0.22:55694: Keepalive watchdog fired. Closing transport.”]
[2021/12/19 01:20:18.899 +08:00] [ERROR] [raft_client.rs:707] [“connection abort”] [addr=192.168.0.11:20160] [store_id=52662884]
[2021/12/19 01:20:18.913 +08:00] [INFO] [client.rs:136] [“TSO stream is closed, reconnect to PD”]
[2021/12/19 01:20:18.913 +08:00] [ERROR] [util.rs:419] [“request failed, retry”] [err_code=KV:PD:gRPC] [err=“Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: “context deadline exceeded”, details: [] }))”]
[2021/12/19 01:20:18.920 +08:00] [ERROR] [raft_client.rs:407] [“connection aborted”] [addr=192.168.0.22:20160] [receiver_err=“Some(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: “keepalive watchdog timeout”, details: [] }))”] [sink_error=“Some(RpcFinished(Some(RpcStatus { code: 14-UNAVAILABLE, message: “keepalive watchdog timeout”, details: [] })))”] [store_id=42827561]
[2021/12/19 01:20:18.921 +08:00] [ERROR] [util.rs:419] [“request failed, retry”] [err_code=KV:PD:Unknown] [err=“Other(”[components/pd_client/src/tso.rs:88]: Timestamp channel is dropped")"]
[2021/12/19 01:20:18.921 +08:00] [ERROR] [util.rs:419] [“request failed, retry”] [err_code=KV:PD:Unknown] [err=“Other(”[components/pd_client/src/tso.rs:88]: Timestamp channel is dropped")"]
[2021/12/19 01:20:18.943 +08:00] [ERROR] [kv.rs:729] [“KvService::batch_raft send response fail”] [err=RemoteStopped]
[2021/12/19 01:20:19.241 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 01:20:19.272 +08:00] [INFO] [tso.rs:148] [“TSO worker terminated”] [receiver_cause=None] [sender_cause=“Some(Grpc(RpcFinished(Some(RpcStatus { code: 14-UNAVAILABLE, message: “keepalive watchdog timeout”, details: [] }))))”]
[2021/12/19 01:20:19.272 +08:00] [ERROR] [util.rs:419] [“request failed, retry”] [err_code=KV:PD:Unknown] [err=“Other(”[components/pd_client/src/tso.rs:88]: Timestamp channel is dropped")"]
[2021/12/19 01:20:19.272 +08:00] [ERROR] [util.rs:419] [“request failed, retry”] [err_code=KV:PD:Unknown] [err=“Other(”[components/pd_client/src/tso.rs:88]: Timestamp channel is dropped")"]
[2021/12/19 01:20:19.272 +08:00] [ERROR] [util.rs:419] [“request failed, retry”] [err_code=KV:PD:Unknown] [err=“Other(”[components/pd_client/src/tso.rs:85]: TimestampRequest channel is closed")"]
[2021/12/19 01:20:19.272 +08:00] [ERROR] [util.rs:419] [“request failed, retry”] [err_code=KV:PD:Unknown] [err=“Other(”[components/pd_client/src/tso.rs:85]: TimestampRequest channel is closed")"]
[2021/12/19 01:20:19.272 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 01:20:19.275 +08:00] [WARN] [kv.rs:1108] [“call GetStoreSafeTS failed”] [err=Grpc(RemoteStopped)]
[2021/12/19 01:20:19.309 +08:00] [INFO] [kv.rs:699] [“batch_raft RPC is called, new gRPC stream established”]
[2021/12/19 01:20:19.318 +08:00] [ERROR] [kv.rs:729] [“KvService::batch_raft send response fail”] [err=RemoteStopped]
[2021/12/19 01:20:19.318 +08:00] [ERROR] [raft_client.rs:407] [“connection aborted”] [addr=192.168.0.66:20160] [receiver_err=“Some(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: “keepalive watchdog timeout”, details: [] }))”] [sink_error=“Some(RpcFinished(Some(RpcStatus { code: 14-UNAVAILABLE, message: “keepalive watchdog timeout”, details: [] })))”] [store_id=443528]
[2021/12/19 01:20:19.318 +08:00] [ERROR] [raft_client.rs:407] [“connection aborted”] [addr=192.168.0.33:20160] [receiver_err=“Some(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: “keepalive watchdog timeout”, details: [] }))”] [sink_error=“Some(RpcFinished(Some(RpcStatus { code: 14-UNAVAILABLE, message: “keepalive watchdog timeout”, details: [] })))”] [store_id=53826365]
[2021/12/19 01:20:19.324 +08:00] [ERROR] [kv.rs:722] [“dispatch raft msg from gRPC to raftstore fail”] [err=Grpc(RpcFinished(None))]
[2021/12/19 01:20:19.325 +08:00] [ERROR] [kv.rs:722] [“dispatch raft msg from gRPC to raftstore fail”] [err=Grpc(RpcFinished(None))]
[2021/12/19 01:20:19.326 +08:00] [ERROR] [kv.rs:729] [“KvService::batch_raft send response fail”] [err=RemoteStopped]
[2021/12/19 01:20:19.327 +08:00] [INFO] [kv.rs:699] [“batch_raft RPC is called, new gRPC stream established”]
[2021/12/19 01:20:19.327 +08:00] [ERROR] [kv.rs:722] [“dispatch raft msg from gRPC to raftstore fail”] [err=Grpc(RpcFinished(None))]
[2021/12/19 01:20:19.327 +08:00] [WARN] [kv.rs:1108] [“call GetStoreSafeTS failed”] [err=Grpc(RemoteStopped)]
[2021/12/19 01:20:19.327 +08:00] [ERROR] [kv.rs:729] [“KvService::batch_raft send response fail”] [err=RemoteStopped]
[2021/12/19 01:20:19.327 +08:00] [INFO] [kv.rs:699] [“batch_raft RPC is called, new gRPC stream established”]
[2021/12/19 01:20:19.334 +08:00] [ERROR] [kv.rs:729] [“KvService::batch_raft send response fail”] [err=RemoteStopped]
[2021/12/19 01:20:19.357 +08:00] [INFO] [kv.rs:699] [“batch_raft RPC is called, new gRPC stream established”]
[2021/12/19 01:20:19.363 +08:00] [WARN] [apply.rs:641] ["[store 435001] handle ready 1 committed entries"] [takes=110699]
[2021/12/19 01:20:19.363 +08:00] [WARN] [apply.rs:641] ["[store 435001] handle ready 1 committed entries"] [takes=189014]
[2021/12/19 01:20:19.430 +08:00] [INFO] [] [“New connected subchannel at 0x7f586fac35d0 for subchannel 0x7f578c64b1c0”]
[2021/12/19 01:20:19.439 +08:00] [WARN] [store.rs:782] ["[store 435001] handle 124 pending peers include 2 ready, 1 entries, 44 messages and 0 snapshots"] [takes=219625]
[2021/12/19 01:20:19.439 +08:00] [WARN] [store.rs:782] ["[store 435001] handle 107 pending peers include 2 ready, 1 entries, 30 messages and 0 snapshots"] [takes=219194]
[2021/12/19 01:20:19.440 +08:00] [INFO] [kv.rs:699] [“batch_raft RPC is called, new gRPC stream established”]
[2021/12/19 01:20:19.483 +08:00] [INFO] [] [“New connected subchannel at 0x7f4a1b10ba10 for subchannel 0x7f57ba8b6080”]
[2021/12/19 01:20:19.524 +08:00] [ERROR] [raft_client.rs:707] [“connection abort”] [addr=192.168.0.55:20160] [store_id=441829]
[2021/12/19 01:20:19.524 +08:00] [ERROR] [raft_client.rs:707] [“connection abort”] [addr=192.168.0.22:20160] [store_id=42827561]
[2021/12/19 01:20:19.524 +08:00] [ERROR] [raft_client.rs:707] [“connection abort”] [addr=192.168.0.66:20160] [store_id=443528]
[2021/12/19 01:20:19.524 +08:00] [ERROR] [raft_client.rs:707] [“connection abort”] [addr=192.168.0.33:20160] [store_id=53826365]
[2021/12/19 01:20:19.558 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.11:2379]
[2021/12/19 01:20:19.558 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.11:2379]
[2021/12/19 01:20:19.577 +08:00] [INFO] [util.rs:668] [“connected to PD member”] [endpoints=http://192.168.0.11:2379]
[2021/12/19 01:20:19.577 +08:00] [INFO] [util.rs:668] [“connected to PD member”] [endpoints=http://192.168.0.11:2379]
[2021/12/19 01:20:19.637 +08:00] [INFO] [util.rs:202] [“heartbeat sender and receiver are stale, refreshing …”]
[2021/12/19 01:20:19.663 +08:00] [INFO] [util.rs:230] [“update pd client”] [via=] [leader=http://192.168.0.11:2379] [prev_via=] [prev_leader=http://192.168.0.11:2379]
[2021/12/19 01:20:19.715 +08:00] [INFO] [util.rs:357] [“trying to update PD client done”] [spend=391.041556ms]
[2021/12/19 01:20:19.715 +08:00] [INFO] [util.rs:202] [“heartbeat sender and receiver are stale, refreshing …”]
[2021/12/19 01:20:19.715 +08:00] [INFO] [util.rs:230] [“update pd client”] [via=] [leader=http://192.168.0.11:2379] [prev_via=] [prev_leader=http://192.168.0.11:2379]
[2021/12/19 01:20:19.715 +08:00] [INFO] [util.rs:357] [“trying to update PD client done”] [spend=758.877831ms]
[2021/12/19 01:20:19.734 +08:00] [ERROR] [client.rs:575] [“failed to send heartbeat”] [err_code=KV:PD:gRPC] [err=“Grpc(RpcFinished(Some(RpcStatus { code: 14-UNAVAILABLE, message: “keepalive watchdog timeout”, details: [] })))”]
[2021/12/19 01:20:19.741 +08:00] [INFO] [tso.rs:148] [“TSO worker terminated”] [receiver_cause=None] [sender_cause=None]
/ERROR
[2021/12/19 03:33:15.416 +08:00] [INFO] [raft.rs:1092] [“became follower at term 13”] [term=13] [raft_id=54376867] [region_id=54376866]
[2021/12/19 03:34:56.400 +08:00] [INFO] [raft.rs:1092] [“became follower at term 0”] [term=0] [raft_id=54445117] [region_id=54291215]
[2021/12/19 03:34:56.400 +08:00] [INFO] [raft.rs:1092] [“became follower at term 8”] [term=8] [raft_id=54445117] [region_id=54291215]
[2021/12/19 03:34:57.519 +08:00] [INFO] [snap.rs:247] [“saving snapshot file”] [file=/tidb_ssd_data/tikv-20160/snap/rev_54291215_8_9_(default|lock|write).sst] [snap_key=54291215_8_9]
[2021/12/19 03:34:57.546 +08:00] [INFO] [raft.rs:2593] [“restored snapshot”] [snapshot_term=8] [snapshot_index=9] [last_term=8] [last_index=9] [commit=9] [raft_id=54445117] [region_id=54291215]
[2021/12/19 03:34:57.549 +08:00] [INFO] [region.rs:328] [“begin apply snap data”] [region_id=54291215]
[2021/12/19 03:34:58.418 +08:00] [INFO] [apply.rs:1954] [“exec ConfChangeV2”] [epoch=“conf_ver: 1872 version: 27069”] [kind=EnterJoint] [peer_id=54445117] [region_id=54291215]
[2021/12/19 03:34:58.428 +08:00] [INFO] [apply.rs:1394] [“execute admin command”] [command=“cmd_type: ChangePeerV2 change_peer_v2 {}”] [index=11] [term=8] [peer_id=54445117] [region_id=54291215]
[2021/12/19 03:34:58.428 +08:00] [INFO] [apply.rs:1954] [“exec ConfChangeV2”] [epoch=“conf_ver: 1874 version: 27069”] [kind=LeaveJoint] [peer_id=54445117] [region_id=54291215]
[2021/12/19 03:34:58.433 +08:00] [INFO] [apply.rs:1773] [“exec ConfChange”] [epoch=“conf_ver: 1876 version: 27069”] [type=RemoveNode] [peer_id=54445117] [region_id=54291215]
[2021/12/19 03:34:58.444 +08:00] [INFO] [apply.rs:2510] [“asking delegate to stop”] [source_region_id=54291215] [peer_id=54403255] [region_id=54142014]
[2021/12/19 03:34:58.448 +08:00] [INFO] [peer.rs:3778] [“require updating max ts”] [initial_status=38654759806] [region_id=54142014]
[2021/12/19 03:34:58.448 +08:00] [INFO] [peer.rs:2075] [“starts destroy”] [merged_by_target=true] [peer_id=54445117] [region_id=54291215]
[2021/12/19 03:34:58.448 +08:00] [INFO] [peer.rs:813] [“begin to destroy”] [peer_id=54445117] [region_id=54291215]
[2021/12/19 03:34:58.448 +08:00] [INFO] [peer_storage.rs:1638] [“finish clear peer meta”] [takes=68.502µs] [raft_key=1] [apply_key=1] [meta_key=1] [region_id=54291215]
[2021/12/19 03:34:58.451 +08:00] [INFO] [peer.rs:853] [“peer destroy itself”] [takes=2.858078ms] [peer_id=54445117] [region_id=54291215]
[2021/12/19 03:34:58.451 +08:00] [INFO] [router.rs:300] ["[region 54291215] shutdown mailbox"]
[2021/12/19 03:35:09.575 +08:00] [INFO] [peer.rs:219] [“create peer”] [peer_id=54445121] [region_id=54445118]
[2021/12/19 03:35:09.575 +08:00] [INFO] [raft.rs:1092] [“became follower at term 5”] [term=5] [raft_id=54445121] [region_id=54445118]
[2021/12/19 03:35:09.575 +08:00] [INFO] [raw_node.rs:315] [“RawNode created with id 54445121.”] [id=54445121] [raft_id=54445121] [region_id=54445118]
[2021/12/19 03:35:09.575 +08:00] [INFO] [raft.rs:1336] [“received a message with higher term from 54445119”] [“msg type”=MsgAppend] [message_term=6] [term=5] [from=54445119] [raft_id=54445121] [region_id=54445118]
[2021/12/19 03:35:09.575 +08:00] [INFO] [raft.rs:1092] [“became follower at term 6”] [term=6] [raft_id=54445121] [region_id=54445118]
[2021/12/19 03:35:20.006 +08:00] [INFO] [store.rs:2170] [“region is disconnected, remove snaps”] [snaps="[(SnapKey { region_id: 54291215, term: 8, idx: 9 }, false)]"] [region_id=54291215]
[2021/12/19 03:40:27.421 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 03:40:27.421 +08:00] [INFO] [] [“New connected subchannel at 0x7f577ee6f1a0 for subchannel 0x7f5875412bc0”]
[2021/12/19 03:50:27.423 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 03:50:27.424 +08:00] [INFO] [] [“New connected subchannel at 0x7f5780347190 for subchannel 0x7f578a03eb80”]
[2021/12/19 04:00:27.425 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 04:00:27.426 +08:00] [INFO] [] [“New connected subchannel at 0x7f577cdbd6d0 for subchannel 0x7f5875412bc0”]
[2021/12/19 04:10:27.427 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 04:10:27.429 +08:00] [INFO] [] [“New connected subchannel at 0x7f586fa2fb90 for subchannel 0x7f578a03eb80”]
[2021/12/19 03:35:09.575 +08:00] [INFO] [peer.rs:219] [“create peer”] [peer_id=54445121] [region_id=54445118]
[2021/12/19 03:35:09.575 +08:00] [INFO] [raft.rs:1092] [“became follower at term 5”] [term=5] [raft_id=54445121] [region_id=54445118]
[2021/12/19 03:35:09.575 +08:00] [INFO] [raw_node.rs:315] [“RawNode created with id 54445121.”] [id=54445121] [raft_id=54445121] [region_id=54445118]
[2021/12/19 03:35:09.575 +08:00] [INFO] [raft.rs:1336] [“received a message with higher term from 54445119”] [“msg type”=MsgAppend] [message_term=6] [term=5] [from=54445119] [raft_id=54445121] [region_id=54445118]
[2021/12/19 03:35:09.575 +08:00] [INFO] [raft.rs:1092] [“became follower at term 6”] [term=6] [raft_id=54445121] [region_id=54445118]
[2021/12/19 03:35:20.006 +08:00] [INFO] [store.rs:2170] [“region is disconnected, remove snaps”] [snaps="[(SnapKey { region_id: 54291215, term: 8, idx: 9 }, false)]"] [region_id=54291215]
[2021/12/19 03:40:27.421 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 03:40:27.421 +08:00] [INFO] [] [“New connected subchannel at 0x7f577ee6f1a0 for subchannel 0x7f5875412bc0”]
[2021/12/19 03:50:27.423 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 03:50:27.424 +08:00] [INFO] [] [“New connected subchannel at 0x7f5780347190 for subchannel 0x7f578a03eb80”]
[2021/12/19 04:00:27.425 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 04:00:27.426 +08:00] [INFO] [] [“New connected subchannel at 0x7f577cdbd6d0 for subchannel 0x7f5875412bc0”]
[2021/12/19 04:10:27.427 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 04:10:27.429 +08:00] [INFO] [] [“New connected subchannel at 0x7f586fa2fb90 for subchannel 0x7f578a03eb80”]
[2021/12/19 04:20:27.429 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 04:20:27.430 +08:00] [INFO] [] [“New connected subchannel at 0x7f5780d54250 for subchannel 0x7f5875412bc0”]
[2021/12/19 04:30:27.432 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 04:30:27.433 +08:00] [INFO] [] [“New connected subchannel at 0x7f5780314c80 for subchannel 0x7f578a03eb80”]
[2021/12/19 04:40:27.435 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 04:40:27.436 +08:00] [INFO] [] [“New connected subchannel at 0x7f578233d9d0 for subchannel 0x7f5875412bc0”]
[2021/12/19 04:50:27.437 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 05:40:27.448 +08:00] [INFO] [util.rs:544] [“connecting to PD endpoint”] [endpoints=http://192.168.0.66:2379]
[2021/12/19 08:56:39.002 +08:00] [ERROR] [kv.rs:729] [“KvService::batch_raft send response fail”] [err=RemoteStopped]
[2021/12/19 08:56:39.002 +08:00] [ERROR] [kv.rs:729] [“KvService::batch_raft send response fail”] [err=RemoteStopped]
[2021/12/19 08:56:39.002 +08:00] [ERROR] [kv.rs:729] [“KvService::batch_raft send response fail”] [err=RemoteStopped]
[2021/12/19 08:56:39.081 +08:00] [INFO] [mod.rs:179] [“Storage stopped.”]
[2021/12/19 08:56:39.110 +08:00] [INFO] [node.rs:419] [“stop raft store thread”] [store_id=435001]
[2021/12/19 08:56:39.110 +08:00] [INFO] [batch.rs:576] [“shutdown batch system apply”]
[2021/12/19 08:56:39.110 +08:00] [INFO] [router.rs:285] [“broadcasting shutdown”]
[2021/12/19 08:56:39.157 +08:00] [ERROR] [kv.rs:729] [“KvService::batch_raft send response fail”] [err=RemoteStopped]
[2021/12/19 08:56:39.157 +08:00] [ERROR] [kv.rs:729] [“KvService::batch_raft send response fail”] [err=RemoteStopped]
[2021/12/19 08:56:39.188 +08:00] [INFO] [batch.rs:589] [“batch system apply is stopped.”]
[2021/12/19 08:56:39.188 +08:00] [INFO] [batch.rs:576] [“shutdown batch system raftstore-435001”]
[2021/12/19 08:56:39.188 +08:00] [INFO] [router.rs:285] [“broadcasting shutdown”]
[2021/12/19 08:56:39.359 +08:00] [INFO] [batch.rs:589] [“batch system raftstore-435001 is stopped.”]
[2021/12/19 08:56:39.846 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:39.846 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:40.846 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:41.846 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:42.846 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:42.986 +08:00] [INFO] [store.rs:479] [“failed to schedule store tick, are we shutting down?”] [err="“SendError(…)”"] [tick=RaftEnginePurge]
[2021/12/19 08:56:42.986 +08:00] [INFO] [store.rs:479] [“failed to schedule store tick, are we shutting down?”] [err="“SendError(…)”"] [tick=PdStoreHeartbeat]
[2021/12/19 08:56:43.847 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:44.850 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:44.850 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:45.850 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:46.850 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:47.850 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:48.850 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:49.853 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:49.853 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:50.853 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:51.853 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:52.854 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:53.854 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:54.857 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:54.857 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:55.857 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:56.857 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:57.857 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:58.857 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:59.861 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:56:59.861 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:00.861 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:01.861 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:02.861 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:03.861 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:04.864 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:04.864 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:05.864 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:06.865 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:07.865 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:08.865 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:09.868 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:09.868 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:10.868 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:11.868 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:12.869 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:13.869 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:14.872 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:14.872 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:15.872 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:16.872 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:17.872 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:18.872 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:19.875 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:19.875 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:20.346 +08:00] [INFO] [store.rs:479] [“failed to schedule store tick, are we shutting down?”] [err="“SendError(…)”"] [tick=SnapGc]
[2021/12/19 08:57:20.875 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:21.876 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:22.876 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:23.876 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:24.879 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:24.879 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:25.879 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:26.879 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:27.879 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:29.882 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:30.883 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:31.883 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:32.883 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:33.883 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:34.886 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:34.886 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:35.886 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:36.886 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:37.887 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:38.887 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:39.890 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:39.890 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:40.890 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:41.890 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:42.890 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:43.891 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:44.894 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:44.894 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:45.894 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:46.894 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:47.894 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:48.894 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:49.897 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:49.897 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:50.897 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:51.898 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:52.898 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:53.898 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:54.901 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:54.901 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:55.901 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:56.901 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:57.901 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:58.901 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:59.905 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:57:59.905 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:58:00.905 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:58:01.905 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:58:02.905 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:58:03.905 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:58:04.908 +08:00] [ERROR] [pd.rs:426] [“failed to send store infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:58:04.908 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:58:05.908 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:58:06.909 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:58:07.909 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:58:08.909 +08:00] [ERROR] [pd.rs:441] [“failed to send split infos to pd worker”] [err=“channel has been closed”]
[2021/12/19 08:58:14.878 +08:00] [INFO] [lib.rs:80] [“Welcome to TiKV”]
[2021/12/19 08:58:14.879 +08:00] [INFO] [lib.rs:85] [“Release Version: 5.2.1”]

1 个赞

学习一下

这个后面的处理方式是什么啊,我现在也是这个情况,要么能拉起来,要么就下架,现在下架都下不了

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。