tikv节点突然下线，状态变成Disconnected

[a398058068]

利用tiup启动无效，最后重启的机器好了。没在日志中分析出原因很慌。

tiup cluster start tidb -R tikv

查看集群状态还是Disconnected，而且日志没有任何新信息

tikv.log

[2021/04/25 14:08:54.829 +08:00] [INFO] [raft.rs:1064] ["became follower at term 32"] [term=32] [raft_id=23312] [region_id=23309]
[2021/04/25 14:08:54.833 +08:00] [INFO] [raft.rs:1064] ["became follower at term 36"] [term=36] [raft_id=23320] [region_id=23317]
[2021/04/25 14:08:56.801 +08:00] [INFO] [raft.rs:1064] ["became follower at term 67"] [term=67] [raft_id=1693] [region_id=1690]
[2021/04/25 14:08:56.829 +08:00] [INFO] [raft.rs:1064] ["became follower at term 31"] [term=31] [raft_id=23006] [region_id=23003]
[2021/04/25 14:10:17.522 +08:00] [WARN] [endpoint.rs:639] [error-response] [err="Key is locked (will clean up) primary_lock: 7480000000000000175F728000000000035112 lock_version: 424497910240772138 key: 7480000000000000175F728000000000035124 lock_ttl: 3925 txn_size: 5 min_commit_ts: 424497910240772139"]
[2021/04/25 14:10:17.522 +08:00] [WARN] [endpoint.rs:639] [error-response] [err="Key is locked (will clean up) primary_lock: 7480000000000000175F728000000000035112 lock_version: 424497910240772138 key: 7480000000000000175F728000000000035116 lock_ttl: 3925 txn_size: 13 min_commit_ts: 424497910240772139"]
[2021/04/25 14:11:35.215 +08:00] [INFO] [pd.rs:856] ["try to transfer leader"] [to_peer="id: 27018 store_id: 1"] [from_peer="id: 27020 store_id: 5"] [region_id=27017]
[2021/04/25 14:11:35.216 +08:00] [INFO] [peer.rs:2460] ["transfer leader"] [peer="id: 27018 store_id: 1"] [peer_id=27020] [region_id=27017]
[2021/04/25 14:11:35.216 +08:00] [INFO] [raft.rs:1719] ["[term 28] starts to transfer leadership to 27018"] [lead_transferee=27018] [term=28] [raft_id=27020] [region_id=27017]
[2021/04/25 14:11:35.216 +08:00] [INFO] [raft.rs:1732] ["sends MsgTimeoutNow to 27018 immediately as 27018 already has up-to-date log"] [lead_transferee=27018] [raft_id=27020] [region_id=27017]
[2021/04/25 14:11:35.220 +08:00] [INFO] [raft.rs:1307] ["received a message with higher term from 27018"] ["msg type"=MsgRequestVote] [message_term=29] [term=28] [from=27018] [raft_id=27020] [region_id=27017]
[2021/04/25 14:11:35.220 +08:00] [INFO] [raft.rs:1064] ["became follower at term 29"] [term=29] [raft_id=27020] [region_id=27017]
[2021/04/25 14:11:35.220 +08:00] [INFO] [raft.rs:1503] ["[logterm: 28, index: 599, vote: 0] cast vote for 27018 [logterm: 28, index: 599] at term 29"] ["msg type"=MsgRequestVote] [term=29] [msg_index=599] [msg_term=28] [from=27018] [vote=0] [log_index=599] [log_term=28] [raft_id=27020] [region_id=27017]
[2021/04/25 14:11:35.411 +08:00] [WARN] [endpoint.rs:639] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 27017, leader may Some(id: 27018 store_id: 1)\" not_leader { region_id: 27017 leader { id: 27018 store_id: 1 } }"]
[2021/04/25 14:11:35.495 +08:00] [WARN] [endpoint.rs:639] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 27017, leader may Some(id: 27018 store_id: 1)\" not_leader { region_id: 27017 leader { id: 27018 store_id: 1 } }"]
[2021/04/25 14:11:36.514 +08:00] [INFO] [pd.rs:856] ["try to transfer leader"] [to_peer="id: 25233 store_id: 4"] [from_peer="id: 25234 store_id: 5"] [region_id=25231]
[2021/04/25 14:11:36.518 +08:00] [INFO] [peer.rs:2460] ["transfer leader"] [peer="id: 25233 store_id: 4"] [peer_id=25234] [region_id=25231]
[2021/04/25 14:11:36.518 +08:00] [INFO] [raft.rs:1719] ["[term 97] starts to transfer leadership to 25233"] [lead_transferee=25233] [term=97] [raft_id=25234] [region_id=25231]
[2021/04/25 14:11:36.518 +08:00] [INFO] [raft.rs:1732] ["sends MsgTimeoutNow to 25233 immediately as 25233 already has up-to-date log"] [lead_transferee=25233] [raft_id=25234] [region_id=25231]
[2021/04/25 14:11:36.521 +08:00] [INFO] [raft.rs:1307] ["received a message with higher term from 25233"] ["msg type"=MsgRequestVote] [message_term=98] [term=97] [from=25233] [raft_id=25234] [region_id=25231]
[2021/04/25 14:11:36.521 +08:00] [INFO] [raft.rs:1064] ["became follower at term 98"] [term=98] [raft_id=25234] [region_id=25231]
[2021/04/25 14:11:36.521 +08:00] [INFO] [raft.rs:1503] ["[logterm: 97, index: 458946, vote: 0] cast vote for 25233 [logterm: 97, index: 458946] at term 98"] ["msg type"=MsgRequestVote] [term=98] [msg_index=458946] [msg_term=97] [from=25233] [vote=0] [log_index=458946] [log_term=97] [raft_id=25234] [region_id=25231]
[2021/04/25 14:11:37.329 +08:00] [WARN] [endpoint.rs:639] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 25231, leader may Some(id: 25233 store_id: 4)\" not_leader { region_id: 25231 leader { id: 25233 store_id: 4 } }"]
[2021/04/25 14:11:37.330 +08:00] [WARN] [endpoint.rs:639] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 25231, leader may Some(id: 25233 store_id: 4)\" not_leader { region_id: 25231 leader { id: 25233 store_id: 4 } }"]
[2021/04/25 14:11:37.330 +08:00] [WARN] [endpoint.rs:639] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 25231, leader may Some(id: 25233 store_id: 4)\" not_leader { region_id: 25231 leader { id: 25233 store_id: 4 } }"]
[2021/04/25 14:11:39.400 +08:00] [INFO] [raft.rs:2151] ["[term 18] received MsgTimeoutNow from 24840 and starts an election to get leadership."] [from=24840] [term=18] [raft_id=24842] [region_id=24839]
[2021/04/25 14:11:39.400 +08:00] [INFO] [raft.rs:1488] ["starting a new election"] [term=18] [raft_id=24842] [region_id=24839]
[2021/04/25 14:11:39.400 +08:00] [INFO] [raft.rs:1088] ["became candidate at term 19"] [term=19] [raft_id=24842] [region_id=24839]
[2021/04/25 14:11:39.400 +08:00] [INFO] [raft.rs:1242] ["broadcasting vote request"] [to="[24840, 24841]"] [log_index=26] [log_term=18] [term=19] [type=MsgRequestVote] [raft_id=24842] [region_id=24839]
[2021/04/25 14:11:39.407 +08:00] [INFO] [raft.rs:2022] ["received votes response"] [term=19] [type=MsgRequestVoteResponse] [approvals=2] [rejections=0] [from=24840] [vote=true] [raft_id=24842] [region_id=24839]
[2021/04/25 14:11:39.407 +08:00] [INFO] [raft.rs:1172] ["became leader at term 19"] [term=19] [raft_id=24842] [region_id=24839]
[2021/04/25 14:11:39.407 +08:00] [INFO] [peer.rs:3563] ["require updating max ts"] [initial_status=81604380122] [region_id=24839]
[2021/04/25 14:11:39.407 +08:00] [INFO] [pd.rs:986] ["succeed to update max timestamp"] [region_id=24839]
[2021/04/25 14:12:05.817 +08:00] [INFO] [raft.rs:2151] ["[term 10] received MsgTimeoutNow from 27030 and starts an election to get leadership."] [from=27030] [term=10] [raft_id=27032] [region_id=27029]
[2021/04/25 14:12:05.817 +08:00] [INFO] [raft.rs:1488] ["starting a new election"] [term=10] [raft_id=27032] [region_id=27029]
[2021/04/25 14:12:05.817 +08:00] [INFO] [raft.rs:1088] ["became candidate at term 11"] [term=11] [raft_id=27032] [region_id=27029]
[2021/04/25 14:12:05.817 +08:00] [INFO] [raft.rs:1242] ["broadcasting vote request"] [to="[27030, 27031]"] [log_index=11] [log_term=10] [term=11] [type=MsgRequestVote] [raft_id=27032] [region_id=27029]
[2021/04/25 14:12:05.826 +08:00] [INFO] [raft.rs:2022] ["received votes response"] [term=11] [type=MsgRequestVoteResponse] [approvals=2] [rejections=0] [from=27030] [vote=true] [raft_id=27032] [region_id=27029]
[2021/04/25 14:12:05.826 +08:00] [INFO] [raft.rs:1172] ["became leader at term 11"] [term=11] [raft_id=27032] [region_id=27029]
[2021/04/25 14:12:05.826 +08:00] [INFO] [peer.rs:3563] ["require updating max ts"] [initial_status=47244641472] [region_id=27029]
[2021/04/25 14:12:05.827 +08:00] [INFO] [pd.rs:986] ["succeed to update max timestamp"] [region_id=27029]
[2021/04/25 14:12:28.918 +08:00] [INFO] [pd.rs:856] ["try to transfer leader"] [to_peer="id: 17100 store_id: 1"] [from_peer="id: 17102 store_id: 5"] [region_id=17099]
[2021/04/25 14:12:28.921 +08:00] [INFO] [peer.rs:2460] ["transfer leader"] [peer="id: 17100 store_id: 1"] [peer_id=17102] [region_id=17099]
[2021/04/25 14:12:28.922 +08:00] [INFO] [raft.rs:1719] ["[term 228] starts to transfer leadership to 17100"] [lead_transferee=17100] [term=228] [raft_id=17102] [region_id=17099]
[2021/04/25 14:12:28.922 +08:00] [INFO] [raft.rs:1732] ["sends MsgTimeoutNow to 17100 immediately as 17100 already has up-to-date log"] [lead_transferee=17100] [raft_id=17102] [region_id=17099]
[2021/04/25 14:12:28.922 +08:00] [INFO] [raft.rs:1307] ["received a message with higher term from 17100"] ["msg type"=MsgRequestVote] [message_term=229] [term=228] [from=17100] [raft_id=17102] [region_id=17099]
[2021/04/25 14:12:28.922 +08:00] [INFO] [raft.rs:1064] ["became follower at term 229"] [term=229] [raft_id=17102] [region_id=17099]
[2021/04/25 14:12:28.922 +08:00] [INFO] [raft.rs:1503] ["[logterm: 228, index: 7531, vote: 0] cast vote for 17100 [logterm: 228, index: 7531] at term 229"] ["msg type"=MsgRequestVote] [term=229] [msg_index=7531] [msg_term=228] [from=17100] [vote=0] [log_index=7531] [log_term=228] [raft_id=17102] [region_id=17099]
[2021/04/25 14:12:30.219 +08:00] [INFO] [pd.rs:856] ["try to transfer leader"] [to_peer="id: 21908 store_id: 1"] [from_peer="id: 21910 store_id: 5"] [region_id=21907]
[2021/04/25 14:12:30.220 +08:00] [INFO] [peer.rs:2460] ["transfer leader"] [peer="id: 21908 store_id: 1"] [peer_id=21910] [region_id=21907]
[2021/04/25 14:12:30.220 +08:00] [INFO] [raft.rs:1719] ["[term 247] starts to transfer leadership to 21908"] [lead_transferee=21908] [term=247] [raft_id=21910] [region_id=21907]
[2021/04/25 14:12:30.220 +08:00] [INFO] [raft.rs:1732] ["sends MsgTimeoutNow to 21908 immediately as 21908 already has up-to-date log"] [lead_transferee=21908] [raft_id=21910] [region_id=21907]
[2021/04/25 14:12:30.223 +08:00] [INFO] [raft.rs:1307] ["received a message with higher term from 21908"] ["msg type"=MsgRequestVote] [message_term=248] [term=247] [from=21908] [raft_id=21910] [region_id=21907]
[2021/04/25 14:12:30.223 +08:00] [INFO] [raft.rs:1064] ["became follower at term 248"] [term=248] [raft_id=21910] [region_id=21907]
[2021/04/25 14:12:30.223 +08:00] [INFO] [raft.rs:1503] ["[logterm: 247, index: 7497, vote: 0] cast vote for 21908 [logterm: 247, index: 7497] at term 248"] ["msg type"=MsgRequestVote] [term=248] [msg_index=7497] [msg_term=247] [from=21908] [vote=0] [log_index=7497] [log_term=247] [raft_id=21910] [region_id=21907]
[2021/04/25 14:12:30.421 +08:00] [WARN] [endpoint.rs:639] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 21907, leader may Some(id: 21908 store_id: 1)\" not_leader { region_id: 21907 leader { id: 21908 store_id: 1 } }"]
[2021/04/25 14:12:32.491 +08:00] [INFO] [raft.rs:2151] ["[term 52] received MsgTimeoutNow from 14252 and starts an election to get leadership."] [from=14252] [term=52] [raft_id=14254] [region_id=14251]
[2021/04/25 14:12:32.491 +08:00] [INFO] [raft.rs:1488] ["starting a new election"] [term=52] [raft_id=14254] [region_id=14251]
[2021/04/25 14:12:32.491 +08:00] [INFO] [raft.rs:1088] ["became candidate at term 53"] [term=53] [raft_id=14254] [region_id=14251]
[2021/04/25 14:12:32.491 +08:00] [INFO] [raft.rs:1242] ["broadcasting vote request"] [to="[14252, 14253]"] [log_index=62] [log_term=52] [term=53] [type=MsgRequestVote] [raft_id=14254] [region_id=14251]
[2021/04/25 14:12:32.497 +08:00] [INFO] [raft.rs:2022] ["received votes response"] [term=53] [type=MsgRequestVoteResponse] [approvals=2] [rejections=0] [from=14252] [vote=true] [raft_id=14254] [region_id=14251]
[2021/04/25 14:12:32.497 +08:00] [INFO] [raft.rs:1172] ["became leader at term 53"] [term=53] [raft_id=14254] [region_id=14251]
[2021/04/25 14:12:32.497 +08:00] [INFO] [peer.rs:3563] ["require updating max ts"] [initial_status=227633267826] [region_id=14251]
[2021/04/25 14:12:32.498 +08:00] [INFO] [pd.rs:986] ["succeed to update max timestamp"] [region_id=14251]
[2021/04/25 14:12:40.533 +08:00] [INFO] [profiler.rs:163] ["starting cpu profiler"]
[2021/04/25 14:12:40.533 +08:00] [INFO] [mod.rs:323] ["start profiling 30 seconds with frequency 99 /s"]
[2021/04/25 14:12:46.870 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.16:20170: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:12:46.872 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.166:20170: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:12:52.084 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.41:20160: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:12:52.098 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.174:20160: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:12:54.652 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.111:60142: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:12:54.653 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.111:60146: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:12:56.122 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.16:53374: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:12:56.309 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.174:36066: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:12:56.542 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.34:50352: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:12:56.542 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.34:50354: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:12:56.545 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.34:50348: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:12:56.621 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.111:60148: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:12:57.758 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.41:58782: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:13:02.076 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.166:54446: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:13:04.629 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.111:60144: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:13:10.607 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.34:51354: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:13:10.607 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.34:50350: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:13:20.189 +08:00] [INFO] [<unknown>] ["ipv4:172.18.150.111:2379: Keepalive watchdog fired. Closing transport."]
[2021/04/25 14:27:47.084 +08:00] [INFO] [signal_handler.rs:20] ["receive signal 15, stopping server..."]
[2021/04/25 14:28:47.276 +08:00] [INFO] [lib.rs:89] ["Welcome to TiKV"]
[2021/04/25 14:28:47.278 +08:00] [INFO] [lib.rs:94] ["Release Version:   5.0.1"]
[2021/04/25 14:28:47.278 +08:00] [INFO] [lib.rs:94] ["Edition:           Community"]
[2021/04/25 14:28:47.278 +08:00] [INFO] [lib.rs:94] ["Git Commit Hash:   e26389a278116b2f61addfa9f15ca25ecf38bc80"]
[2021/04/25 14:28:47.278 +08:00] [INFO] [lib.rs:94] ["Git Commit Branch: heads/refs/tags/v5.0.1"]
[2021/04/25 14:28:47.278 +08:00] [INFO] [lib.rs:94] ["UTC Build Time:    2021-04-23 08:24:13"]
[2021/04/25 14:28:47.278 +08:00] [INFO] [lib.rs:94] ["Rust Version:      rustc 1.51.0-nightly (bc39d4d9c 2021-01-15)"]
[2021/04/25 14:28:47.278 +08:00] [INFO] [lib.rs:94] ["Enable Features:   jemalloc mem-profiling portable sse protobuf-codec test-engines-rocksdb cloud-aws cloud-gcp"]
[2021/04/25 14:28:47.278 +08:00] [INFO] [lib.rs:94] ["Profile:           dist_release"]
[2021/04/25 14:28:47.278 +08:00] [INFO] [mod.rs:65] ["memory limit in bytes: 17055675392, cpu cores quota: 8"]
[2021/04/25 14:28:47.281 +08:00] [WARN] [lib.rs:512] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2021/04/25 14:28:47.281 +08:00] [INFO] [config.rs:590] ["kernel parameters"] [value=0] [param=vm.swappiness]
[2021/04/25 14:28:47.285 +08:00] [WARN] [server.rs:1083] ["check: kernel"] [err="kernel parameters net.core.somaxconn got 128, expect 32768"]
[2021/04/25 14:28:47.285 +08:00] [WARN] [server.rs:1083] ["check: kernel"] [err="kernel parameters net.ipv4.tcp_syncookies got 1, expect 0"]
[2021/04/25 14:28:47.339 +08:00] [INFO] [util.rs:536] ["connecting to PD endpoint"] [endpoints=172.18.150.34:2379]
[2021/04/25 14:28:47.347 +08:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2021/04/25 14:28:47.350 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f1c3062f1b0 for subchannel 0x7f1c34417bc0"]

tikv_stderr.log

<jemalloc>: Malformed conf string
<jemalloc>: Malformed conf string
<jemalloc>: Malformed conf string
<jemalloc>: Malformed conf string
<jemalloc>: Malformed conf string
<jemalloc>: Malformed conf string

/var/log/messages 只有tiup启动时登陆的信息，确定没有OOM

[这道题我不会 (Lizhengyang@PingCAP)]

1.从日志看这个集群是 v5.0.1 ，请问下这个集群是从旧版本升级上来的还是直接上线的 v5.0.1 版本？
2.是否方便提供下完整的 tikv.log 信息？

[a398058068]

1.v5.0.0升级到的5.0.1
2.我重新修改了下tikv的日志。
另外忘记了个关键点。 我14:12从dashboard点击了实例性能分析，总共3个节点，其中1个也就是我提供日志的这台直接挂了。 而且从tiup启动没有任何反应，重启机器后恢复。14:27:47.084之后的日志已经是重启机器后的了。

[这道题我不会 (Lizhengyang@PingCAP)]

1.请问重启的这个 tikv 在时间段 14:12~ 14:27 内是否有日志产生？重启之后在 dashboard 中点击实例性能分析还会出现这个现象吗？
2.麻烦提供下集群的拓扑结构和机器的配置规格

[a398058068]

1.出问题之后没再去继续点实例性能分析。

14:12~ 14:27 时间内用下列命令启动过 tikv 没有任何反应和日志。 tifalsh给的配置比较低，一开始也怀疑过内存问题，按道理如果配置规格问题应该会有error log 或者 message 内应该有日志。

tiup cluster start tidb -N 172.18.150.193:20160

3台 tikv节点
8核 16G

[这道题我不会 (Lizhengyang@PingCAP)]

从日志中看，在点击 dashboard 性能实例分析之后（日志中 start profiling 30 seconds） 出现下面这些日志：

[INFO] [<unknown>] ["ipv4:172.18.150.111:2379: Keepalive watchdog fired. Closing transport."]

麻烦重新点击 dashboard 性能实例分析，看下是否还会出现问题。

[a398058068]

点废了，看了grafana内tidb-tikv-summary面板。没有任何异常

[a398058068]

刚重启了机器上线了。 跟之前情况一样。

[这道题我不会 (Lizhengyang@PingCAP)]

集群拓扑结构麻烦提供下: tiup cluster display {cluster-name}，并在关掉的这个 tikv 节点上执行下 dmesge -T |grep tikv-server，将结果反馈下。

[a398058068]

问题节点

确定下命令是，没有任何信息，我是在重启以后，恢复后正常执行的。

dmesg -T |grep tikv-server

[这道题我不会 (Lizhengyang@PingCAP)]

我这边测试了下并没有复现你的问题，麻烦核实下挂掉的这个 tikv 节点负载情况，看下是否由于负载太高导致在进行 CPU profile 时资源不足挂掉了。

[a398058068]

8核 2.4的load不算高吧

[这道题我不会 (Lizhengyang@PingCAP)]

如果方便的话，麻烦提供下 tikv 宕掉前后半个小时内监控数据，包括 overview/tidb/pd/tikv-details 四个监控面板数据，导出方式参考下：
https://metricstool.pingcap.com/#backup-with-dev-tools

[a398058068]

tidb.zip (2.4 MB)

[这道题我不会 (Lizhengyang@PingCAP)]

从监控数据中看在 tikv 重启之前并没有特别异常的指标，我这边没有复现出你的现象，不知道您这边是否将集群升级到最新的 v5.0.1 ，看下是否还存在这个问题。

[a398058068]

就是v5.0.1出现的这个问题。

[这道题我不会 (Lizhengyang@PingCAP)]

不好意思忘了这个集群已经是 v5.0.1 ，我们这边再分析下。

[这道题我不会 (Lizhengyang@PingCAP)]

请问你这边集群 tikv 节点所在的服务器上有无开启 THP ，另外操作系统版本是多少？

[a398058068]

没开启
系统是 centos7
Linux txy-150-180 3.10.0-1160.21.1.el7.x86_64 #1 SMP Tue Mar 16 18:28:22 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[GangShen]

@a398058068

[2021/04/25 14:27:47.084 +08:00] [INFO] [signal_handler.rs:20] [“receive signal 15, stopping server…”] [2021/04/25 14:28:47.276 +08:00] [INFO] [lib.rs:89] [“Welcome to TiKV”]

日志中看到 tikv-server 是接收到了操作系统的 signal 15 之后关闭进程的。这个问题目前还是能复现么，如果可以的话，可以参考下这篇文章的排查方式排查下看能不能定位问题：

使用 Systemtap 排查隐形 Killer