【 TiDB 使用环境】生产环境
【 TiDB 版本】v5.4.2
【复现路径】无
【遇到的问题:问题现象及影响】
业务低峰期,系统运行好好的,突然某一tikv节点出错后自己退出。导致系统崩溃,出现大量平时不是慢查询的慢查询。看上去像是tidb程序出什么BUG了?
报错如下:
tikv-20160.service: main process exited, code=exited, status=1/FAILURE
[2023/07/24 18:30:55.854 +08:00] [FATAL] [lib.rs:465] [“elapsed=5717168819; when=5717168817”] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/lib.rs:464:18\n 1: std::panicking::rust_panic_with_hook\n at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:626:17\n 2: std::panicking::begin_panic_handler::{{closure}}\n at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:519:13\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys_common/backtrace.rs:141:18\n 4: rust_begin_unwind\n at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:515:5\n 5: std::panicking::begin_panic_fmt\n at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:457:5\n 6: tokio_timer::Wheel::poll\n 7: tokio_timer:
:Timer<T,N>::process\n at /rust/git/checkouts/tokio-8e927faba632ed16/e8ac149/tokio-timer/src/timer/mod.rs:272:33\n 8: tokio_timer:
:Timer<T,N>::turn\n 9: tikv_util:
:start_global_steady_timer::{{closure}}\n at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/timer.rs:196:17\n 10: std::sys_common::backtrace::__rust_begin_short_backtrace\n at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys_common/backtrace.rs:125:18\n 11: std:
:Builder::spawn_unchecked::{{closure}}::{{closure}}\n at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/thread/mod.rs:476:17\n 12: <std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panic.rs:347:9\n 13: std::panicking::try::do_call\n at /rustc/2f
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【附件:截图/日志/监控】
系统日志:
tikv.log
监控图: