【 TiDB 使用环境】生产环境
【 TiDB 版本】V4.0.9
【复现路径】做过哪些操作出现的问题
【遇到的问题:TiKV异常crash重启后恢复正常】
tikv错误日志:
[FATAL] [lib.rs:482] ["index out of bounds: the len is 6 but the index is 6"] [backtrace="stack backtrace:\n 0: tikv_util::set_panic_hook::{{closure}}\n at c
omponents/tikv_util/src/lib.rs:481\n 1: std::panicking::rust_panic_with_hook\n at src/libstd/panicking.rs:475\n 2: rust_begin_unwind\n at src/libstd/panicking.rs:375\n 3: cor
e::panicking::panic_fmt\n at src/libcore/panicking.rs:84\n 4: core::panicking::panic_bounds_check\n at src/libcore/panicking.rs:62\n 5: <usize as core::slice::SliceIndex<[T]>>:
:index_mut\n at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/slice/mod.rs:2806\n core::slice::<impl core::ops::index::IndexMut<I> for [T]>::index_mut\n at /rust
c/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/slice/mod.rs:2661\n <alloc::vec::Vec<T> as core::ops::index::IndexMut<I>>::index_mut\n at /rustc/0de96d37fbcc54978458c18f5067cd98176
69bc8/src/liballoc/vec.rs:1880\n tokio_timer::wheel::Wheel<T>::insert\n at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/wheel/mod.rs:114\n tokio_timer::timer
::Timer<T,N>::add_entry\n at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/timer/mod.rs:324\n 6: tokio_timer::timer::Timer<T,N>::process_queue\n at /rust/
registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/timer/mod.rs:301\n 7: <tokio_timer::timer::Timer<T,N> as tokio_executor::park::Park>::park\n at /rust/registry/src/github.com-
1ecc6299db9ec823/tokio-timer-0.2.11/src/timer/mod.rs:361\n tokio_timer::timer::Timer<T,N>::turn\n at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/timer/mod.rs:256
\n 8: tikv_util::timer::start_global_steady_timer::{{closure}}\n at components/tikv_util/src/timer.rs:192\n 9: std::sys_common::backtrace::__rust_begin_short_backtrace\n at /ru
stc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/sys_common/backtrace.rs:136\n 10: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\n at /rustc/0de96d37fbcc54978458c18f5
067cd9817669bc8/src/libstd/thread/mod.rs:469\n 11: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/lib
std/panic.rs:318\n 12: std::panicking::try::do_call\n at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panicking.rs:292\n std::panicking::try\n at /rustc/0de96d3
7fbcc54978458c18f5067cd9817669bc8//src/libpanic_unwind/lib.rs:78\n std::panic::catch_unwind\n at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:394\n std::thread
::Builder::spawn_unchecked::{{closure}}\n at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:468\n core::ops::function::FnOnce::call_once{{vtable.shim}}\n
at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/ops/function.rs:232\n 13: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\n at /rustc/0de96d37fbcc54978
458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\n 14: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\n at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/
boxed.rs:1022\n std::sys_common::thread::start_thread\n at src/libstd/sys_common/thread.rs:13\n std::sys::unix::thread::Thread::new::thread_start\n at src/libstd/sys/unix
/thread.rs:80\n 15: start_thread\n 16: __clone\n"] [location=/rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/slice/mod.rs:2806] [thread_name=steady-timer]
操作系统日志:
systemd: tikv-20160.service: main process exited, code=exited, status=1/FAILURE
systemd: Unit tikv-20160.service entered failed state.
systemd: tikv-20160.service failed.
systemd: tikv-20160.service holdoff time over, scheduling restart.
systemd: Started tikv service.
systemd: Starting tikv service...
run_tikv.sh: sync ...
run_tikv.sh: real#0110m0.035s
run_tikv.sh: user#0110m0.000s
run_tikv.sh: sys#0110m0.029s
run_tikv.sh: ok
tikv.log (3.7 KB)
【资源配置】 3TiKV 43核 256G 3tidb 3*pd
【附件:截图/日志/监控】