tikv 7.5.1 异常重启

田帅萌7 · 2024 年5 月 4 日 09:43

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】7.5.1
【复现路径】做过哪些操作出现的问题
【遇到的问题：问题现象及影响】
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【附件：截图/日志/监控】

tikv异常重启日志如下
[2024/05/04 17:25:35.249 +08:00] [FATAL] [lib.rs:510] [“called Result::unwrap() on an Err value: Custom { kind: Uncategorized, error: "fdatasync" }”] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n at /workspace/source/tikv/components/tikv_util/src/lib.rs:509:18\n 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2032:9\n std::panicking::rust_panic_with_hook\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:692:13\n 2: std::panicking::begin_panic_handler::{{closure}}\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:579:13\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:137:18\n 4: rust_begin_unwind\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:575:5\n 5: core::panicking::panic_fmt\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:65:14\n 6: core::result::unwrap_failed\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1791:5\n 7: core::result::Result<T,E>::unwrap\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1113:23\n raft_engine::file_pipe_log::log_file::LogFileWriter::sync\n at /workspace/.cargo/git/checkouts/raft-engine-35ec7b0b2c07ddd2/e505d63/src/file_pipe_log/log_file.rs:125:9\n 8: raft_engine::file_pipe_log::pipe::SinglePipe::sync\n at /workspace/.cargo/git/checkouts/raft-engine-35ec7b0b2c07ddd2/e505d63/src/file_pipe_log/pipe.rs:401:9\n <raft_engine::file_pipe_log::pipe::DualPipes as raft_engine::pipe_log::PipeLog>::sync\n at /workspace/.cargo/git/checkouts/raft-engine-35ec7b0b2c07ddd2/e505d63/src/file_pipe_log/pipe.rs:511:9\n raft_engine::engine::Engine<F,P>::write\n at /workspace/.cargo/git/checkouts/raft-engine-35ec7b0b2c07ddd2/e505d63/src/engine.rs:177:21\n 9: <raft_log_engine::engine::RaftLogEngine as engine_traits::raft_engine::RaftEngine>::consume_and_shrink\n at /workspace/source/tikv/components/raft_log_engine/src/engine.rs:671:9\n 10: raftstore::store::async_io::write::Worker<EK,ER,N,T>::write_to_db\n at /workspace/source/tikv/components/raftstore/src/store/async_io/write.rs:754:17\n 11: raftstore::store::async_io::write::Worker<EK,ER,N,T>::run\n at /workspace/source/tikv/components/raftstore/src/store/async_io/write.rs:655:13\n raftstore::store::async_io::write::StoreWriters<EK,ER>::increase_to::{{closure}}::{{closure}}\n at /workspace/source/tikv/components/raftstore/src/store/async_io/write.rs:1036:33\n <std::Builder as tikv_util::sys::StdThreadBuildWrapper>::spawn_wrapper::{{closure}}\n at /workspace/source/tikv/components/tikv_util/src/sys/thread.rs:438:13\n std::sys_common::backtrace::rust_begin_short_backtrace\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:121:18\n 12: std::Builder::spawn_unchecked::{{closure}}::{{closure}}\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:551:17\n <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9\n std::panicking::try::do_call\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:483:40\n std::panicking::try\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:447:19\n std::panic::catch_unwind\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137:14\n std::Builder::spawn_unchecked::{{closure}}\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:550:30\n core::ops::function::FnOnce::call_once{{vtable.shim}}\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:513:5\n 13: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9\n <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9\n std::sys::unix::thread_start\n at /root/.rustup/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/st
[2024/05/04 17:26:00.663 +08:00] [INFO] [lib.rs:88] [“Welcome to TiKV”] [thread_id=0x5]

xfworld · 2024 年5 月 5 日 01:48

kind: Uncategorized, error: “fdatasync”

看起来是刷盘失败了，这个错误有点严重了

WalterWj · 2024 年5 月 6 日 06:03

感觉可以看看系统日志。是不是盘坏了？

田帅萌7 · 2024 年5 月 6 日 06:10

dmesg 没有明显error 我找idc同学看看

WalterWj · 2024 年5 月 7 日 09:13

idc 的同学有啥发现没。

田帅萌7 · 2024 年5 月 7 日 09:14

没～我们在观察观察吧

田帅萌7 · 2024 年5 月 7 日 09:15

他们把日志给厂商了硬件+系统也没看出来啥。在观察观察吧内核层面的log 没打印后面开一下

WalterWj · 2024 年5 月 7 日 09:19

好吧我看看能不能摇个 kv 研发老师看下错误堆栈，是不是也判断是硬件触发吧。

田帅萌7 · 2024 年5 月 7 日 09:20

也不着急这么久都没继续报错。。应该不是咱tidb的事。

WalterWj · 2024 年5 月 7 日 09:27

优秀。

pingyu · 2024 年5 月 12 日 10:54

libc::fdatasync 失败，应该是盘的问题。

Panic 的位置在这里：

github.com

tikv/raft-engine/blob/e505d631c8c6d63f7fc63d83ea6e8fb88cf970a5/src/file_pipe_log/log_file.rs#L125


      
                          });
                      e
                  })?;
                  self.written = new_written;
                  Ok(())
              }
          
              pub fn sync(&mut self) -> IoResult<()> {
                  let _t = StopWatch::new(&*LOG_SYNC_DURATION_HISTOGRAM);
                  // Panic if sync fails, in case of data loss.
                  self.handle.sync().unwrap();
                  Ok(())
              }
          
              #[inline]
              pub fn offset(&self) -> usize {
                  self.written
              }
          }
          
          /// Build a file reader.

里边就是调用 fdatasync：

github.com

tikv/raft-engine/blob/cd1533d79cfea02d005b458bd70fd9675846c9c6/src/env/log_fd/unix.rs#L170


      
                      .map_err(|e| from_nix_error(e, "lseek"))
              }
          
              #[inline]
              fn sync(&self) -> IoResult<()> {
                  fail_point!("log_fd::sync::err", |_| {
                      Err(from_nix_error(nix::Error::EINVAL, "fp"))
                  });
                  #[cfg(target_os = "linux")]
                  {
                      nix::unistd::fdatasync(self.0).map_err(|e| from_nix_error(e, "fdatasync"))
                  }
                  #[cfg(not(target_os = "linux"))]
                  {
                      nix::unistd::fsync(self.0).map_err(|e| from_nix_error(e, "fsync"))
                  }
              }
          }
          
          impl Drop for LogFd {
              fn drop(&mut self) {