tikv节点异常重启

【 TiDB 使用环境】测试
【 TiDB 版本】 V5.1.4
【遇到的问题】tikv节点异常重启
【复现路径】
【问题现象及影响】

【附件】

请提供各个组件的 version 信息,如 cdc/tikv,可通过执行 cdc version/tikv-server --version 获取。
TiKV
Release Version: 5.1.4
Edition: Community
Git Commit Hash: 80ce9341cd9346bf5a956effd15ccf77deb96a0a
Git Commit Branch: heads/refs/tags/v5.1.4
UTC Build Time: 2022-02-22 03:36:42
Rust Version: rustc 1.53.0-nightly (16bf626a3 2021-04-14)
Enable Features: jemalloc mem-profiling portable sse protobuf-codec test-engines-rocksdb cloud-aws cloud-gcp
Profile: dist_release

<!--* 相关日志 和 [监控](https://metricstool.pingcap.com/) ![image|690x231](upload://mb4W1LHvOCg2T2tkmsRaoWGROhg.png) 系统日志 Jul 14 17:28:01 centos7 systemd: Started Session 279529 of user root. Jul 14 17:28:01 centos7 systemd: Started Session 279528 of user root. Jul 14 17:28:03 centos7 systemd: tikv-20160.service: main process exited, code=exited, status=1/FAILURE Jul 14 17:28:03 centos7 systemd: Unit tikv-20160.service entered failed state. Jul 14 17:28:03 centos7 systemd: tikv-20160.service failed. Jul 14 17:28:18 centos7 systemd: tikv-20160.service holdoff time over, scheduling restart. Jul 14 17:28:18 centos7 systemd: Stopped tikv service. Jul 14 17:28:18 centos7 systemd: Started tikv service. Jul 14 17:28:18 centos7 run_tikv.sh: sync ... Jul 14 17:28:18 centos7 run_tikv.sh: real#0110m0.030s Jul 14 17:28:18 centos7 run_tikv.sh: user#0110m0.000s Jul 14 17:28:18 centos7 run_tikv.sh: sys#0110m0.026s Jul 14 17:28:18 centos7 run_tikv.sh: ok Jul 14 17:28:32 centos7 systemd: tikv-20160.service: main process exited, code=exited, status=1/FAILURE Jul 14 17:28:32 centos7 systemd: Unit tikv-20160.service entered failed state. Jul 14 17:28:32 centos7 systemd: tikv-20160.service failed. Jul 14 17:28:47 centos7 systemd: tikv-20160.service holdoff time over, scheduling restart. Jul 14 17:28:47 centos7 systemd: Stopped tikv service. Jul 14 17:28:47 centos7 systemd: Started tikv service. Jul 14 17:28:47 centos7 run_tikv.sh: sync ... Jul 14 17:28:47 centos7 run_tikv.sh: real#0110m0.032s Jul 14 17:28:47 centos7 run_tikv.sh: user#0110m0.001s Jul 14 17:28:47 centos7 run_tikv.sh: sys#0110m0.026s Jul 14 17:28:47 centos7 run_tikv.sh: ok tikv日志 `[2022/07/14 17:28:08.614 +08:00] [FATAL] [lib.rs:463] ["called `Result::unwrap()` on an `Err` value: Error(\"json contains invalid UTF-8 characters\", line: 0, column: 0)"] [backtrace="stack backtrace:\ 0: tikv_util::set_panic_hook::{{closure}}\ at components/tikv_util/src/lib.rs:462\ 1: std::panicking::rust_panic_with_hook\ at library/std/src/panicking.rs:595\ 2: std::panicking::begin_panic_handler::{{closure}}\ at library/std/src/panicking.rs:497\ 3: std::sys_common::backtrace::__rust_end_short_backtrace\ at library/std/src/sys_common/backtrace.rs:141\ 4: rust_begin_unwind\ at library/std/src/panicking.rs:493\ 5: core::panicking::panic_fmt\ at library/core/src/panicking.rs:92\ 6: core::result::unwrap_failed\ at library/core/src/result.rs:1355\ 7: core::result::Result<T,E>::unwrap\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/result.rs:1037\ tidb_query_datatype::codec::mysql::json::serde::<impl alloc::string::ToString for tidb_query_datatype::codec::mysql::json::JsonRef>::to_string\ at components/tidb_query_datatype/src/codec/mysql/json/serde.rs:66\ 8: tidb_query_datatype::codec::mysql::json::comparison::<impl tidb_query_datatype::codec::mysql::json::JsonRef>::as_f64\ at components/tidb_query_datatype/src/codec/mysql/json/comparison.rs:55\ 9: tidb_query_datatype::codec::mysql::json::comparison::<impl core::cmp::PartialOrd for tidb_query_datatype::codec::mysql::json::JsonRef>::partial_cmp\ at components/tidb_query_datatype/src/codec/mysql/json/comparison.rs:147\ 10: tidb_query_expr::impl_compare::compare_json\ at components/tidb_query_datatype/src/codec/mysql/json/comparison.rs:65\ <tidb_query_expr::types::function::Arg<Arg0_,tidb_query_expr::types::function::Arg<Arg1_,tidb_query_expr::types::function::Null>> as tidb_query_expr::impl_compare::CompareJson_Fn<F>>::eval\ at components/tidb_query_expr/src/impl_compare.rs:23\ <tidb_query_expr::impl_compare::CompareJson_Evaluator<F> as tidb_query_expr::types::function::Evaluator>::eval\ at components/tidb_query_expr/src/impl_compare.rs:23\ <tidb_query_expr::types::function::ArgConstructor<A,E> as tidb_query_expr::types::function::Evaluator>::eval\ at components/tidb_query_expr/src/types/function.rs:259\ <tidb_query_expr::types::function::ArgConstructor<A,E> as tidb_query_expr::types::function::Evaluator>::eval\ at components/tidb_query_expr/src/types/function.rs:259\ tidb_query_expr::impl_compare::compare_json_fn_meta::run\ at components/tidb_query_expr/src/impl_compare.rs:23\ 11: tidb_query_expr::types::expr_eval::<impl tidb_query_expr::types::expr::RpnExpression>::eval_decoded\ at components/tidb_query_expr/src/types/expr_eval.rs:248\ 12: tidb_query_executors::selection_executor::BatchSelectionExecutor<Src>::handle_src_result\ at components/tidb_query_expr/src/types/expr_eval.rs:161\ <tidb_query_executors::selection_executor::BatchSelectionExecutor<Src> as tidb_query_executors::interface::BatchExecutor>::next_batch\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/selection_executor.rs:182\ <tidb_query_common::execute_stats::WithSummaryCollector<C,T> as tidb_query_executors::interface::BatchExecutor>::next_batch\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/interface.rs:108\ 13: <alloc::boxed::Box<T> as tidb_query_executors::interface::BatchExecutor>::next_batch\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/interface.rs:77\ tidb_query_executors::runner::BatchExecutorsRunner<SS>::internal_handle_request\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/runner.rs:534\ 14: tidb_query_executors::runner::BatchExecutorsRunner<SS>::handle_request::{{closure}}\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/runner.rs:416\ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80\ <tikv::coprocessor::dag::BatchDAGHandler as tikv::coprocessor::RequestHandler>::handle_request::__handle_request::{{closure}}\ at src/coprocessor/dag/mod.rs:104\ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80\ 15: <core::pin::Pin<P> as core::future::future::Future>::poll\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/future.rs:120\ <tikv::coprocessor::interceptors::deadline::DeadlineChecker<F> as core::future::future::Future>::poll\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/coprocessor/interceptors/deadline.rs:34\ <tikv::coprocessor::interceptors::tracker::Tracker<F> as core::future::future::Future>::poll\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/coprocessor/interceptors/tracker.rs:49\ 16: <tikv::coprocessor::interceptors::concurrency_limiter::ConcurrencyLimiter<PF,F> as core::future::future::Future>::poll\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/coprocessor/interceptors/concurrency_limiter.rs:103\ 17: tikv::coprocessor::endpoint::Endpoint<E>::handle_unary_request_impl::{{closure}}\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/coprocessor/endpoint.rs:427\ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80\ <resource_metering::cpu::future_ext::InTags<T> as core::future::future::Future>::poll\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/resource_metering/src/cpu/future_ext.rs:30\ tikv::read_pool::ReadPoolHandle::spawn_handle::{{closure}}\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/read_pool.rs:145\ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80\ 18: tikv::read_pool::ReadPoolHandle::spawn::{{closure}}\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/read_pool.rs:121\ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80\ 19: <yatp::task::future::Runner as yatp::pool::runner::Runner>::handle\ at /rust/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/future.rs:261\ 20: <tikv_util::yatp_pool::YatpPoolRunner<T> as yatp::pool::runner::Runner>::handle\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/yatp_pool/mod.rs:104\ <yatp::queue::multilevel::MultilevelRunner<R> as yatp::pool::runner::Runner>::handle\ at /rust/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue/multilevel.rs:245\ yatp::pool::worker::WorkerThread<T,R>::run\ at /rust/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48\ yatp::pool::builder::LazyBuilder<T>::build::{{closure}}\ at /rust/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91\ std::sys_common::backtrace::__rust_begin_short_backtrace\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/sys_common/backtrace.rs:125\ 21: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/thread/mod.rs:474\ <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:344\ std::panicking::try::do_call\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:379\ std::panicking::try\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:343\ std:` 还请麻烦帮忙分析下哪里的问题,如果还需要别的信息我再上传

格式乱了 我补充下 系统日志
Jul 14 17:28:01 centos7 systemd: Started Session 279529 of user root.
Jul 14 17:28:01 centos7 systemd: Started Session 279528 of user root.
Jul 14 17:28:03 centos7 systemd: tikv-20160.service: main process exited, code=exited, status=1/FAILURE
Jul 14 17:28:03 centos7 systemd: Unit tikv-20160.service entered failed state.
Jul 14 17:28:03 centos7 systemd: tikv-20160.service failed.
Jul 14 17:28:18 centos7 systemd: tikv-20160.service holdoff time over, scheduling restart.
Jul 14 17:28:18 centos7 systemd: Stopped tikv service.
Jul 14 17:28:18 centos7 systemd: Started tikv service.
Jul 14 17:28:18 centos7 run_tikv.sh: sync …
Jul 14 17:28:18 centos7 run_tikv.sh: real#0110m0.030s
Jul 14 17:28:18 centos7 run_tikv.sh: user#0110m0.000s
Jul 14 17:28:18 centos7 run_tikv.sh: sys#0110m0.026s
Jul 14 17:28:18 centos7 run_tikv.sh: ok
Jul 14 17:28:32 centos7 systemd: tikv-20160.service: main process exited, code=exited, status=1/FAILURE
Jul 14 17:28:32 centos7 systemd: Unit tikv-20160.service entered failed state.
Jul 14 17:28:32 centos7 systemd: tikv-20160.service failed.
Jul 14 17:28:47 centos7 systemd: tikv-20160.service holdoff time over, scheduling restart.
Jul 14 17:28:47 centos7 systemd: Stopped tikv service.
Jul 14 17:28:47 centos7 systemd: Started tikv service.
Jul 14 17:28:47 centos7 run_tikv.sh: sync …
Jul 14 17:28:47 centos7 run_tikv.sh: real#0110m0.032s
Jul 14 17:28:47 centos7 run_tikv.sh: user#0110m0.001s
Jul 14 17:28:47 centos7 run_tikv.sh: sys#0110m0.026s
Jul 14 17:28:47 centos7 run_tikv.sh: ok

tikv日志
[2022/07/14 17:28:08.614 +08:00] [FATAL] [lib.rs:463] [“called Result::unwrap() on an Err value: Error("json contains invalid UTF-8 characters", line: 0, column: 0)”] [backtrace="stack backtrace:
0: tikv_util::set_panic_hook::{{closure}}
at components/tikv_util/src/lib.rs:462
1: std::panicking::rust_panic_with_hook
at library/std/src/panicking.rs:595
2: std::panicking::begin_panic_handler::{{closure}}
at library/std/src/panicking.rs:497
3: std::sys_common::backtrace::rust_end_short_backtrace
at library/std/src/sys_common/backtrace.rs:141
4: rust_begin_unwind
at library/std/src/panicking.rs:493
5: core::panicking::panic_fmt
at library/core/src/panicking.rs:92
6: core::result::unwrap_failed
at library/core/src/result.rs:1355
7: core::result::Result<T,E>::unwrap
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/result.rs:1037
tidb_query_datatype::codec::mysql::json::serde::::to_string
at components/tidb_query_datatype/src/codec/mysql/json/serde.rs:66
8: tidb_query_datatype::codec::mysql::json::comparison::::as_f64
at components/tidb_query_datatype/src/codec/mysql/json/comparison.rs:55
9: tidb_query_datatype::codec::mysql::json::comparison::::partial_cmp
at components/tidb_query_datatype/src/codec/mysql/json/comparison.rs:147
10: tidb_query_expr::impl_compare::compare_json
at components/tidb_query_datatype/src/codec/mysql/json/comparison.rs:65
<tidb_query_expr::types::function::Arg<Arg0
,tidb_query_expr::types::function::Arg<Arg1
,tidb_query_expr::types::function::Null>> as tidb_query_expr::impl_compare::CompareJson_Fn>::eval
at components/tidb_query_expr/src/impl_compare.rs:23
<tidb_query_expr::impl_compare::CompareJson_Evaluator as tidb_query_expr::types::function::Evaluator>::eval
at components/tidb_query_expr/src/impl_compare.rs:23
<tidb_query_expr::types::function::ArgConstructor<A,E> as tidb_query_expr::types::function::Evaluator>::eval
at components/tidb_query_expr/src/types/function.rs:259
<tidb_query_expr::types::function::ArgConstructor<A,E> as tidb_query_expr::types::function::Evaluator>::eval
at components/tidb_query_expr/src/types/function.rs:259
tidb_query_expr::impl_compare::compare_json_fn_meta::run
at components/tidb_query_expr/src/impl_compare.rs:23
11: tidb_query_expr::types::expr_eval::::eval_decoded
at components/tidb_query_expr/src/types/expr_eval.rs:248
12: tidb_query_executors::selection_executor::BatchSelectionExecutor::handle_src_result
at components/tidb_query_expr/src/types/expr_eval.rs:161
<tidb_query_executors::selection_executor::BatchSelectionExecutor as tidb_query_executors::interface::BatchExecutor>::next_batch
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/selection_executor.rs:182
<tidb_query_common::execute_stats::WithSummaryCollector<C,T> as tidb_query_executors::interface::BatchExecutor>::next_batch
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/interface.rs:108
13: <alloc::boxed::Box as tidb_query_executors::interface::BatchExecutor>::next_batch
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/interface.rs:77
tidb_query_executors::runner::BatchExecutorsRunner::internal_handle_request
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/runner.rs:534
14: tidb_query_executors::runner::BatchExecutorsRunner::handle_request::{{closure}}
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/runner.rs:416
<core::future::from_generator::GenFuture as core::future::future::Future>::poll
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80
<tikv::coprocessor::dag::BatchDAGHandler as tikv::coprocessor::RequestHandler>::handle_request::__handle_request::{{closure}}
at src/coprocessor/dag/mod.rs:104
<core::future::from_generator::GenFuture as core::future::future::Future>::poll
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80
15: <core::pin::Pin

as core::future::future::Future>::poll
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/future.rs:120
<tikv::coprocessor::interceptors::deadline::DeadlineChecker as core::future::future::Future>::poll
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/coprocessor/interceptors/deadline.rs:34
<tikv::coprocessor::interceptors::tracker::Tracker as core::future::future::Future>::poll
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/coprocessor/interceptors/tracker.rs:49
16: <tikv::coprocessor::interceptors::concurrency_limiter::ConcurrencyLimiter<PF,F> as core::future::future::Future>::poll
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/coprocessor/interceptors/concurrency_limiter.rs:103
17: tikv::coprocessor::endpoint::Endpoint::handle_unary_request_impl::{{closure}}
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/coprocessor/endpoint.rs:427
<core::future::from_generator::GenFuture as core::future::future::Future>::poll
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80
<resource_metering::cpu::future_ext::InTags as core::future::future::Future>::poll
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/resource_metering/src/cpu/future_ext.rs:30
tikv::read_pool::ReadPoolHandle::spawn_handle::{{closure}}
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/read_pool.rs:145
<core::future::from_generator::GenFuture as core::future::future::Future>::poll
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80
18: tikv::read_pool::ReadPoolHandle::spawn::{{closure}}
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/read_pool.rs:121
<core::future::from_generator::GenFuture as core::future::future::Future>::poll
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80
19: <yatp::task::future::Runner as yatp::pool::runner::Runner>::handle
at /rust/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/future.rs:261
20: <tikv_util::yatp_pool::YatpPoolRunner as yatp::pool::runner::Runner>::handle
at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/yatp_pool/mod.rs:104
<yatp::queue::multilevel::MultilevelRunner as yatp::pool::runner::Runner>::handle
at /rust/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue/multilevel.rs:245
yatp::pool::worker::WorkerThread<T,R>::run
at /rust/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48
yatp::pool::builder::LazyBuilder::build::{{closure}}
at /rust/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91
std::sys_common::backtrace::__rust_begin_short_backtrace
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/sys_common/backtrace.rs:125
21: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/thread/mod.rs:474
<std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:344
std::panicking::try::do_call
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:379
std::panicking::try
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:343
std:

tikv-20160节点,main程序启动失败,一直尝试启动,先检查空间和内存是否充足,这些日志只能看出来这些

CPU/内存/IO监控是否有异常,慢SQL也打捞下

内存 CPU 都正常,没有oom,也没有很严重的慢sql
日志中这一段,不知道为什么会有字符集的报错
[“called Result::unwrap() on an Err value: Error(“json contains invalid UTF-8 characters”, line: 0, column: 0)”]

tikv.log 没有异常吗?