【 TiDB 使用环境】测试
【 TiDB 版本】 V5.1.4
【遇到的问题】tikv节点异常重启
【复现路径】
【问题现象及影响】
【附件】
<!--* 相关日志 和 [监控](https://metricstool.pingcap.com/)  系统日志 Jul 14 17:28:01 centos7 systemd: Started Session 279529 of user root. Jul 14 17:28:01 centos7 systemd: Started Session 279528 of user root. Jul 14 17:28:03 centos7 systemd: tikv-20160.service: main process exited, code=exited, status=1/FAILURE Jul 14 17:28:03 centos7 systemd: Unit tikv-20160.service entered failed state. Jul 14 17:28:03 centos7 systemd: tikv-20160.service failed. Jul 14 17:28:18 centos7 systemd: tikv-20160.service holdoff time over, scheduling restart. Jul 14 17:28:18 centos7 systemd: Stopped tikv service. Jul 14 17:28:18 centos7 systemd: Started tikv service. Jul 14 17:28:18 centos7 run_tikv.sh: sync ... Jul 14 17:28:18 centos7 run_tikv.sh: real#0110m0.030s Jul 14 17:28:18 centos7 run_tikv.sh: user#0110m0.000s Jul 14 17:28:18 centos7 run_tikv.sh: sys#0110m0.026s Jul 14 17:28:18 centos7 run_tikv.sh: ok Jul 14 17:28:32 centos7 systemd: tikv-20160.service: main process exited, code=exited, status=1/FAILURE Jul 14 17:28:32 centos7 systemd: Unit tikv-20160.service entered failed state. Jul 14 17:28:32 centos7 systemd: tikv-20160.service failed. Jul 14 17:28:47 centos7 systemd: tikv-20160.service holdoff time over, scheduling restart. Jul 14 17:28:47 centos7 systemd: Stopped tikv service. Jul 14 17:28:47 centos7 systemd: Started tikv service. Jul 14 17:28:47 centos7 run_tikv.sh: sync ... Jul 14 17:28:47 centos7 run_tikv.sh: real#0110m0.032s Jul 14 17:28:47 centos7 run_tikv.sh: user#0110m0.001s Jul 14 17:28:47 centos7 run_tikv.sh: sys#0110m0.026s Jul 14 17:28:47 centos7 run_tikv.sh: ok tikv日志 `[2022/07/14 17:28:08.614 +08:00] [FATAL] [lib.rs:463] ["called `Result::unwrap()` on an `Err` value: Error(\"json contains invalid UTF-8 characters\", line: 0, column: 0)"] [backtrace="stack backtrace:\ 0: tikv_util::set_panic_hook::{{closure}}\ at components/tikv_util/src/lib.rs:462\ 1: std::panicking::rust_panic_with_hook\ at library/std/src/panicking.rs:595\ 2: std::panicking::begin_panic_handler::{{closure}}\ at library/std/src/panicking.rs:497\ 3: std::sys_common::backtrace::__rust_end_short_backtrace\ at library/std/src/sys_common/backtrace.rs:141\ 4: rust_begin_unwind\ at library/std/src/panicking.rs:493\ 5: core::panicking::panic_fmt\ at library/core/src/panicking.rs:92\ 6: core::result::unwrap_failed\ at library/core/src/result.rs:1355\ 7: core::result::Result<T,E>::unwrap\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/result.rs:1037\ tidb_query_datatype::codec::mysql::json::serde::<impl alloc::string::ToString for tidb_query_datatype::codec::mysql::json::JsonRef>::to_string\ at components/tidb_query_datatype/src/codec/mysql/json/serde.rs:66\ 8: tidb_query_datatype::codec::mysql::json::comparison::<impl tidb_query_datatype::codec::mysql::json::JsonRef>::as_f64\ at components/tidb_query_datatype/src/codec/mysql/json/comparison.rs:55\ 9: tidb_query_datatype::codec::mysql::json::comparison::<impl core::cmp::PartialOrd for tidb_query_datatype::codec::mysql::json::JsonRef>::partial_cmp\ at components/tidb_query_datatype/src/codec/mysql/json/comparison.rs:147\ 10: tidb_query_expr::impl_compare::compare_json\ at components/tidb_query_datatype/src/codec/mysql/json/comparison.rs:65\ <tidb_query_expr::types::function::Arg<Arg0_,tidb_query_expr::types::function::Arg<Arg1_,tidb_query_expr::types::function::Null>> as tidb_query_expr::impl_compare::CompareJson_Fn<F>>::eval\ at components/tidb_query_expr/src/impl_compare.rs:23\ <tidb_query_expr::impl_compare::CompareJson_Evaluator<F> as tidb_query_expr::types::function::Evaluator>::eval\ at components/tidb_query_expr/src/impl_compare.rs:23\ <tidb_query_expr::types::function::ArgConstructor<A,E> as tidb_query_expr::types::function::Evaluator>::eval\ at components/tidb_query_expr/src/types/function.rs:259\ <tidb_query_expr::types::function::ArgConstructor<A,E> as tidb_query_expr::types::function::Evaluator>::eval\ at components/tidb_query_expr/src/types/function.rs:259\ tidb_query_expr::impl_compare::compare_json_fn_meta::run\ at components/tidb_query_expr/src/impl_compare.rs:23\ 11: tidb_query_expr::types::expr_eval::<impl tidb_query_expr::types::expr::RpnExpression>::eval_decoded\ at components/tidb_query_expr/src/types/expr_eval.rs:248\ 12: tidb_query_executors::selection_executor::BatchSelectionExecutor<Src>::handle_src_result\ at components/tidb_query_expr/src/types/expr_eval.rs:161\ <tidb_query_executors::selection_executor::BatchSelectionExecutor<Src> as tidb_query_executors::interface::BatchExecutor>::next_batch\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/selection_executor.rs:182\ <tidb_query_common::execute_stats::WithSummaryCollector<C,T> as tidb_query_executors::interface::BatchExecutor>::next_batch\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/interface.rs:108\ 13: <alloc::boxed::Box<T> as tidb_query_executors::interface::BatchExecutor>::next_batch\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/interface.rs:77\ tidb_query_executors::runner::BatchExecutorsRunner<SS>::internal_handle_request\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/runner.rs:534\ 14: tidb_query_executors::runner::BatchExecutorsRunner<SS>::handle_request::{{closure}}\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tidb_query_executors/src/runner.rs:416\ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80\ <tikv::coprocessor::dag::BatchDAGHandler as tikv::coprocessor::RequestHandler>::handle_request::__handle_request::{{closure}}\ at src/coprocessor/dag/mod.rs:104\ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80\ 15: <core::pin::Pin<P> as core::future::future::Future>::poll\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/future.rs:120\ <tikv::coprocessor::interceptors::deadline::DeadlineChecker<F> as core::future::future::Future>::poll\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/coprocessor/interceptors/deadline.rs:34\ <tikv::coprocessor::interceptors::tracker::Tracker<F> as core::future::future::Future>::poll\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/coprocessor/interceptors/tracker.rs:49\ 16: <tikv::coprocessor::interceptors::concurrency_limiter::ConcurrencyLimiter<PF,F> as core::future::future::Future>::poll\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/coprocessor/interceptors/concurrency_limiter.rs:103\ 17: tikv::coprocessor::endpoint::Endpoint<E>::handle_unary_request_impl::{{closure}}\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/coprocessor/endpoint.rs:427\ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80\ <resource_metering::cpu::future_ext::InTags<T> as core::future::future::Future>::poll\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/resource_metering/src/cpu/future_ext.rs:30\ tikv::read_pool::ReadPoolHandle::spawn_handle::{{closure}}\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/read_pool.rs:145\ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80\ 18: tikv::read_pool::ReadPoolHandle::spawn::{{closure}}\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/read_pool.rs:121\ <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80\ 19: <yatp::task::future::Runner as yatp::pool::runner::Runner>::handle\ at /rust/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/future.rs:261\ 20: <tikv_util::yatp_pool::YatpPoolRunner<T> as yatp::pool::runner::Runner>::handle\ at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/yatp_pool/mod.rs:104\ <yatp::queue::multilevel::MultilevelRunner<R> as yatp::pool::runner::Runner>::handle\ at /rust/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue/multilevel.rs:245\ yatp::pool::worker::WorkerThread<T,R>::run\ at /rust/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48\ yatp::pool::builder::LazyBuilder<T>::build::{{closure}}\ at /rust/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91\ std::sys_common::backtrace::__rust_begin_short_backtrace\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/sys_common/backtrace.rs:125\ 21: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/thread/mod.rs:474\ <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:344\ std::panicking::try::do_call\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:379\ std::panicking::try\ at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:343\ std:` 还请麻烦帮忙分析下哪里的问题，如果还需要别的信息我再上传请提供各个组件的 version 信息，如 cdc/tikv，可通过执行 cdc version/tikv-server --version 获取。
TiKV
Release Version: 5.1.4
Edition: Community
Git Commit Hash: 80ce9341cd9346bf5a956effd15ccf77deb96a0a
Git Commit Branch: heads/refs/tags/v5.1.4
UTC Build Time: 2022-02-22 03:36:42
Rust Version: rustc 1.53.0-nightly (16bf626a3 2021-04-14)
Enable Features: jemalloc mem-profiling portable sse protobuf-codec test-engines-rocksdb cloud-aws cloud-gcp
Profile: dist_release
:BatchExecutorsRunner::internal_handle_request
:Builder::spawn_unchecked::{{closure}}::{{closure}}