dm同步mysql数据到tidb，总是报错9005: Region is unavailable，看到报错peer is not leader for region 45049, leader may Some(id: 45050 sto

lukaiwo · 2021 年8 月 7 日 02:05

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：
【 TiDB 使用环境】v5.1.0

【概述】迁移一个比较大的库，之前迁移到80%的时候始终kvbusy，后面重启了几次tidb集群后，同步到99.86后直接报错region is unavailable了

【备份和数据迁移策略逻辑】
使用dm2.0.1迁移mysql5.7
【背景】做过哪些操作
重启过
【现象】业务和数据库现象

【问题】当前遇到的问题

【业务影响】

【TiDB 版本】
5.1.0
【附件】
tikverr日志
[FATAL] [lib.rs:462] [“called Result::unwrap() on an Err value: channel has been closed”] [backtrace="stack backtrace:
0: tikv_util::set_panic_hook::{{closure}}
at components/tikv_util/src/lib.rs:461
1: std::panicking::rust_panic_with_hook
at library/std/src/panicking.rs:595
2: std::panicking::begin_panic_handler::{{closure}}
at library/std/src/panicking.rs:497
3: std::sys_common::backtrace::__rust_end_short_backtrace
atlibrary/std/src/sys_common/backtrace.rs:141
4: rust_begin_unwind
at library/std/src/panicking.rs:493
5: core::panicking::panic_fmt
at library/core/src/panicking.rs:92
6: core::result::unwrap_failed
at library/core/src/result.rs:1355
7: core::result::Result<T,E>::unwrap
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/result.rs:1037
resolved_ts::endpoint::Endpoint<T,E,C>::build_scan_task::{{closure}}
at /home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tikv/components/resolved_ts/src/endpoint.rs:361
8: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/alloc/src/boxed.rs:1560
resolved_ts::scanner::ScannerPool<T,E>::spawn_task::{{closure}}
at /home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tikv/components/resolved_ts/src/scanner.rs:97
9: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80
tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/core.rs:173
tokio::loom::std::unsafe_cell::UnsafeCell::with_mut
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/loom/std/unsafe_cell.rs:14
tokio::runtime::task::core::Core<T,S>::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/core.rs:158
tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/harness.rs:107
core::ops::function::FnOnce::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/ops/function.rs:227
<std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:344
std::panicking::try::do_call
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:379
std::panicking::try
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:343
std::panic::catch_unwind
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:431
tokio::runtime::task::harness::Harness<T,S>::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/harness.rs:89
tokio::runtime::task::raw::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/raw.rs:104
10: tokio::runtime::task::raw::RawTask::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/raw.rs:66
tokio::runtime::task::Notified::run
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/mod.rs:169
tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:349
tokio::coop::with_budget::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/coop.rs:127
std::local::LocalKey::try_with
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/thread/local.rs:272
std::local::LocalKey::with
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/thread/local.rs:248
tokio::coop::with_budget
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/coop.rs:120
tokio::coop::budget
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/coop.rs:96
tokio::runtime::thread_pool::worker::Context::run_task
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:348
11: tokio::runtime::thread_pool::worker::Context::run
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:320
tokio::runtime::thread_pool::worker::run::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:305
tokio::macros::scoped_tls::ScopedKey::set
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/macros/scoped_tls.rs:63
tokio::runtime::thread_pool::worker::run
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:302
tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:281
<tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/blocking/task.rs:41
tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/core.rs:173
tokio::loom::std::unsafe_cell::UnsafeCell::with_mut
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/loom/std/unsafe_cell.rs:14
tokio::runtime::task::core::Core<T,S>::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/core.rs:158
tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/harness.rs:107
core::ops::function::FnOnce::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/ops/function.rs:227
<std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:344
std::panicking::try::do_call
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:379
std::panicking::try
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:343
std::panic::catch_unwind
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:431
tokio::runtime::task::harness::Harness<T,S>::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/harness.rs:89
tokio::runtime::task::raw::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/raw.rs:104
12: tokio::runtime::task::raw::RawTask::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/raw.rs:66
tokio::runtime::task::Notified::run
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/mod.rs:169
tokio::runtime::blocking::pool::Inner::run
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/blocking/pool.rs:250
tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/blocking/pool.rs:230
tokio::runtime::context::enter
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/context.rs:72
tokio::runtime::handle::Handle::enter
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/handle.rs:76
tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/blocking/pool.rs:229
std::sys_common::backtrace::__rust_begin_short_backtrace
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/sys_common/backtrace.rs:125
13: std::Builder::spawn_unchecked::{{closure}}::{{closure}}
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/thread/mod.rs:474
<std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:344
std::panicking::try::do_call
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:379
std::panicking::try
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:343
std::panic::catch_unwind
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:431
std::Builder::spawn_unchecked::{{closure}}
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/thread/mod.rs:473
core::ops::function::FnOnce::call_once{{vtable.shim}}
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/ops/function.rs:227
14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/alloc/src/boxed.rs:1546
<alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/alloc/src/boxed.rs:1546
std::sys::unix::thread_start
at library/std/src/sys/unix/thread.rs:71
15: start_thread
16: clone
"] [location=/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tikv/components/resolved_ts/src/endpoint.rs:367] [thread_name=inc-scan]
[2021/08/07 02:32:21.731 +08:00] [FATAL] [lib.rs:462] [“called Result::unwrap() on an Err value: channel has been closed”] [backtrace="stack backtrace:
0: tikv_util::set_panic_hook::{{closure}}
at components/tikv_util/src/lib.rs:461
1: std::panicking::rust_panic_with_hook
at library/std/src/panicking.rs:595
2: std::panicking::begin_panic_handler::{{closure}}
at library/std/src/panicking.rs:497
3: std::sys_common::backtrace::__rust_end_short_backtrace
atlibrary/std/src/sys_common/backtrace.rs:141
4: rust_begin_unwind
at library/std/src/panicking.rs:493
5: core::panicking::panic_fmt
at library/core/src/panicking.rs:92
[2021/08/07 03:53:31.041 +08:00] [FATAL] [lib.rs:462] [“calledResult::unwrap() on an Err value: channel has been closed”] [backtrace="stack backtrace:
0: tikv_util::set_panic_hook::{{closure}}
at components/tikv_util/src/lib.rs:461
1: std::panicking::rust_panic_with_hook
at library/std/src/panicking.rs:595
2: std::panicking::begin_panic_handler::{{closure}}
at library/std/src/panicking.rs:497
3: std::sys_common::backtrace::__rust_end_short_backtrace
at library/std/src/sys_common/backtrace.rs:141
4: rust_begin_unwind
at library/std/src/panicking.rs:493
5: core::panicking::panic_fmt
at library/core/src/panicking.rs:92
6: core::result::unwrap_failed
at library/core/src/result.rs:1355
7: core::result::Result<T,E>::unwrap
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/result.rs:1037
resolved_ts::endpoint::Endpoint<T,E,C>::build_scan_task::{{closure}}
at /home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tikv/components/resolved_ts/src/endpoint.rs:361
8: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/alloc/src/boxed.rs:1560
resolved_ts::scanner::ScannerPool<T,E>::spawn_task::{{closure}}
at /home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tikv/components/resolved_ts/src/scanner.rs:97
9: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/future/mod.rs:80
tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/core.rs:173
tokio::loom::std::unsafe_cell::UnsafeCell::with_mut
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/loom/std/unsafe_cell.rs:14
tokio::runtime::task::core::Core<T,S>::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/core.rs:158
tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/harness.rs:107
core::ops::function::FnOnce::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/ops/function.rs:227
<std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:344
std::panicking::try::do_call
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:379
std::panicking::try
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:343
std::panic::catch_unwind
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:431
tokio::runtime::task::harness::Harness<T,S>::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/harness.rs:89
tokio::runtime::task::raw::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/raw.rs:104
10: tokio::runtime::task::raw::RawTask::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/raw.rs:66
tokio::runtime::task::Notified::run
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/mod.rs:169
tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:349
tokio::coop::with_budget::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/coop.rs:127
std::local::LocalKey::try_with
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/thread/local.rs:272
std::local::LocalKey::with
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/thread/local.rs:248
tokio::coop::with_budget
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/coop.rs:120
tokio::coop::budget
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/coop.rs:96
tokio::runtime::thread_pool::worker::Context::run_task
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:348
11: tokio::runtime::thread_pool::worker::Context::run
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:320
tokio::runtime::thread_pool::worker::run::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:305
tokio::macros::scoped_tls::ScopedKey::set
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/macros/scoped_tls.rs:63
tokio::runtime::thread_pool::worker::run
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:302
tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:281
<tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/blocking/task.rs:41
tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/core.rs:173
tokio::loom::std::unsafe_cell::UnsafeCell::with_mut
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/loom/std/unsafe_cell.rs:14
tokio::runtime::task::core::Core<T,S>::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/core.rs:158
tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/harness.rs:107
core::ops::function::FnOnce::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/ops/function.rs:227
<std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:344
std::panicking::try::do_call
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:379
std::panicking::try
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:343
std::panic::catch_unwind
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:431
tokio::runtime::task::harness::Harness<T,S>::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/harness.rs:89
tokio::runtime::task::raw::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/raw.rs:104
12: tokio::runtime::task::raw::RawTask::poll
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/raw.rs:66
tokio::runtime::task::Notified::run
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/mod.rs:169
tokio::runtime::blocking::pool::Inner::run
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/blocking/pool.rs:250
tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/blocking/pool.rs:230
tokio::runtime::context::enter
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/context.rs:72
tokio::runtime::handle::Handle::enter
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/handle.rs:76
tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/blocking/pool.rs:229
std::sys_common::backtrace::__rust_begin_short_backtrace
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/sys_common/backtrace.rs:125
13: std::Builder::spawn_unchecked::{{closure}}::{{closure}}
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/thread/mod.rs:474
<std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:344
std::panicking::try::do_call
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:379
std::panicking::try
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panicking.rs:343
std::panic::catch_unwind
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/panic.rs:431
std::Builder::spawn_unchecked::{{closure}}
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/std/src/thread/mod.rs:473
core::ops::function::FnOnce::call_once{{vtable.shim}}
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/core/src/ops/function.rs:227
14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/alloc/src/boxed.rs:1546
<alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/16bf626a31cb5b121d0bca2baa969b4f67eb0dab/library/alloc/src/boxed.rs:1546
std::sys::unix::thread_start
at library/std/src/sys/unix/thread.rs:71
15: start_thread
16: clone
"] [location=/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tikv/components/resolved_ts/src/endpoint.rs:367] [thread_name=inc-scan]

tikv-log
[2021/08/07 09:56:04.367 +08:00] [INFO] [raft.rs:1300] [“[logterm: 43, index: 2462, vote: 25414] ignored vote from 25415 [logterm: 43, index: 2462]: lease is not expired”] [“msg type”=MsgRequestPreVote] [“remaining ticks”=10] [term=43] [msg_index=2462] [msg_term=43][from=25415] [vote=25414] [log_index=2462] [log_term=43] [raft_id=25414] [region_id=25412]
[2021/08/07 09:56:04.370 +08:00] [INFO] [raft.rs:1300] [“[logterm: 36, index: 51, vote: 39575] ignored vote from 39576 [logterm: 36, index: 51]: lease is not expired”] [“msg type”=MsgRequestPreVote] [“remaining ticks”=10] [term=36] [msg_index=51] [msg_term=36] [from=39576] [vote=39575] [log_index=51] [log_term=36] [raft_id=39575] [region_id=39573]
[2021/08/07 09:56:04.620 +08:00] [INFO] [raft.rs:1088] [“became follower at term 7”] [term=7] [raft_id=62043] [region_id=62041]
[2021/08/07 09:56:04.622 +08:00] [INFO] [raft.rs:1088] [“became follower at term 56”] [term=56] [raft_id=29822] [region_id=29820]
[2021/08/07 09:56:04.622 +08:00] [INFO] [raft.rs:1088] [“became follower at term 13”] [term=13] [raft_id=52415] [region_id=52413]
[2021/08/07 09:56:04.622 +08:00] [INFO] [raft.rs:1088] [“became follower at term 57”] [term=57] [raft_id=23690] [region_id=23688]
[2021/08/07 09:56:04.622 +08:00] [INFO] [raft.rs:1088] [“became follower at term 47”] [term=47] [raft_id=3642] [region_id=3640]
[2021/08/07 09:56:04.622 +08:00] [INFO] [raft.rs:1088] [“became follower at term 53”] [term=53] [raft_id=9774] [region_id=9772]
[2021/08/07 09:56:04.744 +08:00] [INFO] [raft.rs:1513] [“starting a new election”] [term=14] [raft_id=63655] [region_id=63653]
[2021/08/07 09:56:04.744 +08:00] [INFO] [raft.rs:1138] [“became pre-candidate at term 14”] [term=14] [raft_id=63655] [region_id=63653]
[2021/08/07 09:56:04.744 +08:00] [INFO] [raft.rs:1267] [“broadcasting vote request”] [to=“[63656, 63654]”] [log_index=16] [log_term=14] [term=14] [type=MsgRequestPreVote] [raft_id=63655] [region_id=63653]
[2021/08/07 09:56:04.744 +08:00] [INFO] [raft.rs:1513] [“starting a new election”] [term=24] [raft_id=45011] [region_id=45009]
[2021/08/07 09:56:04.744 +08:00] [INFO] [raft.rs:1138] [“became pre-candidate at term 24”] [term=24] [raft_id=45011] [region_id=45009]
[2021/08/07 09:56:04.744 +08:00] [INFO] [raft.rs:1513] [“starting a new election”] [term=47] [raft_id=25930] [region_id=25928]
[2021/08/07 09:56:04.744 +08:00] [INFO] [raft.rs:1267] [“broadcasting vote request”] [to=“[45012, 45010]”] [log_index=38] [log_term=24] [term=24] [type=MsgRequestPreVote] [raft_id=45011] [region_id=45009]
[2021/08/07 09:56:04.744 +08:00] [INFO] [raft.rs:1138] [“became pre-candidate at term 47”] [term=47] [raft_id=25930] [region_id=25928]
[2021/08/07 09:56:04.744 +08:00] [INFO] [raft.rs:1267] [“broadcasting vote request”] [to=“[25929, 25931]”] [log_index=54] [log_term=47] [term=47] [type=MsgRequestPreVote] [raft_id=25930] [region_id=25928]
[2021/08/07 09:56:04.744 +08:00] [INFO] [raft.rs:1513] [“starting a new election”] [term=55] [raft_id=3614] [region_id=3612]
[2021/08/07 09:56:04.744 +08:00] [INFO] [raft.rs:1138] [“became pre-candidate at term 55”] [term=55] [raft_id=3614] [region_id=3612]
[2021/08/07 09:56:04.744 +08:00] [INFO] [raft.rs:1267] [“broadcasting vote request”] [to=“[3613, 3615]”] [log_index=12327] [log_term=55] [term=55] [type=MsgRequestPreVote] [raft_id=3614] [region_id=3612]
[2021/08/07 09:56:04.864 +08:00] [INFO] [scheduler.rs:435] [“get snapshot failed”] [err=“Error(Request(message: "peer is not leader for region 45049, leader may Some(id: 45050 store_id: 1)" not_leader { region_id: 45049 leader { id: 45050 store_id: 1 } }))”] [cid=29910]

dm日志2021/08/07 09:51:47.624 +08:00] [WARN] [task_checker.go:393] [“backoff skip auto resume task”] [component=“task checker”] [task=task-dm-rds-mysql] [latestResumeTime=2021/08/07 09:46:47.624 +08:00] [duration=5m0s]
[2021/08/07 09:51:52.624 +08:00] [INFO] [worker.go:476] [“auto_resume sub task”] [component=“worker controller”] [task=task-dm-rds-mysql]
[2021/08/07 09:51:52.624 +08:00] [INFO] [subtask.go:502] [“resume with unit”] [subtask=task-dm-rds-mysql] [unit=Load]
[2021/08/07 09:51:52.624 +08:00] [INFO] [task_checker.go:401] [“dispatch auto resume task”] [component=“task checker”] [task=task-dm-rds-mysql]
[2021/08/07 09:51:52.769 +08:00] [INFO] [loader.go:1017] [“prepare loading”] [task=task-dm-rds-mysql] [unit=load] [“cost time”=71.199071ms]
[2021/08/07 09:52:33.507 +08:00] [ERROR] [baseconn.go:105] [“query statement failed”] [task=task-dm-rds-mysql] [unit=load] [query=“SELECT filename,cp_schema,cp_table,offset,end_pos from dm_meta.task-dm-rds-mysql_loader_checkpoint where id=?”] [argument=“[mysql-rds]”] [error=“Error 9005: Region is unavailable”]
[2021/08/07 09:52:33.508 +08:00] [ERROR] [db.go:101] [“query statement failed after retry”] [task=task-dm-rds-mysql] [unit=load] [query=“SELECT filename,cp_schema,cp_table,offset,end_pos from dm_meta.task-dm-rds-mysql_loader_checkpoint where id=?”] [argument=“[mysql-rds]”] [error=“[code=10005:class=database:scope=not-set:level=high], Message: query statement failed: SELECT filename,cp_schema,cp_table,offset,end_pos from dm_meta.task-dm-rds-mysql_loader_checkpoint where id=?, RawCause: Error 9005:Region is unavailable”]
[2021/08/07 09:52:33.508 +08:00] [INFO] [checkpoint.go:168] [“load checkpoint”] [task=task-dm-rds-mysql] [unit=load] [component=“remote checkpoint”] [“cost time”=40.738989873s]
[2021/08/07 09:52:33.508 +08:00] [INFO] [loader.go:744] [“stop importing data process”] [task=task-dm-rds-mysql] [unit=load]
[2021/08/07 09:52:33.508 +08:00] [INFO] [subtask.go:289] [“unit process returned”] [subtask=task-dm-rds-mysql] [unit=Load] [stage=Paused] [status=“{"totalBytes":201673205140,"progress":"0.00 %","metaBinlog":"(mysql-bin.000173, 349182009)","metaBinlogGTID":"0828d702-f1b6-11eb-aaca-88df9e38ad80:1-149,9a5d9a23-8e4c-11eb-85e6-88df9e38973c:1-11925684,c9a30b08-993f-11eb-86f3-88df9e389758:1-109543700,f3549e93-993f-11eb-b613-88df9e38ad24:1-1797159524"}”]
[2021/08/07 09:52:33.508 +08:00] [ERROR] [subtask.go:308] [“unit process error”] [subtask=task-dm-rds-mysql] [unit=Load] [“error information”=“{"ErrCode":10005,"ErrClass":"database","ErrScope":"downstream","ErrLevel":"high","Message":"query statement failed: SELECT filename,cp_schema,cp_table,offset,end_pos from dm_meta.task-dm-rds-mysql_loader_checkpoint where id=?","RawCause":"Error 9005: Region is unavailable"}”]
[2021/08/07 09:52:37.624 +08:00] [WARN] [task_checker.go:393] [“backoff skip auto resume task”] [component=“task checker”] [task=task-dm-rds-mysql] [latestResumeTime=2021/08/07 09:51:52.624 +08:00] [duration=5m0s]
[2021/08/07 09:52:42.624 +08:00] [WARN] [task_checker.go:393] [“backoff skip auto resume task”] [component=“task checker”] [task=task-dm-rds-mysql] [latestResumeTime=2021/08/07 09:51:52.624 +08:00] [duration=5m0s]

相关日志、配置文件、Grafana 监控（https://metricstool.pingcap.com/)
Found cluster newer version:

The latest version: v1.5.4
Local installed version: v1.5.3
Update current component: tiup update cluster
Update all components: tiup update --all

Starting component cluster: /root/.tiup/components/cluster/v1.5.3/tiup-cluster display test
Cluster type: tidb
Cluster name: test
Cluster version: v5.1.0
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://10.96.217.125:2379/dashboard
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

10.96.217.123:9093 alertmanager 10.96.217.123 9093/9094 linux/x86_64 Up /data/vols/tidb/tidb-data/alertmanager-9093 /data/vols/tidb/tidb/alertmanager-9093
10.96.217.123:3000 grafana 10.96.217.123 3000 linux/x86_64 Up - /data/vols/tidb/tidb/grafana-3000
10.96.217.123:2379 pd 10.96.217.123 2379/2380 linux/x86_64 Up|L /data/vols/tidb/tidb-data/pd-2379 /data/vols/tidb/tidb/pd-2379
10.96.217.124:2379 pd 10.96.217.124 2379/2380 linux/x86_64 Up /data/vols/tidb/tidb-data/pd-2379 /data/vols/tidb/tidb/pd-2379
10.96.217.125:2379 pd 10.96.217.125 2379/2380 linux/x86_64 Up|UI /data/vols/tidb/tidb-data/pd-2379 /data/vols/tidb/tidb/pd-2379
10.96.217.123:9090 prometheus 10.96.217.123 9090 linux/x86_64 Up /data/vols/tidb/tidb-data/prometheus-9090 /data/vols/tidb/tidb/prometheus-9090
10.96.217.123:4000 tidb 10.96.217.123 4000/10080 linux/x86_64 Up - /data/vols/tidb/tidb/tidb-4000
10.96.217.124:4000 tidb 10.96.217.124 4000/10080 linux/x86_64 Up - /data/vols/tidb/tidb/tidb-4000
10.96.217.125:4000 tidb 10.96.217.125 4000/10080 linux/x86_64 Up - /data/vols/tidb/tidb/tidb-4000
10.96.217.123:20160 tikv 10.96.217.123 20160/20180 linux/x86_64 Up /data/vols/tidb/tidb-data/tikv-20160 /data/vols/tidb/tidb/tikv-20160
10.96.217.124:20160 tikv 10.96.217.124 20160/20180 linux/x86_64 Up /data/vols/tidb/tidb-data/tikv-20160 /data/vols/tidb/tidb/tikv-20160
10.96.217.125:20160 tikv 10.96.217.125 20160/20180 linux/x86_64 Up /data/vols/tidb/tidb-data/tikv-20160 /data/vols/tidb/tidb/tikv-20160

TiUP CLuster Edit config 信息

image1920×891 136 KB
对应模块的 Grafana 监控（如有 BR、TiDB-binlog、TiCDC 等）
tikv.log
对应模块日志（包含问题前后 1 小时日志）

看到peer is not leader for region 45049, leader may Some(id: 45050 sto怎样处理[2021/08/07 19:10:23.051 +08:00] [INFO] [raft.rs:1513] [“starting a new election”] [term=12] [raft_id=74819] [region_id=74817]
[2021/08/07 19:10:23.052 +08:00] [INFO] [raft.rs:1138] [“became pre-candidate at term 12”] [term=12] [raft_id=74819] [region_id=74817]
[2021/08/07 19:10:23.052 +08:00] [INFO] [raft.rs:1267] [“broadcasting vote request”] [to=“[74820, 74818]”] [log_index=11] [log_term=12] [term=12] [type=MsgRequestPreVote] [raft_id=74819] [region_id=74817]
[2021/08/07 19:10:23.069 +08:00] [INFO] [raft.rs:1513] [“starting a new election”] [term=25] [raft_id=64111] [region_id=64109]
[2021/08/07 19:10:23.069 +08:00] [INFO] [raft.rs:1138] [“became pre-candidate at term 25”] [term=25] [raft_id=64111] [region_id=64109]
[2021/08/07 19:10:23.069 +08:00] [INFO] [raft.rs:1267] [“broadcasting vote request”] [to=“[64112, 64110]”] [log_index=29] [log_term=25] [term=25] [type=MsgRequestPreVote] [raft_id=64111] [region_id=64109]
[2021/08/07 19:10:23.122 +08:00] [INFO] [raft.rs:1528] [“[logterm: 20, index: 25, vote: 0] cast vote for 65192 [logterm: 20, index: 25] at term 20”] [“msg type”=MsgRequestPreVote] [term=20] [msg_index=25] [msg_term=20] [from=65192] [vote=0] [log_index=25] [log_term=20] [raft_id=65191] [region_id=65189]
[2021/08/07 19:10:23.123 +08:00] [INFO] [raft.rs:1332] [“received a message with higher term from 65192”] [“msg type”=MsgRequestVote] [message_term=21] [term=20] [from=65192] [raft_id=65191] [region_id=65189]
[2021/08/07 19:10:23.123 +08:00] [INFO] [raft.rs:1088] [“became follower at term 21”] [term=21] [raft_id=65191] [region_id=65189]
[2021/08/07 19:10:23.123 +08:00] [INFO] [raft.rs:1528] [“[logterm: 20, index: 25, vote: 0] cast vote for 65192 [logterm: 20, index: 25] at term 21”] [“msg type”=MsgRequestVote] [term=21] [msg_index=25] [msg_term=20] [from=65192] [vote=0] [log_index=25] [log_term=20] [raft_id=65191] [region_id=65189]
[2021/08/07 19:10:23.152 +08:00] [INFO] [scheduler.rs:435] [“get snapshot failed”] [err=“Error(Request(message: "peer is not leader for region 45049, leader may Some(id: 45050 store_id: 1)" not_leader { region_id: 45049 leader { id: 45050 store_id: 1 } }))”] [cid=3479354]
[2021/08/07 19:10:23.496 +08:00] [WARN] [peer.rs:3898] [“leader missing longer than abnormal_leader_missing_duration”] [expect=10m] [peer_id=62243] [region_id=62241]
[2021/08/07 19:10:23.657 +08:00] [INFO] [scheduler.rs:435] [“get snapshot failed”] [err=“Error(Request(message: "peer is not leader for region 45049, leader may Some(id: 45050 store_id: 1)" not_leader { region_id: 45049 leader { id: 45050 store_id: 1 } }))”] [cid=3479355]
[2021/08/07 19:10:23.752 +08:00] [WARN] [peer.rs:3898] [“leader missing longer than abnormal_leader_missing_duration”] [expect=10m] [peer_id=79567] [region_id=79565]
[2021/08/07 19:10:23.777 +08:00] [WARN] [peer.rs:3898] [“leader missing longer than abnormal_leader_missing_duration”] [expect=10m] [peer_id=77467] [region_id=77465]

hecomlilong · 2021 年8 月 7 日 14:37

TiKV节点写压力过大，导致主region无响应，频繁选主
有可能是mysql数据过于集中，存在热点问题，比如使用的是自增ID作为主键，同步时导致单节点压力过大，关注下流量可视化工具
还有可能是TiKV节点配置过低，cpu被打满，看看是否能，增加一些TiKV节点或TiKV的配置。使得CPU保持在50%左右

lukaiwo · 2021 年8 月 7 日 19:26

一个表14g。64c256g的机器用什么方法负载均衡？

我先
threads: 4
chunk-filesize: 64
skip-tz-utc: true
extra-args: "-B singlemysql --no-locks --consistency none -r 100000
试试看

Lucien · 2021 年8 月 8 日 14:05

可以参考一下这篇文档～读性能慢-热点问题

lukaiwo · 2021 年8 月 9 日 02:31

另外请问max-grpc-send-msg-len : 用于设定 grpc 最大消息体大小，截止 2021.07.10 ，该参数未被在文档中提及，属于「隐藏配置项」，时在排查问题时才从官方研发团队获知的一个参数

这个在什么地方配置？

lukaiwo · 2021 年8 月 10 日 01:48

此问题已解决，是因为有个表超过默认的10mb
我们增加了如下参数
tidb:

开启支持大小写不敏感, 只有在集群初始化时配置才生效, 默认 false

new_collations_enabled_on_first_bootstrap: true
performance.txn-entry-size-limit: 20971520

tikv:
raftstore.raft-entry-max-size: 20971520
server.max-grpc-send-msg-len: 20971520

system · 2022 年10 月 31 日 19:05

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。