【 TiDB 使用环境】 Poc
【 TiDB 版本】6.5.3
【复现路径】
- TiKV 因为性能要求使用单副本, 两个节点出现异常, 做过有损数据恢复.
- Tiflash的 不同步了, replica 置为0 再置为1 不管用.
- scale-in 了 tiflash 然后scale-out tiflash
- 发现tiflash 依旧没有同步tikv的表, 出现了错误
【遇到的问题：问题现象及影响】
2023/09/19 15:04:36.820 +08:00] [WARN] [TiDBSchemaSyncer.h:225] [“apply diff meets exception : DB::TiFlashException: miss table in TiKV : 600140 \n stack is \n 0x15395a4\tDB::TiFlashException::TiFlashException(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, DB::TiFlashError const&) [tiflash+22255012]\n \tdbms/src/Common/TiFlashException.h:250\n 0x60286cc\tDB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyDiff(DB::SchemaDiff const&) [tiflash+100828876]\n \tdbms/src/TiDB/Schema/SchemaBuilder.cpp:521\n 0x5f811d0\tDB::TiDBSchemaSyncer<false, false>::syncSchemas(DB::Context&) [tiflash+100143568]\n \tdbms/src/TiDB/Schema/TiDBSchemaSyncer.h:128\n 0x605d750\tstd::__1::__function::__func<DB::SchemaSyncService::SchemaSyncService(DB::Context&)::$_0, std::__1::allocatorDB::SchemaSyncService::SchemaSyncService(DB::Context&)::$_0, bool ()>::operator()() [tiflash+101046096]\n \t/usr/local/bin/…/include/c++/v1/__functional/function.h:345\n 0x5cc872c\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >)::$_1> >(void*) [tiflash+97290028]\n \t/usr/local/bin/…/include/c++/v1/thread:291\n 0xffffac3b87ac\t [libpthread.so.0+34732]\n 0xffffac0a60fc\t [libc.so.6+876796]”] [source=SchemaSyncer] [thread_id=194]
[2023/09/19 15:53:19.486 +08:00] [WARN] [TiDBSchemaSyncer.h:225] [“apply diff meets exception : DB::TiFlashException: miss table in TiKV : 600153 \n stack is \n 0x15395a4\tDB::TiFlashException::TiFlashException(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, DB::TiFlashError const&) [tiflash+22255012]\n \tdbms/src/Common/TiFlashException.h:250\n 0x60286cc\tDB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyDiff(DB::SchemaDiff const&) [tiflash+100828876]\n \tdbms/src/TiDB/Schema/SchemaBuilder.cpp:521\n 0x5f811d0\tDB::TiDBSchemaSyncer<false, false>::syncSchemas(DB::Context&) [tiflash+100143568]\n \tdbms/src/TiDB/Schema/TiDBSchemaSyncer.h:128\n 0x605d750\tstd::__1::__function::__func<DB::SchemaSyncService::SchemaSyncService(DB::Context&)::$_0, std::__1::allocatorDB::SchemaSyncService::SchemaSyncService(DB::Context&)::$_0, bool ()>::operator()() [tiflash+101046096]\n \t/usr/local/bin/…/include/c++/v1/__functional/function.h:345\n 0x5cc872c\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >)::$_1> >(void*) [tiflash+97290028]\n \t/usr/local/bin/…/include/c++/v1/thread:291\n 0xffffac3b87ac\t [libpthread.so.0+34732]\n 0xffffac0a60fc\t [libc.so.6+876796]”] [source=SchemaSyncer] [thread_id=198]
[2023/09/19 16:12:03.657 +08:00] [WARN] [TiDBSchemaSyncer.h:225] [“apply diff meets exception : DB::TiFlashException: miss table in TiKV : 600158 \n stack is \n 0x15395a4\tDB::TiFlashException::TiFlashException(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, DB::TiFlashError const&) [tiflash+22255012]\n \tdbms/src/Common/TiFlashException.h:250\n 0x60286cc\tDB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyDiff(DB::SchemaDiff const&) [tiflash+100828876]\n \tdbms/src/TiDB/Schema/SchemaBuilder.cpp:521\n 0x5f811d0\tDB::TiDBSchemaSyncer<false, false>::syncSchemas(DB::Context&) [tiflash+100143568]\n \tdbms/src/TiDB/Schema/TiDBSchemaSyncer.h:128\n 0x605d750\tstd::__1::__function::__func<DB::SchemaSyncService::SchemaSyncService(DB::Context&)::$_0, std::__1::allocatorDB::SchemaSyncService::SchemaSyncService(DB::Context&)::$_0, bool ()>::operator()() [tiflash+101046096]\n \t/usr/local/bin/…/include/c++/v1/__functional/function.h:345\n 0x5cc872c\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >)::$_1> >(void*) [tiflash+97290028]\n \t/usr/local/bin/…/include/c++/v1/thread:291\n 0xffffac3b87ac\t [libpthread.so.0+34732]\n 0xffffac0a60fc\t [libc.so.6+876796]”] [source=SchemaSyncer] [thread_id=192]
【资源配置】4* 96core 512G鲲鹏服务器 NVME SSD 磁盘空间剩余充足.
【附件：截图/日志/监控】