tiflash v5.3.0 报错重启

lindoubled · 2023 年1 月 2 日 02:24

【 TiDB 使用环境】
生产环境
【 TiDB 版本】
v5.3.0
【复现路径】
无
【遇到的问题：问题现象及影响】

[2023/01/02 01:55:35.309 +08:00] [ERROR] [<unknown>] ["DeltaMergeStore[db_428.t_2727]: Task MergeDelta on Segment [1] failed. Error msg: DeltaMerge return wrong result, current handle[23793238]version[438459978676699146]@read[31]@pos[3102] is expected >= last_handle[23814470]last_version[438459978624532501]@read[31]@pos[3101]"] [thread_id=21670]
[2023/01/02 01:55:35.331 +08:00] [ERROR] [<unknown>] ["void DB::BackgroundProcessingPool::threadFunction(): Code: 0, e.displayText() = DB::Exception: DeltaMerge return wrong result, current handle[23793238]version[438459978676699146]@read[31]@pos[3102] is expected >= last_handle[23814470]last_version[438459978624532501]@read[31]@pos[3101], e.what() = DB::Exception, Stack trace:\n\n0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x16) [0x370f6a6]\n1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x26) [0x37034a6]\n2. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::beforeReturnBlock(DB::Block const&)+0x53c) [0x75e342c]\n3. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::read()+0xc4) [0x75e6c64]\n4. bin/tiflash/tiflash(bool DB::DM::placeInsert<true, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists> >(std::shared_ptr<DB::DM::SkippableBlockInputStream> const&, DB::Block const&, DB::DM::RowKeyRange const&, bool, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, unsigned long, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul> const&, std::vector<DB::SortColumnDescription, std::allocator<DB::SortColumnDescription> > const&)+0x1ec) [0x75de72c]\n5. bin/tiflash/tiflash(bool DB::DM::Segment::placeUpsert<true>(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, unsigned long, DB::Block&&, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, DB::DM::RowKeyRange const&, bool) const+0x3a1) [0x75e0aa1]\n6. bin/tiflash/tiflash(DB::DM::Segment::ensurePlace(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x496) [0x75c41a6]\n7. bin/tiflash/tiflash(DB::DM::Segment::getReadInfo(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x237) [0x75c54c7]\n8. bin/tiflash/tiflash(DB::DM::Segment::getInputStreamForDataExport(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::RowKeyRange const&, unsigned long, bool) const+0x87) [0x75c72a7]\n9. bin/tiflash/tiflash(DB::DM::Segment::prepareMergeDelta(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::WriteBatches&) const+0xa3) [0x75c75b3]\n10. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::segmentMergeDelta(DB::DM::DMContext&, std::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::TaskRunThread, std::shared_ptr<DB::DM::SegmentSnapshot>)+0x3c0) [0x7596250]\n11. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::handleBackgroundTask(bool)+0x3ae) [0x759e8fe]\n12. bin/tiflash/tiflash(DB::BackgroundProcessingPool::threadFunction()+0x55f) [0x767c45f]\n13. bin/tiflash/tiflash() [0x8b729ef]\n14. /lib64/libpthread.so.0(+0x7ea5) [0x7feb2bdd9ea5]\n15. /lib64/libc.so.6(clone+0x6d) [0x7feb2b8008dd]\n"] [thread_id=21670]
[2023/01/02 01:55:35.403 +08:00] [ERROR] [<unknown>] ["DeltaMergeStore[db_428.t_2727]: Task MergeDelta on Segment [1] failed. Error msg: DeltaMerge return wrong result, current handle[23793238]version[438459978676699146]@read[31]@pos[3102] is expected >= last_handle[23814470]last_version[438459978624532501]@read[31]@pos[3101]"] [thread_id=39624]
[2023/01/02 01:55:35.424 +08:00] [ERROR] [<unknown>] ["void DB::BackgroundProcessingPool::threadFunction(): Code: 0, e.displayText() = DB::Exception: DeltaMerge return wrong result, current handle[23793238]version[438459978676699146]@read[31]@pos[3102] is expected >= last_handle[23814470]last_version[438459978624532501]@read[31]@pos[3101], e.what() = DB::Exception, Stack trace:\n\n0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x16) [0x370f6a6]\n1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x26) [0x37034a6]\n2. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::beforeReturnBlock(DB::Block const&)+0x53c) [0x75e342c]\n3. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::read()+0xc4) [0x75e6c64]\n4. bin/tiflash/tiflash(bool DB::DM::placeInsert<true, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists> >(std::shared_ptr<DB::DM::SkippableBlockInputStream> const&, DB::Block const&, DB::DM::RowKeyRange const&, bool, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, unsigned long, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul> const&, std::vector<DB::SortColumnDescription, std::allocator<DB::SortColumnDescription> > const&)+0x1ec) [0x75de72c]\n5. bin/tiflash/tiflash(bool DB::DM::Segment::placeUpsert<true>(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, unsigned long, DB::Block&&, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, DB::DM::RowKeyRange const&, bool) const+0x3a1) [0x75e0aa1]\n6. bin/tiflash/tiflash(DB::DM::Segment::ensurePlace(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x496) [0x75c41a6]\n7. bin/tiflash/tiflash(DB::DM::Segment::getReadInfo(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x237) [0x75c54c7]\n8. bin/tiflash/tiflash(DB::DM::Segment::getInputStreamForDataExport(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::RowKeyRange const&, unsigned long, bool) const+0x87) [0x75c72a7]\n9. bin/tiflash/tiflash(DB::DM::Segment::prepareMergeDelta(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::WriteBatches&) const+0xa3) [0x75c75b3]\n10. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::segmentMergeDelta(DB::DM::DMContext&, std::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::TaskRunThread, std::shared_ptr<DB::DM::SegmentSnapshot>)+0x3c0) [0x7596250]\n11. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::handleBackgroundTask(bool)+0x3ae) [0x759e8fe]\n12. bin/tiflash/tiflash(DB::BackgroundProcessingPool::threadFunction()+0x55f) [0x767c45f]\n13. bin/tiflash/tiflash() [0x8b729ef]\n14. /lib64/libpthread.so.0(+0x7ea5) [0x7feb2bdd9ea5]\n15. /lib64/libc.so.6(clone+0x6d) [0x7feb2b8008dd]\n"] [thread_id=39624]
[2023/01/02 01:55:35.540 +08:00] [ERROR] [<unknown>] ["DB::EngineStoreApplyRes DB::HandleWriteRaftCmd(const DB::EngineStoreServerWrap*, DB::WriteCmdsView, DB::RaftCmdHeader): Code: 0, e.displayText() = DB::Exception: DeltaMerge return wrong result, current handle[23793238]version[438459978676699146]@read[31]@pos[3102] is expected >= last_handle[23814470]last_version[438459978624532501]@read[31]@pos[3101], e.what() = DB::Exception, Stack trace:\n\n0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x16) [0x370f6a6]\n1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x26) [0x37034a6]\n2. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::beforeReturnBlock(DB::Block const&)+0x53c) [0x75e342c]\n3. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::read()+0xc4) [0x75e6c64]\n4. bin/tiflash/tiflash(bool DB::DM::placeInsert<true, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists> >(std::shared_ptr<DB::DM::SkippableBlockInputStream> const&, DB::Block const&, DB::DM::RowKeyRange const&, bool, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, unsigned long, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul> const&, std::vector<DB::SortColumnDescription, std::allocator<DB::SortColumnDescription> > const&)+0x1ec) [0x75de72c]\n5. bin/tiflash/tiflash(bool DB::DM::Segment::placeUpsert<true>(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, unsigned long, DB::Block&&, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, DB::DM::RowKeyRange const&, bool) const+0x3a1) [0x75e0aa1]\n6. bin/tiflash/tiflash(DB::DM::Segment::ensurePlace(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x496) [0x75c41a6]\n7. bin/tiflash/tiflash(DB::DM::Segment::getReadInfo(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x237) [0x75c54c7]\n8. bin/tiflash/tiflash(DB::DM::Segment::getInputStreamForDataExport(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::RowKeyRange const&, unsigned long, bool) const+0x87) [0x75c72a7]\n9. bin/tiflash/tiflash(DB::DM::Segment::prepareMergeDelta(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::WriteBatches&) const+0xa3) [0x75c75b3]\n10. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::segmentMergeDelta(DB::DM::DMContext&, std::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::TaskRunThread, std::shared_ptr<DB::DM::SegmentSnapshot>)+0x3c0) [0x7596250]\n11. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::checkSegmentUpdate(std::shared_ptr<DB::DM::DMContext> const&, std::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::ThreadType)+0xb0f) [0x7597d4f]\n12. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::waitForWrite(std::shared_ptr<DB::DM::DMContext> const&, std::shared_ptr<DB::DM::Segment> const&)+0x229) [0x7599a79]\n13. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::write(DB::Context const&, DB::Settings const&, DB::Block&&)+0x768) [0x759a308]\n14. bin/tiflash/tiflash() [0x776479c]\n15. bin/tiflash/tiflash() [0x77651be]\n16. bin/tiflash/tiflash(DB::RegionTable::writeBlockByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::vector<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject<false> const> >, std::allocator<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject<false> const> > > >&, Poco::Logger*, bool)+0x102) [0x77655b2]\n17. bin/tiflash/tiflash(DB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&)+0x2fa) [0x777324a]\n18. bin/tiflash/tiflash(DB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&)+0x5b) [0x775d58b]\n19. bin/tiflash/tiflash(HandleWriteRaftCmd+0x31) [0x7769031]\n20. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0x138ae25) [0x7feb2d99ee25]\n21. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0x13954a0) [0x7feb2d9a94a0]\n22. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0x136aad1) [0x7feb2d97ead1]\n23. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0xd3c0db) [0x7feb2d3500db]\n24. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0xd7db51) [0x7feb2d391b51]\n25. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0xdb113f) [0x7feb2d3c513f]\n26. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0x14f757b) [0x7feb2db0b57b]\n27. /lib64/libpthread.so.0(+0x7ea5) [0x7feb2bdd9ea5]\n28. /lib64/libc.so.6(clone+0x6d) [0x7feb2b8008dd]\n"] [thread_id=24]
[2023/01/02 01:55:35.643 +08:00] [ERROR] [<unknown>] ["DB::EngineStoreApplyRes DB::HandleWriteRaftCmd(const DB::EngineStoreServerWrap*, DB::WriteCmdsView, DB::RaftCmdHeader): Code: 0, e.displayText() = DB::Exception: DeltaMerge return wrong result, current handle[23793238]version[438459978676699146]@read[31]@pos[3102] is expected >= last_handle[23814470]last_version[438459978624532501]@read[31]@pos[3101], e.what() = DB::Exception, Stack trace:\n\n0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x16) [0x370f6a6]\n1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x26) [0x37034a6]\n2. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::beforeReturnBlock(DB::Block const&)+0x53c) [0x75e342c]\n3. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::read()+0xc4) [0x75e6c64]\n4. bin/tiflash/tiflash(bool DB::DM::placeInsert<true, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists> >(std::shared_ptr<DB::DM::SkippableBlockInputStream> const&, DB::Block const&, DB::DM::RowKeyRange const&, bool, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, unsigned long, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul> const&, std::vector<DB::SortColumnDescription, std::allocator<DB::SortColumnDescription> > const&)+0x1ec) [0x75de72c]\n5. bin/tiflash/tiflash(bool DB::DM::Segment::placeUpsert<true>(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, unsigned long, DB::Block&&, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, DB::DM::RowKeyRange const&, bool) const+0x3a1) [0x75e0aa1]\n6. bin/tiflash/tiflash(DB::DM::Segment::ensurePlace(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x496) [0x75c41a6]\n7. bin/tiflash/tiflash(DB::DM::Segment::getReadInfo(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x237) [0x75c54c7]\n8. bin/tiflash/tiflash(DB::DM::Segment::getInputStreamForDataExport(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::RowKeyRange const&, unsigned long, bool) const+0x87) [0x75c72a7]\n9. bin/tiflash/tiflash(DB::DM::Segment::prepareMergeDelta(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::WriteBatches&) const+0xa3) [0x75c75b3]\n10. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::segmentMergeDelta(DB::DM::DMContext&, std::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::TaskRunThread, std::shared_ptr<DB::DM::SegmentSnapshot>)+0x3c0) [0x7596250]\n11. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::checkSegmentUpdate(std::shared_ptr<DB::DM::DMContext> const&, std::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::ThreadType)+0xb0f) [0x7597d4f]\n12. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::waitForWrite(std::shared_ptr<DB::DM::DMContext> const&, std::shared_ptr<DB::DM::Segment> const&)+0x229) [0x7599a79]\n13. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::write(DB::Context const&, DB::Settings const&, DB::Block&&)+0x768) [0x759a308]\n14. bin/tiflash/tiflash() [0x776479c]\n15. bin/tiflash/tiflash() [0x77651be]\n16. bin/tiflash/tiflash(DB::RegionTable::writeBlockByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::vector<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject<false> const> >, std::allocator<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject<false> const> > > >&, Poco::Logger*, bool)+0x102) [0x77655b2]\n17. bin/tiflash/tiflash(DB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&)+0x2fa) [0x777324a]\n18. bin/tiflash/tiflash(DB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&)+0x5b) [0x775d58b]\n19. bin/tiflash/tiflash(HandleWriteRaftCmd+0x31) [0x7769031]\n20. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0x138ae25) [0x7feb2d99ee25]\n21. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0x13954a0) [0x7feb2d9a94a0]\n22. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0x136aad1) [0x7feb2d97ead1]\n23. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0xd3c0db) [0x7feb2d3500db]\n24. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0xd7db51) [0x7feb2d391b51]\n25. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0xdb113f) [0x7feb2d3c513f]\n26. /data/tikv/20160/tiflash/bin/tiflash/libtiflash_proxy.so(+0x14f757b) [0x7feb2db0b57b]\n27. /lib64/libpthread.so.0(+0x7ea5) [0x7feb2bdd9ea5]\n28. /lib64/libc.so.6(clone+0x6d) [0x7feb2b8008dd]\n"] [thread_id=25]
[2023/01/02 02:11:58.494 +08:00] [ERROR] [<unknown>] ["ServerErrorHandler: Code: 209, e.displayText() = DB::NetException: Timeout exceeded while reading from socket (10.5.39.194:58651), e.what() = DB::NetException, Stack trace:\n\n0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x16) [0x370f6a6]\n1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x26) [0x37034a6]\n2. bin/tiflash/tiflash(DB::ReadBufferFromPocoSocket::nextImpl()+0x213) [0x7e98c83]\n3. bin/tiflash/tiflash(DB::TCPHandler::runImpl()+0x109a) [0x3749b1a]\n4. bin/tiflash/tiflash(DB::TCPHandler::run()+0x1c) [0x374a75c]\n5. bin/tiflash/tiflash(Poco::Net::TCPServerConnection::start()+0xf) [0x80f77af]\n6. bin/tiflash/tiflash(Poco::Net::TCPServerDispatcher::run()+0x166) [0x80f7b76]\n7. bin/tiflash/tiflash(Poco::PooledThread::run()+0x77) [0x8244ca7]\n8. bin/tiflash/tiflash(Poco::ThreadImpl::runnableEntry(void*)+0x38) [0x8240b58]\n9. bin/tiflash/tiflash() [0x8b729ef]\n10. /lib64/libpthread.so.0(+0x7ea5) [0x7f0e703a8ea5]\n11. /lib64/libc.so.6(clone+0x6d) [0x7f0e6fdcf8dd]\n"] [thread_id=43]

【资源配置】
32C64G
【附件：截图/日志/监控】