tiflash同步副本过程中出现报错，无法启动

🪐 TiDB 技术问题
,
1

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：

【TiDB 版本】
v5.0.1
【问题描述】
同步表时，突然就挂了，状态为 Disconnected，无法连接，无法启动与关闭

2021.05.01 22:44:23.503775 [ 44 ] DB::EngineStoreApplyRes DB::HandleWriteRaftCmd(const DB::EngineStoreServerWrap*, DB::WriteCmdsView, DB::RaftCmdHeader): Code: 9008, e.displayText() = DB::Exception: Raw TiDB PK: 80000001B753972C, Prewrite ts: 424641250168406065 can not found in default cf for key: 748000000000006CFFD25F7280000001B7FF53972C0000000000FAFA1B5F0866BBFF91, e.what() = DB::Exception, Stack trace:

  1. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x367c835]
  2. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x36733c5]
  3. bin/tiflash/tiflash(DB::RegionData::readDataByWriteIt(std::_Rb_tree_const_iterator<std::pair<std::pair<DB::RawTiDBPK, unsigned long> const, std::tuple<std::shared_ptr<DB::StringObject const>, std::shared_ptr<DB::StringObject const>, DB::RecordKVFormat::InnerDecodedWriteCFValue> > > const&, bool) const+0x50c) [0x734ef3c]
  4. bin/tiflash/tiflash(DB::Region::readDataByWriteIt(std::_Rb_tree_const_iterator<std::pair<std::pair<DB::RawTiDBPK, unsigned long> const, std::tuple<std::shared_ptr<DB::StringObject const>, std::shared_ptr<DB::StringObject const>, DB::RecordKVFormat::InnerDecodedWriteCFValue> > > const&, bool) const+0x17) [0x732ddd7]
  5. bin/tiflash/tiflash(DB::ReadRegionCommitCache(std::shared_ptrDB::Region const&, bool)+0x1c1) [0x7322fa1]
  6. bin/tiflash/tiflash(DB::RegionTable::writeBlockByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::vector<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject const> >, std::allocator<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject const> > > >&, Poco::Logger*, bool)+0x111) [0x7326051]
  7. bin/tiflash/tiflash(DB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&)+0x2c4) [0x7330ce4]
  8. bin/tiflash/tiflash(DB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&)+0x5a) [0x73201aa]
  9. bin/tiflash/tiflash(DB::HandleWriteRaftCmd(DB::EngineStoreServerWrap const*, DB::WriteCmdsView, DB::RaftCmdHeader)+0x30) [0x7327a50]
  10. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbe31f7) [0x7f8bec4121f7]
  11. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbe9e9a) [0x7f8bec418e9a]
  12. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbbd541) [0x7f8bec3ec541]
  13. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x51d0d6) [0x7f8bebd4c0d6]
  14. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x5632dd) [0x7f8bebd922dd]
  15. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x596f3f) [0x7f8bebdc5f3f]
  16. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x156ea8f) [0x7f8becd9da8f]
  17. /lib64/libpthread.so.0(+0x7e64) [0x7f8beaff4e64]
  18. /lib64/libc.so.6(clone+0x6c) [0x7f8beaa1b88c]

2021.05.01 22:44:39.177198 [ 1 ] Application: The configuration “path” is deprecated. Check [storage] section for new style.
2021.05.01 22:44:40.286183 [ 26 ] DMFile: Existing dmfile, removed :/opt/ssd/tidb/tidb-data/tiflash-9000/data/t_27858/stable/.tmp.dmf_198
2021.05.01 22:44:43.464557 [ 33 ] DB::EngineStoreApplyRes DB::HandleWriteRaftCmd(const DB::EngineStoreServerWrap*, DB::WriteCmdsView, DB::RaftCmdHeader): Code: 9008, e.displayText() = DB::Exception: Raw TiDB PK: 80000001B753972C, Prewrite ts: 424641250168406065 can not found in default cf for key: 748000000000006CFFD25F7280000001B7FF53972C0000000000FAFA1B5F0866BBFF91, e.what() = DB::Exception, Stack trace:

  1. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x367c835]
  2. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x36733c5]
  3. bin/tiflash/tiflash(DB::RegionData::readDataByWriteIt(std::_Rb_tree_const_iterator<std::pair<std::pair<DB::RawTiDBPK, unsigned long> const, std::tuple<std::shared_ptr<DB::StringObject const>, std::shared_ptr<DB::StringObject const>, DB::RecordKVFormat::InnerDecodedWriteCFValue> > > const&, bool) const+0x50c) [0x734ef3c]
  4. bin/tiflash/tiflash(DB::Region::readDataByWriteIt(std::_Rb_tree_const_iterator<std::pair<std::pair<DB::RawTiDBPK, unsigned long> const, std::tuple<std::shared_ptr<DB::StringObject const>, std::shared_ptr<DB::StringObject const>, DB::RecordKVFormat::InnerDecodedWriteCFValue> > > const&, bool) const+0x17) [0x732ddd7]
  5. bin/tiflash/tiflash(DB::ReadRegionCommitCache(std::shared_ptrDB::Region const&, bool)+0x1c1) [0x7322fa1]
  6. bin/tiflash/tiflash(DB::RegionTable::writeBlockByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::vector<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject const> >, std::allocator<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject const> > > >&, Poco::Logger*, bool)+0x111) [0x7326051]
  7. bin/tiflash/tiflash(DB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&)+0x2c4) [0x7330ce4]
  8. bin/tiflash/tiflash(DB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&)+0x5a) [0x73201aa]
  9. bin/tiflash/tiflash(DB::HandleWriteRaftCmd(DB::EngineStoreServerWrap const*, DB::WriteCmdsView, DB::RaftCmdHeader)+0x30) [0x7327a50]
  10. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbe31f7) [0x7fc7be0431f7]
  11. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbe9e9a) [0x7fc7be049e9a]
  12. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbbd541) [0x7fc7be01d541]
  13. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x51d0d6) [0x7fc7bd97d0d6]
  14. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x5632dd) [0x7fc7bd9c32dd]
  15. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x596f3f) [0x7fc7bd9f6f3f]
  16. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x156ea8f) [0x7fc7be9cea8f]
  17. /lib64/libpthread.so.0(+0x7e64) [0x7fc7bcc25e64]
  18. /lib64/libc.so.6(clone+0x6c) [0x7fc7bc64c88c]

2021.05.01 22:44:58.907561 [ 1 ] Application: The configuration “path” is deprecated. Check [storage] section for new style.
2021.05.01 22:45:01.725301 [ 21 ] DB::EngineStoreApplyRes DB::HandleWriteRaftCmd(const DB::EngineStoreServerWrap*, DB::WriteCmdsView, DB::RaftCmdHeader): Code: 9008, e.displayText() = DB::Exception: Raw TiDB PK: 80000001B753972C, Prewrite ts: 424641250168406065 can not found in default cf for key: 748000000000006CFFD25F7280000001B7FF53972C0000000000FAFA1B5F0866BBFF91, e.what() = DB::Exception, Stack trace:

  1. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x367c835]
  2. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x36733c5]
  3. bin/tiflash/tiflash(DB::RegionData::readDataByWriteIt(std::_Rb_tree_const_iterator<std::pair<std::pair<DB::RawTiDBPK, unsigned long> const, std::tuple<std::shared_ptr<DB::StringObject const>, std::shared_ptr<DB::StringObject const>, DB::RecordKVFormat::InnerDecodedWriteCFValue> > > const&, bool) const+0x50c) [0x734ef3c]
  4. bin/tiflash/tiflash(DB::Region::readDataByWriteIt(std::_Rb_tree_const_iterator<std::pair<std::pair<DB::RawTiDBPK, unsigned long> const, std::tuple<std::shared_ptr<DB::StringObject const>, std::shared_ptr<DB::StringObject const>, DB::RecordKVFormat::InnerDecodedWriteCFValue> > > const&, bool) const+0x17) [0x732ddd7]
  5. bin/tiflash/tiflash(DB::ReadRegionCommitCache(std::shared_ptrDB::Region const&, bool)+0x1c1) [0x7322fa1]
  6. bin/tiflash/tiflash(DB::RegionTable::writeBlockByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::vector<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject const> >, std::allocator<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject const> > > >&, Poco::Logger*, bool)+0x111) [0x7326051]
  7. bin/tiflash/tiflash(DB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&)+0x2c4) [0x7330ce4]
  8. bin/tiflash/tiflash(DB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&)+0x5a) [0x73201aa]
  9. bin/tiflash/tiflash(DB::HandleWriteRaftCmd(DB::EngineStoreServerWrap const*, DB::WriteCmdsView, DB::RaftCmdHeader)+0x30) [0x7327a50]
  10. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbe31f7) [0x7f2e159e51f7]
  11. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbe9e9a) [0x7f2e159ebe9a]
  12. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbbd541) [0x7f2e159bf541]
  13. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x51d0d6) [0x7f2e1531f0d6]
  14. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x5632dd) [0x7f2e153652dd]
  15. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x596f3f) [0x7f2e15398f3f]
  16. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x156ea8f) [0x7f2e16370a8f]
  17. /lib64/libpthread.so.0(+0x7e64) [0x7f2e145c7e64]
  18. /lib64/libc.so.6(clone+0x6c) [0x7f2e13fee88c]

2021.05.01 22:45:01.726700 [ 27 ] BaseDaemon: ########################################
2021.05.01 22:45:01.726742 [ 27 ] BaseDaemon: (from thread 24) Received signal Segmentation fault (11).
2021.05.01 22:45:01.726763 [ 27 ] BaseDaemon: Address: 0x7f2d3c3b776e
2021.05.01 22:45:01.726772 [ 27 ] BaseDaemon: Access: read.
2021.05.01 22:45:01.726780 [ 27 ] BaseDaemon: Address not mapped to object.
2021.05.01 22:45:17.157283 [ 1 ] Application: The configuration “path” is deprecated. Check [storage] section for new style.
2021.05.01 22:45:19.766322 [ 20 ] DB::EngineStoreApplyRes DB::HandleWriteRaftCmd(const DB::EngineStoreServerWrap*, DB::WriteCmdsView, DB::RaftCmdHeader): Code: 9008, e.displayText() = DB::Exception: Raw TiDB PK: 80000001B753972C, Prewrite ts: 424641250168406065 can not found in default cf for key: 748000000000006CFFD25F7280000001B7FF53972C0000000000FAFA1B5F0866BBFF91, e.what() = DB::Exception, Stack trace:

  1. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x367c835]
  2. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x36733c5]
  3. bin/tiflash/tiflash(DB::RegionData::readDataByWriteIt(std::_Rb_tree_const_iterator<std::pair<std::pair<DB::RawTiDBPK, unsigned long> const, std::tuple<std::shared_ptr<DB::StringObject const>, std::shared_ptr<DB::StringObject const>, DB::RecordKVFormat::InnerDecodedWriteCFValue> > > const&, bool) const+0x50c) [0x734ef3c]
  4. bin/tiflash/tiflash(DB::Region::readDataByWriteIt(std::_Rb_tree_const_iterator<std::pair<std::pair<DB::RawTiDBPK, unsigned long> const, std::tuple<std::shared_ptr<DB::StringObject const>, std::shared_ptr<DB::StringObject const>, DB::RecordKVFormat::InnerDecodedWriteCFValue> > > const&, bool) const+0x17) [0x732ddd7]
  5. bin/tiflash/tiflash(DB::ReadRegionCommitCache(std::shared_ptrDB::Region const&, bool)+0x1c1) [0x7322fa1]
  6. bin/tiflash/tiflash(DB::RegionTable::writeBlockByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::vector<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject const> >, std::allocator<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject const> > > >&, Poco::Logger*, bool)+0x111) [0x7326051]
  7. bin/tiflash/tiflash(DB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&)+0x2c4) [0x7330ce4]
  8. bin/tiflash/tiflash(DB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&)+0x5a) [0x73201aa]
  9. bin/tiflash/tiflash(DB::HandleWriteRaftCmd(DB::EngineStoreServerWrap const*, DB::WriteCmdsView, DB::RaftCmdHeader)+0x30) [0x7327a50]
  10. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbe31f7) [0x7f2baf26b1f7]
  11. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbe9e9a) [0x7f2baf271e9a]
  12. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbbd541) [0x7f2baf245541]
  13. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x51d0d6) [0x7f2baeba50d6]
  14. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x5632dd) [0x7f2baebeb2dd]
  15. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x596f3f) [0x7f2baec1ef3f]
  16. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x156ea8f) [0x7f2bafbf6a8f]
  17. /lib64/libpthread.so.0(+0x7e64) [0x7f2bade4de64]
  18. /lib64/libc.so.6(clone+0x6c) [0x7f2bad87488c]

2021.05.01 22:45:19.767778 [ 27 ] BaseDaemon: ########################################
2021.05.01 22:45:19.767823 [ 27 ] BaseDaemon: (from thread 23) Received signal Segmentation fault (11).
2021.05.01 22:45:19.767841 [ 27 ] BaseDaemon: Address: NULL pointer.
2021.05.01 22:45:19.767860 [ 27 ] BaseDaemon: Access: read.
2021.05.01 22:45:19.767903 [ 27 ] BaseDaemon: Unknown si_code.
2021.05.01 22:45:35.153526 [ 1 ] Application: The configuration “path” is deprecated. Check [storage] section for new style.
2021.05.01 22:45:37.757130 [ 21 ] DB::EngineStoreApplyRes DB::HandleWriteRaftCmd(const DB::EngineStoreServerWrap*, DB::WriteCmdsView, DB::RaftCmdHeader): Code: 9008, e.displayText() = DB::Exception: Raw TiDB PK: 80000001B753972C, Prewrite ts: 424641250168406065 can not found in default cf for key: 748000000000006CFFD25F7280000001B7FF53972C0000000000FAFA1B5F0866BBFF91, e.what() = DB::Exception, Stack trace:

  1. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x367c835]
  2. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x36733c5]
  3. bin/tiflash/tiflash(DB::RegionData::readDataByWriteIt(std::_Rb_tree_const_iterator<std::pair<std::pair<DB::RawTiDBPK, unsigned long> const, std::tuple<std::shared_ptr<DB::StringObject const>, std::shared_ptr<DB::StringObject const>, DB::RecordKVFormat::InnerDecodedWriteCFValue> > > const&, bool) const+0x50c) [0x734ef3c]
  4. bin/tiflash/tiflash(DB::Region::readDataByWriteIt(std::_Rb_tree_const_iterator<std::pair<std::pair<DB::RawTiDBPK, unsigned long> const, std::tuple<std::shared_ptr<DB::StringObject const>, std::shared_ptr<DB::StringObject const>, DB::RecordKVFormat::InnerDecodedWriteCFValue> > > const&, bool) const+0x17) [0x732ddd7]
  5. bin/tiflash/tiflash(DB::ReadRegionCommitCache(std::shared_ptrDB::Region const&, bool)+0x1c1) [0x7322fa1]
  6. bin/tiflash/tiflash(DB::RegionTable::writeBlockByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::vector<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject const> >, std::allocator<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject const> > > >&, Poco::Logger*, bool)+0x111) [0x7326051]
  7. bin/tiflash/tiflash(DB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&)+0x2c4) [0x7330ce4]
  8. bin/tiflash/tiflash(DB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&)+0x5a) [0x73201aa]
  9. bin/tiflash/tiflash(DB::HandleWriteRaftCmd(DB::EngineStoreServerWrap const*, DB::WriteCmdsView, DB::RaftCmdHeader)+0x30) [0x7327a50]
  10. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbe31f7) [0x7efdb8dca1f7]
  11. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbe9e9a) [0x7efdb8dd0e9a]
  12. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbbd541) [0x7efdb8da4541]
  13. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x51d0d6) [0x7efdb87040d6]
  14. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x5632dd) [0x7efdb874a2dd]
  15. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x596f3f) [0x7efdb877df3f]
  16. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x156ea8f) [0x7efdb9755a8f]
  17. /lib64/libpthread.so.0(+0x7e64) [0x7efdb79ace64]
  18. /lib64/libc.so.6(clone+0x6c) [0x7efdb73d388c]

2021.05.01 22:45:37.758690 [ 28 ] BaseDaemon: ########################################
2021.05.01 22:45:37.758731 [ 28 ] BaseDaemon: (from thread 26) Received signal Segmentation fault (11).
2021.05.01 22:45:37.758745 [ 28 ] BaseDaemon: Address: 0x102000084
2021.05.01 22:45:37.758758 [ 28 ] BaseDaemon: Access: read.
2021.05.01 22:45:37.758766 [ 28 ] BaseDaemon: Address not mapped to object.
2021.05.01 22:45:53.155226 [ 1 ] Application: The configuration “path” is deprecated. Check [storage] section for new style.
q2021.05.01 22:45:55.964761 [ 20 ] DB::EngineStoreApplyRes DB::HandleWriteRaftCmd(const DB::EngineStoreServerWrap*, DB::WriteCmdsView, DB::RaftCmdHeader): Code: 9008,e.displayText() = DB::Exception: Raw TiDB PK: 80000001B753972C, Prewrite ts: 424641250168406065 can not found in default cf for key: 748000000000006CFFD25F7280000001B7FF53972C0000000000FAFA1B5F0866BBFF91, e.what() = DB::Exception, Stack trace:

  1. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x367c835]
  2. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x36733c5]
  3. bin/tiflash/tiflash(DB::RegionData::readDataByWriteIt(std::_Rb_tree_const_iterator<std::pair<std::pair<DB::RawTiDBPK, unsigned long> const, std::tuple<std::shared_ptr<DB::StringObject const>, std::shared_ptr<DB::StringObject const>, DB::RecordKVFormat::InnerDecodedWriteCFValue> > > const&, bool) const+0x50c) [0x734ef3c]
  4. bin/tiflash/tiflash(DB::Region::readDataByWriteIt(std::_Rb_tree_const_iterator<std::pair<std::pair<DB::RawTiDBPK, unsigned long> const, std::tuple<std::shared_ptr<DB::StringObject const>, std::shared_ptr<DB::StringObject const>, DB::RecordKVFormat::InnerDecodedWriteCFValue> > > const&, bool) const+0x17) [0x732ddd7]
  5. bin/tiflash/tiflash(DB::ReadRegionCommitCache(std::shared_ptrDB::Region const&, bool)+0x1c1) [0x7322fa1]
  6. bin/tiflash/tiflash(DB::RegionTable::writeBlockByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::vector<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject const> >, std::allocator<std::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::shared_ptr<DB::StringObject const> > > >&, Poco::Logger*, bool)+0x111) [0x7326051]
  7. bin/tiflash/tiflash(DB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&)+0x2c4) [0x7330ce4]
  8. bin/tiflash/tiflash(DB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&)+0x5a) [0x73201aa]
  9. bin/tiflash/tiflash(DB::HandleWriteRaftCmd(DB::EngineStoreServerWrap const*, DB::WriteCmdsView, DB::RaftCmdHeader)+0x30) [0x7327a50]
  10. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbe31f7) [0x7fbb10b1d1f7]
  11. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbe9e9a) [0x7fbb10b23e9a]
  12. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xbbd541) [0x7fbb10af7541]
  13. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x51d0d6) [0x7fbb104570d6]
  14. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x5632dd) [0x7fbb1049d2dd]
  15. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x596f3f) [0x7fbb104d0f3f]
  16. /opt/ssd/tidb/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x156ea8f) [0x7fbb114a8a8f]
  17. /lib64/libpthread.so.0(+0x7e64) [0x7fbb0f6ffe64]
  18. /lib64/libc.so.6(clone+0x6c) [0x7fbb0f12688c]

2021.05.01 22:45:55.965923 [ 27 ] BaseDaemon: ########################################
2021.05.01 22:45:55.965968 [ 27 ] BaseDaemon: (from thread 24) Received signal Segmentation fault (11).
2021.05.01 22:45:55.965989 [ 27 ] BaseDaemon: Address: 0xf060001c0
2021.05.01 22:45:55.966003 [ 27 ] BaseDaemon: Access: read.
2021.05.01 22:45:55.966012 [ 27 ] BaseDaemon: Address not mapped to object.

若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出的打印结果，请务必全选并复制粘贴上传。

2

log.zip (7.3 MB)

3

麻烦导出 tiflash 监控面板 TiFlash-Summary 、 TiFlash-Proxy-Summary 和 TiFlash-Proxy-Details。
https://metricstool.pingcap.com/#backup-with-dev-tools。

4

在同步 tiflash 表的时候，集群上游什么操作吗？比如说使用 dm 或者 lightning 同步工具？

5

tidb-app-TiFlash-Proxy-Details_2021-05-02T04_37_25.245Z.json (3.0 MB) tidb-app-TiFlash-Proxy-Summary_2021-05-02T04_36_52.756Z.json (373.9 KB) tidb-app-TiFlash-Summary_2021-05-02T04_36_11.313Z.json (2.8 MB)

6

dm在实时同步上游mysql数据

7

有分析结果，我们会及时更新帖子，感谢理解和等待

8

请问挂掉之前，有对正在同步的表进行查询操作吗？

9

请问

  1. 表结构是怎样的？
  2. dm同步的模式？
  3. 确认一下 tiflash 版本是 5.0.1 还是5.0.0？
10

没有查询操作

11

1、我是一次性同步100多个表，具体是哪个表有问题，我也不清楚；
2、task-mode: incremental
3、先是用了5.0.0，有这个问题，然后升级到了5.0.1,还是有

12

目前已经定位到这是一个 bug
https://github.com/pingcap/tics/issues/1692

13

我打开这个连接提示404，请问目前有什么解决方法吗

14

您这面是线上生产环境吗？
目前 tiflash 的 issue 是不对外的。

15

是的，目前是属于生产环境使用，我可以自行判断同步哪些表会出现问题吗，那我就跳过这些表

16

目前还没有拌饭判断是那个表导致了

在和您确认一下，您以前使用过 lightning 或者 br 导入过数据吗

17

出现了Disconnected，只能是缩容再重新扩容，重做表同步吗？
全量导入用的是 lightning

18

建议您删除所有的 tiflash replica，然后锁绒再扩容。

19

您还能找到当时 lightning 导入数据是的日志文件吗？能查看一下日志文件最后是否有 success 的关键字吗？

20

image
image1887×341 22.3 KB