TiFlash突然宕机，启动不起来

🪐 TiDB 技术问题
1

【 TiDB 使用环境】生产环境
【 TiDB 版本】7.1.0
【复现路径】做过哪些操作出现的问题
【遇到的问题：问题现象及影响】
[2023/10/30 22:54:02.439 +08:00] [WARN] [store.rs:1793] [“set thread priority for raftstore failed”] [error=“Os { code: 13, kind: PermissionDenied, message: "Permission denied" }”]

[2023/10/30 23:05:51.485 +08:00] [ERROR] [Exception.cpp:90] ["Code: 33, e.displayText() = DB::Exception: Cannot read all data, e.what() = DB::Exception, Stack trace:
       0x1bfe42e\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+29353006]
                \tdbms/src/Common/Exception.h:46
       0x1c84ff0\tDB::ReadBuffer::readStrict(char*, unsigned long) [tiflash+29904880]
                \tdbms/src/IO/ReadBuffer.h:161
       0x73bcadf\tDB::DM::readSegmentMetaInfo(DB::ReadBuffer&, DB::DM::Segment::SegmentMetaInfo&) [tiflash+121359071]
                \tdbms/src/Storages/DeltaMerge/Segment.cpp:318
       0x73bcd48\tDB::DM::Segment::restoreSegment(std::__1::shared_ptr<DB::Logger> const&, DB::DM::DMContext&, unsigned long) [tiflash+121359688]
                \tdbms/src/Storages/DeltaMerge/Segment.cpp:349
       0x7348013\tDB::DM::DeltaMergeStore::DeltaMergeStore(DB::Context&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, long, bool, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, DB::DM::ColumnDefine const&, bool, unsigned long, DB::DM::DeltaMergeStore::Settings const&, DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >*) [tiflash+120881171]
                \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:303
       0x7e5ad37\tDB::StorageDeltaMerge::getAndMaybeInitStore(DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >*) [tiflash+132492599]
                \tdbms/src/Storages/StorageDeltaMerge.cpp:1842
       0x7e66d33\tDB::StorageDeltaMerge::getSchemaSnapshotAndBlockForDecoding(DB::TableDoubleLockHolder<false> const&, bool) [tiflash+132541747]
                \tdbms/src/Storages/StorageDeltaMerge.cpp:1204
       0x7f7d614\tDB::writeRegionDataToStorage(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > >&, std::__1::shared_ptr<DB::Logger> const&)::$_2::operator()(bool) const [tiflash+133682708]
                \tdbms/src/Storages/Transaction/PartitionStreams.cpp:126
       0x7f7a18a\tDB::writeRegionDataToStorage(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > >&, std::__1::shared_ptr<DB::Logger> const&) [tiflash+133669258]
                \tdbms/src/Storages/Transaction/PartitionStreams.cpp:181
       0x7f79e4c\tDB::RegionTable::writeBlockByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > >&, std::__1::shared_ptr<DB::Logger> const&, bool) [tiflash+133668428]
                \tdbms/src/Storages/Transaction/PartitionStreams.cpp:359
       0x7fa41ad\tDB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&) [tiflash+133841325]
                \tdbms/src/Storages/Transaction/Region.cpp:721
       0x7f620ce\tDB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&) const [tiflash+133570766]
                \tdbms/src/Storages/Transaction/KVStore.cpp:300
       0x7f82195\tHandleWriteRaftCmd [tiflash+133702037]
                \tdbms/src/Storages/Transaction/ProxyFFI.cpp:97
0x7f8e31e4fb98\t_$LT$engine_store_ffi..observer..TiFlashObserver$LT$T$C$ER$GT$$u20$as$u20$raftstore..coprocessor..QueryObserver$GT$::post_exec_que:

【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【附件：截图/日志/监控】

2

@大飞飞jeffery /tmp/下的tidb 加777权限

3

看报错是磁盘权限问题

4

没有写权限，这个权限应该是tidb用户的权限，不是root权限

5

看报错信息，应该就是没有写权限导致的

1 个赞
6

Permission denied 是哪个组件的日志？

7

应该不是这个问题，同一个问题的另一个帖子

8

大佬怎么定位问题原因？

9

tiflash_tikv.log

10

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。