tiflash突发异常，频繁重启

Hacker_2SoDekMm · 2021 年3 月 24 日 07:22

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：

[TiDB 版本]

[问题描述]
tiflash运行一直正常，今天突然开始频繁的重启，没有端口抢占问题，找不到原因

2021.03.24 15:00:14.708228 [ 47 ] BaseDaemon: ########################################
2021.03.24 15:00:14.708259 [ 47 ] BaseDaemon: (from thread 48) Received signal Segmentation fault (11).
2021.03.24 15:00:14.708272 [ 47 ] BaseDaemon: Address: 0xffffffffffffff70
2021.03.24 15:00:14.708284 [ 47 ] BaseDaemon: Access: read.
2021.03.24 15:00:14.708294 [ 47 ] BaseDaemon: Address not mapped to object.
2021.03.24 15:00:14.766959 [ 47 ] BaseDaemon: 0. bin/tiflash/tiflash(DB::FunctionSubstringUTF8::executeImpl(DB::Block&, std::vector<unsigned long, std::allocator > const&, unsigned long)+0x77c) [0x3da6ddc]
2021.03.24 15:00:14.766984 [ 47 ] BaseDaemon: 1. bin/tiflash/tiflash(DB::IFunctionBase::execute(DB::Block&, std::vector<unsigned long, std::allocator > const&, unsigned long)+0x3c) [0x324086c]
2021.03.24 15:00:14.767000 [ 47 ] BaseDaemon: 2. bin/tiflash/tiflash(DB::ExpressionAction::execute(DB::Block&) const+0x51a) [0x692fc9a]
2021.03.24 15:00:14.767014 [ 47 ] BaseDaemon: 3. bin/tiflash/tiflash(DB::ExpressionActions::execute(DB::Block&) const+0x6a) [0x69344ea]
2021.03.24 15:00:14.767028 [ 47 ] BaseDaemon: 4. bin/tiflash/tiflash(DB::FilterBlockInputStream::readImpl()+0x1e3) [0x680c3d3]
2021.03.24 15:00:14.767049 [ 47 ] BaseDaemon: 5. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>&, bool)+0x3bd) [0x5c38a2d]
2021.03.24 15:00:14.767063 [ 47 ] BaseDaemon: 6. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x5c38c17]
2021.03.24 15:00:14.767075 [ 47 ] BaseDaemon: 7. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1b) [0x680887b]
2021.03.24 15:00:14.767089 [ 47 ] BaseDaemon: 8. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>&, bool)+0x3bd) [0x5c38a2d]
2021.03.24 15:00:14.767102 [ 47 ] BaseDaemon: 9. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x5c38c17]
2021.03.24 15:00:14.767116 [ 47 ] BaseDaemon: 10. bin/tiflash/tiflash(DB::Aggregator::execute(std::shared_ptrDB::IBlockInputStream const&, DB::AggregatedDataVariants&)+0x256) [0x6afff96]
2021.03.24 15:00:14.767129 [ 47 ] BaseDaemon: 11. bin/tiflash/tiflash(DB::AggregatingBlockInputStream::readImpl()+0x3d3) [0x6a6a0c3]
2021.03.24 15:00:14.767143 [ 47 ] BaseDaemon: 12. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>&, bool)+0x3bd) [0x5c38a2d]
2021.03.24 15:00:14.767155 [ 47 ] BaseDaemon: 13. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x5c38c17]
2021.03.24 15:00:14.767185 [ 47 ] BaseDaemon: 14. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1b) [0x680887b]
2021.03.24 15:00:14.767208 [ 47 ] BaseDaemon: 15. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>&, bool)+0x3bd) [0x5c38a2d]
2021.03.24 15:00:14.767221 [ 47 ] BaseDaemon: 16. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x5c38c17]
2021.03.24 15:00:14.767233 [ 47 ] BaseDaemon: 17. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1b) [0x680887b]
2021.03.24 15:00:14.767247 [ 47 ] BaseDaemon: 18. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>&, bool)+0x3bd) [0x5c38a2d]
2021.03.24 15:00:14.767261 [ 47 ] BaseDaemon: 19. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x17) [0x5c38c17]
2021.03.24 15:00:14.767276 [ 47 ] BaseDaemon: 20. bin/tiflash/tiflash(DB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::atomic)+0x5d) [0x68c690d]
2021.03.24 15:00:14.767294 [ 47 ] BaseDaemon: 21. bin/tiflash/tiflash(DB::DAGDriver::execute()+0x592) [0x6bd2912]
2021.03.24 15:00:14.767311 [ 47 ] BaseDaemon: 22. bin/tiflash/tiflash(DB::CoprocessorHandler::execute()+0x640) [0x6c01c70]
2021.03.24 15:00:14.767326 [ 47 ] BaseDaemon: 23. bin/tiflash/tiflash(DB::FlashService::Coprocessor(grpc_impl::ServerContext*, coprocessor::Request const*, coprocessor::Response*)+0x6f3) [0x6bb7fe3]
2021.03.24 15:00:14.767358 [ 47 ] BaseDaemon: 24. bin/tiflash/tiflash(std::_Function_handler<grpc::Status (tikvpb::Tikv::Service*, grpc_impl::ServerContext*, coprocessor::Request const*, coprocessor::Response*), std::_Mem_fn<grpc::Status (tikvpb::Tikv::Service::)(grpc_impl::ServerContext, coprocessor::Request const*, coprocessor::Response*)> >::_M_invoke(std::_Any_data const&, tikvpb::Tikv::Service*&&, grpc_impl::ServerContext*&&, coprocessor::Request const*&&, coprocessor::Response*&&)+0x39) [0x7362f39]
2021.03.24 15:00:14.767384 [ 47 ] BaseDaemon: 25. bin/tiflash/tiflash(grpc::Status grpc::internal::CatchingFunctionHandler<grpc::internal::RpcMethodHandler<tikvpb::Tikv::Service, coprocessor::Request, coprocessor::Response>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}>(grpc::internal::RpcMethodHandler<tikvpb::Tikv::Service, coprocessor::Request, coprocessor::Response>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}&&)+0x54) [0x73b9564]
2021.03.24 15:00:14.767400 [ 47 ] BaseDaemon: 26. bin/tiflash/tiflash(grpc::internal::RpcMethodHandler<tikvpb::Tikv::Service, coprocessor::Request, coprocessor::Response>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)+0x1fe) [0x73f0bae]
2021.03.24 15:00:14.767414 [ 47 ] BaseDaemon: 27. bin/tiflash/tiflash(grpc_impl::Server::SyncRequest::CallData::ContinueRunAfterInterception()+0x161) [0x768ff91]
2021.03.24 15:00:14.767428 [ 47 ] BaseDaemon: 28. bin/tiflash/tiflash(grpc_impl::Server::SyncRequestThreadManager::DoWork(void*, bool, bool)+0x430) [0x7691550]
2021.03.24 15:00:14.767442 [ 47 ] BaseDaemon: 29. bin/tiflash/tiflash(grpc::ThreadManager::MainWorkLoop()+0x9b) [0x769718b]
2021.03.24 15:00:14.767454 [ 47 ] BaseDaemon: 30. bin/tiflash/tiflash(grpc::ThreadManager::WorkerThread::Run()+0xc) [0x769726c]
2021.03.24 15:00:14.767466 [ 47 ] BaseDaemon: 31. bin/tiflash/tiflash() [0x7789903]
2021.03.24 15:00:14.767478 [ 47 ] BaseDaemon: 32. /lib64/libpthread.so.0(+0x7dd5) [0x7f003efccdd5]

若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出的打印结果，请务必全选并复制粘贴上传。

不懂就问 · 2021 年3 月 24 日 09:06

是什么版本的 tiflash？可以看下是否是 OOM 引起的重启，如果可以排除 OOM 问题导致的，可以看下 TiFlash log 下面日志信息或者传一份到帖子里。

Hacker_2SoDekMm · 2021 年3 月 24 日 09:19

不是OOM引起的，tiflash.log 日志如下。我下掉了昨天添加的一个表，tiflash暂时不再重启了，但不知道什么原因

Hacker_2SoDekMm · 2021 年3 月 24 日 09:20

日常tiflash会有以下警告，但也不是影响重启的原因

yilong · 2021 年3 月 26 日 03:05

message 日志里有什么信息吗？

Hacker_2SoDekMm · 2021 年3 月 26 日 06:10

只有这点信息，没有有价值的

yilong · 2021 年3 月 29 日 02:25

好的，多谢，如果下次可以复现问题，麻烦继续反馈。

windtaker · 2021 年5 月 13 日 04:37

请问这个问题现在还存在吗，看这个log是substring函数有bug导致tiflash segment fault了

Lucien · 2021 年5 月 16 日 03:16

@Hacker_2SoDekMm 希望有进展反馈一下，感谢啦～