升级6.1后,TiFlash服务异常

服务异常期间其他的一下错误日志:

[2022/06/23 09:40:53.708 +08:00] [ERROR] [DAGDriver.cpp:197] ["DAGDriver:DB Exception: Failed to write resp

       0x1d272d3\tStackTrace::StackTrace() [tiflash+30569171]
                \tdbms/src/Common/StackTrace.cpp:23
       0x1d248d6\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+30558422]
                \tdbms/src/Common/Exception.h:41
       0x7b9a3f4\tDB::StreamWriter::write(tipb::SelectResponse&, unsigned short) [tiflash+129606644]
                \tdbms/src/Flash/Coprocessor/StreamWriter.h:60
       0x7b97d67\tvoid DB::StreamingDAGResponseWriter<std::__1::shared_ptr<DB::StreamWriter> >::batchWrite<true>() [tiflash+129596775]
                \tdbms/src/Flash/Coprocessor/StreamingDAGResponseWriter.cpp:314
       0x7370176\tDB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::__1::atomic<bool>*) [tiflash+121045366]
                \tdbms/src/DataStreams/copyData.cpp:78
       0x7af74d1\tDB::DAGDriver<true>::execute() [tiflash+128939217]
                \tdbms/src/Flash/Coprocessor/DAGDriver.cpp:142
       0x7ad6522\tDB::BatchCoprocessorHandler::execute() [tiflash+128804130]
                \tdbms/src/Flash/BatchCoprocessorHandler.cpp:78
       0x7ad2d3f\tstd::__1::__function::__func<DB::FlashService::BatchCoprocessor(grpc_impl::ServerContext*, coprocessor::BatchRequest const*, grpc_impl::ServerWriter<coprocessor::BatchResponse>*)::$_23, std::__1::allocator<DB::FlashService::BatchCoprocessor(grpc_impl::ServerContext*, coprocessor::BatchRequest const*, grpc_impl::ServerWriter<coprocessor::BatchResponse>*)::$_23>, grpc::Status ()>::operator()() [tiflash+128789823]
                \t/usr/local/bin/../include/c++/v1/__functional/function.h:345
       0x7ad5968\tstd::__1::__packaged_task_func<std::__1::function<grpc::Status ()>, std::__1::allocator<std::__1::function<grpc::Status ()> >, grpc::Status ()>::operator()() [tiflash+128801128]
                \t/usr/local/bin/../include/c++/v1/future:1687
       0x7ad5a65\tstd::__1::packaged_task<grpc::Status ()>::operator()() [tiflash+128801381]
                \t/usr/local/bin/../include/c++/v1/future:1960
       0x7c75e59\tThreadPool::worker() [tiflash+130506329]
                \tlibs/libcommon/src/ThreadPool.cpp:129
       0x7c76183\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ThreadPool::ThreadPool(unsigned long, std::__1::function<void ()>)::$_1> >(void*) [tiflash+130507139]
                \t/usr/local/bin/../include/c++/v1/thread:291
  0x7f5295810ea5\tstart_thread [libpthread.so.0+32421]
  0x7f52953239fd\t__clone [libc.so.6+1042941]"] [thread_id=18032]


[2022/06/23 09:40:51.204 +08:00] [ERROR] [<unknown>] ["pingcap.tikv: Failed4: Deadline Exceeded"] [thread_id=27897]
[2022/06/23 10:50:38.551 +08:00] [ERROR] [Server.cpp:310] ["grpc:/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tics/contrib/grpc/src/cpp/thread_manager/thread_manager.cc, line number: 38, log msg : Could not create grpc_sync_server worker-thread"] [thread_id=72014]

[2022/06/23 10:50:40.012 +08:00] [ERROR] [BaseDaemon.cpp:570] ["BaseDaemon:
       0x1ed2661\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+32319073]
                \tlibs/libdaemon/src/BaseDaemon.cpp:221
  0x7f5295818630\t<unknown symbol> [libpthread.so.0+63024]
  0x7f529525b387\tgsignal [libc.so.6+222087]
  0x7f529525ca78\t__GI_abort [libc.so.6+227960]
       0x88a843e\tgpr_malloc [tiflash+143295550]
                \tcontrib/grpc/src/core/lib/gpr/alloc.cc:34
       0x85b2f56\tgrpc_resource_user_alloc_slices(grpc_resource_user_slice_allocator*, unsigned long, unsigned long, grpc_slice_buffer*) [tiflash+140193622]
                \tcontrib/grpc/src/core/lib/iomgr/resource_quota.cc:1013
       0x85b5fba\ttcp_handle_read(void*, grpc_error*) [tiflash+140206010]
                \tcontrib/grpc/src/core/lib/iomgr/tcp_posix.cc:619
       0x86dc0f2\tread_action_locked(void*, grpc_error*) [tiflash+141410546]
                \tcontrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:2590
       0x85a39dc\tgrpc_combiner_continue_exec_ctx() [tiflash+140130780]
                \tcontrib/grpc/src/core/lib/iomgr/combiner.cc:236
       0x85a623d\tgrpc_core::ExecCtx::Flush() [tiflash+140141117]
                \tcontrib/grpc/src/core/lib/iomgr/exec_ctx.cc:156
       0x85ac06b\tpollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+140165227]
                \tcontrib/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1136
       0x85cca7e\tcq_pluck(grpc_completion_queue*, void*, gpr_timespec, void*) [tiflash+140298878]
                \tcontrib/grpc/src/core/lib/surface/completion_queue.cc:1281
       0x856ab11\tgrpc_impl::internal::ErrorMethodHandler<(grpc::StatusCode)8>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&) [tiflash+139897617]
                \tcontrib/grpc/include/grpcpp/impl/codegen/method_handler_impl.h:367
       0x8565260\tgrpc_impl::Server::SyncRequest::CallData::ContinueRunAfterInterception() [tiflash+139874912]
                \tcontrib/grpc/src/cpp/server/server_cc.cc:492
       0x8565080\tgrpc_impl::Server::SyncRequest::CallData::Run(std::__1::shared_ptr<grpc_impl::Server::GlobalCallbacks> const&, bool) [tiflash+139874432]
                \tcontrib/grpc/src/cpp/server/server_cc.cc:479
       0x8575bb7\tgrpc::ThreadManager::MainWorkLoop() [tiflash+139942839]
                \tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:211
       0x8576651\tgrpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::__invoke(void*) [tiflash+139945553]
                \tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:35
       0x88aca6c\tgrpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) [tiflash+143313516]
                \tcontrib/grpc/src/core/lib/gprpp/thd_posix.cc:110
  0x7f5295810ea5\tstart_thread [libpthread.so.0+32421]"] [thread_id=72017]
[2022/06/23 10:50:40.012 +08:00] [ERROR] [Server.cpp:310] ["grpc:/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tics/contrib/grpc/src/cpp/thread_manager/thread_manager.cc, line number: 38, log msg : Could not create grpc_sync_server worker-thread"] [thread_id=52616] 

其中 Could not create grpc_sync_server worker-threadDeadline Exceeded错误大量出现。