TPCH性能测试

🪐 TiDB 技术问题 备份&数据迁移
1

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：

【概述】 场景 + 问题概述
TPCH性能测试
【备份和数据迁移策略逻辑】

【背景】 做过哪些操作

【现象】 业务和数据库现象
tiflash 出错，tidb查询出错
【问题】 当前遇到的问题

【业务影响】

【TiDB 版本】
v5.0.0
【附件】

  • 相关日志、配置文件、Grafana 监控（https://metricstool.pingcap.com/)
  • TiUP Cluster Display 信息
  • TiUP CLuster Edit config 信息
  • TiDB-Overview 监控
  • 对应模块的 Grafana 监控（如有 BR、TiDB-binlog、TiCDC 等）
  • 对应模块日志（包含问题前后 1 小时日志）

tpch100G
当查询第五条sql时出现如下问题：

235e8d5503192f168c3e137223bc41c
235e8d5503192f168c3e137223bc41c1529×397 11.2 KB
235e8d5503192f168c3e137223bc41c
235e8d5503192f168c3e137223bc41c1529×397 11.2 KB

若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出的打印结果，请务必全选并复制粘贴上传。

2

测试场景是 TPCH 100GB 么，集群硬件配置如何，是否参考了这里的参数 https://docs.pingcap.com/zh/tidb/stable/v5.0-performance-benchmarking-with-tpch

3

已经按照文档要求

image
image702×74 3.3 KB

4

那发一下报错的 tiflash 日志吧， 包含问题前后 1 小时日志，再补充描述下集群的拓扑信息

5

image
image1466×388 42.8 KB

[2021/06/08 11:14:01.551 +08:00] [INFO] [] [“DAGQueryBlockInterpreter: execution stream size for query block(before aggregation) _QB_1 is 16”] [thread_id=216]
[2021/06/08 11:14:01.561 +08:00] [INFO] [] [“MPPHandler: processing dispatch is over; the time cost is 62 ms”] [thread_id=216]
[2021/06/08 11:14:01.561 +08:00] [INFO] [] [“task 6: task starts running”] [thread_id=217]
[2021/06/08 11:14:01.660 +08:00] [INFO] [] [“FlashService: connection for tunnel3+5 cost 168 ms.”] [thread_id=218]
[2021/06/08 11:14:01.660 +08:00] [INFO] [] [“task 3: task ends, time cost is 166 ms.”] [thread_id=208]
[2021/06/08 11:14:03.409 +08:00] [INFO] [] [“FlashService: connection for tunnel4+5 cost 1914 ms.”] [thread_id=202]
[2021/06/08 11:14:03.409 +08:00] [INFO] [] [“task 4: task ends, time cost is 1882 ms.”] [thread_id=215]
[2021/06/08 11:14:32.623 +08:00] [ERROR] [] ["task 5: task running meets error DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address Stack Trace : 0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x367e645]\

  1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x36751d5]\
  2. bin/tiflash/tiflash(DB::throwFromErrno(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int, int)+0x1ac) [0x367afbc]\
  3. bin/tiflash/tiflash(Allocator::realloc(void*, unsigned long, unsigned long, unsigned long)+0x423) [0x3697353]\
  4. bin/tiflash/tiflash(void DB::PODArrayBase<8ul, 4096ul, Allocator, 15ul, 16ul>::reserveForNextSize<>()+0x55) [0x3765035]\
  5. bin/tiflash/tiflash(DB::ColumnVector::insertFrom(DB::IColumn const&, unsigned long)+0x40) [0x380f2a0]\
  6. bin/tiflash/tiflash() [0x7663bce]\
  7. bin/tiflash/tiflash(void DB::Join::joinBlockImpl<(DB::ASTTableJoin::Kind)0, (DB::ASTTableJoin::Strictness)2, DB::Join::MapsTemplate<DB::Join::WithUsedFlag<false, DB::Join::RowRefList> > >(DB::Block&, DB::Join::MapsTemplate<DB::Join::WithUsedFlag<false, DB::Join::RowRefList> > const&) const+0x100b) [0x7665f5b]\
  8. bin/tiflash/tiflash(DB::Join::joinBlock(DB::Block&) const+0x1d1) [0x7609311]\
  9. bin/tiflash/tiflash(DB::ExpressionAction::execute(DB::Block&) const+0x9e) [0x75b12de]\
  10. bin/tiflash/tiflash(DB::ExpressionActions::execute(DB::Block&) const+0x69) [0x75b7819]\
  11. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x33) [0x7463dd3]\
  12. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  13. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  14. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1a) [0x7463dba]\
  15. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  16. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  17. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1a) [0x7463dba]\
  18. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  19. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  20. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1a) [0x7463dba]\
  21. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  22. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  23. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1a) [0x7463dba]\
  24. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  25. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  26. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1a) [0x7463dba]\
  27. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  28. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  29. bin/tiflash/tiflash(DB::ParallelInputsProcessor<DB::UnionBlockInputStream<(DB::StreamUnionMode)0>::Handler, (DB::StreamUnionMode)0>::loop(unsigned long)+0x144) [0x6f93004]\
  30. bin/tiflash/tiflash(DB::ParallelInputsProcessor<DB::UnionBlockInputStream<(DB::StreamUnionMode)0>::Handler, (DB::StreamUnionMode)0>::thread(MemoryTracker*, unsigned long)+0x230) [0x6f93900]\
  31. bin/tiflash/tiflash() [0x86a760e]
    "] [thread_id=211]
    [2021/06/08 11:14:32.624 +08:00] [INFO] [] [“task 5: task ends, time cost is 31118 ms.”] [thread_id=211]
    [2021/06/08 11:14:32.624 +08:00] [INFO] [] [“FlashService: connection for tunnel5+7 cost 31129 ms.”] [thread_id=219]
    [2021/06/08 11:14:32.625 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=220]
    [2021/06/08 11:14:32.625 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=221]
    [2021/06/08 11:14:32.625 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=222]
    [2021/06/08 11:14:32.625 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=223]
    [2021/06/08 11:14:32.626 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=224]
    [2021/06/08 11:14:32.627 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=225]
    [2021/06/08 11:14:32.627 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=226]
    [2021/06/08 11:14:32.628 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=227]
    [2021/06/08 11:14:32.628 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=228]
    [2021/06/08 11:14:32.628 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=229]
    [2021/06/08 11:14:32.628 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=230]
    [2021/06/08 11:14:32.629 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=231]
    [2021/06/08 11:14:32.629 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=232]
    [2021/06/08 11:14:32.629 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=233]
    [2021/06/08 11:14:32.630 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=234]
    [2021/06/08 11:14:32.630 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=235]
    [2021/06/08 11:14:32.650 +08:00] [ERROR] [] ["task 7: task running meets error DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address Stack Trace : 0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x367e645]\
  32. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x36751d5]\
  33. bin/tiflash/tiflash(DB::TiRemoteBlockInputStreamDB::ExchangeReceiver::fetchRemoteResult()+0x2f8) [0x7569bc8]\
  34. bin/tiflash/tiflash(DB::TiRemoteBlockInputStreamDB::ExchangeReceiver::readImpl()+0x257) [0x756aa07]\
  35. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  36. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  37. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1a) [0x7463dba]\
  38. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  39. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  40. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1a) [0x7463dba]\
  41. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  42. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  43. bin/tiflash/tiflash(DB::ParallelInputsProcessor<DB::UnionBlockInputStream<(DB::StreamUnionMode)0>::Handler, (DB::StreamUnionMode)0>::loop(unsigned long)+0x144) [0x6f93004]\
  44. bin/tiflash/tiflash(DB::ParallelInputsProcessor<DB::UnionBlockInputStream<(DB::StreamUnionMode)0>::Handler, (DB::StreamUnionMode)0>::thread(MemoryTracker*, unsigned long)+0x230) [0x6f93900]\
  45. bin/tiflash/tiflash() [0x86a760e]\
  46. /lib64/libpthread.so.0(+0x7e24) [0x7f043d246e24]\
  47. /lib64/libc.so.6(clone+0x6c) [0x7f043cc6ebac]
    "] [thread_id=212]
    [2021/06/08 11:14:32.650 +08:00] [INFO] [] [“task 7: task ends, time cost is 31143 ms.”] [thread_id=212]
    [2021/06/08 11:14:32.650 +08:00] [INFO] [] [“FlashService: connection for tunnel7+8 cost 31159 ms.”] [thread_id=203]
    [2021/06/08 11:14:32.652 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=236]
    [2021/06/08 11:14:32.652 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=237]
    [2021/06/08 11:14:32.653 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=238]
    [2021/06/08 11:14:32.653 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=239]
    [2021/06/08 11:14:32.653 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=240]
    [2021/06/08 11:14:32.653 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=241]
    [2021/06/08 11:14:32.653 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=242]
    [2021/06/08 11:14:32.654 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=243]
    [2021/06/08 11:14:32.654 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=244]
    [2021/06/08 11:14:32.654 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=245]
    [2021/06/08 11:14:32.654 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=246]
    [2021/06/08 11:14:32.655 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=247]
    [2021/06/08 11:14:32.655 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=248]
    [2021/06/08 11:14:32.655 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=249]
    [2021/06/08 11:14:32.655 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=250]
    [2021/06/08 11:14:32.655 +08:00] [WARN] [] [“TiRemoteBlockInputStream(ExchangeReceiver): remote reader meets error: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address”] [thread_id=251]
    [2021/06/08 11:14:32.674 +08:00] [ERROR] [] ["DB::UnionBlockInputStream::~UnionBlockInputStream() [with DB::StreamUnionMode mode = (DB::StreamUnionMode)0]: Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception:
    Code: 0, e.displayText() = DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address, e.what() = DB::Exception, e.what() = DB::Exception, Stack trace:
    \
  48. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x367e645]\
  49. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x36751d5]\
  50. bin/tiflash/tiflash(DB::TiRemoteBlockInputStreamDB::ExchangeReceiver::fetchRemoteResult()+0x2f8) [0x7569bc8]\
  51. bin/tiflash/tiflash(DB::TiRemoteBlockInputStreamDB::ExchangeReceiver::readImpl()+0x257) [0x756aa07]\
  52. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  53. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  54. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1a) [0x7463dba]\
  55. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  56. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  57. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1a) [0x7463dba]\
  58. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  59. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  60. bin/tiflash/tiflash(DB::ParallelInputsProcessor<DB::UnionBlockInputStream<(DB::StreamUnionMode)0>::Handler, (DB::StreamUnionMode)0>::loop(unsigned long)+0x144) [0x6f93004]\
  61. bin/tiflash/tiflash(DB::ParallelInputsProcessor<DB::UnionBlockInputStream<(DB::StreamUnionMode)0>::Handler, (DB::StreamUnionMode)0>::thread(MemoryTracker*, unsigned long)+0x230) [0x6f93900]\
  62. bin/tiflash/tiflash() [0x86a760e]\
  63. /lib64/libpthread.so.0(+0x7e24) [0x7f043d246e24]\
  64. /lib64/libc.so.6(clone+0x6c) [0x7f043cc6ebac]
    "] [thread_id=203]
    [2021/06/08 11:14:32.675 +08:00] [WARN] [] [“exchange_receiver: Decode packet meet error, exit from ReadLoop”] [thread_id=252]
    [2021/06/08 11:14:32.696 +08:00] [ERROR] [] ["task 8: task running meets error DB::Exception: exchange receiver meet error : DB::Exception: exchange receiver meet error : DB::Exception: Allocator: Cannot mremap memory chunk from 512.00 MiB to 1.00 GiB., errno: 14, strerror: Bad address Stack Trace : 0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x367e645]\
  65. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x36751d5]\
  66. bin/tiflash/tiflash(DB::TiRemoteBlockInputStreamDB::ExchangeReceiver::fetchRemoteResult()+0x2f8) [0x7569bc8]\
  67. bin/tiflash/tiflash(DB::TiRemoteBlockInputStreamDB::ExchangeReceiver::readImpl()+0x257) [0x756aa07]\
  68. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  69. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  70. bin/tiflash/tiflash(DB::ExpressionBlockInputStream::readImpl()+0x1a) [0x7463dba]\
  71. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x3bc) [0x686482c]\
  72. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  73. bin/tiflash/tiflash(DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler, (DB::StreamUnionMode)0>::loop(unsigned long)+0xea) [0x777ef5a]\
  74. bin/tiflash/tiflash(DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler, (DB::StreamUnionMode)0>::thread(MemoryTracker*, unsigned long)+0x20c) [0x777f4ac]\
  75. bin/tiflash/tiflash() [0x86a760e]\
  76. /lib64/libpthread.so.0(+0x7e24) [0x7f043d246e24]\
  77. /lib64/libc.so.6(clone+0x6c) [0x7f043cc6ebac]
    "] [thread_id=206]
    [2021/06/08 11:14:32.697 +08:00] [INFO] [] [“task 8: task ends, time cost is 31205 ms.”] [thread_id=206]
    [2021/06/08 11:14:32.697 +08:00] [INFO] [] [“FlashService: connection for tunnel8±1 cost 31202 ms.”] [thread_id=253]
    [2021/06/08 11:14:32.698 +08:00] [WARN] [] [“TaskManager: Begin cancel query: 425491704428363777”] [thread_id=204]
    [2021/06/08 11:14:32.698 +08:00] [WARN] [] ["TaskManager: Remaining task in query 425491704428363777 are: [425491704428363777,6] "] [thread_id=204]
    [2021/06/08 11:14:32.698 +08:00] [WARN] [] [“task 6: Begin cancel task: [425491704428363777,6]”] [thread_id=254]
    [2021/06/08 11:14:32.709 +08:00] [ERROR] [] ["task 6: task running meets error DB::Exception: Query was cancelled Stack Trace : 0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x367e645]\
  78. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x36751d5]\
  79. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::isCancelledOrThrowIfKilled() const+0x65) [0x685ba55]\
  80. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool)+0x88) [0x68644f8]\
  81. bin/tiflash/tiflash(DB::IProfilingBlockInputStream::read()+0x16) [0x6864a16]\
  82. bin/tiflash/tiflash(DB::MPPTask::runImpl()+0x1ce) [0x78c779e]\
  83. bin/tiflash/tiflash() [0x86a760e]\
  84. /lib64/libpthread.so.0(+0x7e24) [0x7f043d246e24]\
  85. /lib64/libc.so.6(clone+0x6c) [0x7f043cc6ebac]
    "] [thread_id=217]
    [2021/06/08 11:14:32.733 +08:00] [WARN] [] [“task 6: Finish cancel task: [425491704428363777,6]”] [thread_id=254]
    [2021/06/08 11:14:32.733 +08:00] [WARN] [] [“TaskManager: Finish cancel query: 425491704428363777”] [thread_id=204]
    [2021/06/08 11:14:32.737 +08:00] [INFO] [] [“FlashService: connection for tunnel6+7 cost 31236 ms.”] [thread_id=209]
    [2021/06/08 11:14:32.741 +08:00] [ERROR] [] ["task 6: Failed to write error DB::Exception: Query was cancelled to all tunnels: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed., e.what() = DB::Exception, Stack trace:
    \
  86. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x367e645]\
  87. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x36751d5]\
  88. bin/tiflash/tiflash(DB::MPPTunnel::write(mpp::MPPDataPacket const&, bool)+0x426) [0x78af3e6]\
  89. bin/tiflash/tiflash(DB::MPPTask::writeErrToAllTunnel(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)+0x6b) [0x78cce6b]\
  90. bin/tiflash/tiflash(DB::MPPTask::runImpl()+0x10fb) [0x78c86cb]\
  91. bin/tiflash/tiflash() [0x86a760e]\
  92. /lib64/libpthread.so.0(+0x7e24) [0x7f043d246e24]\
  93. /lib64/libc.so.6(clone+0x6c) [0x7f043cc6ebac]
    "] [thread_id=217]
    [2021/06/08 11:14:32.741 +08:00] [INFO] [] [“task 6: task ends, time cost is 31180 ms.”] [thread_id=217]
    [2021/06/08 11:14:32.742 +08:00] [ERROR] [] [“TaskManager: The task [425491704428363777,6] cannot be found and fail to unregister”] [thread_id=217]
6

请问可以提供一下报错前后 grafana 中的内存监控吗？

7

image
image1863×789 78.6 KB