tidb节点到达内存上限不自动重启

🪐 TiDB 技术问题
1

【 TiDB 使用环境】生产环境
【 TiDB 版本】v5.4.3
【复现路径】做过哪些操作出现的问题
【遇到的问题：问题现象及影响】tidb节点内存设置为70G，最近发现tidb节点内存逐渐上涨到70G夯住，并且不重启，重新启停后仍然复现，没有发现有太异常的sql
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【附件：截图/日志/监控】

1693900089472
16939000894721146×404 95.7 KB

2

是混合部署么？有没有看看tidb日志？

1 个赞
3

可以在 tidb.log 中 grep “expensive_query” ，该 log 会记录运行超时、或使用内存超过阈值的 SQL。
如果sql暂时无法优化，设置下 tidb_mem_quota_query参数试试
https://docs.pingcap.com/zh/tidb/v5.4/system-variables#tidb_mem_quota_query

4

tidb 日志有大量这些 报错：
[2023/09/05 04:43:16.303 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“[types:1292]Incorrect time value: ‘{0 0 0 0 0 0 0}’”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/types.(*Time).Sub\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/types/time.go:680\ngithub.com/pingcap/tidb/statistics.convertDatumToScalar\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/scalar.go:77\ngithub.com/pingcap/tidb/statistics.(*Histogram).outOfRangeRowCount\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/histogram.go:896\ngithub.com/pingcap/tidb/statistics.(*Column).GetColumnRowCount\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/histogram.go:1247\ngithub.com/pingcap/tidb/statistics.(*HistColl).GetRowCountByColumnRanges\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/table.go:380\ngithub.com/pingcap/tidb/statistics.(*HistColl).Selectivity\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/selectivity.go:245\ngithub.com/pingcap/tidb/planner/core.(*DataSource).deriveStatsByFilter\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:250\ngithub.com/pingcap/tidb/planner/core.(*DataSource).DeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:406\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:131\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:124\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:124\ngithub.com/pingcap/tidb/planner/core.physicalOptimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/optimizer.go:448\ngithub.com/pingcap/tidb/planner/core.DoOptimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/optimizer.go:284\ngithub.com/pingcap/tidb/planner.optimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/optimize.go:385\ngithub.com/pingcap/tidb/planner.Optimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/optimize.go:225\ngithub.com/pingcap/tidb/executor.(*Compiler).Compile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/compiler.go:77\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1698\ngithub.com/pingcap/tidb/session.(*session).ExecRestrictedStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1632\ngithub.com/pingcap/tidb/bindinfo.(*BindHandle).Update\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/bindinfo/handle.go:142\ngithub.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/domain/domain.go:1110”]
[2023/09/05 04:43:19.303 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“[types:1292]Incorrect time value: ‘{0 0 0 0 0 0 0}’”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/types.(*Time).Sub\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/types/time.go:680\ngithub.com/pingcap/tidb/statistics.convertDatumToScalar\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/scalar.go:77\ngithub.com/pingcap/tidb/statistics.(*Histogram).outOfRangeRowCount\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/histogram.go:896\ngithub.com/pingcap/tidb/statistics.(*Column).GetColumnRowCount\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/histogram.go:1247\ngithub.com/pingcap/tidb/statistics.(*HistColl).GetRowCountByColumnRanges\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/table.go:380\ngithub.com/pingcap/tidb/statistics.(*HistColl).Selectivity\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/selectivity.go:245\ngithub.com/pingcap/tidb/planner/core.(*DataSource).deriveStatsByFilter\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:250\ngithub.com/pingcap/tidb/planner/core.(*DataSource).DeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:406\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:131\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:124\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:124\ngithub.com/pingcap/tidb/planner/core.physicalOptimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/optimizer.go:448\ngithub.com/pingcap/tidb/planner/core.DoOptimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/optimizer.go:284\ngithub.com/pingcap/tidb/planner.optimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/optimize.go:385\ngithub.com/pingcap/tidb/planner.Optimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/optimize.go:225\ngithub.com/pingcap/tidb/executor.(*Compiler).Compile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/compiler.go:77\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1698\ngithub.com/pingcap/tidb/session.(*session).ExecRestrictedStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1632\ngithub.com/pingcap/tidb/bindinfo.(*BindHandle).Update\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/bindinfo/handle.go:142\ngithub.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/domain/domain.go:1110”]
[2023/09/05 04:43:22.304 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“[types:1292]Incorrect time value: ‘{0 0 0 0 0 0 0}’”] [stack=“github.com/pingcap/tidb/parser/terror.Log\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:307\ngithub.com/pingcap/tidb/types.(*Time).Sub\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/types/time.go:680\ngithub.com/pingcap/tidb/statistics.convertDatumToScalar\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/scalar.go:77\ngithub.com/pingcap/tidb/statistics.(*Histogram).outOfRangeRowCount\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/histogram.go:896\ngithub.com/pingcap/tidb/statistics.(*Column).GetColumnRowCount\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/histogram.go:1247\ngithub.com/pingcap/tidb/statistics.(*HistColl).GetRowCountByColumnRanges\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/table.go:380\ngithub.com/pingcap/tidb/statistics.(*HistColl).Selectivity\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/selectivity.go:245\ngithub.com/pingcap/tidb/planner/core.(*DataSource).deriveStatsByFilter\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:250\ngithub.com/pingcap/tidb/planner/core.(*DataSource).DeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:406\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:131\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:124\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/stats.go:124\ngithub.com/pingcap/tidb/planner/core.physicalOptimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/optimizer.go:448\ngithub.com/pingcap/tidb/planner/core.DoOptimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/optimizer.go:284\ngithub.com/pingcap/tidb/planner.optimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/optimize.go:385\ngithub.com/pingcap/tidb/planner.Optimize\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/optimize.go:225\ngithub.com/pingcap/tidb/executor.(*Compiler).Compile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/compiler.go:77\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1698\ngithub.com/pingcap/tidb/session.(*session).ExecRestrictedStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1632\ngithub.com/pingcap/tidb/bindinfo.(*BindHandle).Update\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/bindinfo/handle.go:142\ngithub.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/domain/domain.go:1110”]

5

有TiCDC么？

1 个赞
6

[2023/09/05 04:43:16.303 +08:00] [ERROR] [terror.go:307] [“encountered error”] [error=“[types:1292]Incorrect time value: ‘{0 0 0 0 0 0 0}’”]
库里有这值么？

8

没后续了 :rofl:

9

没有的