TiDB V5 encountered error报错后 tidb-server OOM

【 TiDB 使用环境】测试
【 TiDB 版本】5.0.4
【复现路径】1.28号晚18:30左右 tidb-server oom; 1.29号 09:54-10:03之间多次oom
【遇到的问题:问题现象及影响】

出现多次:[terror.go:291] [“encountered error”] 之后1min内tidb-server 出现oom

[2023/01/28 18:26:30.792 +08:00] [ERROR] [terror.go:291] ["encountered error"] [error="[types:1292]Incorrect time value: '{0 0 0 0 0 0 0}'"] [stack="github.com/pingcap/parser/terror.Log\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20210623034316-5ee95ed0081f/terror/terror.go:291\ngithub.com/pingcap/tidb/types.(*Time).Sub\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/types/time.go:680\ngithub.com/pingcap/tidb/statistics.convertDatumToScalar\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/scalar.go:66\ngithub.com/pingcap/tidb/statistics.(*Histogram).PreCalculateScalar\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/scalar.go:93\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).initStatsBuckets\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/handle/bootstrap.go:369\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).InitStats\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/handle/bootstrap.go:406\ngithub.com/pingcap/tidb/domain.(*Domain).loadStatsWorker\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/domain/domain.go:1177"]
[2023/01/28 18:26:30.792 +08:00] [ERROR] [terror.go:291] ["encountered error"] [error="[types:1292]Incorrect time value: '{0 0 0 0 0 0 0}'"] [stack="github.com/pingcap/parser/terror.Log\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20210623034316-5ee95ed0081f/terror/terror.go:291\ngithub.com/pingcap/tidb/types.(*Time).Sub\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/types/time.go:680\ngithub.com/pingcap/tidb/statistics.convertDatumToScalar\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/scalar.go:66\ngithub.com/pingcap/tidb/statistics.(*Histogram).PreCalculateScalar\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/scalar.go:94\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).initStatsBuckets\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/handle/bootstrap.go:369\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).InitStats\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/statistics/handle/bootstrap.go:406\ngithub.com/pingcap/tidb/domain.(*Domain).loadStatsWorker\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/domain/domain.go:1177"]
[2023/01/28 18:26:30.953 +08:00] [INFO] [domain.go:1181] ["init stats info time"] ["take time"=15.752796356s]
[2023/01/28 18:26:31.265 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1907] ["cost time"=49.349023ms]
[2023/01/28 18:26:31.756 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1821] ["cost time"=73.369494ms]
[2023/01/28 18:26:32.601 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1821] ["cost time"=34.266625ms]
[2023/01/28 18:26:33.310 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1907] ["cost time"=40.109161ms]
[2023/01/28 18:26:35.282 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1907] ["cost time"=50.723493ms]
[2023/01/28 18:26:35.329 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1821] ["cost time"=76.530381ms]
[2023/01/28 18:26:36.450 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1821] ["cost time"=30.648492ms]
[2023/01/28 18:26:37.240 +08:00] [WARN] [pd.go:131] ["get timestamp too slow"] [conn=1821] ["cost time"=37.580602ms]
[2023/01/28 18:26:39.114 +08:00] [WARN] [memory_usage_alarm.go:141] ["tidb-server has the risk of OOM. Running SQLs and heap profile will be recorded in record path"] ["is server-memory-quota set"=false] ["system memory total"=33566154752] ["system memory usage"=26890240000] ["tidb-server memory usage"=8267442624] [memory-usage-alarm-ratio=0.8] ["record path"="/tmp/3013_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage/record"]
[2023/01/28 18:26:39.135 +08:00] [INFO] [manager.go:325] ["get owner"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager da53c763-89c4-47b6-bf64-f4328edd517b"] [ownerID=da53c763-89c4-47b6-bf64-f4328edd517b]

问题
如何找到 Running SQLs and heap profile

在tmp 路径 会有临时文件,保存日志文件

/tmp/3013_tidb/MC4wLjAuMxxxxxxxxxx4wLjA6MTAwODA=/tmp-storage/record

能找到最有可能导致oom的10个sql

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。