tidb 4.0.1,tispark 2.3.0-rc.1,限定条件读取tikv,返回数据集过大，好像没有用到过滤条件

[wfxxh (倔强的蜗牛)]

on yarn的，我用的是sparklauncher
val launcher = new SparkLauncher(environment)
.setMainClass(“cn.com.wf.Source2Solr”)
.setAppName(“Source2Solr”)
.setMaster(“yarn”)
.setDeployMode(“cluster”)
.setSparkHome(“G:\…”)
.setConf(“spark.eventLog.enabled”, “true”)
.setConf(“spark.eventLog.dir”, “hdfs://…”)
.setAppResource(“F:\…”)
.setConf(“spark.yarn.jars”, “hdfs://…”)
.setConf(SparkLauncher.DRIVER_MEMORY, “1g”)
.setConf(“spark.driver.memoryOverhead”, “1g”)
.setConf(“spark.executor.instances”, “6”)
.setConf(SparkLauncher.EXECUTOR_CORES, “3”)
.setConf(SparkLauncher.EXECUTOR_MEMORY, “8g”)
.setConf(“spark.executor.memoryOverhead”, “3g”)
.setConf(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS, “-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/heap-dump-driver.log”)
.setConf(SparkLauncher.EXECUTOR_EXTRA_JAVA_OPTIONS, “-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/heap-dump-exec.log”)
.addJar("…")
.addAppArgs(args: _*)

[marsishandsome (顾亮亮 - PingCAP)]

能从yarn上下载一份全量的日志发给我吗？（包括driver和executor)

[marsishandsome (顾亮亮 - PingCAP)]

应该是权限问题，yarn节点的目录/opt/heap-dump-driver.log没有权限

如果local模式能复现的话，可以用local模式dump出来

[wfxxh (倔强的蜗牛)]

driver.log (8.8 KB)
exec.log (615.0 KB)

[wfxxh (倔强的蜗牛)]

local不行，我们tidb在另一个机房，本地连不上。。。我设置一下权限试试

[marsishandsome (顾亮亮 - PingCAP)]

能否提供一下 executor 13 的日志？（提供的execuror日志貌似正好没有发生oom）

[wfxxh (倔强的蜗牛)]

不好意思，刚才没仔细看，随便找了个exec的日志就拉下来了。我已经修改了刚才的回复，那个exec就是了

[marsishandsome (顾亮亮 - PingCAP)]

请问是运行哪一句sql导致oom？

[wfxxh (倔强的蜗牛)]

select article_id,orig_num,platform,third_url from qk_eng.perio_art_source where primary_partition >= 0 and primary_partition < 10000 and last_modify_time > timestamp ‘2020-06-30 13:41:36’
执行了5句这类的sql，区别是以前是qk_chi库，现在是qk_eng库。

[marsishandsome (顾亮亮 - PingCAP)]

我已经能复现了，多谢

[marsishandsome (顾亮亮 - PingCAP)]

能否试一下加上这个参数，看看是否还会oom？

spark.tispark.partition_per_split 1

[wfxxh (倔强的蜗牛)]

确实是的，加上这个配置就不会oom了

[marsishandsome (顾亮亮 - PingCAP)]

后续我们会fix这个问题，请关注 https://github.com/pingcap/tispark/pull/1519