tispark 整合 java 连接时报错

Tidb版本4.0.0 ,spark版本 2.3.2 ,hadoop版本 2.7.3,tispark的jar2.4.0。在java中创建任务时报错;
2021-12-27 11:36:48.591 INFO 17260 — [nio-8088-exec-1] com.pingcap.tikv.PDClient : Switched to new leader: [leaderInfo: 172.20.131.47:2379]
2021-12-27 11:36:50.539 INFO 17260 — [nio-8088-exec-1] c.pingcap.tispark.utils.ReflectionUtil$ : tispark class url: jar:file:/home/htuser/poc/poc-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/tispark-assembly-2.4.0.jar!/
2021-12-27 11:36:50.540 INFO 17260 — [nio-8088-exec-1] c.pingcap.tispark.utils.ReflectionUtil$ : spark wrapper class url: jar:jar:file:/home/htuser/poc/poc-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/tispark-assembly-2.4.0.jar!/!/resources/spark-wrapper-spark-2_3/
2021-12-27 11:36:50.571 ERROR 17260 — [nio-8088-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.catalog.TiCompositeSessionCatalog] with root cause

java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.catalog.TiCompositeSessionCatalog
at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[na:1.8.0_312]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[na:1.8.0_312]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[na:1.8.0_312]
at com.pingcap.tispark.utils.ReflectionUtil$.newTiCompositeSessionCatalog(ReflectionUtil.scala:108) ~[tispark-assembly-2.4.0.jar!/:na]
at org.apache.spark.sql.TiContext.tiCatalog$lzycompute(TiContext.scala:49) ~[tispark-assembly-2.4.0.jar!/:2.3.2]
at org.apache.spark.sql.TiContext.tiCatalog(TiContext.scala:49) ~[tispark-assembly-2.4.0.jar!/:2.3.2]
at org.apache.spark.sql.extensions.TiResolutionRule.org$apache$spark$sql$extensions$TiResolutionRule$$tiCatalog$lzycompute(rules.scala:34) ~[tispark-assembly-2.4.0.jar!/:na]
at org.apache.spark.sql.extensions.TiResolutionRule.org$apache$spark$sql$extensions$TiResolutionRule$$tiCatalog(rules.scala:34) ~[tispark-assembly-2.4.0.jar!/:na]
at org.apache.spark.sql.extensions.TiResolutionRule$$anonfun$resolveTiDBRelations$1.applyOrElse(rules.scala:68) ~[tispark-assembly-2.4.0.jar!/:na]
at org.apache.spark.sql.extensions.TiResolutionRule$$anonfun$resolveTiDBRelations$1.applyOrElse(rules.scala:61) ~[tispark-assembly-2.4.0.jar!/:na]
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:286) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]
at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304) ~[spark-catalyst_2.11-2.3.2.jar!/:2.3.2]

1 个赞

image
我用的sparksession,可以正常的跑,请问你那的配置与我类似么?

java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.catalog.TiCompositeSessionCatalog

缺包了

貌似你发了重复的…

我给你回到这边把

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。