TiSpark 本地开发问题

[古月水泽]

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：

[TiDB 版本]
TIDB 5.0.1
[问题描述]
本地开发运行时缺少 TiCompositeSessionCatalog 请问这个该如何修改
def main(args: Array[String]): Unit = {
//System.setProperty(“hadoop.home.dir”, “D:\java\hadoop\hadoop-2.7.1\bin\winutils.exe”)
val sparkConf = new SparkConf()
.setIfMissing(“spark.master”, “spark://192.168.0.221:7077”)
.setIfMissing(“spark.app.name”, getClass.getName)
.setIfMissing(“spark.sql.runSQLOnFiles”,“true”)
.setIfMissing(“spark.sql.extensions”, “org.apache.spark.sql.TiExtensions”)
.setIfMissing(“spark.tispark.pd.addresses”, “192.168.0.222:2379,192.168.0.223:2379,192.168.0.224:2379”)
val spark = SparkSession.builder()
.config(sparkConf)
.getOrCreate();
spark.sql(“show databases”).show();
spark.sql(“use test”);
spark.sql(“select count(*) from test”).show();
spark.close();
}

报错：
21/05/05 18:08:18 INFO ReflectionUtil$: tispark class url: file:/D:/java/maven/repository/com/pingcap/tispark/tispark-core-internal/2.4.0/tispark-core-internal-2.4.0.jar
21/05/05 18:08:18 INFO ReflectionUtil$: spark wrapper class url: jar:file:/D:/java/maven/repository/com/pingcap/tispark/tispark-core-internal/2.4.0/tispark-core-internal-2.4.0.jar!/resources/spark-wrapper-spark-2_4/
Exception in thread “main” java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.catalog.TiCompositeSessionCatalog
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at com.pingcap.tispark.utils.ReflectionUtil$.newTiCompositeSessionCatalog(ReflectionUtil.scala:108)
at org.apache.spark.sql.TiContext.tiCatalog$lzycompute(TiContext.scala:49)
at org.apache.spark.sql.TiContext.tiCatalog(TiContext.scala:49)
at org.apache.spark.sql.execution.command.TiCommand.tiCatalog(TiCommand.scala:30)
at org.apache.spark.sql.execution.command.TiShowDatabasesCommand.run(databases.scala:44)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withAction(Dataset.scala:3369)
at org.apache.spark.sql.Dataset.(Dataset.scala:194)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:643)
at com.dgh.hzf.test$.main(test.scala:25)
at com.dgh.hzf.test.main(test.scala)

若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出的打印结果，请务必全选并复制粘贴上传。

[yilong (yi888long)]

请问 spark 和 tispark 是什么版本？

[古月水泽]

tispark 2.4.0 spark 2.4.7

[yilong (yi888long)]

找了下，感觉在这里，请问相关的包都导入了哪些？https://github.com/pingcap/tispark/blob/d076066aab0f55f703b3f0668f290fc694cfdad1/core/src/main/scala/org/apache/spark/sql/TiContext.scala

[古月水泽]

[marsishandsome (顾亮亮 - PingCAP)]

请问 pom 里面是否依赖这个？

<dependencies>
    <dependency>
      <groupId>com.pingcap.tispark</groupId>
      <artifactId>tispark-assembly</artifactId>
      <version>2.4.0</version>
    </dependency>
</dependencies>

[古月水泽]

是的 在Tispark集群里 spark-sql 是可以的
spark-sql> select * from test limit 10;
CN204934249U NULL 13998080
CN204957586U NULL 13998081
CN204698062U NULL 13998082
CN204698133U NULL 13998083
CN204727064U NULL 13998084
CN204727065U NULL 13998085
CN104961436A NULL 13998086
CN104961437A NULL 13998087
CN104961438A NULL 13998088
CN104961479A NULL 13998089
Time taken: 1.391 seconds, Fetched 10 row(s)
spark-sql>

[marsishandsome (顾亮亮 - PingCAP)]

能否提供一下 pom 文件？

[古月水泽]

好的 pom.xml (5.2 KB)

[marsishandsome (顾亮亮 - PingCAP)]

请将依赖修改成如下试试

<dependency>
                    <groupId>com.pingcap.tispark</groupId>
                    <artifactId>tispark-assembly</artifactId>
                    <version>2.4.0</version>
                    <exclusions>
                        <exclusion>
                            <groupId>com.pingcap.tispark</groupId>
                            <artifactId>tispark-core-internal</artifactId>
                        </exclusion>
                    </exclusions>
                </dependency>

[古月水泽]

您好 依赖修改后是可以执行
spark.sql(“show databases”).show(false)
tidb中的数据库可以显示出来了
不过
spark.sql(“use test”)
spark.sql(
“”"
|select count(*)
|from test.test
“”".stripMargin).show(false)

跑sql代码时 一直停留在了 runnning 状态中

[marsishandsome (顾亮亮 - PingCAP)]

select count(*) 在Tispark集群里能执行成功吗？

[古月水泽]

是的 可以的
spark-sql> select count(*) from test.test;
18851659

[marsishandsome (顾亮亮 - PingCAP)]

检查一下本地机器的网络和 tikv 的网络是否联通？

[古月水泽]

网络是通的

[marsishandsome (顾亮亮 - PingCAP)]

看下 task 执行状况？有没有报错？总共多少 task？完成了多少？

[古月水泽]

感谢 可以运行了 因为我在本运行的 造成问题的是本地 spark --driver-url 参数

[奇奇怪怪的猿]

大佬你好，方便看下这个主题的问题吗？

java操作tispark找不到TiCompositeSessionCatalog类

【 TiDB 使用环境】 tidb4.0 ，spark版本2.3.2 【概述】 场景 + 问题概述 使用Java操作tispark报错，找不到TiCompositeSessionCatalog类 [image] 【背景】 做过哪些操作 pom文件中引入了tispark的jar [image]