环境版本 tibd5.0 spark 版本2.3，tispark2.4.2 scala 2.11 jdk1.8
部署spark standlone集群
spark-defaults.conf 增加入
spark.tispark.pd.addresses $your_pd_servers
spark.sql.extensions org.apache.spark.sql.TiExtensions
配置
启动spark-shell 会报错 （不配置启动正常 ），报错如下
021-11-24 13:54:13 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2021-11-24 13:54:13 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V
at org.apache.spark.sql.TiExtensions.(TiExtensions.scala:23)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.apache.spark.sql.SparkSession$Builder.liftedTree1$1(SparkSession.scala:940)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:938)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103)
… 55 elided
:14: error: not found: value spark
import spark.implicits._
^
:14: error: not found: value spark
import spark.sql
^
Welcome to
____ __
/ / ___ / /
\ / _ / _ `/ __/ '/
// .__/_,// //_\ version 2.3.0
//
Using Scala version 2.11.8 (Java HotSpot™ 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
