tispark的spark配置问题

❯ cat /etc/apache-spark/spark-defaults.conf
spark.sql.extensions org.apache.spark.sql.TiExtensions
spark.tispark.pd.addresses 127.0.0.1:2379
spark.sql.catalog.tidb_catalog org.apache.spark.sql.catalyst.catalog.TiCatalog
spark.sql.catalog.tidb_catalog.pd.addresses 127.0.0.1:2379

当我跑tispark的集成测试的时候,会遇到异常

22/04/05 17:14:31 ERROR TiExtensions$: TiSpark must work with TiCatalog. Please add TiCatalog in spark conf.

spark的jar包

❯ ll /opt/apache-spark/jars/tispark-assembly*
.rw-r--r-- 26M root root 2022-04-05 16:33 /opt/apache-spark/jars/tispark-assembly-2.5.0.jar

缺少 TiCatalog 的配置,可以参考这个帖子:


https://tidb.io/blog/b8f902a9