spark任务中在tidb A中查询数据,插入到tidb B中,使用spark connector的方式,报无法找到NoSuchTableException: Table or view 'xxxx' not found in database 'xxxx'

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:

【TiDB 版本】
4.0
【问题描述】
spark任务中在tidb A中查询数据,插入到tidb B中,使用spark connector的方式,报错无法找到NoSuchTableException: Table or view ‘xxxx’ not found in database ‘xxxx’
下面为主要代码,在现有集群里配置了tidbA 的pd节点,查询数据后,插入到tidbB中。
查询代码
val data = spark.sql(“select * from xxxx”);
插入代码
data.write.
format(“tidb”).
option(“tidb.addr”, map(“tidb.addr”)).
option(“tidb.port”, map(“tidb.port”)).
option(“tidb.user”, map(“tidb.user”)).
option(“tidb.password”, map(“tidb.password”)).
option(“database”, “xxxx”).
option(“table”, “xxxx”).
mode(“append”).
save()
目前的现象是,tidbB中无法找到该表提示错误如下
ationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view ‘xxxxxx’ not found in database ‘xxxxxx’;
at com.pingcap.tispark.write.TiBatchWriteTable.(TiBatchWriteTable.scala:81)
at com.pingcap.tispark.write.TiBatchWrite$$anonfun$doWrite$1.apply(TiBatchWrite.scala:140)
at com.pingcap.tispark.write.TiBatchWrite$$anonfun$doWrite$1.apply(TiBatchWrite.scala:138)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at com.pingcap.tispark.write.TiBatchWrite.doWrite(TiBatchWrite.scala:138)
at com.pingcap.tispark.write.TiBatchWrite.com$pingcap$tispark$write$TiBatchWrite$$write(TiBatchWrite.scala:87)
at com.pingcap.tispark.write.TiBatchWrite$.write(TiBatchWrite.scala:45)
at com.pingcap.tispark.write.TiDBWriter$.write(TiDBWriter.scala:40)
at com.pingcap.tispark.TiDBDataSource.createRelation(TiDBDataSource.scala:57)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1$$anonfun$apply$2.apply(SparkPlan.scala:160)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1$$anonfun$apply$2.apply(SparkPlan.scala:160)
at scala.Option.getOrElse(Option.scala:121)

若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

您好,目前 TiSpark 还不支持为多个 TiDB 集群同时提供服务。所以并不能成功写入到对应的表中。

1赞

可以到 TiSpark 提交 feature 申请 https://github.com/pingcap/tispark

1赞