Tispark使用overwrite写入数据后通过tispark sql读取存在延迟

zhengyunpeng · 2019 年8 月 30 日 06:25

写入方式：

dataset
    .write
    .mode(saveMode = "overwrite")
    .format("jdbc")
    .option("driver", "com.mysql.jdbc.Driver")
        .option("url", "jdbc:mysql://192.168.10.201:4000/test?&autoReconnect=true&failOverReadOnly=false")
    .option("useSSL", "false")
    .option("dbtable", "t_order_pay_subtotal")
    .option("isolationLevel", "NONE")
    .option("truncate", "true")
    .option("user", "test")
    .option("password", "test@123")

tisparksql查询：

select * from `hydee`.t_order_pay_subtotal

直接使用mysql客户端数据能实时查询更新后数据，使用tispark查询延迟10-20s才能查到。
尝试saveMode = "append"模式，使用tisparksql也能实时查到更新后的数据

不懂就问 · 2019 年8 月 30 日 07:05

overwrite 是先 drop table 然后再 create table，当时 catalog 是每10s刷新一次，所以需要有一定延迟才能查到，是已知问题。

zhexuany-PingCAP · 2019 年8 月 30 日 08:05

@zhengyunpeng

首先感谢你使用 TiSpark 并且尝试我们的 batch write 功能。

请问是否是 drop and create 同名 table 呢？如果是，这是预期行为。我们正准备重新设计这一模块，优化这一行为。

zhengyunpeng · 2019 年8 月 30 日 09:24

overwrite的时候使用option(“truncate”, “true”)参数，不知道tidb使用什么方式处理truncate语句的，按mysql的truncate是drop and create

zhengyunpeng · 2019 年8 月 30 日 09:27

直接使用mysql客户端查询数据的时候没有延迟的

marsishandsome · 2019 年9 月 18 日 05:49

@zhengyunpeng 您好，这个问题我提了一个PR，不知道您有没有时间帮忙做一下测试？看看是否已经解决了您的问题 https://github.com/pingcap/tispark/pull/1108

zhengyunpeng · 2019 年9 月 18 日 06:49

可以的

marsishandsome · 2019 年9 月 18 日 07:05

jar包可以在这里下载，请使用spark-2.3.x或者spark-2.4.x，感谢

链接:https://pan.baidu.com/s/11J8iQrQy2xuFVYH3LNi1gQ 密码:ctn9

zhengyunpeng · 2019 年9 月 18 日 08:28

可以了，测试结果如下： tispark-assembly-2.3.0-SNAPSHOT-PR-1108.jar版：

scala> var dataset = spark.sql("select 1,'a'")
dataset: org.apache.spark.sql.DataFrame = [1: int, a: string]

scala> spark.sql("select * from test").show
2019-09-18 16:20:59 WARN  ObjectStore:568 - Failed to get database test, returning NoSuchObjectException
2019-09-18 16:20:59 WARN  ObjectStore:568 - Failed to get database test, returning NoSuchObjectException
2019-09-18 16:20:59 WARN  KVErrorHandler:222 - Stale Epoch encountered for region [{Region[2] ConfVer[1] Version[21] Store[1] KeyRange[t200\000\000\000\000\000\000+]:[]}]
2019-09-18 16:20:59 WARN  KVErrorHandler:120 - Failed to send notification back to driver since CacheInvalidateCallBack is null in executor node.
2019-09-18 16:20:59 WARN  RegionStoreClient:464 - Re-splitting region task due to region error:EpochNotMatch current epoch of region 2 is conf_ver: 1 version: 22, but you sent conf_ver: 1 version: 21
+---+---+
|  1|  a|
+---+---+
|  1|  a|
+---+---+


scala> 

scala> 

scala> 

scala> var dataset = spark.sql("select 2,'b'")
dataset: org.apache.spark.sql.DataFrame = [2: int, b: string]

scala> dataset.write.mode(saveMode = "overwrite").format("jdbc").option("driver", "com.mysql.jdbc.Driver").option("url", "jdbc:mysql://192.168.10.111:4000/test?&autoReconnect=true&failOverReadOnly=false").option("useSSL", "false").option("dbtable", "test").option("isolationLevel", "NONE").option("user", "root").option("password", "hydee@123").save()

scala> spark.sql("select * from test").show
2019-09-18 16:21:31 WARN  ObjectStore:568 - Failed to get database test, returning NoSuchObjectException
2019-09-18 16:21:31 WARN  ObjectStore:568 - Failed to get database test, returning NoSuchObjectException
2019-09-18 16:21:32 WARN  KVErrorHandler:222 - Stale Epoch encountered for region [{Region[2] ConfVer[1] Version[22] Store[1] KeyRange[t200\000\000\000\000\000\000.]:[]}]
2019-09-18 16:21:32 WARN  KVErrorHandler:120 - Failed to send notification back to driver since CacheInvalidateCallBack is null in executor node.
2019-09-18 16:21:32 WARN  RegionStoreClient:464 - Re-splitting region task due to region error:EpochNotMatch current epoch of region 2 is conf_ver: 1 version: 23, but you sent conf_ver: 1 version: 22
+---+---+
|  2|  b|
+---+---+
|  2|  b|
+---+---+

将jar包切换到tispark-core-2.1.2-spark_2.3-jar-with-dependencies.jar，重启spark-shell：

scala> spark.sql("select * from test").show
2019-09-18 16:23:22 WARN  ObjectStore:568 - Failed to get database test, returning NoSuchObjectException
2019-09-18 16:23:22 WARN  ObjectStore:568 - Failed to get database test, returning NoSuchObjectException
+---+---+
|  2|  b|
+---+---+
|  2|  b|
+---+---+


scala> var dataset = spark.sql("select 1,'a'")
dataset: org.apache.spark.sql.DataFrame = [1: int, a: string]

scala> dataset.write.mode(saveMode = "overwrite").format("jdbc").option("driver", "com.mysql.jdbc.Driver").option("url", "jdbc:mysql://192.168.10.111:4000/test?&autoReconnect=true&failOverReadOnly=false").option("useSSL", "false").option("dbtable", "test").option("isolationLevel", "NONE").option("user", "root").option("password", "hydee@123").save()
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.

scala> spark.sql("select * from test").show
2019-09-18 16:23:47 WARN  ObjectStore:568 - Failed to get database test, returning NoSuchObjectException
2019-09-18 16:23:47 WARN  ObjectStore:568 - Failed to get database test, returning NoSuchObjectException
2019-09-18 16:23:47 WARN  KVErrorHandler:193 - Stale Epoch encountered for region [{Region[2] ConfVer[1] Version[23] Store[1] KeyRange[t200\000\000\000\000\000\0001]:[]}]
2019-09-18 16:23:47 WARN  KVErrorHandler:104 - Failed to send notification back to driver since CacheInvalidateCallBack is null in executor node.
2019-09-18 16:23:47 WARN  RegionStoreClient:564 - Re-splitting region task due to region error:EpochNotMatch current epoch of region 2 is conf_ver: 1 version: 24, but you sent conf_ver: 1 version: 23
+---+---+
|  2|  b|
+---+---+
|  2|  b|
+---+---+


scala> spark.sql("select * from test").show
2019-09-18 16:25:48 WARN  ObjectStore:568 - Failed to get database test, returning NoSuchObjectException
2019-09-18 16:25:48 WARN  ObjectStore:568 - Failed to get database test, returning NoSuchObjectException
2019-09-18 16:25:49 WARN  KVErrorHandler:193 - Stale Epoch encountered for region [{Region[2] ConfVer[1] Version[23] Store[1] KeyRange[t200\000\000\000\000\000\0001]:[]}]
2019-09-18 16:25:49 WARN  KVErrorHandler:104 - Failed to send notification back to driver since CacheInvalidateCallBack is null in executor node.
2019-09-18 16:25:49 WARN  RegionStoreClient:564 - Re-splitting region task due to region error:EpochNotMatch current epoch of region 2 is conf_ver: 1 version: 24, but you sent conf_ver: 1 version: 23
+---+---+
|  1|  a|
+---+---+
|  1|  a|
+---+---+

zhengyunpeng · 2019 年9 月 18 日 08:37

目前这两个tispark-assembly-2.3.0-SNAPSHOT的jar只支持读取tidb版本3.0及以上的数据吗？我们环境tidb 2.1.15的，使用这两个jar都报错，把spark升级了也不行。我都是单机环境测试，TiDB-v3.0.0-rc.1

marsishandsome · 2019 年9 月 18 日 09:42

这个jar包应该可以读取tidb-2.1.15，请问具体报什么错？建议可以用local模式运行一下

zhengyunpeng · 2019 年9 月 18 日 09:48

好的，我试一下。
tidb-2.1.15 spark 2.3.3 报错信息:

16-09-2019 18:19:50 CST StructuredStream INFO - 2019-09-16 18:19:50 WARN  TaskSetManager:66 - Lost task 0.0 in stage 0.0 (TID 0, 192.168.10.204, executor 2): java.io.InvalidClassException: org.apache.spark.sql.tispark.TiRDD; local class incompatible: stream classdesc serialVersionUID = -6650751149050193034, local class serialVersionUID = 1833683067262546845
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.lang.reflect.Method.invoke(Method.java:498)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.lang.reflect.Method.invoke(Method.java:498)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.lang.reflect.Method.invoke(Method.java:498)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.lang.reflect.Method.invoke(Method.java:498)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.lang.reflect.Method.invoke(Method.java:498)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:80)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at org.apache.spark.scheduler.Task.run(Task.scala:109)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
16-09-2019 18:19:50 CST StructuredStream INFO - 	at java.lang.Thread.run(Thread.java:745)
16-09-2019 18:19:50 CST StructuredStream INFO -

marsishandsome · 2019 年9 月 18 日 09:49

请问是local模式运行的吗？

zhengyunpeng · 2019 年9 月 18 日 10:31

您是说spark运行模式吗？不是local，是standalone模式

marsishandsome · 2019 年9 月 18 日 10:42

建议先用local运行试试 (standalone模式的话需要先升级集群上的tispark，并重启)

zhengyunpeng · 2019 年9 月 19 日 01:38

使用local模式可以的，谢谢

marsishandsome · 2019 年9 月 23 日 02:08

非常感谢，等我们发布下个fix版本，可以考虑升级一下集群

zhengyunpeng · 2019 年9 月 23 日 02:18

好的，谢谢

system · 2022 年10 月 31 日 19:10

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。