用cdh和zeppline开启kerberos,启用tispark读数据，通不过认证的问题

[Hacker_5YEF49J7]

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：

【TiDB 版本】
4.0
【问题描述】

CDH：6.3.2
zepplin:0.9.0
tispark 2.3.14
tidb:4.0

spark-submit --master yarn --deploy-mode cluster --principal jzyc/hadoop@JOIN.COM --keytab /hadoop/jzyc.keytab --class App --jars hdfs://bigdser1:8020/sparklib/* JZTanalyse-1.0-SNAPSHOT.jar 127.0.0.1 1 1 key

我这里运行是没有问题的。但是我在zeppline里面添加了spark.jars hdfs://bigdser1:8020/sparklib/*
在zeppline里运行spark 用tispark读取数据的时候会报错
java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: “bigdser4/10.3.87.26”; destination host is: “bigdser1”:8020;

zeppline中已经添加了spark.yarn.principal 和 spark.yarn.keytab这两个参数

我发现如果我直接spark-submit
spark的yarn日志中
YARN executor launch context:
resources:
resources:
tispark-assembly-2.3.14.jar -> resource { scheme: “hdfs” host: “nameservice1” port: -1 file: “/user/jzyc/.sparkStaging/application_1618988436199_0020/tispark-assembly-2.3.14.jar” } size: 24497987 timestamp: 1619593796714 type: FILE visibility: PRIVATE
app.jar -> resource { scheme: “hdfs” host: “nameservice1” port: -1 file: “/user/jzyc/.sparkStaging/application_1618988436199_0020/JZTanalyse-1.0-SNAPSHOT.jar” } size: 44009 timestamp: 1619593796342 type: FILE visibility: PRIVATE
spark_conf -> resource { scheme: “hdfs” host: “nameservice1” port: -1 file: “/user/jzyc/.sparkStaging/application_1618988436199_0020/spark_conf.zip” } size: 171568 timestamp: 1619593796927 type: ARCHIVE visibility: PRIVATE

但用zeppline调用的时候会多出
spark_conf -> resource { scheme: “hdfs” host: “nameservice1” port: -1 file: “/user/jzyc/.sparkStaging/application_1618988436199_0002/spark_conf.zip” } size: 161768 timestamp: 1619149044240 type: ARCHIVE visibility: PRIVATE
log4j_yarn_cluster.properties -> resource { scheme: “hdfs” host: “nameservice1” port: -1 file: “/user/jzyc/.sparkStaging/application_1618988436199_0002/log4j_yarn_cluster.properties” } size: 1018 timestamp: 1619149044023 type: FILE visibility: PRIVATE
tispark-assembly-2.3.14.jar -> resource { scheme: “hdfs” host: “nameservice1” port: -1 file: “/user/jzyc/.sparkStaging/application_1618988436199_0002/tispark-assembly-2.3.14.jar” } size: 24497987 timestamp: 1619149043707 type: FILE visibility: PRIVATE

多出来的这两句不知道是不是造成kerberos认证失败的原因

我现在不清楚是zeppline还是tispark的原因

[Hacker_5YEF49J7]

我还发现。就算我不去读取tidb的数据

val df1 = spark.createDataFrame(Seq((1, “andy”, 20, “USA”), (2, “jeff”, 23, “China”), (3, “james”, 18, “USA”))).toDF(“id”, “name”, “age”, “country”)

val df2 = df1.groupBy(“country”).count()
df2.show

而且是在spark任务启动以后在运行过和中报的错误

在日志中我可看到
INFO [2021-04-26 08:56:36,952] ({FIFOScheduler-interpreter_724716275-Worker-1} Logging.scala[logInfo]:57) - Attempting to login to KDC using principal: jzyc/hadoop@JOIN.COM
INFO [2021-04-26 08:56:37,105] ({FIFOScheduler-interpreter_724716275-Worker-1} Logging.scala[logInfo]:57) - Successfully logged into KDC.
INFO [2021-04-26 08:56:37,668] ({FIFOScheduler-interpreter_724716275-Worker-1} Logging.scala[logInfo]:57) - getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-2141307523_32, ugi=jzyc/hadoop@JOIN.COM (auth:KERBEROS)]] with renewer yarn/bigdser1@JOIN.COM
INFO [2021-04-26 08:56:37,864] ({FIFOScheduler-interpreter_724716275-Worker-1} DFSClient.java[getDelegationToken]:700) - Created token for jzyc: HDFS_DELEGATION_TOKEN owner=jzyc/hadoop@JOIN.COM, renewer=yarn, realUser=, issueDate=1619398597784, maxDate=1620003397784, sequenceNumber=340, masterKeyId=44 on ha-hdfs:nameservice1
INFO [2021-04-26 08:56:37,874] ({FIFOScheduler-interpreter_724716275-Worker-1} Logging.scala[logInfo]:57) - getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-2141307523_32, ugi=jzyc/hadoop@JOIN.COM (auth:KERBEROS)]] with renewer jzyc/hadoop@JOIN.COM
INFO [2021-04-26 08:56:37,879] ({FIFOScheduler-interpreter_724716275-Worker-1} DFSClient.java[getDelegationToken]:700) - Created token for jzyc: HDFS_DELEGATION_TOKEN owner=jzyc/hadoop@JOIN.COM, renewer=jzyc, realUser=, issueDate=1619398597875, maxDate=1620003397875, sequenceNumber=341, masterKeyId=44 on ha-hdfs:nameservice1
INFO [2021-04-26 08:56:37,890] ({dispatcher-event-loop-0} Logging.scala[logInfo]:57) - Got an error when resolving hostNames. Falling back to /default-rack for all
INFO [2021-04-26 08:56:37,929] ({FIFOScheduler-interpreter_724716275-Worker-1} Logging.scala[logInfo]:57) - Renewal interval is 86400050 for token HDFS_DELEGATION_TOKEN
证明认证是对的。但到了任务阶段时

INFO [2021-04-26 08:56:55,057] ({dispatcher-event-loop-0} Logging.scala[logInfo]:57) - Starting task 1.0 in stage 0.0 (TID 0, bigdser2, executor 2, partition 1, RACK_LOCAL, 9987 bytes)
INFO [2021-04-26 08:56:55,061] ({dispatcher-event-loop-0} Logging.scala[logInfo]:57) - Starting task 0.0 in stage 0.0 (TID 1, bigdser2, executor 1, partition 0, RACK_LOCAL, 9987 bytes)
WARN [2021-04-26 08:56:56,020] ({task-result-getter-0} Logging.scala[logWarning]:69) - Lost task 1.0 in stage 0.0 (TID 0, bigdser2, executor 2): java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: “bigdser2/10.3.87.24”; destination host is: “bigdser1”:8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:808)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1503)
at org.apache.hadoop.ipc.Client.call(Client.java:1445)

就这样的操作都会报认证失败

[yilong (yi888long)]

看报错都是HDFS，或者其他组件的报错，感觉如果 tispark 可以正常写入，应该没什么问题了。其他组件目前没有相关测试。可以到相关社区询问下，如果有明确的需要 tispark 查看的麻烦继续反馈，多谢。

[Hacker_5YEF49J7]

spark.jars 只能是具体的jar包。不能用通配符
hdfs://xxxxxx:8020/sparklib/txxxxxx-2.3.14.jar

[yilong (yi888long)]

请问您的意思是 zeppline里面添加了spark.jars hdfs://bigdser1:8020/sparklib/* 这里改为 spark.jars hdfs://xxxxxx:8020/sparklib/txxxxxx-2.3.14.jar 就可以解决吗？

[Hacker_5YEF49J7]

是的！！！