kylo问题

原文链接

2018-08-24 15:42:39 ERROR spark-shell-pool-0:SparkShellProcessBuilder:549 - Unable to determine Spark version
java.lang.IllegalStateException: Unable to determine Spark home
        at com.thinkbiganalytics.spark.shell.SparkClientUtil.lambda$getSparkHome$2(SparkClientUtil.java:84)
        at java.util.Optional.orElseThrow(Optional.java:290)
        at com.thinkbiganalytics.spark.shell.SparkClientUtil.getSparkHome(SparkClientUtil.java:84)
        at com.thinkbiganalytics.spark.shell.SparkClientUtil.getVersion(SparkClientUtil.java:109)
        at com.thinkbiganalytics.spark.shell.SparkClientUtil.getMajorVersion(SparkClientUtil.java:65)
        at com.thinkbiganalytics.spark.shell.SparkShellProcessBuilder.getSparkVersion(SparkShellProcessBuilder.java:547)
        at com.thinkbiganalytics.spark.shell.SparkShellProcessBuilder.getAppResource(SparkShellProcessBuilder.java:493)
        at com.thinkbiganalytics.spark.shell.SparkShellProcessBuilder.build(SparkShellProcessBuilder.java:465)
        at com.thinkbiganalytics.spark.shell.AbstractProcessManager.getSystemProcess(AbstractProcessManager.java:155)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-08-24 15:44:23 WARN  http-nio-8420-exec-1:Errors:191 - The following warnings have been detected: WARNING: The (sub)resource method findJobs in com.thinkbiganalytics.jobrepo.rest.controller.JobsRestController contains empty path annotation.
WARNING: The (sub)resource method getAssessments in com.thinkbiganalytics.jobrepo.rest.controller.ServiceLevelAssessmentsController contains empty path annotation.
WARNING: The (sub)resource method getProperties in com.thinkbiganalytics.feedmgr.rest.controller.FeedMetadataPropertiesRestController contains empty path annotation.

编辑application.properties文件：

$ vim /opt/kylo/kylo-services/conf/application.properties
添加或确保以下属性：

spring.profiles.include = auth-spark, ...

hive.userImpersonation.enabled=true
hive.userImpersonation.cache.expiry.duration=4
hive.userImpersonation.cache.expiry.time-unit=HOURS

编辑spark.properties文件：

vim /opt/kylo/kylo-services/conf/spark.properties

加入sparkhome就可以了

# Ensure these two properties are commented out
#spark.shell.server.host
#spark.shell.server.port

# Executes both driver and executors as the user
spark.shell.deployMode = cluster
spark.shell.master = yarn
# Enables user impersonation
spark.shell.proxyUser = true
# Reduces memory requirements and allows Kerberos user impersonation
spark.shell.sparkArgs = --driver-memory 512m --executor-memory 512m --driver-java-options -Djavax.security.auth.useSubjectCredsOnly=false

kerberos.spark.kerberosEnabled = true
kerberos.spark.kerberosPrincipal = kylo
kerberos.spark.keytabLocation = /etc/security/keytabs/kylo.headless.keytab

image

这个错误是由于spark-shell启动不起来
查看配置，nifi服务的principal认证失败造成

---

image

[root@kylo2 soft]# beeline 
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Beeline version 1.1.0-cdh5.15.0 by Apache Hive
beeline>
beeline> !connect jdbc:hive2://10.88.88.120:10000/default;principal=hive/kylo1.hypers.cc@KYLO.CC
scan complete in 2ms
Connecting to jdbc:hive2://10.88.88.120:10000/default;principal=hive/kylo1.hypers.cc@KYLO.CC
18/09/30 17:10:54 [main]: ERROR transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
      
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
              ... 35 more
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://10.88.88.120:10000/default;principal=hive/kylo1.hypers.cc@KYLO.CC: GSS initiate failed (state=08S01,code=0)