Spark On Yarn之指定hive-site.xml找不到metastore
来源:互联网 发布:卓睿安位软件 编辑:程序博客网 时间:2024/06/05 03:35
先上代码:
def main(args: Array[String]) { val appName = "SparkApp" val conf = new SparkConf().setAppName(appName) val sc = new SparkContext(conf) val hiveContext = new HiveContext(sc) val df = hiveContext.sql("show databases") LOG.info("count is " + df.count()) df.show() sc.stop }场景一、
spark-submit \--master yarn \--deploy-mode cluster \--class Spark spark.jar结果如下图:
场景二、
spark-submit \--master yarn \--deploy-mode client \--class Spark spark.jar结果如下图:
场景三、
spark-submit \--master yarn \--deploy-mode cluster \--files $HIVE_HOME/hive-site.xml \--class Spark spark.jar结果报错:
17/03/23 22:47:33 ERROR ApplicationMaster: User class threw exception: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/hadoopdata/hadoop/yarn/local/usercache/root/appcache/application_1489988829960_0121/container_e72_1489988829960_0121_02_000001/__app__.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/hadoopdata/hadoop/yarn/local/filecache/160/spark-hdp-assembly.jar!/javax/ws/rs/ext/RuntimeDelegate.classjava.lang.LinkageError: ClassCastException: attempting to castjar:file:/hadoopdata/hadoop/yarn/local/usercache/root/appcache/application_1489988829960_0121/container_e72_1489988829960_0121_02_000001/__app__.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/hadoopdata/hadoop/yarn/local/filecache/160/spark-hdp-assembly.jar!/javax/ws/rs/ext/RuntimeDelegate.classat javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:116)at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91)at javax.ws.rs.core.MediaType.<clinit>(MediaType.java:44)at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:64)at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175)at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)at com.sun.jersey.api.client.Client.init(Client.java:342)at com.sun.jersey.api.client.Client.access$000(Client.java:118)at com.sun.jersey.api.client.Client$1.f(Client.java:191)at com.sun.jersey.api.client.Client$1.f(Client.java:187)at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)at com.sun.jersey.api.client.Client.<init>(Client.java:187)at com.sun.jersey.api.client.Client.<init>(Client.java:170)at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceInit(TimelineClientImpl.java:335)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)at org.apache.hadoop.hive.ql.hooks.ATSHook.<init>(ATSHook.java:67)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:526)at java.lang.Class.newInstance(Class.java:383)at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:60)at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1309)at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1293)at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1347)at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:484)at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:473)at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$withHiveState$1.apply(ClientWrapper.scala:279)at org.apache.spark.sql.hive.client.ClientWrapper.liftedTree1$1(ClientWrapper.scala:226)at org.apache.spark.sql.hive.client.ClientWrapper.retryLocked(ClientWrapper.scala:225)at org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:268)at org.apache.spark.sql.hive.client.ClientWrapper.runHive(ClientWrapper.scala:473)at org.apache.spark.sql.hive.client.ClientWrapper.runSqlHive(ClientWrapper.scala:463)at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:605)at org.apache.spark.sql.hive.execution.HiveNativeCommand.run(HiveNativeCommand.scala:33)at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:145)at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130)at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52)at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817)at com.spark.jdbc.SparkSQLHiveOnYarn$.main(SparkSQLHiveOnYarn.scala:23)at com.spark.jdbc.SparkSQLHiveOnYarn.main(SparkSQLHiveOnYarn.scala)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:546)17/03/23 22:47:33 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/hadoopdata/hadoop/yarn/local/usercache/root/appcache/application_1489988829960_0121/container_e72_1489988829960_0121_02_000001/__app__.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/hadoopdata/hadoop/yarn/local/filecache/160/spark-hdp-assembly.jar!/javax/ws/rs/ext/RuntimeDelegate.class)
解决问题:
查看 Driver的Classpath:
抱着试试的态度,在目录/usr/hdp/current/hadoop-client/conf/新建文件hive-ste.xml,并修改内容为:
<configuration> <property> <name>hive.metastore.uris</name> <value>thrift://ip:9083</value> </property> </configuration>再次执行场景一,结果正常,哈哈!!!!
0 0
- Spark On Yarn之指定hive-site.xml找不到metastore
- hive on spark on yarn
- spark 使用hive metastore
- spark2.0.1 on yarn with hue 集群搭建部署(四)hive、hive metastore安装
- Spark学习之14:Spark on Yarn
- yarn-site.xml
- yarn-site.xml
- yarn-site.xml
- [spark]spark/hive配置metastore为postgresql
- hive on spark通过YARN-client提交任务不成功
- yarn-site.xml配置参数
- hadoop yarn-site.xml配置
- yarn-site.xml的配置
- hive-site.xml
- spark sql 无法访问 hive metastore问题解决
- Spark连接Hive的metastore异常
- 搭建Hive on Spark 遇见的问题File /home/daxin/bigdata/hive/scripts/metastore/upgrade/MySQL/upgrade.order.MyS
- Spark on Yarn部署
- 在ubuntu中使用virtualenv创建python2和python3的虚拟环境
- Linux Kernel source code of all versions
- Cadence Allegro学习之Add connnect+slide+copy
- Log.v(“ThreeTips”, ”#13”)
- Oracle PL/SQL开发基础(第四弹:索引)
- Spark On Yarn之指定hive-site.xml找不到metastore
- Centos下yum安装LAMP环境
- Java集合---ConcurrentHashMap原理分析
- NET 项目多人沟通记录
- HDU 1012 u Calculate e
- 特征描述子——SIFT
- 这个人的财富可超比尔盖茨,却为何富有争议
- MyEclipse中右键项目Build Path菜单无子菜单的解决办法
- 利用Maven构建Redis示例