#org.apache.hadoop.conf.Configuration.addDeprecations

来源:互联网 发布:ios横版动作游戏 知乎 编辑:程序博客网 时间:2024/05/29 04:53

若在spark-dafult.conf里配置spark.driver.extraClassPath /usr/local/spark/lib/*
会报:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDelta;)V    at org.apache.hadoop.yarn.conf.YarnConfiguration.addDeprecatedKeys(YarnConfiguration.java:79)    at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:73)    at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.newConfiguration(YarnSparkHadoopUtil.scala:61)    ......

若spark-dafult.conf配置为spark.driver.extraClassPath [其他路径]就不会报错

若spark-dafult.conf配置为
spark.driver.extraClassPath /usr/local/spark/lib/spark-1.5.1-yarn-shuffle.jar
还是会报classpath冲突的错
所以报org.apache.hadoop.conf.Configuration.addDeprecations有可能是spark-assembly-1.5.1-hadoop2.6.0.jar双重引用

0 0
原创粉丝点击