spark一个版本问题引发的血案(java.lang.NoSuchMethodError: org.apache.spark.ui.SparkUI.addStaticHandler)
来源:互联网 发布:开淘宝店怎么代理货源 编辑:程序博客网 时间:2024/05/29 10:40
先看异常:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.propertiesSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/home/jerry/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/local/data0/program/spark-1.6.1-bin-hadoop2.6.4/lib/spark-assembly-1.6.1-hadoop2.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]16/07/09 19:17:42 INFO SparkContext: Running Spark version 1.3.016/07/09 19:17:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable16/07/09 19:17:43 INFO SecurityManager: Changing view acls to: jerry16/07/09 19:17:43 INFO SecurityManager: Changing modify acls to: jerry16/07/09 19:17:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jerry); users with modify permissions: Set(jerry)16/07/09 19:17:44 INFO Slf4jLogger: Slf4jLogger started16/07/09 19:17:44 INFO Remoting: Starting remoting16/07/09 19:17:44 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@localhost:4090]16/07/09 19:17:44 INFO Utils: Successfully started service 'sparkDriver' on port 4090.16/07/09 19:17:44 INFO SparkEnv: Registering MapOutputTracker16/07/09 19:17:44 INFO SparkEnv: Registering BlockManagerMaster16/07/09 19:17:44 INFO DiskBlockManager: Created local directory at /tmp/spark-a250cf38-ad2c-4f20-8cb2-22f33bd7c391/blockmgr-6a6dbba7-a1de-41fc-99d0-bf0856c81fcf16/07/09 19:17:44 INFO MemoryStore: MemoryStore started with capacity 945.5 MB16/07/09 19:17:44 INFO HttpFileServer: HTTP File server directory is /tmp/spark-ce6f474b-70a6-4edf-be1f-ab6e904f29c8/httpd-1ef8f1f4-4a16-4f59-b99a-6eeceb9ece3116/07/09 19:17:44 INFO HttpServer: Starting HTTP Server16/07/09 19:17:44 INFO Server: jetty-8.y.z-SNAPSHOT16/07/09 19:17:44 INFO AbstractConnector: Started SocketConnector@0.0.0.0:5326916/07/09 19:17:44 INFO Utils: Successfully started service 'HTTP file server' on port 53269.16/07/09 19:17:44 INFO SparkEnv: Registering OutputCommitCoordinator16/07/09 19:17:44 INFO Server: jetty-8.y.z-SNAPSHOT16/07/09 19:17:44 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:404016/07/09 19:17:44 INFO Utils: Successfully started service 'SparkUI' on port 4040.16/07/09 19:17:44 INFO SparkUI: Started SparkUI at http://debian0:404016/07/09 19:17:44 INFO Executor: Starting executor ID <driver> on host localhost16/07/09 19:17:44 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@localhost:4090/user/HeartbeatReceiver16/07/09 19:17:45 INFO NettyBlockTransferService: Server created on 4681916/07/09 19:17:45 INFO BlockManagerMaster: Trying to register BlockManager16/07/09 19:17:45 INFO BlockManagerMasterActor: Registering block manager localhost:46819 with 945.5 MB RAM, BlockManagerId(<driver>, localhost, 46819)16/07/09 19:17:45 INFO BlockManagerMaster: Registered BlockManagerException in thread "main" java.lang.NoSuchMethodError: org.apache.spark.ui.SparkUI.addStaticHandler(Ljava/lang/String;Ljava/lang/String;)Vat org.apache.spark.sql.execution.ui.SQLTab.<init>(SQLTab.scala:34)at org.apache.spark.sql.SQLContext$$anonfun$createListenerAndUI$1.apply(SQLContext.scala:1369)at org.apache.spark.sql.SQLContext$$anonfun$createListenerAndUI$1.apply(SQLContext.scala:1369)at scala.Option.foreach(Option.scala:236)at org.apache.spark.sql.SQLContext$.createListenerAndUI(SQLContext.scala:1369)at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:77)at com.spark.test.sql.SparkSqlJDBC2Mysql$.main(SparkSqlJDBC2Mysql.scala:21)at com.spark.test.sql.SparkSqlJDBC2Mysql.main(SparkSqlJDBC2Mysql.scala)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
一开始想到的是scala版本的问题,所以就将scala版本由 2.10.6 修改成了 2.10.4,修改完以后,问题依旧。
然后就开始想,这是 spark 方法的问题,所以是spark版本的问题,由于自己导入了spark的lib库是1.6.1的,但是项目的pom文件中还是1.3.0版本的,所以修改成1.6.1以后,问题迎刃而解。
总结:注意版本版本版本的统一,重要的事情说三遍,一般scala遇到
java.lang.NoSuchMethodError
异常都是版本的问题,切记,切记,切记,重要的事情再说三遍。
0 0
- spark一个版本问题引发的血案(java.lang.NoSuchMethodError: org.apache.spark.ui.SparkUI.addStaticHandler)
- Spark 与 Kafka 集成出错: Apache Spark: java.lang.NoSuchMethodError
- java.lang.NoClassDefFoundError: org/apache/commons/lang3/StringUtils 一个错误引发的血案 struts2-blank的所有 包
- java.lang.NoClassDefFoundError: org/apache/commons/lang3/StringUtils 一个错误引发的血案 struts2-blank的所有 包
- Spark问题[一]:java.lang.ClassNotFoundException: org.apache.spark.launcher.Main
- java.lang.NoSuchMethodError和kafka.common.OffsetOutOfRangeException(Spark)
- Spark java.lang.NoSuchMethodError: scala.Predef$.refArrayOps
- 一个“java.lang.NoSuchMethodError”异常引发的思考
- 关于java.lang.NoSuchMethodError: org.apache.poi.POIDocument问题的详细说明!
- SSH框架搭建的问题---- java.lang.NoSuchMethodError: org.apache.log4j.Logger.isTraceEnabled()Z
- Spark java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32
- idea sbt java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
- java.lang.ClassCastException:org.apache.spark.rdd.MapPartitionsRDD
- java.lang.NoSuchMethodError: org.apache.poi.POIDocument
- NoSuchMethodError:org.apache.spark.rdd.RDD.mapPartitionsInternal$default$2()Z
- spark sql 中 java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.Gener
- zk的jar包冲突:java.lang.NoSuchMethodError: org.apache.zookeeper.ZooKeeper.getChildren(Ljava/lang/String;
- spark连接es的问题java.lang.ClassNotFoundException:org.elasticsearch.client.transport.NoNodeAvailableExcep
- eclipse一些实用小技巧
- android debug sp方法
- 复杂链表的复制
- Android中Application类用法
- C++,C,java面试例题
- spark一个版本问题引发的血案(java.lang.NoSuchMethodError: org.apache.spark.ui.SparkUI.addStaticHandler)
- 新生练习3
- Codeforces Round #325 (Div. 2) D bfs
- 125. Valid Palindrome
- PE文件详解之区块表
- 华为机试题【9】-整数分割为2的幂次
- 我对CONTAINING_RECORD宏的详细解释
- hdu 5178(尺取法)
- <React Native> 学习日记(1) 成功运行、编译RN-android的爬坑记录