spark一个版本问题引发的血案(java.lang.NoSuchMethodError: org.apache.spark.ui.SparkUI.addStaticHandler)

来源:互联网 发布:开淘宝店怎么代理货源 编辑:程序博客网 时间:2024/05/29 10:40

先看异常:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.propertiesSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/home/jerry/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/local/data0/program/spark-1.6.1-bin-hadoop2.6.4/lib/spark-assembly-1.6.1-hadoop2.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]16/07/09 19:17:42 INFO SparkContext: Running Spark version 1.3.016/07/09 19:17:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable16/07/09 19:17:43 INFO SecurityManager: Changing view acls to: jerry16/07/09 19:17:43 INFO SecurityManager: Changing modify acls to: jerry16/07/09 19:17:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jerry); users with modify permissions: Set(jerry)16/07/09 19:17:44 INFO Slf4jLogger: Slf4jLogger started16/07/09 19:17:44 INFO Remoting: Starting remoting16/07/09 19:17:44 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@localhost:4090]16/07/09 19:17:44 INFO Utils: Successfully started service 'sparkDriver' on port 4090.16/07/09 19:17:44 INFO SparkEnv: Registering MapOutputTracker16/07/09 19:17:44 INFO SparkEnv: Registering BlockManagerMaster16/07/09 19:17:44 INFO DiskBlockManager: Created local directory at /tmp/spark-a250cf38-ad2c-4f20-8cb2-22f33bd7c391/blockmgr-6a6dbba7-a1de-41fc-99d0-bf0856c81fcf16/07/09 19:17:44 INFO MemoryStore: MemoryStore started with capacity 945.5 MB16/07/09 19:17:44 INFO HttpFileServer: HTTP File server directory is /tmp/spark-ce6f474b-70a6-4edf-be1f-ab6e904f29c8/httpd-1ef8f1f4-4a16-4f59-b99a-6eeceb9ece3116/07/09 19:17:44 INFO HttpServer: Starting HTTP Server16/07/09 19:17:44 INFO Server: jetty-8.y.z-SNAPSHOT16/07/09 19:17:44 INFO AbstractConnector: Started SocketConnector@0.0.0.0:5326916/07/09 19:17:44 INFO Utils: Successfully started service 'HTTP file server' on port 53269.16/07/09 19:17:44 INFO SparkEnv: Registering OutputCommitCoordinator16/07/09 19:17:44 INFO Server: jetty-8.y.z-SNAPSHOT16/07/09 19:17:44 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:404016/07/09 19:17:44 INFO Utils: Successfully started service 'SparkUI' on port 4040.16/07/09 19:17:44 INFO SparkUI: Started SparkUI at http://debian0:404016/07/09 19:17:44 INFO Executor: Starting executor ID <driver> on host localhost16/07/09 19:17:44 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@localhost:4090/user/HeartbeatReceiver16/07/09 19:17:45 INFO NettyBlockTransferService: Server created on 4681916/07/09 19:17:45 INFO BlockManagerMaster: Trying to register BlockManager16/07/09 19:17:45 INFO BlockManagerMasterActor: Registering block manager localhost:46819 with 945.5 MB RAM, BlockManagerId(<driver>, localhost, 46819)16/07/09 19:17:45 INFO BlockManagerMaster: Registered BlockManagerException in thread "main" java.lang.NoSuchMethodError: org.apache.spark.ui.SparkUI.addStaticHandler(Ljava/lang/String;Ljava/lang/String;)Vat org.apache.spark.sql.execution.ui.SQLTab.<init>(SQLTab.scala:34)at org.apache.spark.sql.SQLContext$$anonfun$createListenerAndUI$1.apply(SQLContext.scala:1369)at org.apache.spark.sql.SQLContext$$anonfun$createListenerAndUI$1.apply(SQLContext.scala:1369)at scala.Option.foreach(Option.scala:236)at org.apache.spark.sql.SQLContext$.createListenerAndUI(SQLContext.scala:1369)at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:77)at com.spark.test.sql.SparkSqlJDBC2Mysql$.main(SparkSqlJDBC2Mysql.scala:21)at com.spark.test.sql.SparkSqlJDBC2Mysql.main(SparkSqlJDBC2Mysql.scala)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

一开始想到的是scala版本的问题,所以就将scala版本由 2.10.6 修改成了 2.10.4,修改完以后,问题依旧。

然后就开始想,这是 spark 方法的问题,所以是spark版本的问题,由于自己导入了spark的lib库是1.6.1的,但是项目的pom文件中还是1.3.0版本的,所以修改成1.6.1以后,问题迎刃而解。

总结:注意版本版本版本的统一,重要的事情说三遍,一般scala遇到 

java.lang.NoSuchMethodError

异常都是版本的问题,切记,切记,切记,重要的事情再说三遍。

0 0