安装spark-1.5.0-cdh5.5.2所踩过的坑
来源:互联网 发布:中国四大财阀 知乎 编辑:程序博客网 时间:2024/05/29 18:38
我一开始想安装spark-1.5.0-cdh5.5.2的standalone模式,于是乎(已安装有hadoop集群):
[hadoop@h71 ~]$ tar -zxvf spark-1.5.0-cdh5.5.2.tar.gz
[hadoop@h71 ~]$ vi spark-1.5.0-cdh5.5.2/conf/spark-env.sh
export JAVA_HOME=/usr/jdk1.7.0_25
export SPARK_MASTER_IP=h71
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_CORES=1
export SPARK_EXECUTOR_INSTANCES=1
export SPARK_WORKER_MEMORY=1g
[hadoop@h71 ~]$ vi spark-1.5.0-cdh5.5.2/conf/slaves
h72
h73
[hadoop@h71 ~]$ scp -r spark-1.5.0-cdh5.5.2/ h72:/home/hadoop/
[hadoop@h71 ~]$ scp -r spark-1.5.0-cdh5.5.2/ h73:/home/hadoop/
然后启动spark集群却报错:
[hadoop@h71 spark-1.5.0-cdh5.5.2]$ sbin/start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /home/hadoop/spark-1.5.0-cdh5.5.2/sbin/../logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-h71.out
failed to launch org.apache.spark.deploy.master.Master:
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 6 more
full log in /home/hadoop/spark-1.5.0-cdh5.5.2/sbin/../logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-h71.out
h72: starting org.apache.spark.deploy.worker.Worker, logging to /home/hadoop/spark-1.5.0-cdh5.5.2/sbin/../logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-h72.out
h73: starting org.apache.spark.deploy.worker.Worker, logging to /home/hadoop/spark-1.5.0-cdh5.5.2/sbin/../logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-h73.out
h72: failed to launch org.apache.spark.deploy.worker.Worker:
h72: at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
h72: ... 6 more
h72: full log in /home/hadoop/spark-1.5.0-cdh5.5.2/sbin/../logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-h72.out
h73: failed to launch org.apache.spark.deploy.worker.Worker:
h73: at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
h73: ... 6 more
h73: full log in /home/hadoop/spark-1.5.0-cdh5.5.2/sbin/../logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-h73.out
查看日志:
[hadoop@h71 spark-1.5.0-cdh5.5.2]$ cat logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-h71.out
Spark Command: /usr/jdk1.7.0_25/bin/java -cp /home/hadoop/spark-1.5.0-cdh5.5.2/sbin/../conf/:/home/hadoop/spark-1.5.0-cdh5.5.2/lib/spark-assembly-1.5.0-cdh5.5.2-hadoop2.6.0-cdh5.5.2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/etc/hadoop/ -Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.master.Master --ip h71 --port 7077 --webui-port 8080
========================================
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)
at java.lang.Class.getMethod0(Class.java:2764)
at java.lang.Class.getMethod(Class.java:1653)
at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 6 more
原因:无法加载slf4j相关jar包
解决:在spark-env.sh中添加
export HADOOP_HOME=/home/hadoop/hadoop-2.6.0-cdh5.5.2for f in $HADOOP_HOME/share/hadoop/common/lib/*.jar; do if [ "$SPARK_CLASSPATH" ]; then export SPARK_CLASSPATH=$SPARK_CLASSPATH:$f else export SPARK_CLASSPATH=$f fidone for f in $HADOOP_HOME/share/hadoop/common/*.jar; do if [ "$SPARK_CLASSPATH" ]; then export SPARK_CLASSPATH=$SPARK_CLASSPATH:$f else export SPARK_CLASSPATH=$f fidone并复制到其他两个从节点:
[hadoop@h71 spark-1.5.0-cdh5.5.2]$ scp conf/spark-env.sh h72:/home/hadoop/spark-1.5.0-cdh5.5.2/conf/
[hadoop@h71 spark-1.5.0-cdh5.5.2]$ scp conf/spark-env.sh h73:/home/hadoop/spark-1.5.0-cdh5.5.2/conf/
再启动spark集群:
[hadoop@h71 spark-1.5.0-cdh5.5.2]$ sbin/start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /home/hadoop/spark-1.5.0-cdh5.5.2/sbin/../logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-h71.out
h73: starting org.apache.spark.deploy.worker.Worker, logging to /home/hadoop/spark-1.5.0-cdh5.5.2/sbin/../logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-h73.out
h72: starting org.apache.spark.deploy.worker.Worker, logging to /home/hadoop/spark-1.5.0-cdh5.5.2/sbin/../logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-h72.out
用jps命令查看你会在主节点上看到Master进程在从节点上看到Worker进程(可能你都看不到,因为它很快就又shut down了),查看日志:
[hadoop@h71 spark-1.5.0-cdh5.5.2]$ cat logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-h71.outSpark Command: /usr/jdk1.7.0_25/bin/java -cp /home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/avro-1.7.6-cdh5.5.2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-el-1.0.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/hadoop-annotations-2.6.0-cdh5.5.2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/hadoop-auth-2.6.0-cdh5.5.2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/htrace-core4-4.0.1-incubating.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jetty-6.1.26.cloudera.4.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/junit-4.11.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/logredactor-1.0.3.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/xz-1.0.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/lib/zookeeper-3.4.5-cdh5.5.2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/hadoop-common-2.6.0-cdh5.5.2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/hadoop-common-2.6.0-cdh5.5.2-tests.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/share/hadoop/common/hadoop-nfs-2.6.0-cdh5.5.2.jar:/home/hadoop/spark-1.5.0-cdh5.5.2/sbin/../conf/:/home/hadoop/spark-1.5.0-cdh5.5.2/lib/spark-assembly-1.5.0-cdh5.5.2-hadoop2.6.0-cdh5.5.2.jar:/home/hadoop/hadoop-2.6.0-cdh5.5.2/etc/hadoop/ -Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.master.Master --ip h71 --port 7077 --webui-port 8080========================================17/03/17 17:26:27 INFO master.Master: Registered signal handlers for [TERM, HUP, INT]17/03/17 17:26:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable17/03/17 17:26:28 INFO spark.SecurityManager: Changing view acls to: hadoop17/03/17 17:26:28 INFO spark.SecurityManager: Changing modify acls to: hadoop17/03/17 17:26:28 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)17/03/17 17:26:29 INFO slf4j.Slf4jLogger: Slf4jLogger started17/03/17 17:26:29 INFO Remoting: Starting remoting17/03/17 17:26:31 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@h71:7077]17/03/17 17:26:31 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkMaster@h71:7077]17/03/17 17:26:31 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.17/03/17 17:26:32 INFO master.Master: Starting Spark master at spark://h71:707717/03/17 17:26:32 INFO master.Master: Running Spark version 1.5.0-cdh5.5.217/03/17 17:26:34 INFO server.Server: jetty-8.y.z-SNAPSHOT17/03/17 17:26:39 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:808017/03/17 17:26:39 INFO util.Utils: Successfully started service 'MasterUI' on port 8080.17/03/17 17:26:39 INFO ui.MasterWebUI: Started MasterWebUI at http://192.168.8.71:808017/03/17 17:26:39 INFO server.Server: jetty-8.y.z-SNAPSHOT17/03/17 17:26:40 INFO server.AbstractConnector: Started SelectChannelConnector@h71:606617/03/17 17:26:40 INFO util.Utils: Successfully started service on port 6066.17/03/17 17:26:40 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 606617/03/17 17:26:40 ERROR actor.ActorSystemImpl: Uncaught fatal error from thread [sparkMaster-akka.actor.default-dispatcher-15] shutting down ActorSystem [sparkMaster]java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/Module at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at org.apache.spark.util.Utils$.classForName(Utils.scala:172) at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:190) at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:186) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) at org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:186) at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:100) at org.apache.spark.deploy.master.Master.onStart(Master.scala:152) at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$preStart$1.apply$mcV$sp(AkkaRpcEnv.scala:100) at org.apache.spark.rpc.akka.AkkaRpcEnv.org$apache$spark$rpc$akka$AkkaRpcEnv$$safelyCall(AkkaRpcEnv.scala:197) at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.preStart(AkkaRpcEnv.scala:99) at akka.actor.ActorCell.create(ActorCell.scala:562) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:425) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262) at akka.dispatch.Mailbox.run(Mailbox.scala:218) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.Module at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 26 more17/03/17 17:26:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/driver/kill,null}17/03/17 17:26:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/app/kill,null}17/03/17 17:26:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}17/03/17 17:26:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}17/03/17 17:26:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/json,null}17/03/17 17:26:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}17/03/17 17:26:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/history/not-found/json,null}17/03/17 17:26:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/history/not-found,null}17/03/17 17:26:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/app/json,null}17/03/17 17:26:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/app,null}17/03/17 17:26:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}17/03/17 17:26:40 WARN metrics.MetricsSystem: Stopping a MetricsSystem that is not running17/03/17 17:26:40 ERROR akka.AkkaRpcEnv: Ignore error: nulljava.lang.NullPointerException at org.apache.spark.deploy.master.Master.onStop(Master.scala:198) at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$postStop$1.apply$mcV$sp(AkkaRpcEnv.scala:143) at org.apache.spark.rpc.akka.AkkaRpcEnv.org$apache$spark$rpc$akka$AkkaRpcEnv$$safelyCall(AkkaRpcEnv.scala:197) at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.postStop(AkkaRpcEnv.scala:142) at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:201) at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:163) at akka.actor.ActorCell.terminate(ActorCell.scala:338) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:431) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262) at akka.dispatch.Mailbox.run(Mailbox.scala:218) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)17/03/17 17:26:40 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.17/03/17 17:26:40 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.17/03/17 17:26:41 INFO Remoting: Remoting shut down17/03/17 17:26:41 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.原因:CLASSPATH中缺少jackson包
解决:在spark-env.sh中再添加
for f in $HADOOP_HOME/share/hadoop/mapreduce*/lib/*.jar; do if [ "$SPARK_CLASSPATH" ]; then export SPARK_CLASSPATH=$SPARK_CLASSPATH:$f else export SPARK_CLASSPATH=$f fidone并复制到其他两个从节点后再启动spark集群则一切正常。
测试(运行这个spark自带的程序可成功执行):
[hadoop@h71 spark-1.5.0-cdh5.5.2]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://h71:7077 --executor-memory 1G --total-executor-cores 1 ./lib/spark-examples-1.5.0-cdh5.5.2-hadoop2.6.0-cdh5.5.2.jar 100
注意:
在网上https://zhidao.baidu.com/question/2269421342018900708.html有看到解决报错一的另一种方法,即在spark-env.sh中添加如下内容
export HADOOP_HOME=/home/hadoop/hadoop-2.6.0-cdh5.5.2
export SPARK_DIST_CLASSPATH=$(/home/hadoop/hadoop-2.6.0-cdh5.5.2/bin/hadoop classpath)
按这种方法解决报错一后再添加相应内容解决报错二后也是虽然能正常启动但过一段时间相应的进程还是会挂掉,查看日志有:
17/03/17 18:43:02 ERROR actor.ActorSystemImpl: Uncaught fatal error from thread [sparkMaster-akka.actor.default-dispatcher-7] shutting down ActorSystem [sparkMaster]java.lang.VerifyError: (class: org/jboss/netty/channel/socket/nio/NioWorkerPool, method: createWorker signature: (Ljava/util/concurrent/Executor;)Lorg/jboss/netty/channel/socket/nio/AbstractNioWorker;) Wrong return type in function at akka.remote.transport.netty.NettyTransport.<init>(NettyTransport.scala:282) at akka.remote.transport.netty.NettyTransport.<init>(NettyTransport.scala:239) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78) at scala.util.Try$.apply(Try.scala:161) at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73) at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) at scala.util.Success.flatMap(Try.scala:200) at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84) at akka.remote.EndpointManager$$anonfun$8.apply(Remoting.scala:618) at akka.remote.EndpointManager$$anonfun$8.apply(Remoting.scala:610) at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721) at akka.remote.EndpointManager.akka$remote$EndpointManager$$listens(Remoting.scala:610) at akka.remote.EndpointManager$$anonfun$receive$2.applyOrElse(Remoting.scala:450) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) at akka.actor.ActorCell.invoke(ActorCell.scala:456) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)17/03/17 18:43:02 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.17/03/17 18:43:02 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.17/03/17 18:43:02 INFO Remoting: Remoting shut down17/03/17 18:43:03 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.[ERROR] [03/17/2017 18:43:12.812] [main] [Remoting] Remoting error: [Startup timed out] [akka.remote.RemoteTransportException: Startup timed out at akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:129) at akka.remote.Remoting.start(Remoting.scala:191) at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184) at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579) at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577) at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588) at akka.actor.ActorSystem$.apply(ActorSystem.scala:111) at akka.actor.ActorSystem$.apply(ActorSystem.scala:104) at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121) at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:52) at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1912) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1903) at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:55) at org.apache.spark.rpc.akka.AkkaRpcEnvFactory.create(AkkaRpcEnv.scala:253) at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:53) at org.apache.spark.deploy.master.Master$.startRpcEnvAndEndpoint(Master.scala:1074) at org.apache.spark.deploy.master.Master$.main(Master.scala:1058) at org.apache.spark.deploy.master.Master.main(Master.scala)Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10000 milliseconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at akka.remote.Remoting.start(Remoting.scala:173) ... 18 more]Exception in thread "main" java.util.concurrent.TimeoutException: Futures timed out after [10000 milliseconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at akka.remote.Remoting.start(Remoting.scala:173) at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184) at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579) at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577) at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588) at akka.actor.ActorSystem$.apply(ActorSystem.scala:111) at akka.actor.ActorSystem$.apply(ActorSystem.scala:104) at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121) at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:52) at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1912) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1903) at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:55) at org.apache.spark.rpc.akka.AkkaRpcEnvFactory.create(AkkaRpcEnv.scala:253) at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:53) at org.apache.spark.deploy.master.Master$.startRpcEnvAndEndpoint(Master.scala:1074) at org.apache.spark.deploy.master.Master$.main(Master.scala:1058) at org.apache.spark.deploy.master.Master.main(Master.scala)后来经过自己反复尝试,应该将SPARK_DIST_CLASSPATH改为SPARK_CLASSPATH方可好使,真尼玛坑!
说明:
我安装从spark官网下载的spark-1.6.3-bin-hadoop2.6或者spark-1.3.1-bin-hadoop2.6也没这么多的事啊,按我的这篇文章http://blog.csdn.net/m0_37739193/article/details/74908865中的安装步骤就可以,这安装个cdh版的咋尼玛这么费事啊。。。
补充:
将spark-1.6.3-bin-hadoop2.6.tgz解压到相应的目录后什么都不用做就可以运行local单机模式:
[hadoop@h71 spark-1.6.3-bin-hadoop2.6]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[1] ./lib/spark-examples-1.6.3-hadoop2.6.0.jar 100
Yarn集群模式:
client模式(可在SecureCRT终端中看见结果):
[hadoop@h71 spark-1.6.3-bin-hadoop2.6]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --executor-memory 1G --num-executors 1 ./lib/spark-examples-1.6.3-hadoop2.6.0.jar 100
cluster模式(结果只能在h71:8088里面可见!):
[hadoop@h71 spark-1.6.3-bin-hadoop2.6]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --executor-memory 1G --num-executors 1 ./lib/spark-examples-1.6.3-hadoop2.6.0.jar 100
注意:
我见网上Yarn集群模式需要在spark-env.sh添加如下配置:
export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export SPARK_HOME=/usr/hadoopsoft/spark-1.6.3-bin-hadoop2.6
export SPARK_JAR=/usr/hadoopsoft/spark-1.6.3-bin-hadoop2.6/lib/spark-assembly-1.6.3-hadoop2.6.0.jar
export PATH=$SPARK_HOME/bin:$PATH
可是我在h71主节点中将spark-1.6.3-bin-hadoop2.6.tgz解压后什么都没有做,Yarn集群模式却可以正常使用。。。
standalone集群模式:
client模式(可在SecureCRT终端中看见结果):
[hadoop@h71 spark-1.6.3-bin-hadoop2.6]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://h71:7077 --executor-memory 1G --total-executor-cores 1 ./lib/spark-examples-1.6.3-hadoop2.6.0.jar 100
cluster模式(结果只能在h71:8080里面可见!):
[hadoop@h71 spark-1.6.3-bin-hadoop2.6]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://h71:7077 --deploy-mode cluster --supervise --executor-memory 1G --total-executor-cores 1 ./lib/spark-examples-1.6.3-hadoop2.6.0.jar 100
- 安装spark-1.5.0-cdh5.5.2所踩过的坑
- cdh4.0升级cdh5.6踩过的坑
- 搭建Spark所遇过的坑
- 搭建Spark所遇过的坑
- 【Spark学习】Spark 1.1.0 with CDH5.2 安装部署
- Spark 1.1.0 with CDH5.2 安装部署
- spark程序踩过的坑
- spark SQL上踩过的坑
- Oozie-4.1.0-cdh5.5.2 安装部署
- CDH5.0安装【1】
- 趟过spark的坑
- 那些年学习注解所踩过的坑
- #使用bintray-release 上传Jcenter 所踩过的坑#
- 最靠谱的CDH5安装指南
- 最靠谱的CDH5安装指南
- 最靠谱的CDH5安装指南
- 最靠谱的CDH5安装指南
- spark 2.0 踩过的SparkSession的坑
- 沟通能力之PPT综合能力方法
- C++ std::string 不可初始化为NULL及基本用法
- [在线Demo]使用Hibernate多租户实现SaaS服务
- 举例说明JMETER的用法-某网站压力测试
- 移动端适配方案
- 安装spark-1.5.0-cdh5.5.2所踩过的坑
- ftp自动上传
- RCNN学习路线推荐(4)---深度学习
- 严重: Failed to initialize end point associated with ProtocolHandler ["http-bio-8080"]
- Maven Dependencies 的.jar路径错误
- 【LeetCode】234. Palindrome Linked List
- 使用AndroidStudio直接运行无法启动,但是 Gradle Build 出来的安装包可以使用
- Android ApiClound 自定义模块未绑定解决方案
- Java中的static关键字解析