spark on yarn启动异常

一个困扰了我好久的异常：

17/05/31 23:53:23 ERROR spark.SparkContext: Error initializing SparkContext.org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)    at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)    at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)    at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)    at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)    at scala.Option.getOrElse(Option.scala:121)    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)    at org.apache.spark.repl.Main$.createSparkSession(Main.scala:96)    at $line3.$read$$iw$$iw.<init>(<console>:15)    at $line3.$read$$iw.<init>(<console>:42)at $line3.$read.<init>(<console>:44)at $line3.$read$.<init>(<console>:48)at $line3.$read$.<clinit>(<console>)at $line3.$eval$.$print$lzycompute(<console>:7)at $line3.$eval$.$print(<console>:6)at $line3.$eval.$print(<console>)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)    at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)    at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)    at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)    at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)    at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)    at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)    at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)    at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)    at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)    at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38)at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)    at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37)at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:105)at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)    at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)    at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)    at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)    at org.apache.spark.repl.Main$.doMain(Main.scala:69)    at org.apache.spark.repl.Main$.main(Main.scala:52)    at org.apache.spark.repl.Main.main(Main.scala)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke(Method.java:498)    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)17/05/31 23:53:23 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!17/05/31 23:53:24 WARN metrics.MetricsSystem: Stopping a MetricsSystem that is not runningorg.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.  at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)  at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)  at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)  at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)  at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)  at scala.Option.getOrElse(Option.scala:121)  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)  at org.apache.spark.repl.Main$.createSparkSession(Main.scala:96)  ... 47 elided

公司的测试集群不能通过web页面访问，所以一直通过各种手段查找不到原因，这次在家搭了个环境，找到了异常

找不到或无法加载主类 org.apache.spark.deploy.yarn.ExecutorLauncher

网上很多原因是要配置 spark.yarn.jars, 配置了不起作用，后来在一个网页旮旯里找到一个解释，要在文件夹后面加个*,如下：

spark.yarn.jars hdfs://localhost/user/madong/spark/jars/*

后面接着出了另一个异常：

scala> 17/06/01 00:07:39 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!17/06/01 00:07:39 ERROR client.TransportClient: Failed to send RPC 8640096737034642920 to /10.0.0.5:59400: java.nio.channels.ClosedChannelExceptionjava.nio.channels.ClosedChannelException    at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)17/06/01 00:07:39 ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map()) to AM was unsuccessfuljava.io.IOException: Failed to send RPC 8640096737034642920 to /10.0.0.5:59400: java.nio.channels.ClosedChannelException    at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:249)    at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:233)    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:488)    at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)    at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:438)    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)    at java.lang.Thread.run(Thread.java:748)Caused by: java.nio.channels.ClosedChannelException    at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)17/06/01 00:07:39 ERROR util.Utils: Uncaught exception in thread Yarn application state monitororg.apache.spark.SparkException: Exception thrown in awaitResult    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)    at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)    at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:512)    at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:93)    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:151)    at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:467)    at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1588)    at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1833)    at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1283)    at org.apache.spark.SparkContext.stop(SparkContext.scala:1832)    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:108)Caused by: java.io.IOException: Failed to send RPC 8640096737034642920 to /10.0.0.5:59400: java.nio.channels.ClosedChannelException    at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:249)    at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:233)    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:488)    at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)    at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:438)    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)    at java.lang.Thread.run(Thread.java:748)Caused by: java.nio.channels.ClosedChannelException    at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)

原因貌似因为用java8的原因，在yarn-site.xml配置

<property>    <name>yarn.nodemanager.pmem-check-enabled</name>    <value>false</value></property><property>    <name>yarn.nodemanager.vmem-check-enabled</name>    <value>false</value></property>