Spark的submit命令

在win系统下的Idea编译spark自带的JavaWordCount代码，到apache的官网下载对应的spark包spark-1.5.0-bin-hadoop2.6.tgz，从
spark-1.5.0-bin-hadoop2.6\examples\src\main\java\org\apache\spark\examples找到JavaWordCount源码在Idea上进行编译，编译的时候要导入spark-1.5.0-bin-hadoop2.6\lib下的jar包。当代码没有错误后在将其打成jar包。然后发送的集群上。
用submit命令进行提交。
命令参数如下：
一些常用选项：

--class:  #你的应用的启动类 (如 org.apache.spark.examples.SparkPi)--master: #集群的master URL (如 spark://23.195.26.187:7077)--deploy-mode: #是否发布你的驱动到worker节点(cluster) 或者作为一个本地客户端 (client) (默认就是 client模式)--conf:  # 任意的Spark配置属性， 格式key=value. 如果值包含空格，可以加引号“key=value”.--application-jar:  # 打包好的应用jar,包含依赖--application-arguments:  # 传给main()方法的参数

具体的提交命令如下：

---

运行在本地：

spark-submit --master local --class org.apache.spark.examples.JavaWordCount --executor-memory 1G --total-executor-cores 2 ~/JavaWordCount.jar hdfs://Master:9000/user/hadoop/input/text

结果能够在输出打印在屏幕上。

---

运行在集群上：

spark-submit --master yarn  --deploy-mode client --class org.apache.spark.examples.JavaWordCount --executor-memory 1G --total-executor-cores 2 ~/JavaWordCount.jar hdfs://Master:9000/user/hadoop/input/text

这是以交互模式运行在集群上结果能够输出到屏幕上。

---

当不指定–deploy-mode的内容时，默认采用client，即交互模式。

spark-submit --master yarn  --deploy-mode cluster --class org.apache.spark.examples.JavaWordCount --executor-memory 1G --total-executor-cores 2 ~/JavaWordCount.jar hdfs://Master:9000/user/hadoop/input/text

这是运行在集群上，结果并不会打印在屏幕上，结果在Master：8088上对应的application下面的对应的slave的logs内。
当采用交互模式（–deploy-mode client）的时候，Java8会出现错误，错误提示如下：

16/05/11 22:45:46 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!Exception in thread "main" java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext    at org.apache.spark.SparkContext.org$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:103)    at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:817)at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)    at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)    at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)    at org.apache.spark.api.java.JavaSparkContext.textFile(JavaSparkContext.scala:191)    at org.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:31)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke(Method.java:498)    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

在网上查找了很多资料都不解决问题。最后在stackoverflow上有网友说是Java8的问题，更换成Java7就没有问题了，链接如下：stackoverflow解决
附图：