Spark1.01运行例子以及问题解决

1. 直接下载编译好的 

export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

SPARK_JAR=./assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar \./bin/spark-class org.apache.spark.deploy.yarn.Client \--jar ./examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar \--class org.apache.spark.examples.JavaSparkPi \--args yarn-standalone \--num-workers 1 \--master-memory 512m \--worker-memory 512m \--worker-cores 1

改改对应的jar的位置

从上图可知 yarnAppstate: Finished运行失败

查看：http://resourcemanager:8088/cluster

可知运行失败，继续跟进查看日志

http://euca-clc:8042/node/containerlogs/container_1405925244858_0002_01_000001/hadoop/stderr/?start=-4096

Caused by: java.lang.NumberFormatException: For input string: "yarn-standalone"at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)at java.lang.Integer.parseInt(Integer.java:449)at java.lang.Integer.parseInt(Integer.java:499)at org.apache.spark.examples.JavaSparkPi.main(JavaSparkPi.java:40)

查看源代码可知 JavaSparkPi.java被修改了需要接收一个整型参数，不在是运行模式

所以用如下命令去掉 --args yarn-standalone

执行状态

yarnAppState: Succeeded成功了.

访问http://resourcemanager:8088/cluster并查看结果

运行成功。