Spark计算Pi---Spark学习笔记2

来源：互联网发布：网络推广专员是干嘛的编辑：程序博客网时间：2024/06/05 05:24

话接上回Spark学习笔记1-编译源代码，已经成功编译了spark on yarn

启动yarn

victor@victor-ubuntu:~/software/hadoop-2.2.0/sbin$ ./start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.shStarting namenodes on [localhost]localhost: starting namenode, logging to /home/victor/software/hadoop-2.2.0/logs/hadoop-victor-namenode-victor-ubuntu.outlocalhost: starting datanode, logging to /home/victor/software/hadoop-2.2.0/logs/hadoop-victor-datanode-victor-ubuntu.outStarting secondary namenodes [0.0.0.0]0.0.0.0: starting secondarynamenode, logging to /home/victor/software/hadoop-2.2.0/logs/hadoop-victor-secondarynamenode-victor-ubuntu.outstarting yarn daemonsstarting resourcemanager, logging to /home/victor/software/hadoop-2.2.0/logs/yarn-victor-resourcemanager-victor-ubuntu.outlocalhost: starting nodemanager, logging to /home/victor/software/hadoop-2.2.0/logs/yarn-victor-nodemanager-victor-ubuntu.outvictor@victor-ubuntu:~/software/hadoop-2.2.0/sbin$ jps22896 SecondaryNameNode23383 Jps22317 NameNode22578 DataNode23078 ResourceManager23342 NodeManager

运行脚本，准备运行的是spark的计算Pi的例子。先来看下准备条件：1.spark客户端目录，那几个文件夹放编译好的jar包。2. 运行的shell

victor@victor-ubuntu:~/software/incubator-spark-0.8.1-incubating$ pwd/home/victor/software/incubator-spark-0.8.1-incubating

victor@victor-ubuntu:~/software/incubator-spark-0.8.1-incubating/spark_compiled_client$ pwd/home/victor/software/incubator-spark-0.8.1-incubating/spark_compiled_client

victor@victor-ubuntu:~/software/incubator-spark-0.8.1-incubating/spark_compiled_client$ lltotal 28drwxrwxr-x  5 victor victor 4096  3月 28 21:38 ./drwxrwxr-x 24 victor victor 4096  3月 28 22:47 ../drwxrwxr-x  3 victor victor 4096  3月 28 02:41 assembly/drwxrwxr-x  2 victor victor 4096 12月 11 06:35 conf/drwxrwxr-x  3 victor victor 4096  3月 28 02:42 examples/-rwxr-xr-x  1 victor victor 4802 12月 11 06:35 spark-class*

#!/bin/shexport YARN_CONF_DIR=/home/victor/software/hadoop-2.2.0/etc/hadoopSPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop2.2.0.jar  \   ./spark-class org.apache.spark.deploy.yarn.Client   \--jar ./examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar \--class org.apache.spark.examples.JavaSparkPi \--args yarn-standalone  \--num-workers 2  \--master-memory 400m  \--worker-memory 512m  \--worker-cores 1

我的机器配置比较差，这里设置worker和master都比较保守。。。还是比较卡运行的时候。

执行截图：

<原创，转载请注明出处http://blog.csdn.net/oopsoom/article/details/22419597>

0 0