最新版spark1.1.0集群安装配置
来源:互联网 发布:淘宝网限时秒杀 编辑:程序博客网 时间:2024/06/05 09:23
和分布式文件系统和NoSQL数据库相比而言,spark集群的安装配置还算是比较简单的:
很多教程提到要安装java和scala,但我发现spark最新版本是包含scala的,JRE采用linux内嵌的版本也是可以的!
- 在主节点(bluejoe0)上安装spark1.1.0:
wget http://mirror.bit.edu.cn/apache/spark/spark-1.1.0/spark-1.1.0-bin-hadoop2.3.tgz
tar -zxvf spark-1.1.0-bin-hadoop2.3.tgz
ln -s spark-1.1.0-bin-hadoop2.3 spark - 启动spark-shell:
cd /usr/local/spark/bin
./spark-shell
可以看到spark已经自带了scala 2.10: - 输入测试程序:
scala> val data = Array(1, 2, 3, 4, 5)
data: Array[Int] = Array(1, 2, 3, 4, 5)
scala> val distData = sc.parallelize(data)
distData: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:14
scala> distData.reduce(_+_) - 可以观察4040端口:
- 也可以测试PI的计算:
./bin/run-example SparkPi
14/11/23 16:08:25 INFO SparkContext: Job finished: reduce at SparkPi.scala:35, took 1.008332384 s
Pi is roughly 3.1403 - 也可以采用spark-submit来提交任务:
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[6] /usr/local/spark/lib/spark-examples-1.1.0-hadoop2.3.0.jar 1000
14/11/23 16:07:30 INFO SparkContext: Job finished: reduce at SparkPi.scala:35, took 46.220537186 s
Pi is roughly 3.14172056 - 现在安装几个从节点,scp spark.tgz文件到其它节点,如:bluejoe4,bluejoe5,bluejoe9
- 注意设置好ssh无密码登录;
- 修改conf/slaves
# A Spark Worker will be started on each of the machines listed below.
bluejoe4
bluejoe5
bluejoe9 - 在bluejoe0上启动spark集群:
./sbin/start-all.sh
此时可以在浏览器上观察到3个从节点的情况: - 再测试在集群上计算PI的程序:
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://bluejoe0:7077 /usr/local/spark/lib/spark-examples-1.1.0-hadoop2.3.0.jar 1000
14/11/23 16:05:00 INFO SparkContext: Job finished: reduce at SparkPi.scala:35, took 26.322514766 s
Pi is roughly 3.14159516
此时观察浏览器的显示:
0 0
- 最新版spark1.1.0集群安装配置
- Spark1.0.0 集群配置
- hadoop2.4.1集群安装spark1.1.0
- Spark1.3.1集群安装
- spark1.6.0集群安装
- spark1.6.0集群安装
- spark1.6.0集群安装
- spark1.0.2分布式集群安装
- spark1.3.1安装和集群的搭建
- hadoop2.4+spark1.3.0集群安装
- 最新版scala2.11.8与spark1.6.1一步到位安装
- 最新版RockMongo安装配置
- Spark1.5.2 on Hadoop2.4.0 安装配置
- Spark1.4.1单机版安装配置
- Ubuntu Linux64 安装配置Spark1.6.1
- Spark1.3.0安装配置及WordCount示例
- spark1.6.1及scala2.11.8安装配置
- MySQL最新版安装配置教程
- 求一组数字最小回文
- linux学习资源
- 数值的整数次方 【微软面试100题 第七十一题】
- 逻辑与(&和&&)运算符的区别
- 【贪心】HDU-1789 Doing Homework again
- 最新版spark1.1.0集群安装配置
- Session对性能测试的影响
- 【转】PC-Lint的使用方法
- 太恶心了,竟是这个原因导致Android程序UI无法预览
- HDU-#5104 Primes Problem
- ARM寄存器的别名、以及关于APCS
- W3Help知识库(web兼容性问题解决方案知识库)
- 目标识别器的设计-TargetMarker-1
- 图像处理中的卷积与模板