Spark伪分布式安装（一）

笔者是在已安装好的伪分布式Hadoop环境下安装的Spark。虚拟机环境为：centos7。

Hadoop版本号为：

[centosm@centosm spark]$ hadoop versionHadoop 2.7.2

一、Spark伪分布式安装

1、确定hadoop环境已安装好

2、下载spark
http://spark.apache.org/downloads.html

3、解压spark压缩包并重命名解压后的文件名为spark

     tar -zvxf  spark-2.1.1-bin-hadoop2.7.tgz 

4、修改配置文件

     cd /usr/local/spark     cp ./conf/spark-env.sh.template ./conf/spark-env.sh     spark-env.sh 文件后追加如下语句vi spark-env.shexport SPARK_DIST_CLASSPATH=$(/home/centosm/hadoopM/bin/hadoop classpath)

5、启动spark，运行时出现下列问题

解决：

vi  spark-config.shexport JAVA_HOME=/usr/java/jdk1.8.0_91

6、运行spark自带的一些小例子，如下计算 π 的近似值

[centosm@centosm spark]$ ./bin/run-example SparkPi 2>&1 | grep "Pi is roughly"Pi is roughly 3.1428357141785708

7、进入spark交互模式

[centosm@centosm spark]$ ./bin/spark-shell            Setting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/home/centosm/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/home/centosm/hadoopM/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]17/06/22 14:51:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable17/06/22 14:52:15 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectExceptionSpark context Web UI available at http://192.168.22.125:4040Spark context available as 'sc' (master = local[*], app id = local-1498114310652).Spark session available as 'spark'.Welcome to      ____              __     / __/__  ___ _____/ /__    _\ \/ _ \/ _ `/ __/  '_/   /___/ .__/\_,_/_/ /_/\_\   version 2.1.1      /_/Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_91)Type in expressions to have them evaluated.Type :help for more information.scala> 

在交互模式的情启动的情况可以进入web页面查看相关信息，如下：

8、yum 安装sbt

[centosm@centosm test]$ curl https://bintray.com/sbt/rpm/rpm > bintray-sbt-rpm.repo  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100   159    0   159    0     0     23      0 --:--:--  0:00:06 --:--:--    37[centosm@centosm test]$[centosm@centosm test]$ sudo mv bintray-sbt-rpm.repo /etc/yum.repos.d/[root@centosm test]# yum install sbt[centosm@centosm bin]$ sbt --versionGetting org.scala-sbt sbt 0.13.15  (this may take some time)...