Spark源码学习——在Linux环境下使用IDEA看Spark源码

本篇文章主要解决的问题 
1.Spark在Linux下实验环境的搭建

---

一、Spark源码阅读环境的准备

本文介绍的是Centos下的各项配置方法

下面列举搭建此环境需要的各个组件:

• JDK的安装（JDK1.7）
• Scala的安装
• SBT的安装
• Git的安装
• IDEA的安装

最后根据安装的环境，导入源码，进行源码的学习，整片文章会讲的非常细致，适合新手学习参考。

---

二、JDK的安装

1.在安装CentOS时，系统会自动安装jdk，需要先把它卸载掉，装上自定义的jdk 1.7。

卸载步骤如下：

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># rpm -qa|grep jdk</span>java-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.6</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-openjdk-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.6</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.50</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.11</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>.el6_3.x86_64java-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-openjdk-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.3</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4.1</span>.el6_3.x86_64[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># rpm -qa|grep gcj</span>java-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>-gcj-compat-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">40</span>jpp.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">115</span>  libgcj-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4.1</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">48</span>.el5  [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># yum -y remove java java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64</span>[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># yum -y remove java java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64</span>[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># yum -y remove java java-1.4.2-gcj-compat-1.4.2.0-40jpp.115  </span>[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># yum -y remove libgcj-4.1.2-48.el5</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li></ul>

操作完成后，输入java -version后会显示无法查询到Java版本，说明卸载成功。

2.首先从官网下载我们需要的JDK版本，为JDK1.7， 
下载链接为：JDK1.7版本下载链接

会涉及到版本选择的问题，我们需要查询一下当前系统的版本， 
在命令行输入输入 
Uname -a 或 more /proc/version

在内核版本后面会有一个X86_64就是表示64位系统啦 
否则就为32位系统，然后根据系统的版本下载相应的JDK。我这边下载的是jdk-7u45-linux-x64.tar.gz。

3.进行JDK的安装 
解压源码包 
通过终端在/usr/local目录下新建java文件夹，命令行：

<code class="hljs perl has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">sudo <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">mkdir</span> /usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">local</span>/java</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

然后将下载到压缩包拷贝到java文件夹中，命令行： 
进入jdk源码包所在目录

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">cp</span> jdk-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>u45-linux-x64<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tar</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.gz</span> /usr/local/java</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

然后进入java目录，命令行：

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">cd</span> /usr/local/java</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

解压压缩包，命令行：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">sudo tar xvf jdk-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>u45-linux-x64<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tar</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.gz</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

然后可以把压缩包删除，命令行：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">sudo rm jdk-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>u45-linux-x64<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tar</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.gz</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

设置JDK的环境变量 
这里采用全局设置方法，就是修改etc/profile，它是是所有用户的共用的环境变量

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">sudo</span> gedit /etc/profile</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

没有图形界面的请用vi打开。

打开之后在末尾添加

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> JAVA_HOME=/usr/local/java/jdk1.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7.0</span>_79 <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> JRE_HOME=/usr/local/java/jdk1.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7.0</span>_79/jre<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> CLASSPATH=.:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JAVA_HOME</span>/lib/dt.jar:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JAVA_HOME</span>/lib/tools.jar:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JRE_HOME</span>/lib:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$CLASSPATH</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> PATH=<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JAVA_HOME</span>/bin:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$PATH</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>

请记住，在上述添加过程中，等号两侧不要加入空格，不然会出现“不是有效的标识符”，因为source /etc/profile 时不能识别多余到空格，会理解为是路径一部分。 
然后保存

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">source</span> /etc/profile</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

使profile生效 
最后在终端输入java -verson，检验是否安装成功。 
成功则显示 
java version “1.7.0_79” 
Java(TM) SE Runtime Environment (build 1.7.0_79-b18) 
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) 
至此，JDK已经安装完成。

---

三、Scala的安装

1.scala下载 
首先下载安装包scala下载链接 
我下载的是2.10.6版本。

2.解压 
解压下载的安装包，并把它移动到指定目录。

<code class="hljs vala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#tar -zxf scala-2.10.6.tgz</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#sudo mv scala-2.10.6 /usr/local</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

3.配置环境变量

<code class="hljs vala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;"># sudo gedit/etc/profile</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

在最后加

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> SCALA_HOME=/usr/local/scala-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.10</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> PATH=<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$PATH</span>:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SCALA_HOME</span>/bin</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

使配置文件立即生效

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">source</span> /etc/profile</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

4.测试 
输入scala -version进行测试。 
显示：Scala code runner version 2.10.6…安装成功

---

四、SBT的安装

1.下载sbt的安装包 
安装包地址：sbt安装包地址

2.建立目录，解压文件到所建立目录

<code class="hljs perl has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">$ sudo <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">mkdir</span> /usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">local</span>/sbt$ sudo tar zxvf sbt-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">13.5</span>.tgz -C /usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">local</span>/</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

3.建立启动sbt的脚本文件 
/选定一个位置，建立启动sbt的脚本文本文件，如/usr/local/sbt/ 目录下面新建文件名为sbt的文本文件/

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>cd /usr/local/sbt/<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>vim sbt</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

在sbt文本文件中添加

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">BT_OPTS<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">=</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"</span>java <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SBT_OPTS</span> <span class="hljs-attribute" style="box-sizing: border-box;">-jar</span> /opt/scala/sbt/bin/sbt<span class="hljs-attribute" style="box-sizing: border-box;">-launch</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"$@"</span> </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

然后按esc键 输入 :wq 保存退出，注意路径需要能够正确的定位到解压的sbt文件包中的sbt-launch.jar文件即可

/×修改sbt文件权限×/

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>chmod u+x sbt </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

4.配置PATH环境变量，保证在控制台中可以使用sbt命令 
$ vim ~/.bashrc 
/在文件尾部添加如下代码后，保存退出/

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">export <span class="hljs-constant" style="box-sizing: border-box;">PATH</span>=<span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/opt/scala</span><span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/sbt/</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:</span><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$PATH</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

使配置文件立刻生效

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">$ <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">source</span> ~/.bashrc</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

5.测试sbt是否安装成功 
输入sbt -version 
第一次执行时，会下载一些文件包，然后才能正常使用，要确保联网了，安装成功后显示如下 
[info] Set current project to sbt (in build file:/opt/scala/sbt/) 
[info] 0.13.5 
至此，sbt已经安装成功。

---

五、Git的安装

新版本的centos能够通过命令行直接安装

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>yum install git</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

但是这种方法虽然简单，但是一般仓库里的版本更新不及时，比如 CentOS 仓库中的 git 最新版是1.7.1，但是 git 官方已经到2.x 的版本了。对于想要获取最新git的系统，只能下rpm包或者用源码。

步骤如下：

1.下载git的安装包 
下载安装包：git安装包

2.解压源文件

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">tar <span class="hljs-attribute" style="box-sizing: border-box;">-zxvf</span> git<span class="hljs-attribute" style="box-sizing: border-box;">-latest</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>tar<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>gz或者xz <span class="hljs-attribute" style="box-sizing: border-box;">-d</span> git<span class="hljs-attribute" style="box-sizing: border-box;">-latest</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>tar<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>xztar <span class="hljs-attribute" style="box-sizing: border-box;">-xvf</span> git<span class="hljs-attribute" style="box-sizing: border-box;">-latest</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>tar</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

3.编译安装 
进入git的目录下，依次运行以下命令

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>autoconf<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>./configure<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>make<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>make install</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

可能会遇到无法识别autoconf这个命令，则需要先安装下

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>yum -y install autoconf</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

在执行make操作时如果报错：缺少gcc，则通过yum安装gcc

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>yum -y install gcc</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

重新执行make操作，还报错，缺少zlib.h。我们可以查看是否存在zlib.h

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>whereis zlib</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

如果没有，则需要安装zlib，这儿不再详细讲述，安装完zlib后，重新执行上面的命令。

4.验证是否成功

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@sl</span> git-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># whereis git</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">git:</span>/usr/local/bin/git[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@sl</span> git-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># git -version</span>git version <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>至此，git已经安装完成。</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li></ul>

---

六、IDEA的安装

1.下载安装包 
首先需要从官网下载IntelliJ IDEA的安装包。 
下载地址为：IDEA下载链接

2.安装 
将下载的安装包拷贝到需要安装的位置， 
然后进行解压，解压的过程其实就是安装的过程，

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">$  sudo tar -xvf ideaIU-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tar</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.gz</span> </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

安装比较简单，至此已经安装完成。

---

七、在IDEA中导入源码

从 Github 导入 Spark 工程 
打开IntelliJ IDEA 后，在菜单栏中选择 VCS→Check out from Version Control→Git，之后在 Git Repository URL 中填入 Spark 项目的地址，并指定好本地路径。

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-label" style="box-sizing: border-box;">https:</span>//github<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.com</span>/apache/spark</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

点击该窗口中的的 Clone 后，开始从 Github 中 clone 该项目，该过程试你网速而定，大概需要3-10分钟。 
编译 Spark 
当 clone 完毕后，IntelliJ IDEA 会自动提示你该项目有对应的 pom.xml 文件，是否打开。这里直接选择 Open 该 pom.xml 文件，然后系统会自动解析项目的相关依赖，该步骤也会因你的网络和系统相关环境，所需时间不同。 
该步骤完成之后，请手动编辑 Spark 根目录下的 pom.xml 文件，找到其中指定 java 版本的那一行(java.version)，根据你的系统环境，如果使用的是 jdk1.7 ，那么或许需要你将其值改成1.7（默认是1.6）。 
之后打开 shell 终端，在命令行中进入刚才导入的 spark 项目根目录，执行

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>sbt/sbt assembly</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

该编译命令将全部采用默认的配置来编译 Spark，若想指定相关组件的版本，可以查看 Spark 官网中的 Build-Spark（http://spark.apache.org/docs/latest/building-spark.html），查看所有常用的编译选项。

至此，如果能够看完整篇文章，应该能够完成整个环境的安装。