Spark1.5.2安装

一直想学习着写博客做个备忘，但是一直懒也没有开始做，终于最近虚拟机的东西太多了想重新装一个虚拟机，由于自己是学数据挖掘的，就写一篇关于单机版spark的安装攻略来练练手。

安装包准备

spark原理什么的这里就不介绍了，有问题大家最好参考官方文档，网址为：http://spark.apache.org/。 
本次安装主要用到的安装包如下： 
1.spark-1.5.2-bin-hadoop2.6.tgz 
2.hadoop-2.6.0.tar.gz 
3.jdk-8u65-linux-x64.gz

一、Linux环境准备

1.给linux用户添加sudo权限

因为我安装过程中不会在root用户下安装，因此需要给自己的用户添加sudo权限。 
（1）切换到root用户

<code class="hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">su root</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

（2）给用户添加sudoers写权限

<code class="hljs perl has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">chmod</span> u+w sudoers</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

（3）打开sudoers

<code class="hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">vim /etc/sudoers</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

（4）添加权限

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">user     <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">ALL</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">=</span>(<span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">ALL</span>)       <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">ALL</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//user替换为自己的linux用户</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

（5）修改sudoers权限

<code class="hljs perl has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">chmod</span> u-w sudoers</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

2.网络配置修改

本步骤主要设置操作系统的网络连接以及hostname 
（1）设置静态ip 
由于在学校需要使用客户端上网，所以虚拟机用桥接方式无法连接外网，所以需要设置NAT模式静态IP。此处偷懒直接用linux网络设置快捷方法，不修改文件。

 
各项配置主要参考虚拟机设置配置 
（2）修改hostname

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">sudo</span> vim /etc/hosts</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

在hosts中添加 192.168.171.120 spark。ip根据自己设置的各不相同。 
（3）重启网卡生效

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">sudo</span> service network restart</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

终端中输入命令hostname查看是否修改正确，最好重启虚拟机。 

二、安装配置

1.安装jdk

（1）直接解压gz文件

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">tar zxvf jdk<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>u65<span class="hljs-attribute" style="box-sizing: border-box;">-linux</span><span class="hljs-attribute" style="box-sizing: border-box;">-x64</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>gz sudo mv jdk1<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.8</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span>_65<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span> /opt</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

（2）修改jdk环境变量

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">sudo</span> vim /etc/profile</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

在profile末尾添加如下配置

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> JAVA_HOME=/opt/jdk1.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8.0</span>_65<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> CLASSPATH=.:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$CLASSPATH</span>:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JAVA_HOME</span>/lib<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> PATH=<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$PATH</span>:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JAVA_HOME</span>/bin</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

source profile文件

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">source</span> /etc/profile</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

查看环境变量是否已设好,如果出现对应版本即jdk环境变量设置完毕

<code class="hljs livecodeserver has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">java -<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">version</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

2.设置SSH登录

虽然本教程不是搭建集群，理论上不需要使用SSH，但是为了在开启hadoop的时候省略输入密码这一步骤设置本机无密码登录。

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">ssh<span class="hljs-attribute" style="box-sizing: border-box;">-keygen</span> <span class="hljs-attribute" style="box-sizing: border-box;">-t</span> rsa</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

一路回车

<code class="hljs vbnet has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">Generating <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span>/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> rsa <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">key</span> pair.Enter file <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> which <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> save the <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">key</span> (/home/luo/.ssh/id_rsa): Enter passphrase (empty <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> no passphrase): Enter same passphrase again: Your identification has been saved <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> /home/luo/.ssh/id_rsa.Your <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">key</span> has been saved <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> /home/luo/.ssh/id_rsa.pub.The <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">key</span> fingerprint <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">is</span>:f5:d8:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>c:b4:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>:d5:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>e:ce:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>a:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">41</span>:a3:e6:bc:a8:fc luo@sparkThe <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">key</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">'s randomart image is:</span>+--[ RSA <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2048</span>]----+|        .o+.     ||         oo..    ||        o.++     ||       +.+oBo    ||        S.+++.   ||       . .  o    ||      . .        ||   . .           ||    o.E          |+-----------------+</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li></ul>

进入.ssh文件夹cd ~/.ssh

<code class="hljs haml has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">-<span class="ruby" style="box-sizing: border-box;">rw-------. <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> luo luo <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1675</span> <span class="hljs-constant" style="box-sizing: border-box;">Dec</span>  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">03</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">36</span> id_rsa</span>-<span class="ruby" style="box-sizing: border-box;">rw-r--r--. <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> luo luo  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">391</span> <span class="hljs-constant" style="box-sizing: border-box;">Dec</span>  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">03</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">36</span> id_rsa.pub</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

将id_rsa.pub复制到新文件cat id_rsa.pub > authorized_keys

修改authorized_keys权限sudo chmod 600 authorized_keys 
验证是否需要无密码登录是否成功ssh spark 此处spark替换为自己的hostname。

3.Hadoop安装

（1）解压安装gz文件

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">tar zxvf hadoop-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.6</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tar</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.gz</span>sudo mv hadoop-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.6</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span> /opt/</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

（2）环境变量修改

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">sudo</span> vim /etc/profile</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

修改环境变量

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> JAVA_HOME=/opt/jdk1.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8.0</span>_65<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> HADOOP_HOME=/opt/hadoop-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.6</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> CLASSPATH=.:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$CLASSPATH</span>:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JAVA_HOME</span>/lib<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> PATH=<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$PATH</span>:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JAVA_HOME</span>/bin/:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$HADOOP_HOME</span>/bin</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

（3）修改hadoop配置 
配置文件存放在/opt/hadoop-2.6.0/etc/hadoop中。

修改hadoop.env.sh,将JAVA_HOME修改为自己jdk路径。

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> JAVA_HOME=/opt/jdk1.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8.0</span>_65</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

修改core.site.xml

<code class="hljs xml has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span>  <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>fs.default.name<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>hdfs://spark:9000<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>  <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

修改hdfs-site.xml

<code class="hljs xml has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span>  <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>dfs.replication<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>1<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>  <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>  <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>dfs.namenode.name.dir<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>/opt/hadoop/dfs/name<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>  <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>  <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>dfs.datanode.data.dir<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>/opt/hadoop/dfs/data<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>  <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul>

注意：配置文件中/opt/hadoop/dfs/name和/opt/hadoop/dfs/data文件夹需要自己创建，并且权限需要和自己的用户权限一致。不然hadoop无法开启。修改文件夹权限代码如下：

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">sudo</span> chown -R luo:luo /opt/hadoop</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

修改map-site.xml

<code class="hljs xml has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span>  <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>mapreduce.framework.name<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>yarn<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>  <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

修改yarn-site.xml

<code class="hljs xml has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"><!-- Site specific YARN configuration properties --></span>  <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>yarn.nodemanager.aux-services<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>mapreduce_shuffle<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>  <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li></ul>

（4）格式话HDFS

<code class="hljs rsl has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"> bin/hdfs namenode -<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">format</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

出现如下提示，如status=0一般就成功了，如果等于1便失败，此时删除/opt/hadoop/dfs/name文件夹并重新新建格式化。记住namenode只能格式化一次。

<code class="hljs markdown has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">15/12/07 04:25:37 INFO util.ExitUtil: Exiting with status 015/12/07 04:25:37 INFO namenode.NameNode: SHUTDOWN_MSG: /<span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span>SHUTDOWN_MSG: Shutting down NameNode at spark/192.168.171.120<span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span><span class="hljs-strong" style="box-sizing: border-box;">*****</span>/</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>

（5）开启hdaoop

最好关闭掉防火墙sudo service iptables stop

<code class="hljs sql has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">sbin/<span class="hljs-operator" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">start</span>-<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">all</span>.sh</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

使用jps密令查看是否各个组件均开启。

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">[luo<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@spark</span> hadoop-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.6</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>]<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>jps<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4567</span> <span class="hljs-constant" style="box-sizing: border-box;">DataNode</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5272</span> <span class="hljs-constant" style="box-sizing: border-box;">Jps</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4954</span> <span class="hljs-constant" style="box-sizing: border-box;">NodeManager</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4859</span> <span class="hljs-constant" style="box-sizing: border-box;">ResourceManager</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4446</span> <span class="hljs-constant" style="box-sizing: border-box;">NameNode</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4719</span> <span class="hljs-constant" style="box-sizing: border-box;">SecondaryNameNode</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li></ul>

使用浏览其打开spark:50070查看是否正常启动，此处的spark代表自己的hostname

至此，伪分布hadoop安装完成。

4.Spark安装

（1）解压安装包gz文件

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"> tar zxvf spark<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-attribute" style="box-sizing: border-box;">-bin</span><span class="hljs-attribute" style="box-sizing: border-box;">-hadoop2</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.6</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>tgz  sudo mv spark<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-attribute" style="box-sizing: border-box;">-bin</span><span class="hljs-attribute" style="box-sizing: border-box;">-hadoop2</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.6</span> /opt<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

（2）添加spark环境变量

在/etc/profile中添加spark路径

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> JAVA_HOME=/opt/jdk1.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8.0</span>_65<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> HADOOP_HOME=/opt/hadoop-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.6</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> SPARK_HOME=/opt/spark-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>-bin-hadoop2.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> CLASSPATH=.:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$CLASSPATH</span>:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JAVA_HOME</span>/lib:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>/lib:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$HADOOP_HOME</span>/lib<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> PATH=<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$PATH</span>:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JAVA_HOME</span>/bin/:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$HADOOP_HOME</span>/bin:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SPARK_HOME</span>/bin</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>

（3）修改配置文件

进入/conf文件复制cp spark-env.sh.template spark-env.sh 

用vim 打开spark-env.sh添加配置

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> JAVA_HOME=/opt/jdk1.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8.0</span>_65<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> SPARK_MASTER_IP=spark<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> SPARK_WORKER_MEMORY=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>g<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> SPARK_MASTER_PORT=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7077</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

（4）启动spark 
sbin/start-all.sh 
使用jps查看worker和master是否开启

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">[luo<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@spark</span> spark-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>-bin-hadoop2.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>]<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>jps<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5862</span> <span class="hljs-constant" style="box-sizing: border-box;">Worker</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4567</span> <span class="hljs-constant" style="box-sizing: border-box;">DataNode</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4954</span> <span class="hljs-constant" style="box-sizing: border-box;">NodeManager</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4859</span> <span class="hljs-constant" style="box-sizing: border-box;">ResourceManager</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5931</span> <span class="hljs-constant" style="box-sizing: border-box;">Jps</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5709</span> <span class="hljs-constant" style="box-sizing: border-box;">Master</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4446</span> <span class="hljs-constant" style="box-sizing: border-box;">NameNode</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4719</span> <span class="hljs-constant" style="box-sizing: border-box;">SecondaryNameNode</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li></ul>

使用浏览器登录spark：8080验证spark是否开启，此处spark换为自己的hostname 
至此spark配置完毕。 
文中如若有误，欢迎指正交流。