Hadoop集群安装

集群安装：

1、下载编译

下载hadoop2.6.0，解压

2、修改配置文件

打开hadoop-env.sh(一般在hadoop-2.6.0/etc/hadoop/文件夹下)

修改：

export JAVA_HOME=/A/soft/jdk1.7.0_79

修改core-site.xml,将下面的内容copy到core-site.xml中

<configuration>

<!--指定namenode地址-->

    <property>

        <name>fs.defaultFS</name>

        <value>hdfs://59.67.152.31:8020</value>

    </property>

<property>

        <name>hadoop.tmp.dir</name>

        <value>/A/soft/hadoop-2.6.0/data/tmp</value>

    </property>

<property>

<name>fs.trash.interval</name>

<value>10080</value>

</property>

</configuration>

修改 hdfs-site.xml，将下面的内容copy到dfs-site.xml中

<configuration>

<property>

        <name>dfs.replication</name>

        <value>3</value>

    </property>

<property>

        <name>dfs.permissions.enabled</name>

        <value>false</value>

    </property>

<property>

<name>dfs.namenode.http-address</name>

<value>59.67.152.31:50070</value>

</property>

<property>

<name>dfs.namenode.secondary.http-address</name>

<value>59.67.152.33:50090</value>

</property>

</configuration>

将mapred-site.xml.template修改为mapred-site.xml，修改mapred-site.xml，将将下面的内容copy到mapred-site.xml中

 

<configuration>

<property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

    </property>

<property>

        <name>mapreduce.jobhistory.address</name>

        <value>59.67.152.31:10020</value>

    </property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>59.67.152.31:19888</value>

</property>

<property>

<name>mapreduce.job.ubertask.enable</name>

<value>true</value>

</property>

</configuration>

修改yarn-site.xml，将将下面的内容copy到mapred-site.xml中

<configuration>

<!-- Site specific YARN configuration properties -->

<property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

<property>

        <name>yarn.resourcemanager.hostname</name>

        <value>59.67.152.32</value>

    </property>

<property>

<name>yarn.web-proxy.address</name>

<value>59.67.152.32:8888</value>

</property>

<property>

<name>yarn.log-aggregation-enable</name>

<value>true</value>

</property>

<property>

<name>yarn.log-aggregation.retain-seconds</name>

<value>604800</value>

</property>

<!--yarn的内存-->

<property>

 <name>yarn.nodemanager.resource.memory-mb</name>

<value>8192</value>

</property>

<!--YARN CPU-->

<property>

<name>yarn.nodemanager.resource.cpu-vcores</name>

<value>8</value>

</property>

</configuration>

修改slave，指定计算节点

59.67.152.31

59.67.152.32

59.67.152.33

3、格式化

进入当前安装目录下cd /A/soft/hadoop-2.6.0/

格式化：bin/hadoop namenode -format

格式化完成标志：successfully formated

4、向其他机器发送安装文件

 scp -r hadoop-2.6.0/ 59.67.152.32:/A/soft/

 scp -r hadoop-2.6.0/ 59.67.152.33:/A/soft/

5、Hadoop启动

在59.67.152.31机器启动服务HDFS

进入Hadoop目录àsbin/start-dfs.sh

在59.67.152.32机器启动yarn

进入Hadoop目录àsbin/start-yarn.sh

在59.67.152.31机器启动jobhistory

进入Hadoop目录àsbin/mr-jobhistory-daemon.sh start historyserver

在59.67.152.32机器启动防护进程

进入Hadoop目录àsbin/yarn-daemon.sh start proxyserver

jps可以查看是否启动成功

首次启动按照以上步骤，之后启动直接可以进入Hadoop目录àsbin/start-all.sh即可

6、集群测试

上传文件file1：bin/hdfs dfs -put /etc/profile /file1 /profile

查看59.67.152.32:8080/cluster/nodes是否存在上传的文件

运行wordcount：bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /profile /out

查看结果bin/hdfs dfs -cat /out/part-r-00000