5、hadoop的分布式安装

基本信息

版本2.7.3安装机器三台机器账号hadoop源路径/opt/software/hadoop-2.7.3.tar.gz目标路径/opt/hadoop -> /opt/hadoop-2.7.3依赖关系zookeeper

安装过程

1).切换到hadoop账户，通过tar -zxvf命令将hadoop解压缩至目的安装目录：

[root@bgs-5p173-wangwenting opt]# su hadoop[hadoop@bgs-5p173-wangwenting opt]$ cd /opt/software[hadoop@bgs-5p173-wangwenting software]$  tar -zxvf hadoop-${version}.tar.gz  -C /opt[hadoop@bgs-5p173-wangwenting software]$ cd /opt[hadoop@bgs-5p173-wangwenting opt]$ ln -s /opt/hadoop-${version} /opt/hadoop

2).创建tmpdir目录：

[hadoop@bgs-5p173-wangwenting opt]$ cd  /opt/hadoop[hadoop@bgs-5p173-wangwenting hadoop]$ mkdir -p tmpdir

3).配置hadoop-env.sh文件:

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/[hadoop@bgs-5p173-wangwenting hadoop]$ mkdir -p /opt/hadoop/pids[hadoop@bgs-5p173-wangwenting hadoop]$ vim hadoop-env.sh在hadoop-env.sh文件中添加如下配置：export JAVA_HOME=/opt/javaexport HADOOP_PID_DIR=/opt/hadoop/pids

4.配置mapred-env.sh文件:

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/[hadoop@bgs-5p173-wangwenting hadoop]$ vim mapred-env.sh在mapred-env.sh文件中添加如下配置：export JAVA_HOME=/opt/java

5.配置core-site.xml文件

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/[hadoop@bgs-5p173-wangwenting hadoop]$  vim core-site.xml

在core-site.xml文件中添加如下配置：

<configuration>    <property>        <name>fs.defaultFS</name>        <value>hdfs://bgs-5p173-wangwenting:8020</value>    </property>    <property>        <name>hadoop.tmp.dir</name>        <value>/opt/hadoop/tmpdir</value>    </property>    <property>        <name>fs.file.impl</name>        <value>org.apache.hadoop.fs.LocalFileSystem</value>        <description>The FileSystem for file: uris.</description>    </property>    <property>       <name>fs.hdfs.impl</name>       <value>org.apache.hadoop.hdfs.DistributedFileSystem</value>       <description>The FileSystem for hdfs: uris.</description>    </property>    <property>        <name>io.compression.codecs</name>        <value>               org.apache.hadoop.io.compress.GzipCodec,               org.apache.hadoop.io.compress.DefaultCodec,               org.apache.hadoop.io.compress.BZip2Codec,               org.apache.hadoop.io.compress.SnappyCodec        </value>    </property>        <property>         <name>io.file.buffer.size</name>            <value>131072</value>     </property>         <property>       <name>fs.trash.interval</name>         <value>1440</value>      </property></configuration>

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/[hadoop@bgs-5p173-wangwenting hadoop]$ vim hdfs-site.xml

在hdfs-site.xml文件中添加如下配置：

<configuration><property>    <name>dfs.namenode.name.dir</name>    <value>file:/opt/hadoop/data/namenode</value></property><property>    <name>dfs.datanode.data.dir</name>    <value>file:/opt/hadoop/data/datanode</value></property><property>     <name>dfs.webhdfs.enabled</name>     <value>true</value> </property><property>    <name>dfs.replication</name>    <value>2</value></property><property>    <name>dfs.namenode.handler.count</name>    <value>200</value></property><property>    <name>dfs.blocksize</name>    <value>134217728</value></property><property>    <name>dfs.permissions.enabled</name>    <value>true</value></property><property>    <name>dfs.permissions</name>    <value>true</value></property><property>    <name>dfs.secondary.http.address</name>    <value>bgs-5p174-wangwenting:50090</value></property></configuration>

7.配置mapred-site.xml文件 mapred-site.xml

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/[hadoop@bgs-5p173-wangwenting hadoop]$ vim mapred-site.xml

在mapred-site.xml文件中添加如下配置：

<configuration>    <property>        <name>mapreduce.framework.name</name>        <value>yarn</value>    </property>    <property>        <name>mapred.job.history.server.embedded</name>        <value>true</value>    </property>    <property>        <name>mapreduce.jobhistory.address</name>        <value>bgs-5p173-wangwenting:10020</value>    </property>    <property>        <name>jobhistory.webapp.address</name>        <value>bgs-5p173-wangwenting:19888</value>    </property>    <property>        <name>hadoop.job.history.user.location</name>        <value>/mapred/userhistory</value>    </property>    <property>        <name>mapred.local.dir</name>        <value>/tmp/local</value>    </property>    <property>       <name>mapreduce.reduce.shuffle.memory.limit.percent</name>       <value>0.05</value>    </property>   <property>   　　<name>mapreduce.map.memory.mb</name>   　　<value>1536</value>   </property>   <property>   　　<name>mapreduce.map.java.opts</name>   　　<value>-Xmx1024M</value>   </property>   <property>   　　<name>mapreduce.reduce.memory.mb</name>   　　<value>3072</value>   </property>   <property>   　　<name>mapreduce.reduce.java.opts</name>   　　<value>-Xmx2560M</value>   </property></configuration>

8.配置yarn-site.xml文件: yarn-site.xml

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/[hadoop@bgs-5p173-wangwenting hadoop]$ vim yarn-site.xml

在yarn-site.xml文件中添加如下配置：

<configuration>    <property>        <name>yarn.nodemanager.aux-services</name>        <value>mapreduce_shuffle</value>    </property>    <property>        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>        <value>org.apache.hadoop.mapred.ShuffleHandler</value>    </property>    <property>          <name>yarn.log-aggregation-enable</name>          <value>true</value>      </property>      <property>        <name>yarn.nodemanager.remote-app-log-dir</name>          <value>/logs</value>     </property>    <property>        <name>yarn.log-aggregation.retain-seconds</name>        <value>2592000</value>    </property>    <property>        <name>yarn.nodemanager.resource.memory-mb</name>        <value>61440</value>    </property>    <property>        <name>yarn.nodemanager.resource.cpu-vcores</name>        <value>22</value>    </property>    <property>        <name>mapreduce.map.output.compress</name>          <value>true</value>    </property>            <property>        <name>mapred.map.output.compress.codec</name>          <value>org.apache.hadoop.io.compress.SnappyCodec</value>    </property>    <property>          <name>yarn.app.mapreduce.am.env</name>          <value>LD_LIBRARY_PATH=$HADOOP_HOME/lib/native</value>      </property>      <property>         <name>yarn.log.server.url</name>         <value>http://bgs-5p173-wangwenting:19888/jobhistory/logs/</value>     </property>     <property>        <name>yarn.nodemanager.delete.debug-delay-sec</name>        <value>3600</value>    </property>    <property>        <name>yarn.nodemanager.local-dirs</name>        <value>/tmp/nm-local-dir</value>    </property>    <property>        <name>yarn.scheduler.maximum-allocation-vcores</name>        <value>22</value>    </property>    <property>        <name>yarn.resourcemanager.scheduler.class</name>        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>    </property>    <property>        <name>yarn.scheduler.minimum-allocation-mb</name>        <value>512</value>    </property>    <property>        <name>yarn.scheduler.maximum-allocation-mb</name>        <value>10240</value>    </property>    <property>        <name>yarn.resourcemanager.am.max-attempts</name>        <value>4</value>    </property>    <!--RM HA-->    <property>        <name>yarn.resourcemanager.connect.retry-interval.ms</name>        <value>2000</value>    </property>    <property>        <name>yarn.resourcemanager.ha.enabled</name>        <value>true</value>    </property>    <property>        <name>yarn.resourcemanager.cluster-id</name>                                               <value>rm-cluster</value>                                                                  <description>集群名称，确保HA选举时对应的集群</description>                            </property>    <property>        <name>yarn.resourcemanager.ha.id</name>        <value>rm1</value>    </property>    <property>        <name>yarn.resourcemanager.ha.rm-ids</name>        <value>rm1,rm2</value>    </property>    <property>        <name>yarn.resourcemanager.zk-address</name>        <value>bgs-5p173-wangwenting:2181,bgs-5p174-wangwenting:2181,bgs-5p175-wangwenting:2181</value>    </property>    <!--rm1-->    <property>        <name>yarn.resourcemanager.address.rm1</name>        <value>bgs-5p173-wangwenting:8032</value>    </property>    <property>        <name>yarn.resourcemanager.scheduler.address.rm1</name>        <value>bgs-5p173-wangwenting:8030</value>    </property>    <property>        <name>yarn.resourcemanager.webapp.https.address.rm1</name>        <value>bgs-5p173-wangwenting:8090</value>    </property>    <property>        <name>yarn.resourcemanager.webapp.address.rm1</name>        <value>bgs-5p173-wangwenting:8088</value>    </property>    <property>        <name>yarn.resourcemanager.resource-tracker.address.rm1</name>        <value>bgs-5p173-wangwenting:8031</value>    </property>    <property>        <name>yarn.resourcemanager.admin.address.rm1</name>        <value>bgs-5p173-wangwenting:8033</value>    </property>    <!--rm2-->    <property>        <name>yarn.resourcemanager.address.rm2</name>        <value>bgs-5p174-wangwenting:8032</value>    </property>    <property>        <name>yarn.resourcemanager.scheduler.address.rm2</name>        <value>bgs-5p174-wangwenting:8030</value>    </property>    <property>        <name>yarn.resourcemanager.webapp.https.address.rm2</name>        <value>bgs-5p174-wangwenting:8090</value>    </property>    <property>        <name>yarn.resourcemanager.webapp.address.rm2</name>        <value>bgs-5p174-wangwenting:8088</value>    </property>    <property>        <name>yarn.resourcemanager.resource-tracker.address.rm2</name>        <value>bgs-5p174-wangwenting:8031</value>    </property>    <property>        <name>yarn.resourcemanager.admin.address.rm2</name>        <value>bgs-5p174-wangwenting:8033</value>    </property>    <property>          <name>yarn.client.failover-proxy-provider</name>          <value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>      </property>       <property>          <name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name>          <value>/yarn-leader-election</value>      </property>      <property>        <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>        <value>true</value>    </property>    <property>        <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>        <value>true</value>    </property></configuration>

9.配置hadoop运行的环境变量

[hadoop@bgs-5p173-wangwenting hadoop]$ vim /etc/profileexport HADOOP_HOME=/opt/hadoopexport PATH=$HADOOP_HOME/bin:$PATH

配置成功后，执行source  /etc/profile使配置生效

[hadoop@bgs-5p173-wangwenting hadoop]$ source /etc/profile

10.修改slaves文件:

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop[hadoop@bgs-5p173-wangwenting hadoop]$ vim slaves在slaves文件中添加//datanode的节点的位置bgs-5p174-wangwentingbgs-5p175-wangwenting

11.在bgs-5p173-wangwenting上复制hadoop-2.7.3到hadoop@bgs-5p174-wangwenting和hadoop@bgs-5p174-wangwenting机器并按照步骤9修改环境变量并执行以下操作:

[hadoop@bgs-5p173-wangwenting hadoop]$ scp -r /opt/hadoop-${version} hadoop@bgs-5p174-wangwenting:/opt/[hadoop@bgs-5p173-wangwenting hadoop]$ ln -s /opt/hadoop-${version} /opt/hadoop[hadoop@bgs-5p173-wangwenting hadoop]$ scp -r /opt/hadoop-${version} hadoop@bgs-5p175-wangwenting:/opt/[hadoop@bgs-5p173-wangwenting hadoop]$ ln -s /opt/hadoop-${version} /opt/hadoop

12.格式化namenode（仅第一次启动需要格式化！），启动hadoop,并启动jobhistory服务:  

# 格式化 namenode ，仅第一次启动需要格式化！！[hadoop@bgs-5p173-wangwenting hadoop]$ hadoop namenode -format # 启动[hadoop@bgs-5p173-wangwenting hadoop]$ ${HADOOP_HOME}/sbin/start-all.sh[hadoop@bgs-5p173-wangwenting hadoop]$ ${HADOOP_HOME}/sbin/mr-jobhistory-daemon.sh start historyserverstart-all.sh包含dfs和yarn两个模块的启动，分别为start-dfs.sh 、 start-yarn.sh，所以dfs和yarn可以单独启动。注意：如果datanode没有启动起来，看看是不是tmpdir中有之前的脏数据，删除这个目录其他两台机器也要删除。

13.检查每台机器的服务，bgs-5p173-wangwenting、bgs-5p174-wangwenting、bgs-5p175-wangwenting三台机器上分别输入jps：

[hadoop@bgs-5p173-wangwenting ~]$ jps24429 Jps22898 ResourceManager24383 JobHistoryServer22722 SecondaryNameNode22488 NameNode[ahdoop@bgs-5p174-wangwenting ~]$ jps7650 DataNode7788 NodeManager8018 Jps[hadoop@bgs-5p175-wangwenting ~]$ jps28407 Jps28038 DataNode28178 NodeManager如果三台机器正常输出上述内容，则表示hadoop集群的服务正常工作。

访问hadoop的服务页面：在浏览器中输入如下地址

http://bgs-5p173-wangwenting:8088http://bgs-5p173-wangwenting:50070http://bgs-5p173-wangwenting:19888