Hadoop2.6.5安装部署

系统环境：

四台虚拟机

192.168.1.167 vm4.com
192.168.1.31 vm3.com
192.168.1.62 vm2.com
192.168.1.39 vm1.com

系统版本

[root@vm1 ~]# cat /etc/centos-releaseCentOS Linux release 7.0.1406 (Core)

部署前准备

vm1 可以免密登陆其他三台机器

四台机器互相解析

[root@vm1 ~]# cat /etc/hosts192.168.1.167   vm4.com192.168.1.31    vm3.com192.168.1.62    vm2.com192.168.1.39    vm1.com

防火墙关闭状态

systemctl stop firewalld.servicesystemctl disable firewalld.service

新建hadoop用户

useradd -u 5000 hadoop设置密码echo "hadoop"|passwd --stdin hadoop

JDK环境部署（四台机器都需要）：

在官网下载 jdk-8u151-linux-x64.tar.gz

tar zxvf /root/jdk-8u151-linux-x64.tar.gzmv /root/jdk1.8.0_151 /usr/local/jdk1.8

配置环境变量

vim /etc/profile.d/java.sh####JAVA_HOME=/usr/local/jdk1.8JAVA_BIN=/usr/local/jdk1.8/binJRE_HOME=/usr/local/jdk1.8/jrePATH=$PATH:/usr/local/jdk1.8/bin:/usr/local/jdk1.8/jre/binCLASSPATH=/usr/local/jdk1.8/jre/lib:/usr/local/jdk1.8/lib:/usr/local/jdk1.8/jre/lib/charsets.jar#####source生效source /etc/profile.d/java.sh

测试

[root@vm1 /]# java -versionjava version "1.8.0_151"Java(TM) SE Runtime Environment (build 1.8.0_151-b12)Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

JDK环境部署ok

Hadoop环境部署：

1.下载安装hadoop

hadoop官网：http://hadoop.apache.org

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.5/hadoop-2.6.5.tar.gztar zxvf hadoop-2.6.5.tar.gzcd hadoop-2.6.5vim etc/hadoop/hadoop-env.sh修改这一行：export JAVA_HOME=/usr/local/jdk1.8然后测试[root@vm2 hadoop]# ./bin/hadoop versionHadoop 2.6.5Subversion https://github.com/apache/hadoop.git -r e8c9fe0b4c252caf2ebf1464220599650f119997Compiled by sjlee on 2016-10-02T23:43ZCompiled with protoc 2.5.0From source with checksum f05c9fa095a395faa9db9f7ba5d754This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.6.5.jar

2.集群分布式环境配置

hadoop环境变量配置
vim ~/.bashrc
添加一行

export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/hadoop/sbin

source 生效

source ~/.bashrc

mv hadoop-2.6.5 /usr/local/hadoopchown -R hadoop.hadoop /usr/local/hadoop-2.6.5/

编辑配置文件

配置文件目录

[root@vm2 hadoop]# pwd/usr/local/hadoop/etc/hadoop

1.编辑slaves文件

将作为 DataNode 的主机名写入该文件，每行一个

[root@vm2 hadoop]# cat slavesvm2.comvm3.comvm4.com

2.编辑core-site.xml

<configuration>        <property>                <name>fs.defaultFS</name>                <value>hdfs://vm1.com:9000</value>        </property>        <property>                <name>hadoop.tmp.dir</name>                <value>file:/usr/local/hadoop/tmp</value>                <description>Abase for other temporary directories.</description>        </property></configuration>

3.编辑hdfs-site.xml

dfs.replication 为 Slave 节点个数

<configuration>    <property>        <name>dfs.namenode.secondary.http-address</name>        <value>vm1.com:50090</value>    </property>    <property>        <name>dfs.replication</name>        <value>3</value>    </property>    <property>        <name>dfs.namenode.name.dir</name>        <value>file:/usr/local/hadoop/tmp/dfs/name</value>    </property>    <property>        <name>dfs.datanode.data.dir</name>        <value>file:/usr/local/hadoop/tmp/dfs/data</value>    </property></configuration>

4.编辑mapred-site.xml

cp mapred-site.xml.template mapred-site.xml<configuration>        <property>                <name>mapreduce.framework.name</name>                <value>yarn</value>        </property>        <property>                <name>mapreduce.jobhistory.address</name>                <value>vm1.com:10020</value>        </property>        <property>                <name>mapreduce.jobhistory.webapp.address</name>                <value>vm1.com:19888</value>        </property></configuration>

5.编辑yarn-site.xml

<configuration>        <property>                <name>yarn.resourcemanager.hostname</name>                <value>vm1.com</value>        </property>        <property>                <name>yarn.nodemanager.aux-services</name>                <value>mapreduce_shuffle</value>        </property></configuration>

至此 hadoop启动所必需的配置已全部配置好

将整个hadoop目录打包用scp命令发送至其他节点

启动hadoop

首次启动需要先在 Master 节点执行 NameNode 的格式化：

hdfs namenode -format

成功的话，会看到 “successfully formatted” 和 “Exitting with status 0” 的提示，若为 “Exitting with status 1” 则是出错。

接着开启 NameNode 和 DataNode 守护进程。
以下三个脚本的存放路径在

[root@vm1 sbin]# pwd/usr/local/hadoop/sbin

前面已经添加到了环境变量 可以直接执行

start-dfs.shstart-yarn.shmr-jobhistory-daemon.sh start historyserver

通过命令 jps 可以查看各个节点所启动的进程。正确的话，在 vm1.com节点上可以看到 NameNode、ResourceManager、SecondrryNameNode、JobHistoryServer 进程如下：

[root@vm1 ~]# jps4050 NameNode4229 SecondaryNameNode4663 JobHistoryServer4378 ResourceManager7498 Jps

在其他三个datanode节点可以看到 DataNode 和 NodeManager 进程

[root@vm2 sbin]# jps12301 NodeManager13358 Jps12207 DataNode

以上 缺少任一进程都表示出错

另外还需要在 Master 节点上通过命令 hdfs dfsadmin -report 查看 DataNode 是否正常启动

[root@vm1 ~]# hdfs dfsadmin -reportConfigured Capacity: 160982630400 (149.93 GB)Present Capacity: 154752630784 (144.12 GB)DFS Remaining: 154750676992 (144.12 GB)DFS Used: 1953792 (1.86 MB)DFS Used%: 0.00%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0-------------------------------------------------Live datanodes (3):Name: 192.168.1.167:50010 (vm4.com)Hostname: vm4.comDecommission Status : NormalConfigured Capacity: 53660876800 (49.98 GB)DFS Used: 651264 (636 KB)Non DFS Used: 2076315648 (1.93 GB)DFS Remaining: 51583909888 (48.04 GB)DFS Used%: 0.00%DFS Remaining%: 96.13%Configured Cache Capacity: 0 (0 B)Cache Used: 0 (0 B)Cache Remaining: 0 (0 B)Cache Used%: 100.00%Cache Remaining%: 0.00%Xceivers: 1Last contact: Thu Nov 09 17:40:28 CST 2017Name: 192.168.1.31:50010 (vm3.com)Hostname: vm3.comDecommission Status : NormalConfigured Capacity: 53660876800 (49.98 GB)DFS Used: 651264 (636 KB)Non DFS Used: 2076905472 (1.93 GB)DFS Remaining: 51583320064 (48.04 GB)DFS Used%: 0.00%DFS Remaining%: 96.13%Configured Cache Capacity: 0 (0 B)Cache Used: 0 (0 B)Cache Remaining: 0 (0 B)Cache Used%: 100.00%Cache Remaining%: 0.00%Xceivers: 1Last contact: Thu Nov 09 17:40:28 CST 2017Name: 192.168.1.62:50010 (vm2.com)Hostname: vm2.comDecommission Status : NormalConfigured Capacity: 53660876800 (49.98 GB)DFS Used: 651264 (636 KB)Non DFS Used: 2076778496 (1.93 GB)DFS Remaining: 51583447040 (48.04 GB)DFS Used%: 0.00%DFS Remaining%: 96.13%Configured Cache Capacity: 0 (0 B)Cache Used: 0 (0 B)Cache Remaining: 0 (0 B)Cache Used%: 100.00%Cache Remaining%: 0.00%Xceivers: 1Last contact: Thu Nov 09 17:40:28 CST 2017