Hadoop 2.5.2分布式集群配置(VirtualBox虚拟机模拟)

本教程使用VirtualBox模拟一共有四台机器，系统为CentOS 7 x86-64，其中一台为master 其余三台为slave。

1.使用VirtualBox新建master和三台slave，配置网络

四台机器网络设置为桥接模式，网卡选择主机上网的真实网卡。

分别修改/etc/sysconfig/network-scripts/ifcfg-enp0s3

设置静态ip上网模式和固定ip及相应网关配置（与真实主机一致）

# vi /etc/sysconfig/network-scripts/ifcfg-enp0s3#####  master机器IP设置 #####TYPE=Ethernet#BOOTPROTO=dhcp 注释默认自动获取ip上网模式，添加静态 static 模式BOOTPROTO=static DEFROUTE=yesPEERDNS=yesPEERROUTES=yesIPV4_FAILURE_FATAL=noIPV6INIT=yesIPV6_AUTOCONF=yesIPV6_DEFROUTE=yesIPV6_PEERDNS=yesIPV6_PEERROUTES=yesIPV6_FAILURE_FATAL=noNAME=enp0s3UUID=1c78b02d-5a3a-42f0-a83f-a17fd8623cdfDEVICE=enp0s3ONBOOT=yes             # 开机加载IPADDR=192.168.0.200   # master上网固定IPNETMASK=255.255.255.0  # 子网掩码GATEWAY=192.168.0.1    # 网关DNS1=223.5.5.5         # DNS服务器1DNS2=8.8.8.8           # DNS服务器2#####  slave机器IP设置 #####TYPE=Ethernet#BOOTPROTO=dhcp 注释默认自动获取ip上网模式，添加静态 static 模式BOOTPROTO=static DEFROUTE=yesPEERDNS=yesPEERROUTES=yesIPV4_FAILURE_FATAL=noIPV6INIT=yesIPV6_AUTOCONF=yesIPV6_DEFROUTE=yesIPV6_PEERDNS=yesIPV6_PEERROUTES=yesIPV6_FAILURE_FATAL=noNAME=enp0s3UUID=1c78b02d-5a3a-42f0-a83f-a17fd8623cdfDEVICE=enp0s3ONBOOT=yes             # 开机加载IPADDR=192.168.0.201   # slave1上网固定IP 其余两台IP相应为192.168.0.202，192.168.0.203NETMASK=255.255.255.0  # 子网掩码GATEWAY=192.168.0.1    # 网关DNS1=223.5.5.5         # DNS服务器1DNS2=8.8.8.8           # DNS服务器2

以上修改完成后均需执行命苦重启网卡服务使配置生效

# service network restart

互相ping，看是否测试成功，若不成功，注意防火墙的影响。

关闭windows或虚拟机的防火墙。

centos关闭firewall：

systemctl stop firewalld.service #停止firewall

systemctl disable firewalld.service #禁止firewall开机启动

firewall-cmd –state #查看默认防火墙状态（关闭后显示notrunning，开启后显示running）

更多防火墙高级设置参考CentOS 7.0关闭默认防火墙启用iptables防火墙

2.修改master和三台slave主机名并配置/etc/hosts

master重命名为Hmaster (主机名千万不能有下划线！):

# hostnamectl set-hostname Hmaster

slave重命名:

Hslave1

# hostnamectl set-hostname Hslave1# service network restart

Hslave2

# hostnamectl set-hostname Hslave2# service network restart

Hslave3

# hostnamectl set-hostname Hslave3# service network restart

然后四台机器分别修改/etc/hosts，目的是使集群机器之间互相认识，可以使用主机名直接通讯

# vi /etc/hosts192.168.0.200 Hmaster192.168.0.201 HSlave1192.168.0.202 HSlave2192.168.0.203 HSlave3# service network restart

可以直接使用主机名互相ping，看是否配置成功。

3.配置master到slave配置ssh无密码验证配置

在master机器下

cd ~cd .ssh/ssh-keygen -t rsa #连续敲四个回车完成 ，会用rsa算法生成私钥id_rsa和公钥id_rsa.pub

.ssh目录下多出两个文件 私钥文件：id_rsa公钥文件：id_rsa.pub

方式一：
手动copy方式

复制id_rsa.pub文件为authorized_keys

cp id_rsa.pub authorized_keys

将公钥文件authorized_keys分发到节点Hslave1、Hslave2、Hslave3上：

scp authorized_keys root@Hslave1:/root/.ssh/ scp authorized_keys root@Hslave2:/root/.ssh/scp authorized_keys root@Hslave3:/root/.ssh/

注意：如果当前用户目录下没有.ssh目录，可以自己创建一个该目录，该目录的权限最好设置为700，authorized_keys权限最好设置为600

方式二：ssh-copy-id命令

ssh-copy-id命令可以把本地主机的公钥复制到远程主机的authorized_keys文件上，ssh-copy-id命令也会给远程主机的用户主目录（home）和~/.ssh, 和~/.ssh/authorized_keys设置合适的权限。

来自: http://man.linuxde.net/ssh-copy-id

ssh-copy-id Hmaster  # copy自己一份ssh-copy-id Hslave1ssh-copy-id Hslave2ssh-copy-id Hslave3

验证ssh无密码登录：

# ssh Hslave1Last login: Sat Nov  4 15:39:29 2017 from hmaster#

4.配置每台集群机器的JDK环境

由于master机器上已经安装过java，安装目录为/usr/lib/jvm/jdk1.8.0_60，所以直接将安装目录发到其他的slave节点，如果没有java，就要去官网下载解压安装。

具体linux安装JDK的方法可以参考：

Linux上Java JDK环境的部署和配置

# scp -r /usr/java/default root@Hslave1:/usr/java# scp -r /usr/java/default root@Hslave2:/usr/java# scp -r /usr/java/default root@Hslave3:/usr/java

修改/etc/profile文件 配置java环境变量

# vi /etc/profile# set java jdkexport JAVA_HOME=/usr/java/defaultexport JRE_HOME=${JAVA_HOME}/jreexport CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/libexport PATH=${JAVA_HOME}/bin:$PATH

5.安装hadoop

hadoop官网下载地址

可以选择一个自己需要的版本，这里选择的是hadoop-2.5.2
先下载一个到master服务器的/usr/local路径下，

# cd /usr/local# wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz

下载完成后解压

# tar -zxvf hadoop-2.5.2.tar.gz# cd hadoop-2.5.2# mkdir data  //用于制定hadoop的hadoop.tmp.dir目录

修改hadoop core-site.xml文件配置

# vim /usr/local/hadoop-2.5.2/etc/hadoop/core-site.xml

修改etc/hadoop/core-site.xml 配置如下

<configuration>        <property>                <name>fs.default.name</name>                <value>hdfs://Hmaster:9000</value>        </property>        <property>                <name>io.file.buffer.size</name>                <value>131072</value>        </property>        <property>                 <name>hadoop.tmp.dir</name>                 <value>/usr/local/hadoop-2.5.2/data</value>        </property></configuration>

修改etc/conf/mapred-site.xml 配置如下

<configuration>  <property>    <name>mapreduce.framework.name</name>    <value>yarn</value>  </property>  <property>    <name>mapreduce.jobtracker.http.address</name>    <value>Hmaster:50030</value>  </property>  <property>    <name>mapreduce.jobhistory.address</name>    <value>Hmaster:10020</value>  </property>  <property>    <name>mapreduce.jobhistory.webapp.address</name>    <value>Hmaster:19888</value>  </property></configuration>

conf/hdfs-site.xml 配置如下,注意文件路径中不要包含一些点、逗号等特殊字符,文件路径需要写成完全路径，以file:开头

<configuration>    <property>        <name>dfs.replication</name>        <value>3</value>    </property>    <property>        <name>dfs.namenode.name.dir</name>        <value>file:/usr/local/hadoop-2.5.2/dfs/name</value>    </property>    <property>        <name>dfs.datanode.data.dir</name>        <value>file:/usr/local/hadoop-2.5.2/dfs/data</value>    </property>    <property>        <name>dfs.permissions</name>        <value>false</value>       </property>    <property>        <name>dfs.webhdfs.enabled</name>        <value>true</value>    </property>    <property>        <name>dfs.namenode.rpc-address</name>        <value>Hmaster:9000</value>    </property>    <property>        <name>dfs.block.size</name>        <value>67108864</value>    </property></configuration

修改etc/hadoop/yarn-site.xml

<configuration>     <property>        <name>yarn.nodemanager.aux-services</name>        <value>mapreduce_shuffle</value>     </property>     <property>        <name>yarn.resourcemanager.address</name>        <value>Hmaster:8032</value>     </property>     <property>        <name>yarn.resourcemanager.scheduler.address</name>        <value>Hmaster:8030</value>     </property>     <property>        <name>yarn.resourcemanager.resource-tracker.address</name>        <value>Hmaster:8031</value>     </property>     <property>        <name>yarn.resourcemanager.admin.address</name>        <value>Hmaster:8033</value>     </property>     <property>        <name>yarn.resourcemanager.webapp.address</name>        <value>Hmaster:8088</value>     </property></configuration>

修改 etc/hadoop/slaves

Hslave1Hslave2Hslave3

etc/hadoop/hadoop-env.sh和yarn-env.sh中配置Java环境变量

export JAVA_HOME=/usr/java/default

使用scp 直接把以上配置copy到另外的集群上

# scp -r hadoop-2.5.2 root@Hslave1:/usr/local# scp -r hadoop-2.5.2 root@Hslave2:/usr/local# scp -r hadoop-2.5.2 root@Hslave3:/usr/local

修改/etc/profile文件 配置hadoop环境变量

#HADOOP VARIABLES START export HADOOP_HOME=/usr/local/hadoop-2.5.2export HADOOP_COMMON_HOME=$HADOOP_HOMEexport HADOOP_HDFS_HOME=$HADOOP_HOMEexport HADOOP_MAPRED_HOME=$HADOOP_HOMEexport HADOOP_YARN_HOME=$HADOOP_HOMEexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/libexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexport HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_HOME/lib/native"#HADOOP VARIABLES END 

分布式hadoop环境搭建完毕

6.启动验证hadoop

（1）格式化文件系统

# cd /usr/local/hadoop-2.5.2# ./bin/hdfs namenode -format17/11/04 22:19:13 INFO namenode.NameNode: STARTUP_MSG: /************************************************************STARTUP_MSG: Starting NameNodeSTARTUP_MSG:   host = Hmaster/192.168.0.200STARTUP_MSG:   args = [-format]STARTUP_MSG:   version = 2.5.2STARTUP_MSG:   classpath = /usr/local/hadoop-2.5.2/etc/hadoop:/usr/local/hadoop-2.5.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop-2.5.2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/local/hadoop-2.5.2/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop-......STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r cc72e9b000545b86b75a61f4835eb86d57bfafc0; compiled by 'jenkins' on 2014-11-14T23:45ZSTARTUP_MSG:   java = 1.8.0_91************************************************************/17/11/04 22:19:13 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]17/11/04 22:19:13 INFO namenode.NameNode: createNameNode [-format]Formatting using clusterid: CID-c7182aaa-a763-4276-b7eb-f11b4bd86a6317/11/04 22:19:14 INFO namenode.FSNamesystem: fsLock is fair:true17/11/04 22:19:14 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=100017/11/04 22:19:14 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true17/11/04 22:19:14 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.00017/11/04 22:19:14 INFO blockmanagement.BlockManager: The block deletion will start around 2017 十一月 04 22:19:1417/11/04 22:19:14 INFO util.GSet: Computing capacity for map BlocksMap17/11/04 22:19:14 INFO util.GSet: VM type       = 64-bit17/11/04 22:19:14 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB17/11/04 22:19:14 INFO util.GSet: capacity      = 2^21 = 2097152 entries17/11/04 22:19:15 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false17/11/04 22:19:15 INFO blockmanagement.BlockManager: defaultReplication         = 317/11/04 22:19:15 INFO blockmanagement.BlockManager: maxReplication             = 51217/11/04 22:19:15 INFO blockmanagement.BlockManager: minReplication             = 117/11/04 22:19:15 INFO blockmanagement.BlockManager: maxReplicationStreams      = 217/11/04 22:19:15 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false17/11/04 22:19:15 INFO blockmanagement.BlockManager: replicationRecheckInterval = 300017/11/04 22:19:15 INFO blockmanagement.BlockManager: encryptDataTransfer        = false17/11/04 22:19:15 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 100017/11/04 22:19:15 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)17/11/04 22:19:15 INFO namenode.FSNamesystem: supergroup          = supergroup17/11/04 22:19:15 INFO namenode.FSNamesystem: isPermissionEnabled = false17/11/04 22:19:15 INFO namenode.FSNamesystem: HA Enabled: false17/11/04 22:19:15 INFO namenode.FSNamesystem: Append Enabled: true17/11/04 22:19:15 INFO util.GSet: Computing capacity for map INodeMap17/11/04 22:19:15 INFO util.GSet: VM type       = 64-bit17/11/04 22:19:15 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB17/11/04 22:19:15 INFO util.GSet: capacity      = 2^20 = 1048576 entries17/11/04 22:19:15 INFO namenode.NameNode: Caching file names occuring more than 10 times17/11/04 22:19:15 INFO util.GSet: Computing capacity for map cachedBlocks17/11/04 22:19:15 INFO util.GSet: VM type       = 64-bit17/11/04 22:19:15 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB17/11/04 22:19:15 INFO util.GSet: capacity      = 2^18 = 262144 entries17/11/04 22:19:15 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.999000012874603317/11/04 22:19:15 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 017/11/04 22:19:15 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 3000017/11/04 22:19:15 INFO namenode.FSNamesystem: Retry cache on namenode is enabled17/11/04 22:19:15 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis17/11/04 22:19:15 INFO util.GSet: Computing capacity for map NameNodeRetryCache17/11/04 22:19:15 INFO util.GSet: VM type       = 64-bit17/11/04 22:19:15 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB17/11/04 22:19:15 INFO util.GSet: capacity      = 2^15 = 32768 entries17/11/04 22:19:15 INFO namenode.NNConf: ACLs enabled? false17/11/04 22:19:15 INFO namenode.NNConf: XAttrs enabled? true17/11/04 22:19:15 INFO namenode.NNConf: Maximum size of an xattr: 1638417/11/04 22:19:15 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1165534849-192.168.0.200-150980515560417/11/04 22:19:15 INFO common.Storage: Storage directory /usr/local/hadoop-2.5.2/dfs/name has been successfully formatted.17/11/04 22:19:15 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 017/11/04 22:19:15 INFO util.ExitUtil: Exiting with status 017/11/04 22:19:15 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************SHUTDOWN_MSG: Shutting down NameNode at Hmaster/192.168.0.200************************************************************/

如果启动失败 则需要手动创建目录

# mkdir /usr/local/hadoop-2.5.2/dfs

成功会显示 INFO common.Storage: Storage directory /usr/local/hadoop-2.5.2/dfs/name has been successfully formatted.

...省略17/11/04 22:19:15 INFO namenode.NNConf: ACLs enabled? false17/11/04 22:19:15 INFO namenode.NNConf: XAttrs enabled? true17/11/04 22:19:15 INFO namenode.NNConf: Maximum size of an xattr: 1638417/11/04 22:19:15 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1165534849-192.168.0.200-150980515560417/11/04 22:19:15 INFO common.Storage: Storage directory /usr/local/hadoop-2.5.2/dfs/name has been successfully formatted.17/11/04 22:19:15 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 017/11/04 22:19:15 INFO util.ExitUtil: Exiting with status 017/11/04 22:19:15 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************SHUTDOWN_MSG: Shutting down NameNode at Hmaster/192.168.0.200************************************************************/

(2) 启动hadoop

# cd /usr/local/hadoop-2.5.2# sbin/start-all.sh

启动日志

This script is Deprecated. Instead use start-dfs.sh and start-yarn.shStarting namenodes on [Hmaster]Hmaster: starting namenode, logging to /usr/local/hadoop-2.5.2/logs/hadoop-root-namenode-hmaster.outHslave2: starting datanode, logging to /usr/local/hadoop-2.5.2/logs/hadoop-root-datanode-hmaster.outHslave3: starting datanode, logging to /usr/local/hadoop-2.5.2/logs/hadoop-root-datanode-hmaster.outHslave1: starting datanode, logging to /usr/local/hadoop-2.5.2/logs/hadoop-root-datanode-hmaster.outStarting secondary namenodes [0.0.0.0]0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.5.2/logs/hadoop-root-secondarynamenode-hmaster.outstarting yarn daemonsstarting resourcemanager, logging to /usr/local/hadoop-2.5.2/logs/yarn-root-resourcemanager-hmaster.outHslave3: starting nodemanager, logging to /usr/local/hadoop-2.5.2/logs/yarn-root-nodemanager-hmaster.outHslave1: starting nodemanager, logging to /usr/local/hadoop-2.5.2/logs/yarn-root-nodemanager-hmaster.outHslave2: starting nodemanager, logging to /usr/local/hadoop-2.5.2/logs/yarn-root-nodemanager-hmaster.out

在master和三台slave上执行命令jps查看java进程，成功启动情况如下，

# Hmaster 情况# jps5346 ResourceManager5619 Jps5206 SecondaryNameNode5032 NameNode# Hslave1 Hslave2 Hslave3 情况4291 NodeManager4133 DataNode4460 Jps

如果出现以下输出使其卡着不动，则要在/etc/ssh/ssh_config 文件中添加
StrictHostKeyChecking no 然后重启ssh服务/etc/init.d/ssh restart

...The authenticity of host 'localhost (127.0.0.1)' can't be established.ECDSA key fingerprint is 08:1d:db:e4:d2:e0:87:89:ed:ca:69:82:17:6a:83:57 ...

7.遇到的问题

(1)start-all.sh集群启动过程中出现一些服务进程启动失败,首先检查排除防火墙的问题，再去查看相应服务的启动日志的报错信息。

(2)Initialization failed for Block pool(Datanode Uuid unassigned)

FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool (Datanode Uuid unassigned) service to master/xxx. Exiting. java.io.IOException: Incompatible clusterIDs

问题定位：所有namenode目录、所有datanode目录、从节点临时目录
问题原因：
1) 主节点的namenode clusterID与从节点的datanode clusterID不一致
2) 多次格式化了namenode跟datanode之后的结果，格式化之后从节点生成了新的ID，造成不一致
解决办法：
在格式化之前，先把所有的服务停掉（stop-dfs.sh、stop-yarn.sh或者stop-all.sh），确保都停掉了之后，分别到所有节点的namenode目录、datanode目录、临时目录，把以上目录里面的所有内容都删除掉。然后再重新启动就可以了。一个个机器删除比较麻烦 附上一个脚本可以在各台机器上批量删除，参考该博客Linux 集群上批量执行同一命令 shell 脚本改写的

创建脚本：allcmd.sh

if [ "$#" -ne 2 ] ; then    echo "USAGE: $0 -f server_list_file cmd"               exit -1fifile_name=$1cmd_str=$2cwd=$(pwd)cd $cwdserverlist_file="$cwd/$file_name"cmdlist_file="$cwd/$cmd_str"if [ ! -e $serverlist_file ] ; then    echo 'server.list not exist';    exit 0fiif [ ! -e $cmdlist_file ] ; then    echo 'cmd.list not exist';    exit 0fiwhile read line                                        do    #echo $line                                            if [ -n "$line" ] ; then        echo "DOING--->>>>>" $line "<<<<<<<"            while read cmd_str        do        ssh $line $cmd_str < /dev/null > /dev/null        if [ $? -eq 0 ] ; then            echo "$cmd_str done!"                              else            echo "error: " $?                                  fi        done<$cmdlist_file    fidone < $serverlist_file

创建完执行chmod +x allcmd.sh
创建命令文件 cmdList

rm -r /usr/local/hadoop-2.5.2/dfs/*rm -r /usr/local/hadoop-2.5.2/data/*rm -r /usr/local/hadoop-2.5.2/logs/*

创建服务器列表文件 serverList

HmasterHslave1Hslave2Hslave3

使用方法：在脚本所在目录下建立cmdList文件和serverList
然后调用：./allcmd.sh serverList cmdList 即可