【配置】Hadoop单节点集群搭建

准备工作(电脑是Windows64位的操作系统)

• VMware Workstation(CentOS-6.5-x86_64-bin-DVD1.iso);
• Xshell;
• jdk-7u79-linux-x64.tar.gz;
• hadoop-2.6.0.tar.gz;

关闭防火墙

重启之后查看防火墙状态

绑定hostname与IP

[root@node4 ~]# vi /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.70.104 node4 /* IP hostname */

创建hadoop用户和用户组

[root@node4 ~]# useradd -m hadoop /*创建hadoop用户及其家目录*/[root@node4 ~]# cd /home[root@node4 home]# lshadoop[root@node4 home]# su root[root@node4 home]# passwd hadoop /*为hadoop用户设置密码*/Changing password for user hadoop.New password:BAD PASSWORD: it is based on a dictionary wordRetype new password:passwd: all authentication tokens updated successfully.[root@node4 home]#

配置ssh免密码登录

切换到hadoop用户的根目录，
创建.ssh文件夹，ss-keygen生成、管理和转换认证秘钥，-t指定秘钥类型。

[hadoop@node4 ~]$ cd .ssh[hadoop@node4 .ssh]$ lsid_rsa  id_rsa.pub[hadoop@node4 .ssh]$ cat id_rsa.pub >> authorized_keys /*复制*/[hadoop@node4 .ssh]$ lsauthorized_keys  id_rsa  id_rsa.pub[hadoop@node4 .ssh]$ cat authorized_keys /*查看*/ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA7t8/VWOdBRq7y2r5iiorTB4WWYit+MCI3FeMtyhD9txbJf0n1rFcoOkszEO2JPFBGyPdXmd7hjOGGhs9aHvj6XT92WsdmrHBoJeZ+XPtxr2OPqLjK1AdSbLwY+kKh22Ko+UVOJbO3AChilQI4ziWrpJ5+r3IvwKFNGFnuNsTcg4Q3+so6TsPX5Mkh2NAoxBG6F982BAet7cKBgA9YcrhXEzyr8daFrEo8Hpf7103PJjqJvKXvFVpwgbpa6m1suRzEu3KU6ZihiV65nU6WqV/78ZKLP1nqqLtdNNNaqbhFlToCHKdhgM1uLC6QMy5eVEVhm/Hw/EPxg+OT5jBpHM7WQ== hadoop@node4[hadoop@node4 .ssh]$ cd ..[hadoop@node4 ~]$ chmod 700 .ssh /*赋予user读写操作.ssh文件的目录权限*/[hadoop@node4 ~]$ chmod 600 .ssh/*

ps:

JDK安装

[hadoop@node4 ~]$ pwd/home/hadoop[hadoop@node4 ~]$ mkdir app /*创建app目录*/[hadoop@node4 ~]$ cd app[hadoop@node4 app]$ rz /*上传*/[hadoop@node4 app]$ lsjdk-7u79-linux-x64.tar.gz[hadoop@node4 app]$tar -zxvf jdk-7u79-linux-x64.tar.gz /*解压*/[hadoop@node4 app]$ ln  -s jdk1.7.0_79 jdk/*创建软连接*/[hadoop@node4 app]$ lsjdk  jdk1.7.0_79[hadoop@node4 app]$

配置环境变量

[hadoop@node4 app]$  vi ~/.bashrc# .bashrc# Source global definitionsif [ -f /etc/bashrc ]; then        . /etc/bashrcfi# User specific aliases and functionsJAVA_HOME=/home/hadoop/app/jdkCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarPATH=$JAVA_HOME/bin:$PATHexport JAVA_HOME CLASSPATH PATHhadoop@node4 app]$ source ~/.bashrc/*使配置生效*/[hadoop@node4 app]$ java -version /*出现以下版本信息说明安装成功*/java version "1.7.0_79"Java(TM) SE Runtime Environment (build 1.7.0_79-b15)Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)[hadoop@node4 app]$

安装hadoop
将hadoop安装包上传到/home/hadoop/app目录下，解压（同上）。

[hadoop@node4 app]$ lshadoop-2.6.0  jdk  jdk1.7.0_79[hadoop@node4 app]$ cd hadoop-2.6.0[hadoop@node4 hadoop-2.6.0]$ lsbin  etc  include  lib  libexec  LICENSE.txt  NOTICE.txt  README.txt  sbin  share[hadoop@node4 hadoop-2.6.0]$ bin/hadoop versionHadoop 2.6.0Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1Compiled by jenkins on 2014-11-13T21:10ZCompiled with protoc 2.5.0From source with checksum 18e43357c8f927c0695f1e9522859d6aThis command was run using /home/hadoop/app/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar[hadoop@node4 hadoop-2.6.0]$ vi hello.txt/*自行编辑一个txt文件*/[hadoop@node4 hadoop-2.6.0]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount hello.txt output/*运行hadoop自带的Wordcount实例*/

hello.txt

运行结果

修改并替换core-site.xml、hdfs-site.xml、hadoop-env.sh、mapred-site.xml、yarn-site.xml、slaves这些文件。

配置环境变量

[hadoop@node4 ~]$ vi ~/.bashrc

[hadoop@node4 ~]$ source ~/.bashrc

创建Hadoop相关数据目录

[hadoop@node4 ~]$ mkdir -p /home/hadoop/data/tmp[hadoop@node4 ~]$ mkdir -p /home/hadoop/data/dfs/name[hadoop@node4 ~]$ mkdir -p /home/hadoop/data/dfs/data

格式化namenode

[hadoop@node4 hadoop]$ lsbin  etc  hello.txt  include  lib  libexec  LICENSE.txt  NOTICE.txt  output  README.txt  sbin  share[hadoop@node4 hadoop]$ bin/hdfs  namenode -format/*格式化namenode*/[hadoop@node4 hadoop]$ sbin/start-all.sh/*启动集群*/[hadoop@node4 hadoop]$ jps1685 NameNode2204 Jps2174 NodeManager1769 DataNode2088 ResourceManager1947 SecondaryNameNode

启动集群的部分信息

使用webUI查看namenode和yarn信息
在C盘的Windows32目录下找到hosts文件，编辑IP 对应的hostname

测试运行hadoop伪分布式集群

[hadoop@node4 hadoop]$ bin/hdfs dfs -ls /[hadoop@node4 hadoop]$ bin/hdfs dfs -mkdir /abc[hadoop@node4 hadoop]$ bin/hdfs dfs -ls /Found 1 itemsdrwxr-xr-x   - hadoop supergroup         0 2017-11-22 15:41 /abc[hadoop@node4 hadoop]$ lsbin  etc  hello.txt  include  lib  libexec  LICENSE.txt  logs  NOTICE.txt  output  README.txt  sbin  share[hadoop@node4 hadoop]$ bin/hdfs dfs -put hello.txt /abc/[hadoop@node4 hadoop]$ bin/hdfs dfs -ls /abc/Found 1 items-rw-r--r--   1 hadoop supergroup         23 2017-11-22 15:43 /abc/hello.txt[hadoop@node4 hadoop]$ bin/hdfs dfs -cat /abc/hello.txthello hello iiii[hadoop@node4 hadoop]$

以上说明单机版的hadoop集群安装成功