Hadoop安装

（本例以安装Hadoop-2.6.4为例）

 

1、 配置主机名

Master 节点: master

Slave 节点: Slave1

Slave 节点: Slave2

*自定义：#vi /etc/sysconfig/network

NETWORKING=yes #启动网络

HOSTNAME=master  #主机名

#hostname master  

* Slave 节点，类同。

 

 

2、 配置 hosts 列表

#vi /etc/hosts

192.168.6.250 master

192.168.6.251 slave1

192.168.6.252 slave2

 

*测试：ping slave1  ctrl+c结束

Ping slave2

 

****复制/etc/hosts到slaves

scp -r /etc/hosts slave1:/etc/

scp -r /etc/hosts slave2:/etc/

3、 关闭防火墙(三台)

#service iptables status

# service iptables stop

#chkconfig iptables off

 

#vi /etc/selinux/config

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

# enforcing - SELinux security policy is enforced.

# permissive - SELinux prints warnings instead of enforcing.

# disabled - SELinux is fully disabled.

SELINUX=disabled

# SELINUXTYPE= type of policy in use. Possible values are:

# targeted - Only targeted network daemons are protected.

# strict - Full SELinux protection.

SELINUXTYPE=targeted

 

 

4、 做时间同步：分别在四台机器上

- #yum install -y ntpdate.x86_64

- #/usr/sbin/ntpdate 1.cn.pool.ntp.org

- #crontab -e

0 1 * * * /usr/sbin/ntpdate 1.cn.pool.ntp.org

 

5、 免密钥登录

该部分操作要在自定义用户目录下操作

#useradd hadoop

#passwd hadoop

      密码：111111

#su hadoop

[hadoop@master root]$cd

[hadoop@master ~]$ssh-keygen –t rsa

 

[hadoop@master ~]$cd .ssh

[hadoop@master .ssh]$ cat id_rsa.pub >> authorized_keys

[hadoop@master .ssh]$chmod 600 authorized_keys

 

****以上操作在所有机器上做*******

 

Master:

[hadoop@master .ssh]$ scp authorized_keys hadoop@slave1:~/.ssh

 

[hadoop@master .ssh]$ scp authorized_keys hadoop@slave2:~/.ssh

 

 

*测试：#ssh slave1

#exit

 

 

6、 安装 JDK （master和slave都需要）

建立文件夹：mkdir /usr/software

（上传jdk软件包：jar.tz   rpm两种，到/usr/software

[root@master~]$tar xzvf …（解压tar格式）

[root@master software]$rpm –ivh …(会安装到/usr/java)（解压rpm格式））

 

 

****在slave上，如果有权限问题，就转到root

 

 

 

配置环境变量

#cd

$vi .bashrc  （或  /etc/profile）

export JAVA_HOME=/usr/java/jdk1.7.0_67

export PATH=$PATH:$JAVA_HOME/bin

 

[hadoop@master ~]$. .bashrc（点与点之间有空格）

[hadoop@master ~]$ java -version

 

复制到其它机器：

- [hadoop@master ~]$ scp .bashrc hadoop@slave1:~/   (slave2)

- [hadoop@master ~]$ scp -r /usr/java hadoop@slave1:/usr/ (slave2)

在其他机器上：#cd

# . .bashrc

#java –version

 

 

 

Hadoop安装配置

全部在hadoop用户下操作

1、 新建software文件夹mkdir /home/hadoop/software

2、 上传hadoop包hadoop-2.6.4.tar.gz到/home/hadoop/software

3、 解压：

[hadoop@master ~]$cd; cd /home/hadoop/software

 [hadoop@master ~]$tar xzvf hadoop-2.6.4.tar.gz

 

4、 环境变量配置

[hadoop@master ~]$cd /home/hadoop/software/hadoop-2.6.4/etc/hadoop

- 修改 hadoop-env.sh

- [hadoop@master hadoop]$vi hadoop-env.sh

 

在文件靠前的部分找到：export JAVA_HOME=${JAVA_HOME}

改为：export JAVA_HOME=/usr/java/jdk1.7.0_67

 

 

- 修改 yarn-env.sh

找到：# export JAVA_HOME=/home/y/libexec/jdk1.6.0/

改为：export JAVA_HOME=/usr/java/jdk1.7.0_67

#去掉，未删除操作无用

 

- 修改 core-site.xml

 

<configuration>

        <property>

                <name>fs.defaultFS</name>

                <value>hdfs://master:9000</value>

        </property>

        <property>

                <name>hadoop.tmp.dir</name>

                <value>/home/hadoop/software/hadoopdata</value>

        </property>

</configuration>

 

- 修改 hdfs-site.xml

<configuration>

        <property>

                <name>dfs.replication</name>

                <value>2</value>

        </property>             

</configuration>

 

- 修改 yarn-site.xml

- <configuration>

<property>

    <name>yarn.nodemanager.aux-services</name>

    <value>mapreduce_shuffle</value>

  </property>

  <property>

                <name>yarn.resourcemanager.address</name>

                <value>master:18040</value>

        </property>

  <property>

                <name>yarn.resourcemanager.scheduler.address</name>

                <value>master:18030</value>

        </property>

  <property>

                <name>yarn.resourcemanager.resource-tracker.address</name>

                <value>master:18025</value>

        </property>

  <property>

                <name>yarn.resourcemanager.admin.address</name>

                <value>master:18141</value>

        </property>

  <property>

                <name>yarn.resourcemanager.webapp.address</name>

                <value>master:18088</value>

       </property>

</configuration>

 

- 修改 mapred-site.xml

1.cp mapred-site.xml.template mapred-site.xml

2.vi mapred-site.xml

 

<property>

    <name>mapreduce.framework.name</name>

    <value>yarn</value>

    </property>

 

- vi slaves

- 编辑slaves

文件删除，添加：

slave1

slave2

                                                                                                 

5、 复制到从节点

使用下面的命令将已经配置完成的 Hadoop 复制到从节点 Hadoop Slave 上：

[hadoop@master hadoop-2.6.4]$ cd /home/hadoop/software

[hadoop@master software]$ scp -r hadoop-2.6.4 slave1:/home/hadoop/software/

[hadoop@master software]$ scp -r hadoop-2.6.4 slave2:/home/hadoop/software/

 

注意：因为之前已经配置了免密钥登录，这里可以直接远程复制。

6、 启动 Hadoop 集群

 

- 配置 Hadoop 启动的系统环境变量

[hadoop@master ~]$vi .bashrc

 

将下面的代码追加到.bashrc 末尾：

export HADOOP_HOME=/home/hadoop/software/hadoop-2.6.4

export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

 

然后执行命令：

[hadoop@master ~]$ . .bashrc

启动hadoop集群

1、 创建数据目录

该节的配置需要同时在HadoopMaster 和 HadoopSlave上分别进行操作。

 在 hadoop的用户主目录下，创建数据目录，命令如下：

[hadoop@master ~]$ mkdir /home/hadoop/software/hadoopdata

[hadoop@salve1 ~]$ mkdir /home/hadoop/software/hadoopdata

[hadoop@slave2 ~]$ mkdir /home/hadoop/software/hadoopdata

 

2、 格式化hdfs文件系统

[hadoop@master ~]$ hdfs namenode -format

3、 启动hadoop

[hadoop@master ~]$start-all.sh

 

/Hadoop-2.6.4/sbin/start-all.sh

 

 

[hadoop@master ~]$ jps

29657 SecondaryNameNode

29478 NameNode

29792 ResourceManager

30052 Jps

 

[hadoop@slave1 ~]$ jps

6872 Jps

6740 DataNode

6838 NodeManager