Hadoop installation Local (Standalone) Mode

Now you are ready to start your Hadoopcluster in one of the three supported modes:

• Local (Standalone) Mode
• Pseudo-Distributed Mode
• Fully-Distributed Mode

 

 

 

 

# Single Node Setup

# Standalone Operation

# Local (Standalone) Mode

 

 

安装jdk，配置环境变量

# su – root

[root@localhost ~]# cd /usr/local

# 上传jdk安装包jdk-8u121-linux-x64.tar.gz

[root@localhost local]# rz

[root@localhost local]# ll

# 解压jdk-8u121-linux-x64.tar.gz

[root@localhost local]# tar -zxvf jdk-8u121-linux-x64.tar.gz

# 可建立软连接，可跳过（路径根据自己需要来设置）

# 建立一个链接以节省目录长度

ln -s /usr/java/jdk1.8.0_60//usr/jdk

# 删除安装包

[root@localhost local]# rm -rf jdk-8u121-linux-x64.tar.gz

# 重命名

[root@localhost local]# mv jdk1.8.0_121 jdk

# 用root用户设置java环境变量

[root@localhost local]# vi /etc/profile

在 最后一行后面添加：

#java environment variables

JAVA_HOME=/usr/local/jdk

CLASSPATH=$JAVA_HOME/lib/

PATH=$PATH:$JAVA_HOME/bin

export PATH JAVA_HOME CLASSPATH

 

# 使新环境变量生效

[root@localhost local]# source /etc/profile

# 检查jdk环境

[root@localhost local]# java -version

javaversion "1.7.0_45"

OpenJDKRuntime Environment (rhel-2.4.3.3.el6-x86_64u45-b15)

OpenJDK64-Bit Server VM (build 24.45-b08, mixed mode)

 

 

创建hadoop账号和组

1. groupadd hadoop2. useradd hadoop -g hadoop;3. passwd hadoop4. mkdir /home/hadoop5. chown -R hadoop:hadoop /home/hadoop

 

# 查看hadoop用户

[root@localhost ~]# id hadoop

uid=500(hadoop)gid=500(hadoop) 组=500(hadoop)

 

 

免密码ssh设置

[hadoop@localhost ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key(/home/hadoop/.ssh/id_rsa):

Created directory '/home/hadoop/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in/home/hadoop/.ssh/id_rsa.

Your public key has been saved in/home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

78:80:cd:16:42:ce:fd:2d:7a:00:6e:de:67:be:3a:[e-mail]

The key's randomart image is:

+--[ RSA 2048]----+

|   .o .          |

|   o * .         |

|    = *          |

|   . o + .       |

|    o o S.      |

|   o . + .       |

|    . o +        |

|      E=         |

|     .o+o.       |

+-----------------+

[hadoop@localhost ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[hadoop@localhost ~]$ ll -a

总用量 40

drwx------. 6 hadoop hadoop 4096 5月   6 14:29 .

drwxr-xr-x. 3 root  root   4096 5月   6 12:18 ..

-rw-------. 1 hadoop hadoop  476 5月  6 14:26 .bash_history

-rw-r--r--. 1 hadoop hadoop   18 7月 18 2013 .bash_logout

-rw-r--r--. 1 hadoop hadoop  176 7月 18 2013 .bash_profile

-rw-r--r--. 1 hadoop hadoop  124 7月 18 2013 .bashrc

drwxr-xr-x. 2 hadoop hadoop 4096 11月 122010 .gnome2

drwxr-xr-x. 9 hadoop hadoop 4096 11月 142014 hadoop

drwxrwxr-x. 2 hadoop hadoop 4096 5月   6 12:55 .oracle_jre_usage

drwx------. 2 hadoop hadoop 4096 5月   6 14:29 .ssh

[hadoop@localhost ~]$ chmod 0600 ~/.ssh/authorized_keys

# 现在确认能否不输入口令就用ssh登录localhost: $ ssh localhost

[hadoop@localhost ~]$ ssh localhost

The authenticity of host 'localhost (::1)' can't beestablished.

RSA key fingerprint is46:73:c3:15:e8:5c:a9:14:c3:db:d6:33:05:64:6b:d6.

Are you sure you want to continue connecting (yes/no)?yes

Warning: Permanently added 'localhost' (RSA) to thelist of known hosts.

Last login: Sat May 6 14:26:50 2017 from localhost

[hadoop@localhost ~]$

 

 

安装hadoop

[root@localhost local]# su - hadoop

[hadoop@localhost ~]$ pwd

/home/hadoop

# 上传hadoop-2.6.0.tar.gz

[hadoop@localhost ~]$ rz

[hadoop@localhost ~]$ tar -zxvf hadoop-2.6.0.tar.gz

[hadoop@localhost ~]$ rm -rf hadoop-2.6.0.tar.gz

[hadoop@localhost ~]$ mv hadoop-2.6.0hadoop

[hadoop@localhost ~]$ ll

总用量 4

drwxr-xr-x. 9 hadoop hadoop 4096 11月 142014 hadoop

 

#

[hadoop@localhost bin]$ /home/hadoop/hadoop/bin/hadoopversion

Hadoop 2.6.0

Subversionhttps://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1

Compiled by jenkins on 2014-11-13T21:10Z

Compiled with protoc 2.5.0

From source with checksum18e43357c8f927c0695f1e9522859d6a

This command was run using/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.6.0.jar

[hadoop@localhost bin]$

# 修改配置文件

 

[hadoop@localhost ~]$ cd ~/hadoop/etc/hadoop/

 

 

 

Hadoop可以在单节点上以伪分布式的方式运行，Hadoop进程以分离的 Java进程来运行，节点既作为 NameNode也作为 DataNode，同时，读取的是 HDFS中的文件。

Hadoop的配置文件位于 /usr/local/hadoop/etc/hadoop/中，伪分布式需要修改2个配置文件 core-site.xml 和 hdfs-site.xml 。Hadoop的配置文件是 xml 格式，每个配置以声明 property的 name和 value的方式来实现。

修改配置文件 core-site.xml (通过 gedit编辑会比较方便: gedit./etc/hadoop/core-site.xml)，将当中的

1.         <configuration>

2.         </configuration>

XML

修改为下面配置：

1.         <configuration>

2.                 <property>

3.                      <name>hadoop.tmp.dir</name>

4.                      <value>file:/home/hadoop/hadoop/tmp</value>

5.                      <description>Abase for other temporary directories.</description>

6.                 </property>

7.                 <property>

8.                      <name>fs.defaultFS</name>

9.                      <value>hdfs://localhost:9000</value>

10.            </property>

11.    </configuration>

XML

同样的，修改配置文件 hdfs-site.xml：

1.         <configuration>

2.                 <property>

3.                      <name>dfs.replication</name>

4.                      <value>1</value>

5.                 </property>

6.                 <property>

7.                      <name>dfs.namenode.name.dir</name>

8.                      <value>file:/home/hadoop/hadoop/tmp/dfs/name</value>

9.                 </property>

10.            <property>

11.                 <name>dfs.datanode.data.dir</name>

12.                 <value>file:/home/hadoop/hadoop/tmp/dfs/data</value>

13.            </property>

14.    </configuration>

 

 

# 创建相关文件夹

mkdir/home/hadoop/hadoop/tmp

mkdir/home/hadoop/hadoop/tmp/dfs

mkdir/home/hadoop/hadoop/tmp/dfs/data

mkdir/home/hadoop/hadoop/tmp/dfs/name

cd~/hadoop/tmp/dfs

 

 

 

 

# 格式化一个新的分布式文件系统：$ bin/hadoop namenode -format

[hadoop@localhost hadoop]$ ~/hadoop/bin/hadoop namenode -format

17/05/0614:48:13 INFO common.Storage: Storage directory/home/hadoop/hadoop/tmp/dfs/name has been successfully formatted.

17/05/0614:48:13 INFO namenode.NNStorageRetentionManager: Going to retain 1 images withtxid >= 0

17/05/0614:48:13 INFO util.ExitUtil: Exiting with status 0

17/05/0614:48:13 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG:Shutting down NameNode at localhost/127.0.0.1

************************************************************/

[hadoop@localhost hadoop]$

 

 

[hadoop@localhost ~]$ ~/hadoop/sbin/start-all.sh

[hadoop@localhost ~]$ jps

26640 ResourceManager

26737 NodeManager

26502 SecondaryNameNode

26343 DataNode

27034 Jps

26221 NameNode

[hadoop@localhost ~]$

 

 

# 切换到root用户

su – root

# 开放监控端口

[root@localhost local]# /sbin/iptables -I INPUT -p tcp --dport 50070 -j ACCEPT

[root@localhost local]# /sbin/iptables -I INPUT -p tcp --dport 9000 -j ACCEPT

[root@localhost local]# /sbin/iptables -I INPUT -p tcp --dport 8088 -j ACCEPT

# iptables：将防火墙规则保存到 /etc/sysconfig/iptables

[root@localhost local]# /etc/rc.d/init.d/iptables save

# [root@localhost local]# /etc/init.d/iptables status

 

 

页面查看：

http://192.168.56.101:50070

http://192.168.56.101:8088

设置Hadoop

可以通过附加下面的命令到~/.bashrc文件中设置Hadoop环境变量。

export HADOOP_HOME=/usr/local/hadoop export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin export HADOOP_INSTALL=$HADOOP_HOME 

现在，提交所有更改到当前正在运行的系统。

$ source ~/.bashrc 

参考：

Hadoop快速入门

Hadoop安装教程_单机/伪分布式配置_Hadoop2.6.0/Ubuntu14.04_给力星

初学Hadoop之单机模式环境搭建 - 何海洋 - 博客园

Hadoop环境安装设置 - Hadoop教程™

Hadoop快速入门

超详细单机版搭建hadoop环境图文解析_VickyZhang_新浪博客

RHadoop实践系列之一:Hadoop环境搭建