Hadoop/spark安装实战（系列篇3） Hadoop 伪分布模式安装

Hadoop/spark安装实战（系列篇3） Hadoop 伪分布模式安装

 

 Hadoop 的安装与配置 伪分布模式安装

1/解压缩hadoop
  tar  -zxvf    hadoop-1.2.1-bin.tar.gz
2/  将文件移到/usr/local
mv  hadoop-1.2.1  /usr/local

3/设置环境变量HADOOP_HOME，修改文件“/etc/profile”
[root@localhost local]# vi /etc/profile
export HADOOP_HOME=/usr/local/hadoop-1.2.1
export PATH=.:$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
 生效
[root@localhost local]# source /etc/profile
检验
[root@localhost local]# echo $PATH
.:.:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/l

ocal/jdk1.7.0_79/bin:/root/bin:/usr/local/jdk1.7.0_79/bin:/usr/local/ha

doop-1.2.1/bin
[root@localhost local]#

4 修改hadoop 环境变量脚本文件hadoop-env.sh
  增加
 
   export JAVA_HOME=/usr/local/jdk1.7.0_79

5 修改hadoop 核心配置文件core-site.xml

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/tmp</value>
<description> hadoop.tmp.dir</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://spark0:9000</value>
<description>HDFS fs.default.name</description>
</property>
</configuration>

6 hdfs 配置文件hdfs-site.xml，存储副本数1份

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>1</description>
</property>
</configuration>

7 MapReduce 配置文件mapred-site.xml

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>spark0:9001</value>
<description>JobTracker </description>
</property>
</configuration>

8格式化文件系统
在第一次使用之前需要进行格式化，只在第一次启动的时候格式化，不要每次启

动都格式化
[root@localhost bin]# hadoop namenode -format

15/09/12 06:31:11 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost.localdomain/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.2.1
STARTUP_MSG:   build =

https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r

1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG:   java = 1.7.0_79
************************************************************/
15/09/12 06:31:13 INFO util.GSet: Computing capacity for map BlocksMap
15/09/12 06:31:13 INFO util.GSet: VM type       = 32-bit
15/09/12 06:31:13 INFO util.GSet: 2.0% max memory = 1013645312
15/09/12 06:31:13 INFO util.GSet: capacity      = 2^22 = 4194304

entries

9 启动hadoop 程序

[root@localhost bin]# start-all.sh

[root@localhost bin]# start-all.sh
Warning: $HADOOP_HOME is deprecated.

starting namenode, logging to /usr/local/hadoop-

1.2.1/libexec/../logs/hadoop-root-namenode-localhost.localdomain.out
localhost: starting datanode, logging to /usr/local/hadoop-

1.2.1/libexec/../logs/hadoop-root-datanode-localhost.localdomain.out
localhost: starting secondarynamenode, logging to /usr/local/hadoop-

1.2.1/libexec/../logs/hadoop-root-secondarynamenode-

localhost.localdomain.out
starting jobtracker, logging to /usr/local/hadoop-

1.2.1/libexec/../logs/hadoop-root-jobtracker-localhost.localdomain.out
localhost: starting tasktracker, logging to /usr/local/hadoop-

1.2.1/libexec/../logs/hadoop-root-tasktracker-localhost.localdomain.out

检验

[root@localhost bin]# jps

[root@localhost bin]# jps
27375 Jps
26949 DataNode
27046 SecondaryNameNode
26851 NameNode
27240 TaskTracker
27124 JobTracker
[root@localhost bin]#

 

10  检验 在本地物理机浏览器输入
http://192.168.2.100:50070/dfshealth.jsp

NameNode 'spark0:9000'

   
  Started:   Sat Sep 12 06:34:38 PDT 2015
   Version:   1.2.1, r1503152
   Compiled:   Mon Jul 22 15:23:09 PDT 2013 by mattf
   Upgrades:   There are no upgrades in progress.
 
 Browse the filesystem
 Namenode Logs
-----------------------------------------------------------------------

---------
 
Cluster Summary
 7 files and directories, 1 blocks = 8 total. Heap Size is 31.32 MB /

966.69 MB (3%)

  Configured Capacity  :  5.65 GB   DFS Used  :  28.01 KB   Non DFS

Used  :  3.17 GB   DFS Remaining  :  2.47 GB   DFS Used%  :  0 %   DFS

Remaining%  :  43.82 %   Live Nodes   :  1   Dead Nodes   :  0  

Decommissioning Nodes   :  0   Number of Under-Replicated Blocks  :  0

-----------------------------------------------------------------------

---------
 
NameNode Storage:

Storage Directory Type State /home/hadoop/tmp/dfs/name IMAGE_AND_EDITS

Active

-----------------------------------------------------------------------

---------
 This is Apache Hadoop release 1.2.1

至此 hadoop安装完成