windows 7 32位下安装 Hadoop 2.7.3 (2)

11. 安装
这里解压 C:\hadoop-2.7.3\hadoop-dist\target\hadoop-2.7.3.tar.gz 到
c:\deploy_hadoop273\

将如下路径加入到path变量：
c:\deploy_hadoop273\bin;c:\deploy_hadoop273\sbin;

12.启动单节点（伪分布式）集群 Starting a Single Node (pseudo-distributed) Cluster
12.1配置HDFS
a. 编辑 C:\deploy_hadoop273\etc\hadoop\hadoop-env.cmd
在文件结尾加入如下的行

set HADOOP_PREFIX=c:\deploy_hadoop273
set HADOOP_CONF_DIR=%HADOOP_PREFIX%\etc\hadoop
set YARN_CONF_DIR=%HADOOP_CONF_DIR%
set PATH=%PATH%;%HADOOP_PREFIX%\bin

b. 编辑 core-site.xml ，将原始文件 修改为：

fs.default.name
hdfs://0.0.0.0:19000

c. 编辑 hdfs-site.xml，将原始文件 修改为：
dfs.replication
1

12.2 配置YARN
a.创建 mapred-site.xml 文件
b.创建 yarn-site.xml 文件

12.3 初始化环境变量
run c:\deploy_hadoop273\etc\hadoop\hadoop-env.cmd to setup environment variables that will be used by the startup scripts and the daemons.

12.4 格式化文件系统
Format the filesystem with the following command:

%HADOOP_PREFIX%\bin\hdfs namenode -format

This command will print a number of filesystem parameters. Just look for the following two strings to ensure that the format command succeeded.

14/01/18 08:36:23 INFO namenode.FSImage: Saving image file \tmp\hadoop-username\dfs\name\current\fsimage.ckpt_0000000000000000000 using no compression

14/01/18 08:36:23 INFO namenode.FSImage: Image file \tmp\hadoop-username\dfs\name\current\fsimage.ckpt_0000000000000000000 of size 200 bytes saved in 0 seconds.

这里实际跑出的结果：
16/09/23 23:46:45 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/09/23 23:46:45 INFO util.GSet: VM type = 32-bit
16/09/23 23:46:45 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
16/09/23 23:46:45 INFO util.GSet: capacity = 2^16 = 65536 entries
16/09/23 23:46:45 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1006469441-192.168.1.102-1474645605961
16/09/23 23:46:46 INFO common.Storage: Storage directory \tmp\hadoop-username\dfs\name has been successfully formatted.
16/09/23 23:46:46 INFO namenode.FSImageFormatProtobuf: Saving image file \tmp\hadoop-username\dfs\name\current\fsimage.ckpt_0
000000000000000000 using no compression
16/09/23 23:46:46 INFO namenode.FSImageFormatProtobuf: Image file \tmp\hadoop-username\dfs\name\current\fsimage.ckpt_00000000
00000000000 of size 355 bytes saved in 0 seconds.
16/09/23 23:46:46 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/09/23 23:46:46 INFO util.ExitUtil: Exiting with status 0
16/09/23 23:46:46 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at A23348748/192.168.1.102
************************************************************/

12.5 启动 HDFS守护进程 Start HDFS Daemons 运行测试

Run the following command to start the NameNode and DataNode on localhost.
%HADOOP_PREFIX%\sbin\start-dfs.cmd

a. hdfs deprecated 错误
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

解决方式：
编辑 sbin\start-dfs.cmd
找到如下两行
start “Apache Hadoop Distribution” hadoop namenode
start “Apache Hadoop Distribution” hadoop datanode
替换为
start “Apache Hadoop Distribution” hdfs namenode
start “Apache Hadoop Distribution” hdfs datanode

b.测试 HDFS 守护进程是否在运行
在 c:\deploy_hadoop273 创建 test.txt 文件用于测试

c:\deploy_hadoop273>hdfs dfs -put test.txt /

c:\deploy_hadoop273>hdfs dfs -ls /
Found 1 items
-rw-r–r– 1 username supergroup 8 2016-09-24 08:21 /test.txt

12.6 YARN Daemons and run a YARN job
运行 start-yarn.cmd 命令，这会启动 resourcemanager 和 nodemanager

测试：
To verify, we can run a simple wordcount job on the text file we just copied to HDFS.
yarn jar c:\deploy_hadoop273\share\hadoop\mapreduce\hadoop-mapreduce-examples-2.7.3.jar wordcount /test.txt /out

• 原文链接: https://dshowto.com/archives/109