Hadoop-2.4.0安装和wordcount运行验证

此文转载自：http://blog.csdn.net/u012432778/article/details/23964873

Hadoop-2.4.0安装和wordcount运行验证

以下描述了64位centos6.5机器下,安装32位hadoop-2.4.0,并通过运行

系统自带的WordCount例子来验证服务正确性的步骤。

建立目录

/home/QiumingLu/hadoop-2.4.0,以后这个是hadoop的安装目录。

安装hadoop-2.4.0,解压hadoop-2.4.0.tar.gz到目录

/home/QiumingLu/hadoop-2.4.0即可

[root@localhosthadoop-2.4.0]# ls

bin etc lib LICENSE.txt NOTICE.txt sbin synthetic_control.data

dfs include libexec logs README.txt share

配置etc/hadoop/hadoop-env.sh

[root@localhosthadoop-2.4.0]#

[html] view plaincopy

1. cat etc/hadoop/hadoop-env.sh  

[html] view plaincopy

1. #The java implementation to use.  
2. exportJAVA_HOME=/home/QiumingLu/mycloud/jdk/jdk1.7.0_51  

因为hadoop是默认32位的，所以要加这个：

[html] view plaincopy

1. exportHADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native  
2. exportHADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"  

否则，可能出现一下错误：

<span style="font-family:DejaVu Sans Mono,monospace;"><span style="font-size:14px;"><span lang="en-US">Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</span></span></span>

<span style="font-family:DejaVu Sans Mono,monospace;"><span lang="en-US">Starting namenodes on [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /home/hadoop/2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.</span></span>

<span style="font-family:DejaVu Sans Mono,monospace;"><span lang="en-US">It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.</span></span>

<span style="font-family:DejaVu Sans Mono,monospace;"><span lang="en-US">localhost]</span></span>

<span style="font-family:DejaVu Sans Mono,monospace;"><span lang="en-US">sed: -e expression #1, char 6: unknown option to `s'</span></span>

<span style="font-family:DejaVu Sans Mono,monospace;"><span lang="en-US">HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Name or service not known</span></span>

<span style="font-family:DejaVu Sans Mono,monospace;"><span lang="en-US">64-Bit: ssh: Could not resolve hostname 64-Bit: Name or service not known</span></span>

<span style="font-family:DejaVu Sans Mono,monospace;"><span lang="en-US">Java: ssh: Could not resolve hostname Java: Name or service not known</span></span>

<span style="font-family:DejaVu Sans Mono,monospace;"><span lang="en-US">Server: ssh: Could not resolve hostname Server: Name or service not known</span></span>

<span style="font-family:DejaVu Sans Mono,monospace;"><span lang="en-US">VM: ssh: Could not resolve hostname VM: Name or service not known</span></span>

配置etc/hadoop/hdfs-site.xml

[root@localhosthadoop-2.4.0]# cat etc/hadoop/hdfs-site.xml

[html] view plaincopy

1. <configuration>  
2.   
3. <property>  
4. <name>dfs.replication</name>  
5. <value>1</value>  
6. </property>  
7. <property>  
8. <name>dfs.namenode.name.dir</name>  
9. <value>file:/home/QiumingLu/hadoop-2.4.0/dfs/name</value>  
10. </property>  
11. <property>  
12. <name>dfs.datanode.data.dir</name>  
13. <value>file:/home/QiumingLu/hadoop-2.4.0/dfs/data</value>  
14. </property>  
15.   
16. </configuration>  

配置etc/hadoop/core-site.xml

[html] view plaincopy

1. <configuration>  
2.  <property>    
3.       <name>fs.default.name</name>    
4.       <value>hdfs://localhost:9000</value>    
5.    </property>    
6. </configuration>  

配置etc/hadoop/yarn-site.xml

[html] view plaincopy

1. <configuration>  
2.   
3. <!--Site specific YARN configuration properties -->  
4.   
5. <property>  
6. <name>mapreduce.framework.name</name>  
7. <value>yarn</value>  
8. </property>  
9. <property>  
10. <name>yarn.nodemanager.aux-services</name>  
11. <value>mapreduce_shuffle</value>  
12. </property>  
13.   
14. </configuration>  

配置etc/hadoop/mapred-site.xml.template

[html] view plaincopy

1. [root@localhosthadoop-2.4.0]# cat etc/hadoop/mapred-site.xml.template  
2.   
3. <configuration>  
4.   
5. <property>  
6. <name>mapreduce.framework.name</name>  
7. <value>yarn</value>  
8. </property>  
9.   
10. </configuration>  

格式化文件系统

[root@localhosthadoop-2.4.0]#

[html] view plaincopy

1. ./bin/hadoop namenode -format  

启动服务,这里使用root用户,需要输入密码的时候,输入root用户密码

如果使用非root,并假设分布式服务,需要先解决ssh登录问题,此处不详

细描述。

[root@localhosthadoop-2.4.0]#

[html] view plaincopy

1. sbin/start-all.sh  

查看启动状态：

[root@localhosthadoop-2.4.0]#

[html] view plaincopy

1. ./bin/hadoop dfsadmin -report  

DEPRECATED:Use of this script to execute hdfs command is deprecated.

Insteaduse the hdfs command for it.

14/04/1805:15:30 WARN util.NativeCodeLoader: Unable to load native-hadooplibrary for your platform... using builtin-java classes whereapplicable

ConfiguredCapacity: 135938813952 (126.60 GB)

PresentCapacity: 126122217472 (117.46 GB)

DFSRemaining: 126121320448 (117.46 GB)

DFSUsed: 897024 (876 KB)

DFSUsed%: 0.00%

Underreplicated blocks: 0

Blockswith corrupt replicas: 0

Missingblocks: 0

-------------------------------------------------

Datanodesavailable: 1 (1 total, 0 dead)

Livedatanodes:

Name:127.0.0.1:50010 (localhost)

Hostname:localhost

DecommissionStatus : Normal

ConfiguredCapacity: 135938813952 (126.60 GB)

DFSUsed: 897024 (876 KB)

NonDFS Used: 9816596480 (9.14 GB)

DFSRemaining: 126121320448 (117.46 GB)

DFSUsed%: 0.00%

DFSRemaining%: 92.78%

ConfiguredCache Capacity: 0 (0 B)

CacheUsed: 0 (0 B)

CacheRemaining: 0 (0 B)

CacheUsed%: 100.00%

CacheRemaining%: 0.00%

Lastcontact: Fri Apr 18 05:15:29 CST 2014

[root@localhosthadoop-2.4.0]# jps

3614DataNode

3922ResourceManager

3514NameNode

9418Jps

4026NodeManager 

http://localhost:50070

http://localhost:8088

构造数据文件(file1.txt,file2.txt)

[root@localhosthadoop-2.4.0]# cat example/file1.txt 

[html] view plaincopy

1. hello world  
2. hello markhuang  
3. hello hadoop  

[root@localhosthadoop-2.4.0]# cat example/file2.txt

[html] view plaincopy

1. hadoop ok  
2. hadoop fail  
3. hadoop 2.4  

[root@localhosthadoop-2.4.0]#

[html] view plaincopy

1. ./bin/hadoop fs -mkdir /data  

把数据文件加入到hadoop系统。

[root@localhosthadoop-2.4.0]#

[html] view plaincopy

1. ./bin/hadoop fs -put -f example/file1.txtexample/file2.txt /data  

运行WordCount(java)版本。

[root@localhosthadoop-2.4.0]#

[html] view plaincopy

1. ./bin/hadoop jar./share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.4.0-sources.jarorg.apache.hadoop.examples.WordCount /data /output  

查看结果。

[root@localhosthadoop-2.4.0]#

[html] view plaincopy

1. ./bin/hadoop fs -cat /output/part-r-00000  

2.4 1

fail 1

hadoop 4

hello 3

markhuang 1

ok 1

world 1