HadoopWord第一个实例运行的介绍（甲壳虫）

来源：互联网发布：深圳迈瑞工资待遇知乎编辑：程序博客网时间：2024/04/29 23:02

首先确保hadoop已经正确安装、配置以及运行。

拷贝WordCount.java到我们的文件夹，下载的hadoop里带有WordCount.java，路径为：

hadoop-0.20.203.0/src/examples/org/apache/hadoop/examples/WordCount.java

进行拷贝操作

[plain] view plaincopy
[hadoop@localhost~]$ cp hadoop-0.20.203.0/src/examples/org/apache/hadoop/examples/WordCount.java ~  

[plain] view plaincopy
[hadoop@localhost~]$ ls  

Desktop hadoop-0.20.203.0 WordCount.java

在当前目录下创建一个用来存放WordCount.class的文件夹

[plain] view plaincopy
[hadoop@localhost~]$ mkdir classes  

编译WordCount.java

[plain] view plaincopy
[hadoop@localhost~]$ javac -classpath hadoop-0.20.203.0/hadoop-core-0.20.203.0.jar -d classesWordCount.java  

上面的方法是按照hadoop自带的doc文档进行编译的，如果发现报错，出现如下异常

WordCount.java:53:error: cannot access Options

String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();

class file for org.apache.commons.cli.Optionsnot found

1 error

则按如下进行编译

[plain] view plaincopy
[hadoop@localhost~]$ javac -classpath hadoop-0.20.203.0/hadoop-core-0.20.203.0.jar:hadoop-0.20.203.0/lib/commons-cli-1.2.jar -d classes WordCount.java  

编译成功，classes下会出现一个org的文件夹

[plain] view plaincopy
[hadoop@localhost~]$ ls classes  

org

对编译好的class进行打包

[plain] view plaincopy
[hadoop@localhost~]$ jar -cvf wordcount.jar -C classes/ .  

added manifest

adding: org/(in =0) (out= 0)(stored 0%)

adding:org/apache/(in = 0) (out= 0)(stored 0%)

adding:org/apache/hadoop/(in = 0) (out= 0)(stored 0%)

adding: org/apache/hadoop/examples/(in= 0) (out= 0)(stored 0%)

adding:org/apache/hadoop/examples/WordCount$TokenizerMapper.class(in = 1790) (out=765)(deflated 57%)

adding:org/apache/hadoop/examples/WordCount$IntSumReducer.class(in = 1793) (out=746)(deflated 58%)

adding:org/apache/hadoop/examples/WordCount.class(in = 1911) (out= 996)(deflated 47%)

[plain] view plaincopy
[hadoop@localhost~]$ ls  

classes Desktop hadoop-0.20.203.0 wordcount.jar WordCount.java

到此为止，java文件的编译工作已经完成

现在为测试工作准备需要的测试文件，创建2个文件file01、file02，内容如下

[plain] view plaincopy
[hadoop@localhost~]$ ls  

classes Desktop file01 file02 hadoop-0.20.203.0 wordcount.jar WordCount.java

[plain] view plaincopy
[hadoop@localhost~]$ cat file01  

Hello World Bye World

[plain] view plaincopy
[hadoop@localhost~]$ cat file02  

Hello Hadoop Goodbye Hadoop

启动hadoop，在hadoop中创建input文件夹

[plain] view plaincopy
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop dfs -ls  
  
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop dfs -mkdir input  
  
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop dfs -ls  

Found 1 items

drwxr-xr-x - hadoop supergroup 0 2011-11-23 05:20 /user/hadoop/input

把file01、file02上传input中

[plain] view plaincopy
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -put file01 input  
  
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -put file02 input  
  
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -ls input  

Found 2 items

-rw-r--r-- 1 hadoop supergroup 22 2011-11-23 05:22/user/hadoop/input/file01

-rw-r--r-- 1 hadoop supergroup 28 2011-11-23 05:22/user/hadoop/input/file02

运行及输出过程如下

[plain] view plaincopy
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop jar wordcount.jar org.apache.hadoop.examples.WordCount input output  


11/11/23 05:25:11INFO input.FileInputFormat: Total input paths to process : 2
11/11/23 05:25:12INFO mapred.JobClient: Running job: job_201111230519_0001
11/11/23 05:25:13INFO mapred.JobClient:  map 0% reduce 0%
11/11/23 05:25:45INFO mapred.JobClient:  map 50% reduce 0%
11/11/23 05:25:48INFO mapred.JobClient:  map 100% reduce0%
11/11/23 05:26:03INFO mapred.JobClient:  map 100% reduce 100%
11/11/23 05:26:07INFO mapred.JobClient: Job complete: job_201111230519_0001
11/11/23 05:26:07INFO mapred.JobClient: Counters: 25
11/11/23 05:26:07INFO mapred.JobClient:   Job Counters
11/11/23 05:26:07INFO mapred.JobClient:     Launchedreduce tasks=1
11/11/23 05:26:07INFO mapred.JobClient:    SLOTS_MILLIS_MAPS=48562
11/11/23 05:26:07INFO mapred.JobClient:     Total timespent by all reduces waiting after reserving slots (ms)=0
11/11/23 05:26:07INFO mapred.JobClient:     Total timespent by all maps waiting after reserving slots (ms)=0
11/11/23 05:26:07INFO mapred.JobClient:     Launched maptasks=2
11/11/23 05:26:07INFO mapred.JobClient:     Data-local maptasks=2
11/11/23 05:26:07INFO mapred.JobClient:    SLOTS_MILLIS_REDUCES=16678
11/11/23 05:26:07INFO mapred.JobClient:   File OutputFormat Counters
11/11/23 05:26:07INFO mapred.JobClient:     BytesWritten=41
11/11/23 05:26:07INFO mapred.JobClient:  FileSystemCounters
11/11/23 05:26:07INFO mapred.JobClient:    FILE_BYTES_READ=79
11/11/23 05:26:07INFO mapred.JobClient:    HDFS_BYTES_READ=272
11/11/23 05:26:07INFO mapred.JobClient:    FILE_BYTES_WRITTEN=63583
11/11/23 05:26:07INFO mapred.JobClient:    HDFS_BYTES_WRITTEN=41
11/11/23 05:26:07INFO mapred.JobClient:   File Input FormatCounters
11/11/23 05:26:07INFO mapred.JobClient:     Bytes Read=50
11/11/23 05:26:07INFO mapred.JobClient:   Map-ReduceFramework
11/11/23 05:26:07INFO mapred.JobClient:     Reduce inputgroups=5
11/11/23 05:26:07INFO mapred.JobClient:     Map outputmaterialized bytes=85
11/11/23 05:26:07INFO mapred.JobClient:     Combine outputrecords=6
11/11/23 05:26:07INFO mapred.JobClient:     Map inputrecords=2
11/11/23 05:26:07INFO mapred.JobClient:     Reduce shufflebytes=85
11/11/23 05:26:07INFO mapred.JobClient:     Reduce outputrecords=5
11/11/23 05:26:07INFO mapred.JobClient:     SpilledRecords=12
11/11/23 05:26:07INFO mapred.JobClient:     Map outputbytes=82
11/11/23 05:26:07INFO mapred.JobClient:     Combine inputrecords=8
11/11/23 05:26:07INFO mapred.JobClient:     Map outputrecords=8
11/11/23 05:26:07INFO mapred.JobClient:    SPLIT_RAW_BYTES=222
11/11/23 05:26:07INFO mapred.JobClient:     Reduce inputrecords=6
进行结果的查看
[plain] view plaincopy
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -ls   
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2011-11-23 05:22 /user/hadoop/input
drwxr-xr-x   - hadoop supergroup          0 2011-11-23 05:26/user/hadoop/output
发现在hadoop中多了一个output文件夹，查看output中的文件信息
[plain] view plaincopy
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -ls output  
Found 3 items
-rw-r--r--   1 hadoop supergroup          0 2011-11-23 05:26/user/hadoop/output/_SUCCESS
drwxr-xr-x   - hadoop supergroup          0 2011-11-23 05:25/user/hadoop/output/_logs
-rw-r--r--   1 hadoop supergroup         41 2011-11-23 05:25/user/hadoop/output/part-r-00000
对WordCount的运行结果进行查看
[plain] view plaincopy
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -cat output/part-r-00000  
Bye     1
Goodbye 1
Hadoop  2
Hello   2
World   2
至此，hadoop下的WordCount实例运行结束，如果还想重新运行一遍，这需把hadoop下的output文件夹删除，因为hadoop为了保证结果的正确性，存在输出的文件夹的话，就会报异常，异常如下
[plain] view plaincopy
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop jar wordcount.jar org.apache.hadoop.examples.WordCount input output  
11/11/23 05:33:05INFO mapred.JobClient: Cleaning up the staging areahdfs://localhost:9000/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201111230519_0002
Exception inthread "main" org.apache.hadoop.mapred.FileAlreadyExistsException:Output directory output already exists
        atorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:134)
        atorg.apache.hadoop.mapred.JobClient$2.run(JobClient.java:830)
        atorg.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791)
        atjava.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        atorg.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:791)
        atorg.apache.hadoop.mapreduce.Job.submit(Job.java:465)
        atorg.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:494)
        atorg.apache.hadoop.examples.WordCount.main(WordCount.java:67)
        atsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        atjava.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
删除output文件夹的操作为
[plain] view plaincopy
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop dfs -rmr output  
Deletedhdfs://localhost:9000/user/hadoop/output