Hadoop 实战之运行CitationHistogram

环境：Vmware 8.0 和Ubuntu11.04

Hadoop 实战之运行CitationHistogram---计算不同引用次数专利的数目

第一步：首先创建一个工程命名为HadoopTest.目录结构如下图:

第二步: 在/home/tanglg1987目录下新建一个start.sh脚本文件，每次启动虚拟机都要删除/tmp目录下的全部文件，重新格式化namenode，代码如下：

sudo rm -rf /tmp/*rm -rf /home/tanglg1987/hadoop-0.20.2/logshadoop namenode -formathadoop datanode -formatstart-all.shhadoop fs -mkdir input hadoop dfsadmin -safemode leave

第三步：给start.sh增加执行权限并启动hadoop伪分布式集群，代码如下:

chmod 777 /home/tanglg1987/ start.sh./start.sh 

执行过程如下：

第四步：把我上一篇博客：linux下运行MyJob3的运行结果result当成这里的输入input

博客地址：http://blog.csdn.net/xiaotom5/article/details/8088304

五步：新建一个CitationHistogram.java，代码如下：

package com.baison.action;import java.io.IOException;import java.util.Iterator;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.conf.Configured;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapred.FileInputFormat;import org.apache.hadoop.mapred.FileOutputFormat;import org.apache.hadoop.mapred.JobClient;import org.apache.hadoop.mapred.JobConf;import org.apache.hadoop.mapred.KeyValueTextInputFormat;import org.apache.hadoop.mapred.MapReduceBase;import org.apache.hadoop.mapred.Mapper;import org.apache.hadoop.mapred.OutputCollector;import org.apache.hadoop.mapred.Reducer;import org.apache.hadoop.mapred.Reporter;import org.apache.hadoop.mapred.TextOutputFormat;import org.apache.hadoop.util.Tool;import org.apache.hadoop.util.ToolRunner;public class CitationHistogram extends Configured implements Tool {       public static class MapClass extends MapReduceBase        implements Mapper<Text, Text, IntWritable, IntWritable> {               private final static IntWritable uno = new IntWritable(1);        private IntWritable citationCount = new IntWritable();             public void map(Text key, Text value,                        OutputCollector<IntWritable, IntWritable> output,                        Reporter reporter) throws IOException {                                  citationCount.set(Integer.parseInt(value.toString()));            output.collect(citationCount, uno);        }    }    public static class Reduce extends MapReduceBase        implements Reducer<IntWritable,IntWritable,IntWritable,IntWritable>    {              public void reduce(IntWritable key, Iterator<IntWritable> values,                           OutputCollector<IntWritable, IntWritable>output,                           Reporter reporter) throws IOException {                                   int count = 0;            while (values.hasNext()) {                count += values.next().get();            }            output.collect(key, new IntWritable(count));        }    }     public int run(String[] args) throws Exception {        Configuration conf = getConf();            JobConf job = new JobConf(conf, CitationHistogram.class);          Path in = new Path(args[0]);        Path out = new Path(args[1]);        FileInputFormat.setInputPaths(job, in);        FileOutputFormat.setOutputPath(job, out);           job.setJobName("CitationHistogram");        job.setMapperClass(MapClass.class);        job.setReducerClass(Reduce.class);         job.setInputFormat(KeyValueTextInputFormat.class);        job.setOutputFormat(TextOutputFormat.class);        job.setOutputKeyClass(IntWritable.class);        job.setOutputValueClass(IntWritable.class);         JobClient.runJob(job);         return 0;    }    public static void main(String[] args) throws Exception {     String [] arg={"hdfs://localhost:9100/user/tanglg1987/output","hdfs://localhost:9100/user/tanglg1987/output/CitationHistogram"};             int res = ToolRunner.run(new Configuration(),                                  new CitationHistogram(),                                  arg);            System.exit(res);    }}

第六步：Run On Hadoop，运行过程如下：

12/10/18 23:26:49 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/10/18 23:26:49 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/10/18 23:26:49 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 23:26:50 INFO mapred.JobClient: Running job: job_local_0001
12/10/18 23:26:50 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 23:26:50 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 23:26:50 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 23:26:50 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 23:26:50 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 23:26:50 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:50 INFO mapred.MapTask: bufstart = 0; bufend = 2097152; bufvoid = 99614720
12/10/18 23:26:50 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 23:26:51 INFO mapred.JobClient: map 0% reduce 0%
12/10/18 23:26:51 INFO mapred.MapTask: Finished spill 0
12/10/18 23:26:51 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:51 INFO mapred.MapTask: bufstart = 2097152; bufend = 4194296; bufvoid = 99614720
12/10/18 23:26:51 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 23:26:52 INFO mapred.MapTask: Finished spill 1
12/10/18 23:26:52 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:52 INFO mapred.MapTask: bufstart = 4194296; bufend = 6291440; bufvoid = 99614720
12/10/18 23:26:52 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 23:26:52 INFO mapred.MapTask: Finished spill 2
12/10/18 23:26:53 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:53 INFO mapred.MapTask: bufstart = 6291440; bufend = 8388584; bufvoid = 99614720
12/10/18 23:26:53 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 23:26:53 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:26:53 INFO mapred.MapTask: Finished spill 3
12/10/18 23:26:53 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:53 INFO mapred.MapTask: bufstart = 8388584; bufend = 10485736; bufvoid = 99614720
12/10/18 23:26:53 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 23:26:54 INFO mapred.JobClient: map 31% reduce 0%
12/10/18 23:26:54 INFO mapred.MapTask: Finished spill 4
12/10/18 23:26:54 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:54 INFO mapred.MapTask: bufstart = 10485736; bufend = 12582880; bufvoid = 99614720
12/10/18 23:26:54 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 23:26:55 INFO mapred.MapTask: Finished spill 5
12/10/18 23:26:55 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:55 INFO mapred.MapTask: bufstart = 12582880; bufend = 14680024; bufvoid = 99614720
12/10/18 23:26:55 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 23:26:56 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:26:56 INFO mapred.MapTask: Finished spill 6
12/10/18 23:26:56 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:56 INFO mapred.MapTask: bufstart = 14680024; bufend = 16777168; bufvoid = 99614720
12/10/18 23:26:56 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 23:26:57 INFO mapred.JobClient: map 58% reduce 0%
12/10/18 23:26:57 INFO mapred.MapTask: Finished spill 7
12/10/18 23:26:57 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:57 INFO mapred.MapTask: bufstart = 16777168; bufend = 18874312; bufvoid = 99614720
12/10/18 23:26:57 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 23:26:58 INFO mapred.MapTask: Finished spill 8
12/10/18 23:26:58 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:58 INFO mapred.MapTask: bufstart = 18874312; bufend = 20971464; bufvoid = 99614720
12/10/18 23:26:58 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 23:26:58 INFO mapred.MapTask: Finished spill 9
12/10/18 23:26:59 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:26:59 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:59 INFO mapred.MapTask: bufstart = 20971464; bufend = 23068608; bufvoid = 99614720
12/10/18 23:26:59 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 23:26:59 INFO mapred.MapTask: Finished spill 10
12/10/18 23:26:59 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 23:26:59 INFO mapred.MapTask: bufstart = 23068608; bufend = 25165752; bufvoid = 99614720
12/10/18 23:26:59 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 23:27:00 INFO mapred.JobClient: map 88% reduce 0%
12/10/18 23:27:00 INFO mapred.MapTask: Finished spill 11
12/10/18 23:27:00 INFO mapred.MapTask: Starting flush of map output
12/10/18 23:27:00 INFO mapred.MapTask: Finished spill 12
12/10/18 23:27:00 INFO mapred.Merger: Merging 13 sorted segments
12/10/18 23:27:00 INFO mapred.Merger: Merging 4 intermediate segments out of a total of 13
12/10/18 23:27:01 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 32589860 bytes
12/10/18 23:27:02 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:27:03 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 23:27:05 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:27:05 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/18 23:27:05 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/output/part-00000:0+32858613
12/10/18 23:27:05 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
12/10/18 23:27:05 INFO mapred.LocalJobRunner: 
12/10/18 23:27:05 INFO mapred.Merger: Merging 1 sorted segments
12/10/18 23:27:05 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 32589842 bytes
12/10/18 23:27:05 INFO mapred.LocalJobRunner: 
12/10/18 23:27:06 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/10/18 23:27:06 INFO mapred.LocalJobRunner: 
12/10/18 23:27:06 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/10/18 23:27:06 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://localhost:9100/user/tanglg1987/output/CitationHistogram
12/10/18 23:27:06 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 23:27:06 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
12/10/18 23:27:07 INFO mapred.JobClient: map 100% reduce 100%
12/10/18 23:27:07 INFO mapred.JobClient: Job complete: job_local_0001
12/10/18 23:27:07 INFO mapred.JobClient: Counters: 15
12/10/18 23:27:07 INFO mapred.JobClient: FileSystemCounters
12/10/18 23:27:07 INFO mapred.JobClient: FILE_BYTES_READ=118774812
12/10/18 23:27:07 INFO mapred.JobClient: HDFS_BYTES_READ=65717226
12/10/18 23:27:07 INFO mapred.JobClient: FILE_BYTES_WRITTEN=151399190
12/10/18 23:27:07 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1721
12/10/18 23:27:07 INFO mapred.JobClient: Map-Reduce Framework
12/10/18 23:27:07 INFO mapred.JobClient: Reduce input groups=258
12/10/18 23:27:07 INFO mapred.JobClient: Combine output records=0
12/10/18 23:27:07 INFO mapred.JobClient: Map input records=3258984
12/10/18 23:27:07 INFO mapred.JobClient: Reduce shuffle bytes=0
12/10/18 23:27:07 INFO mapred.JobClient: Reduce output records=258
12/10/18 23:27:07 INFO mapred.JobClient: Spilled Records=10825525
12/10/18 23:27:07 INFO mapred.JobClient: Map output bytes=26071872
12/10/18 23:27:07 INFO mapred.JobClient: Map input bytes=32858613
12/10/18 23:27:07 INFO mapred.JobClient: Combine input records=0
12/10/18 23:27:07 INFO mapred.JobClient: Map output records=3258984
12/10/18 23:27:07 INFO mapred.JobClient: Reduce input records=3258984

第七步：查看结果集，运行结果如下：