Hadoop 实战之运行AveragingWithCombiner（二）

环境：Vmware 8.0 和ubuntu11.04

Hadoop 实战之运行AveragingWithCombiner---使用计数器统计缺失值个数的MapClass

第一步：首先创建一个工程命名为HadoopTest.目录结构如下图:

第二步: 在/home/tanglg1987目录下新建一个start.sh脚本文件，每次启动虚拟机都要删除/tmp目录下的全部文件，重新格式化namenode，代码如下：

sudo rm -rf /tmp/*rm -rf /home/tanglg1987/hadoop-0.20.2/logshadoop namenode -formathadoop datanode -formatstart-all.shhadoop fs -mkdir input hadoop dfsadmin -safemode leave

第三步：给start.sh增加执行权限并启动hadoop伪分布式集群，代码如下:

chmod 777 /home/tanglg1987/start.sh./start.sh 

执行过程如下：

第四步：上传本地文件到hdfs

在专利局http://data.nber.org/patents/网站下载专利数据

http://data.nber.org/patents/apat63_99.zip

hadoop fs -put /home/tanglg1987/apat63_99.txt input

五步：新建一个AveragingWithCombiner2.java，代码如下：

package com.baison.action;import java.io.IOException;import java.util.Iterator;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.conf.Configured;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.DoubleWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapred.*;import org.apache.hadoop.util.Tool;import org.apache.hadoop.util.ToolRunner;public class AveragingWithCombiner2 extends Configured implements Tool {       public static class MapClass extends MapReduceBase        implements Mapper<LongWritable, Text, Text, Text> {              static enum ClaimsCounters { MISSING, QUOTED };              public void map(LongWritable key, Text value,                        OutputCollector<Text, Text> output,                        Reporter reporter) throws IOException {                                 String fields[] = value.toString().split(",", -20);            String country = fields[4];            String numClaims = fields[8];            if (numClaims.length() == 0) {                reporter.incrCounter(ClaimsCounters.MISSING, 1);            } else if (numClaims.startsWith("\"")) {                reporter.incrCounter(ClaimsCounters.QUOTED, 1);            } else {                output.collect(new Text(country), new Text(numClaims + ",1"));            }                   }    }       public static class Combine extends MapReduceBase        implements Reducer<Text, Text, Text, Text> {             public void reduce(Text key, Iterator<Text> values,                           OutputCollector<Text, Text> output,                           Reporter reporter) throws IOException {                                     double sum = 0;            int count = 0;            while (values.hasNext()) {                String fields[] = values.next().toString().split(",");                sum += Double.parseDouble(fields[0]);                count += Integer.parseInt(fields[1]);            }            output.collect(key, new Text(sum + "," + count));        }    }       public static class Reduce extends MapReduceBase        implements Reducer<Text, Text, Text, DoubleWritable> {             public void reduce(Text key, Iterator<Text> values,                           OutputCollector<Text, DoubleWritable> output,                           Reporter reporter) throws IOException {                                    double sum = 0;            int count = 0;            while (values.hasNext()) {                String fields[] = values.next().toString().split(",");                sum += Double.parseDouble(fields[0]);                count += Integer.parseInt(fields[1]);            }            output.collect(key, new DoubleWritable(sum/count));        }    }       public int run(String[] args) throws Exception {        // Configuration processed by ToolRunner        Configuration conf = getConf();            // Create a JobConf using the processed conf        JobConf job = new JobConf(conf, AveragingWithCombiner.class);            // Process custom command-line options        Path in = new Path(args[0]);        Path out = new Path(args[1]);        FileInputFormat.setInputPaths(job, in);        FileOutputFormat.setOutputPath(job, out);           // Specify various job-specific parameters             job.setJobName("AveragingWithCombiner");        job.setMapperClass(MapClass.class);        job.setCombinerClass(Combine.class);        job.setReducerClass(Reduce.class);          job.setInputFormat(TextInputFormat.class);        job.setOutputFormat(TextOutputFormat.class);        job.setOutputKeyClass(Text.class);        job.setOutputValueClass(Text.class);             // Submit the job, then poll for progress until the job is complete        JobClient.runJob(job);         return 0;    }        public static void main(String[] args) throws Exception {        // Let ToolRunner handle generic command-line options      String [] arg={"hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt" ,"hdfs://localhost:9100/user/tanglg1987/output"};        int res = ToolRunner.run(new Configuration(), new AveragingWithCombiner(), arg);              System.exit(res);    }}

第六步：Run On Hadoop，运行过程如下：

12/10/19 20:36:29 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/10/19 20:36:29 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/10/19 20:36:29 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/19 20:36:30 INFO mapred.JobClient: Running job: job_local_0001
12/10/19 20:36:30 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/19 20:36:30 INFO mapred.MapTask: numReduceTasks: 1
12/10/19 20:36:30 INFO mapred.MapTask: io.sort.mb = 100
12/10/19 20:36:30 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/19 20:36:30 INFO mapred.MapTask: record buffer = 262144/327680
12/10/19 20:36:31 INFO mapred.JobClient: map 0% reduce 0%
12/10/19 20:36:33 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:0+67108864
12/10/19 20:36:34 INFO mapred.JobClient: map 28% reduce 0%
12/10/19 20:36:36 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:0+67108864
12/10/19 20:36:37 INFO mapred.JobClient: map 80% reduce 0%
12/10/19 20:36:37 INFO mapred.MapTask: Starting flush of map output
12/10/19 20:36:37 INFO mapred.MapTask: Finished spill 0
12/10/19 20:36:37 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/19 20:36:37 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:0+67108864
12/10/19 20:36:37 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
12/10/19 20:36:38 INFO mapred.MapTask: numReduceTasks: 1
12/10/19 20:36:38 INFO mapred.MapTask: io.sort.mb = 100
12/10/19 20:36:38 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/19 20:36:38 INFO mapred.MapTask: record buffer = 262144/327680
12/10/19 20:36:38 INFO mapred.JobClient: map 100% reduce 0%
12/10/19 20:36:40 INFO mapred.MapTask: Spilling map output: record full = true
12/10/19 20:36:40 INFO mapred.MapTask: bufstart = 0; bufend = 2467890; bufvoid = 99614720
12/10/19 20:36:40 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/19 20:36:41 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:67108864+67108864
12/10/19 20:36:41 INFO mapred.JobClient: map 67% reduce 0%
12/10/19 20:36:41 INFO mapred.MapTask: Finished spill 0
12/10/19 20:36:43 INFO mapred.MapTask: Spilling map output: record full = true
12/10/19 20:36:43 INFO mapred.MapTask: bufstart = 2467890; bufend = 4942611; bufvoid = 99614720
12/10/19 20:36:43 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/19 20:36:43 INFO mapred.MapTask: Finished spill 1
12/10/19 20:36:44 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:67108864+67108864
12/10/19 20:36:44 INFO mapred.JobClient: map 89% reduce 0%
12/10/19 20:36:45 INFO mapred.MapTask: Starting flush of map output
12/10/19 20:36:45 INFO mapred.MapTask: Finished spill 2
12/10/19 20:36:45 INFO mapred.Merger: Merging 3 sorted segments
12/10/19 20:36:45 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 4411 bytes
12/10/19 20:36:45 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
12/10/19 20:36:45 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:67108864+67108864
12/10/19 20:36:45 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000001_0' done.
12/10/19 20:36:45 INFO mapred.MapTask: numReduceTasks: 1
12/10/19 20:36:45 INFO mapred.MapTask: io.sort.mb = 100
12/10/19 20:36:45 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/19 20:36:45 INFO mapred.MapTask: record buffer = 262144/327680
12/10/19 20:36:46 INFO mapred.JobClient: map 100% reduce 0%
12/10/19 20:36:47 INFO mapred.MapTask: Spilling map output: record full = true
12/10/19 20:36:47 INFO mapred.MapTask: bufstart = 0; bufend = 2496693; bufvoid = 99614720
12/10/19 20:36:47 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/19 20:36:48 INFO mapred.MapTask: Finished spill 0
12/10/19 20:36:48 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:134217728+67108864
12/10/19 20:36:49 INFO mapred.JobClient: map 81% reduce 0%
12/10/19 20:36:49 INFO mapred.MapTask: Spilling map output: record full = true
12/10/19 20:36:50 INFO mapred.MapTask: bufstart = 2496693; bufend = 5000178; bufvoid = 99614720
12/10/19 20:36:50 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/19 20:36:51 INFO mapred.MapTask: Finished spill 1
12/10/19 20:36:51 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:134217728+67108864
12/10/19 20:36:52 INFO mapred.JobClient: map 94% reduce 0%
12/10/19 20:36:52 INFO mapred.MapTask: Starting flush of map output
12/10/19 20:36:53 INFO mapred.MapTask: Finished spill 2
12/10/19 20:36:53 INFO mapred.Merger: Merging 3 sorted segments
12/10/19 20:36:53 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 4656 bytes
12/10/19 20:36:53 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000002_0 is done. And is in the process of commiting
12/10/19 20:36:53 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:134217728+67108864
12/10/19 20:36:53 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000002_0' done.
12/10/19 20:36:53 INFO mapred.MapTask: numReduceTasks: 1
12/10/19 20:36:53 INFO mapred.MapTask: io.sort.mb = 100
12/10/19 20:36:53 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/19 20:36:53 INFO mapred.JobClient: map 100% reduce 0%
12/10/19 20:36:53 INFO mapred.MapTask: record buffer = 262144/327680
12/10/19 20:36:54 INFO mapred.MapTask: Spilling map output: record full = true
12/10/19 20:36:54 INFO mapred.MapTask: bufstart = 0; bufend = 2526823; bufvoid = 99614720
12/10/19 20:36:54 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/19 20:36:55 INFO mapred.MapTask: Finished spill 0
12/10/19 20:36:56 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:201326592+35576587
12/10/19 20:36:56 INFO mapred.JobClient: map 98% reduce 0%
12/10/19 20:36:56 INFO mapred.MapTask: Starting flush of map output
12/10/19 20:36:56 INFO mapred.MapTask: Finished spill 1
12/10/19 20:36:56 INFO mapred.Merger: Merging 2 sorted segments
12/10/19 20:36:56 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2613 bytes
12/10/19 20:36:56 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000003_0 is done. And is in the process of commiting
12/10/19 20:36:56 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/apat63_99.txt:201326592+35576587
12/10/19 20:36:56 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000003_0' done.
12/10/19 20:36:56 INFO mapred.LocalJobRunner:
12/10/19 20:36:56 INFO mapred.Merger: Merging 4 sorted segments
12/10/19 20:36:56 INFO mapred.Merger: Down to the last merge-pass, with 4 segments left of total size: 7858 bytes
12/10/19 20:36:56 INFO mapred.LocalJobRunner:
12/10/19 20:36:56 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/10/19 20:36:56 INFO mapred.LocalJobRunner:
12/10/19 20:36:56 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/10/19 20:36:56 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://localhost:9100/user/tanglg1987/output
12/10/19 20:36:56 INFO mapred.LocalJobRunner: reduce > reduce
12/10/19 20:36:56 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
12/10/19 20:36:57 INFO mapred.JobClient: map 100% reduce 100%
12/10/19 20:36:57 INFO mapred.JobClient: Job complete: job_local_0001
12/10/19 20:36:57 INFO mapred.JobClient: Counters: 15
12/10/19 20:36:57 INFO mapred.JobClient: FileSystemCounters
12/10/19 20:36:57 INFO mapred.JobClient: FILE_BYTES_READ=129472
12/10/19 20:36:57 INFO mapred.JobClient: HDFS_BYTES_READ=876508703
12/10/19 20:36:57 INFO mapred.JobClient: FILE_BYTES_WRITTEN=236890
12/10/19 20:36:57 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=2658
12/10/19 20:36:57 INFO mapred.JobClient: Map-Reduce Framework
12/10/19 20:36:57 INFO mapred.JobClient: Reduce input groups=151
12/10/19 20:36:57 INFO mapred.JobClient: Combine output records=1063
12/10/19 20:36:57 INFO mapred.JobClient: Map input records=2923923
12/10/19 20:36:57 INFO mapred.JobClient: Reduce shuffle bytes=0
12/10/19 20:36:57 INFO mapred.JobClient: Reduce output records=151
12/10/19 20:36:57 INFO mapred.JobClient: Spilled Records=1721
12/10/19 20:36:57 INFO mapred.JobClient: Map output bytes=18862764
12/10/19 20:36:57 INFO mapred.JobClient: Map input bytes=236903179
12/10/19 20:36:57 INFO mapred.JobClient: Combine input records=1984625
12/10/19 20:36:57 INFO mapred.JobClient: Map output records=1984055
12/10/19 20:36:57 INFO mapred.JobClient: Reduce input records=493

第七步：查看结果集，运行结果如下：