Hadoop的MultipleOutputFormat使用

一、背景

    Hadoop的MapReduce中多文件输出默认是TextOutFormat，输出为part-r- 00000和part-r-00001依次递增的文件名。hadoop提供了

MultipleOutputFormat类，重写该类可实现定制自定义的文件名。

二、技术细节

1.环境：hadoop 0.19（目前hadoop 0.20.2对MultipleOutputFormat支持不好），linux。

2.实现MultipleOutputFormat代码例子如下：

 

public class WordCount {   public static class TokenizerMapper extends MapReduceBase  implements Mapper<LongWritable, Text, Text, IntWritable> {     private final static IntWritable count = new IntWritable(1);     private Text word = new Text();     public void map(LongWritable key, Text value,          OutputCollector<Text, IntWritable> output, Reporter  reporter)          throws IOException {        StringTokenizer itr = new StringTokenizer(value.toString());        while (itr.hasMoreTokens()) {          word.set(itr.nextToken());          output.collect(word, count);        }     }   }   public static class IntSumReducer extends MapReduceBase implements        Reducer<Text, IntWritable, Text, IntWritable> {     private IntWritable result = new IntWritable();     public void reduce(Text key, Iterator<IntWritable> values,          OutputCollector<Text, IntWritable> output, Reporter  reporter)          throws IOException {        int sum = 0;        while (values.hasNext()) {          sum += values.next().get();        }        result.set(sum);        output.collect(key, result);     }   }   public static class WordCountOutputFormat extends        MultipleOutputFormat<Text, IntWritable> {     private TextOutputFormat<Text, IntWritable> output = null;     @Override     protected RecordWriter<Text, IntWritable> getBaseRecordWriter(          FileSystem fs, JobConf job, String name, Progressable arg3)          throws IOException {        if (output == null) {          output = new TextOutputFormat<Text, IntWritable>();        }        return output.getRecordWriter(fs, job, name, arg3);     }     @Override     protected String generateFileNameForKeyValue(Text key,          IntWritable value, String name) {        char c = key.toString().toLowerCase().charAt(0);        if (c >= 'a' && c <= 'z') {          return c + ".txt";        }        return "result.txt";     }   }   public static void main(String[] args) throws Exception {     JobConf job = new JobConf(WordCount.class);     job.setJobName("wordcount");     String[] otherArgs = new GenericOptionsParser(job, args)          .getRemainingArgs();     if (otherArgs.length != 2) {        System.err.println("Usage: wordcount <in> <out>");        System.exit(2);     }     job.setJarByClass(WordCount.class);     job.setMapperClass(TokenizerMapper.class);     job.setCombinerClass(IntSumReducer.class);     job.setReducerClass(IntSumReducer.class);     job.setOutputKeyClass(Text.class);     job.setOutputValueClass(IntWritable.class);     job.setOutputFormat(WordCountOutputFormat.class);// 设置输出格式     FileInputFormat.addInputPath(job, new Path(otherArgs[0]));     FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));     JobClient.runJob(job);   }}

3.在main函数中设置输出格式，job.setOutputFormat(WordCountOutputFormat.class)；实现WordCountOutputFormat类继承MultipleOutputFormat类，重写getBaseRecordWriter和generateFileNameForKeyValue函数，在generateFileNameForKeyValue函数中参数String name为默认的输出part-00000：

public static class WordCountOutputFormat extends        MultipleOutputFormat<Text, IntWritable> {     private TextOutputFormat<Text, IntWritable> output = null;     @Override     protected RecordWriter<Text, IntWritable> getBaseRecordWriter(          FileSystem fs, JobConf job, String name, Progressable arg3)          throws IOException {        if (output == null) {          output = new TextOutputFormat<Text, IntWritable>();        }        return output.getRecordWriter(fs, job, name, arg3);     }     @Override     protected String generateFileNameForKeyValue(Text key,          IntWritable value, String name) {        char c = key.toString().toLowerCase().charAt(0);        if (c >= 'a' && c <= 'z') {          return c + ".txt";        }        return "result.txt";     }    }

4.程序结果为：

-rw-r--r--   2 root supergroup          7 2010-08-07 17:44  /hua/multipleoutput1/c.txt-rw-r--r--   2 root supergroup          6 2010-08-07 17:44  /hua/multipleoutput1/h.txt-rw-r--r--   2 root supergroup          7 2010-08-07 17:44  /hua/multipleoutput1/k.txt-rw-r--r--   2 root supergroup          6 2010-08-07 17:44  /hua/multipleoutput1/m.txt-rw-r--r--   2 root supergroup         28 2010-08-07 17:44  /hua/multipleoutput1/result.txt-rw-r--r--   2 root supergroup          6 2010-08-07 17:44  /hua/multipleoutput1/t.txt

如果generateFileNameForKeyValue返回return c + "_" + name + ".txt";结果为：

-rw-r--r--   2 root supergroup          7 2010-08-07 17:23  /hua/multipleoutput/c_part-00000.txt-rw-r--r--   2 root supergroup          6 2010-08-07 17:23  /hua/multipleoutput/h_part-00000.txt-rw-r--r--   2 root supergroup          7 2010-08-07 17:23  /hua/multipleoutput/k_part-00000.txt-rw-r--r--   2 root supergroup          6 2010-08-07 17:23  /hua/multipleoutput/m_part-00000.txt-rw-r--r--   2 root supergroup         28 2010-08-07 17:23  /hua/multipleoutput/result.txt-rw-r--r--   2 root supergroup          6 2010-08-07 17:23  /hua/multipleoutput/t_part-00000.txt

三、总结

虽然API用的是0.19的，但是使用0.20的API一样可用，只是会提示方法已过时而已。