hadoop编程(3)-MapReduce案例：文本去重

来源：互联网发布：java线程机制编辑：程序博客网时间：2024/06/03 14:38

思路

把每行文本全部作为键，忽视值。

因为相同的键会发送给同一个Reduce任务进行归并，归并时同样key的所有value将归并为一个列表，当然我们并不关注这个列表，我们继续把key写出即可。

这个过程利用了MapReduce归并的特性自然对数据集去重。

代码

  public class DistinctMapper extends Mapper<LongWritable,Text,Text,NullWritable>{    @Override    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {      context.write(value,NullWritable.get());    }  }

public  class DistinctReducer extends Reducer<Text,NullWritable,Text,NullWritable> {  @Override  protected void reduce(Text key, Iterable<NullWritable> values, Context context) throws IOException, InterruptedException {    context.write(key,NullWritable.get());  }}

单元测试代码

public class DistinctTest {  MapReduceDriver<LongWritable, Text, Text, NullWritable, Text, NullWritable> mapReduceDriver;  @Before  public void setUp() {    mapReduceDriver = MapReduceDriver.newMapReduceDriver(new DistinctMapper(), new DistinctReducer());  }  @Test  public void map() throws Exception {    mapReduceDriver        .withInput(new LongWritable(0), new Text("hello"))        .withInput(new LongWritable(1), new Text("hello"))        .withInput(new LongWritable(2), new Text("hi"))        .withOutput(new Text("hello"), NullWritable.get())        .withOutput(new Text("hi"), NullWritable.get())        .runTest();  }}

阅读全文

0 0