MR--WordCount的MapReduce程序注释
来源:互联网 发布:建筑工程软件有哪些 编辑:程序博客网 时间:2024/05/29 14:26
程序基于Hadoop2.7.4开发, 可运行
public class WordCount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); @Override protected void map(Object key, Text value, Context context) throws IOException, InterruptedException { //使用StringTokenizer而不使用split()方法是为了提升效率 StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()){ //set text to contain a content of string word.set(itr.nextToken()); //generate a output pair context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { //计次 int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); //job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); }}
阅读全文
0 0
- MR--WordCount的MapReduce程序注释
- MR--MaxTemperature的Mapreduce程序注释
- 简单的MapReduce程序wordCount
- hadoop hdfs搭建 mapreduce环境搭建 wordcount程序简单注释
- WordCount:Hadoop中MapReduce的HelloWorld程序
- mapreduce中最简单的wordcount程序
- MapReduce程序初探 -------------- WordCount
- MapReduce WordCount程序
- 大数据之MapReduce详解(MR的运行机制及配合WordCount实例来说明运行机制)
- wordcount的mr java代码
- WordCount,第一个MapReduce程序
- 第一个mapreduce程序WordCount
- MapReduce程序——wordCount
- MapReduce基本原理与WordCount程序
- 第一个MapReduce程序-WordCount
- Hadoop MapReduce WordCount程序编写
- MapReduce--1--入门程序WordCount
- Hadoop2.4.1 简单的wordCount的MapReduce程序
- React-Native调试工具
- 表达式计算(栈)
- [Paper Reading]EFFICIENT VECTOR REPRESENTATION FOR DOCUMENTS THROUGH CORRUPTION
- addrinfo& sockaddr结构解析
- HDOH 2309 ICPC Score Totalizer Software(水题)
- MR--WordCount的MapReduce程序注释
- DTE与DCE
- 单点登录原理与简单实现
- java编程思想-内部类
- 基数排序
- Lua中function总结
- ajax方式和修改form属性action(提交路径)方式提交form表单
- 了解Handler,Looper, MessageQueue,Message的工作流程
- c语言 用数组找最小数 消除重复