新建一个map/Reduce工程实例WordCount
来源:互联网 发布:华为云计算认证培训 编辑:程序博客网 时间:2024/06/16 19:40
新建一个map/Reduce工程实例WordCount
新建一个工程:
2.选择 map/reduce
3.
4. 会自定加载 相关jar 包:
5. 代码,运行即可:
import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;public class WordCount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { System.out.println("key: "+key.toString()); System.out.println("value:"+value.toString()); StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] ioArgs = new String[] { "hdfs://192.168.233.128:9000/user/wangxiaoming/input", "hdfs://192.168.233.128:9000/user/wangxiaoming/output1"}; String[] otherArgs = new GenericOptionsParser(conf, ioArgs).getRemainingArgs(); if (otherArgs.length != 2) { System.err.println(otherArgs.length); System.err.println("Usage: wordcount <in> <out>"); System.exit(2); } Job job = new Job(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); }}
怎么运行:
运行结果: 就忽略吧。。。
0 0
- 新建一个map/Reduce工程实例WordCount
- perl实现map-reduce框架实例-wordcount
- WordCount-Map/Reduce示例
- Hadoop中map/reduce之WordCount实例——分解vs汇总
- Hadoop的Map-Reduce例子 WordCount
- Map-Reduce下使用Streaming实现 WordCount
- WordCount------自己写的第一个map/reduce程序------
- 使用SAS实现HADOOP Map/Reduce程序-wordcount
- Hadoop 2.6 以WordCount为例理解Map Reduce
- maven新建一个工程
- 新建一个Xtext工程
- 2.5 新建一个工程
- 实例讲解Hadoop Map-Reduce过程
- 新建一个实例
- 新建一个安卓工程
- IntelliJ新建一个web工程
- IAR-stm32新建一个工程
- 如何新建一个cc3200工程
- 产品:产品经理的能力等级,你属于哪一级?
- 环形队列的应用
- 给大学生的良心建议
- QT串口发送错误
- route
- 新建一个map/Reduce工程实例WordCount
- bzoj1478/1815[Shoi2006]color 有色图
- 如何将Eclipse中java工程上传到Github?
- Linux 线程分离状态
- React Native 启动白屏问题解决方案,教程
- java中的查找子字符串
- webx request注入单例增强实现
- hadoop job 重要性能参数
- CSS Positioning(定位)