20170924MapReduce学习总结
来源:互联网 发布:淘宝指数官网下载 编辑:程序博客网 时间:2024/06/08 10:40
Map过程
IntWritable one = new IntWritable(1);public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{String []res=value.toString().split("\t");String url=res[28];context.write(new Text(url), one);}
Reduce过程
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException{int sum = 0; for(IntWritable value:values){ sum+=value.get(); } context.write(new Text(key.toString()+":"+sum),NullWritable.get());}
job过程
public static void main(String[] args) throws Exception{ Configuration conf =new Configuration(); Job job= Job.getInstance(conf,"Pv"); job.setJarByClass(PvUvjob.class); Path in=new Path("/user/input/pv"); Path out=new Path("/user/output/pv"); FileInputFormat.addInputPath(job, in); FileOutputFormat.setOutputPath(job, out); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); job.setMapperClass(PvUvMap.class); job.setReducerClass(PvUvReduce.class); //map输出类型 job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class); //reduce 输出类型 job.setOutputKeyClass(Text.class); job.setOutputValueClass(NullWritable.class); job.setNumReduceTasks(1); job.waitForCompletion(true);}
阅读全文
0 0
- 20170924MapReduce学习总结
- MapReduce学习总结
- MapReduce学习总结
- Hadoop MapReduce 初步学习总结
- MapReduce总结
- MapReduce总结
- MapReduce总结
- mapreduce学习
- MapReduce--学习
- mapreduce学习
- MapReduce学习
- MapReduce学习
- MapReduce学习
- Hadoop学习笔记2:eclipse运行Mapreduce程序问题总结
- MapReduce源码分析总结
- mapreduce的总结
- MapReduce源码分析总结
- MapReduce源码分析总结
- 用struts2+session写一个注册,登陆小程序
- 关于springmvc注解扫描报错500的解决方案
- json 爬虫
- android Drawable
- BZOJ1104: [POI2007]洪水pow
- 20170924MapReduce学习总结
- 基于51单片机和L298N的小车制作(一)
- Session&&Cookie
- 背包问题
- 重温TCP-IP学习笔记——1/3
- spring中aop
- 贪心算法
- Overload 和 和 Override 的区别
- linux----redis