hadoop学习4
来源:互联网 发布:如何修改tomcat端口 编辑:程序博客网 时间:2024/06/06 00:47
利用mapreduce进行排序
import java.io.IOException;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class Sort { public static class Map extends Mapper<Object,Text,IntWritable,IntWritable>{ private static IntWritable data = new IntWritable(); @Override protected void map(Object key, Text value, Mapper<Object, Text, IntWritable, IntWritable>.Context context) throws IOException, InterruptedException { String line = value.toString(); data.set(Integer.parseInt(line)); context.write(data, new IntWritable(1)); } } public static class Reduce extends Reducer<IntWritable, IntWritable, IntWritable, IntWritable>{ private static IntWritable lineSum = new IntWritable(1); @Override protected void reduce(IntWritable key, Iterable<IntWritable> values, Reducer<IntWritable, IntWritable, IntWritable, IntWritable>.Context context) throws IOException, InterruptedException { for(IntWritable value : values){ context.write(lineSum,key); lineSum = new IntWritable(lineSum.get() + 1); } } } public static void main(String[] args) throws Exception{ Configuration conf = new Configuration(); Job job = new Job(conf,"sort"); job.setOutputKeyClass(IntWritable.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); FileInputFormat.setInputPaths(job,new Path("hdfs://node1:9000/afan/input")); FileOutputFormat.setOutputPath(job,new Path("hdfs://node1:9000/afan/output")); boolean ret = job.waitForCompletion(true); System.exit(ret? 0:1); }}
阅读全文
0 0
- Hadoop学习4
- Hadoop学习笔记(4)
- Hadoop学习笔记 4
- hadoop学习4
- hadoop学习笔记4:hadoop、spark概念
- 【hadoop学习】Hadoop配置
- Hadoop学习-hadoop常用命令
- hadoop正式学习---hadoop
- hadoop 学习
- Hadoop学习
- hadoop学习
- Hadoop学习
- hadoop学习
- Hadoop学习
- hadoop学习
- hadoop学习
- hadoop学习
- hadoop学习
- 回归程序猿科学的生活方式——总结一些开发经验
- Java Jedis操作Redis示例(四)——Redis和Mysql的结合方案演进
- Python Basic
- JavaWeb学习总结(二)——Tomcat服务器学习和使用(二)
- 日常笔记2
- hadoop学习4
- C++参考demo-dll
- 安卓EditText输入时自动编号
- 基于Swift的iOS应用程序开发:以Base64字符串形式上传文件——从iOS选择文件并上传
- Java 静态语句块、语句块、构造函数执行顺序
- Android 开发常用的adb命令
- Android Fragment接口传值
- CODECHEF Sereja and Commands
- javaweb学习总结(四)——Http协议