Hadoop中wordCount功能实现Demo
来源:互联网 发布:离线网络驱动 编辑:程序博客网 时间:2024/06/03 07:58
wordCount是hadoop下mapReduce下的一个基本的入门程序,用来统计一系列文本中单词出现的次数(最基本的用法)。Mapreduce是一个分布式运算程序的编程框架,是用户开发“基于hadoop的数据分析应用”的核心框架。采用“分而治之”的办法,Mapper负责“分”,Reducer负责统计汇总。
#自定义Mapper部分
package com.vm.mapreduce;import java.io.IOException;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Mapper;/** * Map部分 * * @author 1 * */public class MyMapper extends Mapper<LongWritable, Text, Text, LongWritable> {@Overrideprotected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, LongWritable>.Context context)throws IOException, InterruptedException {// value表示,每一行的文本内容String line = value.toString();String[] split = line.split(" ");// 以空格来分割单词for (String word : split) {context.write(new Text(word), new LongWritable(1L));}}}#自定以Reducer部分
package com.vm.mapreduce;import java.io.IOException;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Reducer;/** * Reducer部分 * * @author 1 * */public class MyReducer extends Reducer<Text, LongWritable, Text, LongWritable> {@Overrideprotected void reduce(Text key, Iterable<LongWritable> values,Reducer<Text, LongWritable, Text, LongWritable>.Context context) throws IOException, InterruptedException {long count = 0L;for (LongWritable value : values) {count += value.get();}context.write(key, new LongWritable(count));}}#代码测试
package com.vm.mapreduce;import java.net.URI;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.conf.Configured;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.Tool;import org.apache.hadoop.util.ToolRunner;/** * 测试代码2 * * @author 1 * */public class WordCount2 extends Configured implements Tool {//文件输入路径public static final String INPUT_PATH = "hdfs://mini1:9000/wordcount/input/words.txt";//文件输出路径public static final String OUTPUT_PATH = "hdfs://mini1:9000/wordcount/output8";@Overridepublic int run(String[] arg0) throws Exception {// 判断并删除已经存在的路径和文件System.setProperty("HADOOP_USER_NAME", "hadoop");FileSystem fs = FileSystem.get(new URI(OUTPUT_PATH), getConf(), "hadoop");Path outPath = new Path(OUTPUT_PATH);if (fs.exists(outPath)) {fs.delete(outPath, true);System.out.println("删除路径。。。。OK");}@SuppressWarnings("deprecation")Job job = new Job(getConf(), "wdCount");// 指定输入目录FileInputFormat.setInputPaths(job, new Path(INPUT_PATH));// 置顶自定义的mapper类job.setMapperClass(MyMapper.class);// 置顶mapper输出的<k,v>类型job.setMapOutputKeyClass(Text.class);job.setMapOutputValueClass(LongWritable.class);// 置顶要运行的Reducer类job.setReducerClass(MyReducer.class);// 置顶reduce的输出的<k,v>类型job.setOutputKeyClass(Text.class);job.setOutputValueClass(LongWritable.class);// 设置输出目录FileOutputFormat.setOutputPath(job, new Path(OUTPUT_PATH));// *********提交作业***********boolean success = job.waitForCompletion(true);System.exit(success ? 0 : 1);return success ? 0 : 1;}public static void main(String[] args) throws Exception {Configuration conf = new Configuration();int res = ToolRunner.run(conf, new WordCount2(), args);System.out.println("=================");System.exit(res);}}
阅读全文
1 0
- Hadoop中wordCount功能实现Demo
- run hadoop wordcount demo
- hadoop wordcount demo
- hadoop-wordcount demo
- Hadoop中WordCount例子的实现
- centos6.4安装hadoop-1.2.1,实现wordcount功能
- Eclipse实现Hadoop WordCount
- Hadoop的WordCount实现
- Hadoop中运行WordCount
- Hadoop中WordCount
- Scala实现WordCount功能
- Hadoop Demo 1 ——WordCount 统计文章中单词的个数
- hadoop学习1 实现WordCount
- HADOOP中WORDCOUNT源码分析
- Hadoop中WordCount程序详解
- hadoop wordcount demo 编译、打包、运行(自己的经历)
- Hadoop Pipes编程之C++实现WordCount
- hadoop的第一个程序wordcount实现
- 项目管理【一】
- Mysql 数据库开发规范
- okhttp下载图片
- kafka学习
- 作业
- Hadoop中wordCount功能实现Demo
- 仿ios的滑轮控件,可以自己监听滚动停止的事件
- http请求过程概述及长连接、短连接简述
- String对象不可变
- Android使用okhttp封装多文件批量下载 (带进度条,取消下载)
- vm安装centos7
- Android获取手机联系人匹配用户表并按字母A-Z排序展示
- 实验2.1 顺序表实现成绩管理
- Synonyms-Chinese Synonyms for Natural Language Processing and Understanding