Hadoop2.2.0源码分析(一)——Eclipse运行WordCount.java
来源:互联网 发布:中国自己的域名服务器 编辑:程序博客网 时间:2024/05/01 06:12
学习一门技术,并不仅仅要会用,还要知道它的原理,这里简单分析一下Hadoop样例程序源码,便于后边的学习(分析的不到位,还望各位指教)。
在hadoop-2.2.0.tar.gz文件下没有找到源码(新版本不但没有Eclipse插件,也没有源码,只有.class字节码文件),可以下载hadoop-2.2.0-src.tar.gz,解压,然后在hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples目录下获取源码。
/** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */package org.apache.hadoop.examples;import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;public class WordCount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); // value已经是文件内容的一行 public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); if (otherArgs.length != 2) { System.err.println("Usage: wordcount <in> <out>"); System.exit(2); } Job job = new Job(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); }}
在Eclipse中创建一个MapReduce Project,然后新建Java类,例如创建一个MyWordCount 类,然后将WordCount.java程序代码拷贝到MyWordCount.java文件中。然后点击Run-->Run Configurations…,在弹出的对话框中左边栏选择Java Application,选中MyWordCount,在右边栏中对Arguments进行配置。
在Program arguments中配置输入输出目录参数
/home/jack/Desktop/in /home/jack/Desktop/out
在VM arguments中配置VM arguments的参数
-Xms512m -Xmx1024m -XX:MaxPermSize=256m
注:
- in文件夹是需要在程序运行前创建的,并且要放入需要统计词频的文件,out文件夹是不能提前创建的,要由系统自动生成,否则运行时会出现Output directory file:/home/jack/Desktop/out already exists错误。
- 文件输入和输出目录为本地文件系统中的文件。
- 程序运行需要点击菜单栏上的Run。
程序运行结束后,可以在/home/jack/Desktop/out目录下的part-r-00000文件查看到词频统计的结果。
2 0
- Hadoop2.2.0源码分析(一)——Eclipse运行WordCount.java
- fedora17中hadoop2.2.0在eclipse下运行wordcount
- hadoop2.2.0配置eclipse运行wordcount程序问题及解决方法
- CentOS+eclipse+hadoop2.5.1 运行wordcount
- hadoop2.7.3 编译运行WordCount.java
- Hadoop2.0入门——伪分布式运行WordCount
- win10+eclipse+hadoop2.7.2+maven直接通过Run as Java Application运行wordcount
- Hadoop2.5.1 运行wordcount
- Hadoop2.7.1运行wordcount
- SparkStreaming的WordCount示例及源码分析(一)
- 关于Hadoop2.7.2运行wordcount
- hadoop2.7运行wordcount程序
- Hadoop2.6.4运行Wordcount程序
- Hadoop与Spark算法分析(一)——WordCount
- ibatis源码分析—运行流程解析(一)
- 在Hadoop2.2.0上运行Wordcount小程序
- hadoop2.2.0单机伪分布,运行Wordcount时候出错
- Hadoop2.5.1测试(运行自带的wordcount)
- 第二十五天【java虐我千百遍,我待java如初恋】
- Wireshark捕获的outgoing_TCP包的IP_header_checksum_error问题
- 关于Server.MapPath 出现未将对象引用设置到对象的实例
- URI和URL的区别(转自博客源 just happy)
- apache mysql django 开发平台搭建
- Hadoop2.2.0源码分析(一)——Eclipse运行WordCount.java
- 拓胜第四天(请假)
- 对第三张卡片中的功能实现要点和注意的细节!
- 2013.12.20
- C#中 TreeView 控件的使用
- java线程池中的shutdown()与shutdownNow()
- 基于visual Studio2013解决面试题之1204大数组查找
- 实践是最好的老师
- GitHub使用说明