MapRedcue例程编译和执行
来源:互联网 发布:mac版spss23 编辑:程序博客网 时间:2024/05/01 11:55
MapRedcue例程编译和执行
简介
例程的编译过程参考MapReduce Tutorial[1],历程WordCount.java内容见附录。
Linux平台
开发环境
- JDK1.8.0
- Hadoop2.5.2
环境变量
- export JAVA_HOME=/usr/java/default
- export PATH=$JAVA_HOME/bin:\$PATH
- export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar
编译和打包
- hadoop com.sun.tools.javac.Main WordCount.java
- jar cf wc.jar WordCount*.class
准备输入文件
- echo Hello World Bye World>file01
- echo Hello Hadoop Goodbye Hadoop>file02
- hdfs dfs -mkdir -p /user/joe/wordcount/input
- hdfs dfs -put file0? /user/joe/wordcount/input/
- hdfs dfs -ls /user/joe/wordcount
- hdfs dfs -cat /user/joe/wordcount/input/file01
- hdfs dfs -cat /user/joe/wordcount/input/file02
执行并检查结果
- hadoop jar wc.jar WordCount /user/joe/wordcount/input /user/joe/wordcount/output
- hdfs dfs -cat /user/joe/wordcount/output/part-r-00000
Windows平台
Windows平台的只是完成例程编译和打包过程,仍需将程序包传回linux平台执行。
开发环境
- JDK1.8.0
环境变量
- export JAVA_HOME=/usr/java/default
- export PATH=$JAVA_HOME/bin:\$PATH
Hadoop引用包
- hadoop-common-2.7.3.2.5.0.0-1245.jar
- hadoop-mapreduce-client-core-2.7.3.2.5.0.0-1245.jar
编译和打包
- javac -cp hadoop-common-2.7.3.2.5.0.0-1245.jar;hadoop-mapreduce-client-core-2.7.3.2.5.0.0-1245.jar WordCount.java
- jar cf wc.jar WordCount*.class
附录
- WordCount.java
import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class WordCount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); }}
[1] http://hadoop.apache.org/docs/r2.5.2/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html
0 0
- MapRedcue例程编译和执行
- 解释执行和编译执行
- 编译执行和解释执行
- 解释执行和编译执行
- java编译和执行
- thrift例程编译报错原因和解决方法总结
- Ardupilot编译和运行libraries下面的example例程代码
- 编译执行和解释执行的区别
- 编译执行语言和解释执行语言
- 编译执行和解释执行的区别
- 编译执行和解释执行的区别
- 解释执行和编译执行区别
- 编译执行和解释执行的区别
- 编译执行和解释执行的区别
- 解释执行和编译执行的区别
- 编译执行和解释执行的区别
- CGI的编译和执行
- JAVA编译和执行整个过程
- 2278 音量调节 2012年省队选拔赛河南
- JS数组方法汇总 array数组元素的添加和删除
- [Stochastic]--Basic Concepts
- java.lang.UnsupportedClassVersionError
- 海底高铁(差分)
- MapRedcue例程编译和执行
- dubbo 使用 学习二(spring+dubbo+zookeeper单机服务)
- What is aliasing and what causes it?
- 完美兼容 英雄联盟 穿越火线 DNF 等游戏专用 装机员win10 64位系统
- #import #include @class的区别
- 瀑布流实现
- awt和Swing
- const限定符
- 第八天学习笔记