Ubuntu配置java/hadoop环境变量以及运行hadoop的helloworld实例
来源:互联网 发布:系统重装软件哪个好 编辑:程序博客网 时间:2024/05/17 04:06
Ubuntu配置java/hadoop环境变量以及运行hadoop的helloworld实例
标签(空格分隔): 安装配置 hadoop
配置JAVA_HOMEexport JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:
export PATH=
java -version
配置HADOOP_HOME
export HADOOP_HOME=/home/latham/hadoop-2.7.4
export PATH=
三个文件分别为WordMapper/WordReducer/WordMain
package hadoop.helloword;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;public class WordMain { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args) .getRemainingArgs(); if (otherArgs.length != 2) { System.out.println("Usage:wordcount<in> <out>"); System.exit(2); } Job job = new Job(conf, "word count"); job.setJarByClass(WordMain.class); job.setMapperClass(WordMapper.class); job.setCombinerClass(WordReducer.class); job.setReducerClass(WordReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); }}
package hadoop.helloword;import java.io.IOException;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Reducer;public class WordReducer extends Reducer<Text, IntWritable, Text, IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); }}
package hadoop.helloword;import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Mapper;public class WordMapper extends Mapper<Object, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } }}
需要设置的是输入文件和输出文件的本地信息,下拉后EditConfiguration…中输入
运行即可得到结果
阅读全文
0 0
- Ubuntu配置java/hadoop环境变量以及运行hadoop的helloworld实例
- 安装java和hadoop的过程以及配置环境变量
- Ubuntu与Centos的Hadoop安装以及编译运行MapReduce,Hadoop集群安装配置教程
- 配置java和hadoop环境变量
- ubuntu hadoop安装以及配置
- hadoop实例之HELLOWORLD
- hadoop实例之HELLOWORLD
- 完整配置hadoop的环境变量
- Hadoop 在Ubuntu下的单机配置及运行示例
- Maven, HelloWorld, Java, Hadoop
- Hadoop 2.6.4配置并运行实例
- hadoop运行程序详解-helloworld
- Ubuntu 设置环境变量 以及 snmp配置实例
- Centos配置Hadoop环境变量
- Hadoop JVM环境变量配置
- Hadoop配置PATH环境变量
- hadoop环境变量配置
- Ubuntu下Hadoop以及Spark配置
- 使用OpenFiler来模拟存储配置RAC中ASM共享盘及多路径(multipath)的测试
- 【二项式剩余】Cipolla(模意义下开根)
- 手机CPU知识扫盲:谈谈手机CPU架构与原理
- Python numpy函数hstack() vstack() stack() dstack() vsplit() concatenate()
- BioSolveIT SeeSAR.v7.1 Win32 1CD
- Ubuntu配置java/hadoop环境变量以及运行hadoop的helloworld实例
- Android逆向之利用Xposed绕过悬浮窗权限
- Java 8 新增特性Lambda表达式
- 139. Word Break
- IO流(2)--定义小数组拷贝
- 交换两个整数的值
- Ciclop开源3D扫描仪软件---Horus源码分析之src\horus\engine\calibration\laser_triangulation.py
- USB转串口TTL板RX电平异常导致无法收到数据
- Geometric.Glovius.Pro.v4.4.0.489.Win32_64 2CD