eclipse hadoop远程调试
来源:互联网 发布:游戏产业 知乎 编辑:程序博客网 时间:2024/05/07 11:58
一、插件安装
我用的hadoop-2.6.0,插件下载地址
下载后扔到eclipse/dropins目录下即可,当然eclipse/plugins也是可以的,前者更为轻便,推荐;重启Eclipse,即可在透视图(Perspective)中看到Map/Reduce
二、配置
点击蓝色的小象图标,新建一个Hadoop连接:
注意,一定要填写正确,修改了某些端口,以及默认运行的用户名等
正常情况下,可以在项目区域可以看到
这样可以正常的进行HDFS分布式文件系统的管理:上传,删除等操作。
为下面测试做准备,需要先建了一个目录 user/root/input,然后上传两个txt文件到此目录:
三、Hadoop工程
新建一个Map/Reduce Project工程,设定好本地的hadoop目录
新建一个测试类HadoopTest:
import java.io.IOException;import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;import org.apache.log4j.Logger;public class HadoopTest{ private static final Logger log = Logger.getLogger(HadoopTest.class); public static class Mapper1 extends Mapper<Object, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { log.info("Map key : " + key); log.info("Map value : " + value); StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { String wordStr = itr.nextToken(); word.set(wordStr); log.info("Map word : " + wordStr); context.write(word, one); } } } public static class Reduce1 extends Reducer<Text, IntWritable, Text, IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { log.info("Reduce key : " + key); log.info("Reduce value : " + values); int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); log.info("Reduce sum : " + sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args) .getRemainingArgs(); if (otherArgs.length != 2) { System.err.println("Usage: WordCountTest <in> <out>"); System.exit(2); } Job job = new Job(conf, "word count"); job.setJarByClass(HadoopTest.class); job.setMapperClass(Mapper1.class); job.setCombinerClass(Reducer1.class); job.setReducerClass(Reducer1.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); }}
右键,选择“Run Configurations”,弹出窗口,点击“Arguments”选项卡,在“Program argumetns”处预先输入参数:
备注:参数为了在本地调试使用,而非真实环境。
然后,点击“Apply”,然后“Close”。现在可以右键,选择“Run on Hadoop”,运行。
成功之后,在Eclipse下刷新HDFS目录,可以看到生成了ouput目录:
0 0
- eclipse hadoop远程调试
- eclipse远程调试hadoop
- [Hadoop] Eclipse 远程调试 Hadoop
- Eclipse远程调试hadoop源码
- eclipse 远程调试hadoop代码
- eclipse远程调试hadoop程序
- 在Eclipse中远程调试Hadoop
- eclipse远程方式调试hadoop-yarn
- Eclipse远程调试Hadoop接续上篇
- eclipse远程调试Tomcat, Hadoop集群等
- 在Eclipse中远程调试Hadoop
- hadoop学习(六)--------eclipse远程调试
- eclipse使用插件远程调试Hadoop
- Hadoop学习笔记之在Eclipse中远程调试Hadoop
- Hadoop学习笔记之在Eclipse中远程调试Hadoop
- Hadoop学习笔记之在Eclipse中远程调试Hadoop
- Hadoop学习之配置Eclipse远程调试Hadoop
- Hadoop学习笔记之在Eclipse中远程调试Hadoop
- Hdu 今年暑假不Ac?
- css页面布局基础
- hdu2544 最短路(Dijkstra) 解题报告
- JAVA国际化程序实现
- CSS盒模型理解
- eclipse hadoop远程调试
- Formatting
- mac上的http请求监控软件
- PAT (Advanced Level) 1096. Consecutive Factors (20) 连续质因数
- 数字图像学习3
- hdu 5292 Pocket Cube
- presentmodalviewcontroller, dismissmodalviewcontroller详解
- 今年暑假不AC
- 黑马程序员-C语言const关键字的使用小结