[Hadoop] 50030/jobtracker.jsp监控不到eclipse提交的作业
来源:互联网 发布:淘宝店铺图片上传 编辑:程序博客网 时间:2024/09/21 09:20
搭建好eclipse hadoop开发环境后,一般会用hadoop自带的WordCount做个测试,其自带的WordCount.java如下:
package org.apache.hadoop.examples;import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;import org.apache.log4j.PropertyConfigurator;public class WordCount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); //set inpath,outpath Path inpath=new Path("/test/input/file1"); Path outpath=new Path("/test/output/output4"); Job job = new Job(conf, "mango"); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); //job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, inpath); FileOutputFormat.setOutputPath(job,outpath); System.out.println("开始执行"); System.exit(job.waitForCompletion(true) ? 0 : 1); }}
如果此时你run java aplication,你的程序只会在eclipse中虚拟的一个云环境中运行,而不会跑上云端去运行,所以无法再master:50070/jobtracker.jsp页面中监控到该作业;需要在main方法中添加几行代码,代码附录如下:
//在你的文件地址前自动添加:hdfs://master:9000/ conf.set("fs.default.name", "hdfs://master:9000/"); conf.set("hadoop.job.user","mango"); //指定jobtracker的ip和端口号,master在/etc/hosts中可以配置 conf.set("mapred.job.tracker","master:9001");
- [Hadoop] 50030/jobtracker.jsp监控不到eclipse提交的作业
- 50030(jobtracker.jsp)监控不到eclipse提交的作业
- 学习总结八:在eclipse中的写的代码如何提交作业到JobTracker中的?
- JobTracker健康监控测试作业
- 【Mapred】jobtracker & tasktracker架构作业是怎么提交的
- window下eclipse提交hadoop作业遇到的坑
- hadoop任务监控页面namenode:50030(在hadoop配置中查找集群jobtracker的ip,访问50030)
- Hadoop的作业提交过程
- Hadoop作业提交的方法
- hadoop作业调度详解之JobTracker
- Hadoop JobTracker提交job源码浅析
- 在win7上的eclipse向hadoop提交作业异常-权限/设置调度器
- 在Eclipse中提交作业至远程的Hadoop集群上执行
- Hadoop作业提交的几种方案
- 自动提交hadoop作业的脚本
- hadoop作业的几种提交方式
- hadoop命令行运行wordcount程序 web端监控不到作业状态
- JobTracker节点上的作业恢复RecoveryManager
- Android高级开发第四讲--API之Service
- 程序员找工作应该看一下的书
- init.rc分析
- markdown笔记(一)
- 黑马程序员_正则表达式去除字符串重复的字
- [Hadoop] 50030/jobtracker.jsp监控不到eclipse提交的作业
- 武汉群硕面试简历
- oracle PROFILE的管理
- 无知者无畏
- r+centos
- How to implement an algorithm from a scientific paper
- 武汉群硕软件实习协议
- 黑马程序员_去除ArrayList集合中的重复元素。
- oracle中profile