远程提交Map/Reduce任务
来源:互联网 发布:2017网络流行语英文 编辑:程序博客网 时间:2024/06/05 09:03
1. 将开发好MR代码打包成jar。添加到distributed cache中。
- bin/hadoop fs -copyFromLocal /root/stat-analysis-mapred-1.0-SNAPSHOT.jar /user/root/lib
- bin/hadoop fs -copyFromLocal /root/stat-analysis-mapred-1.0-SNAPSHOT.jar /user/root/lib
2. 在服务器端创建和你客户端一模一样的用户。创建目录 /tmp/hadoop-root/stagging/用户
3. 客户端提交job的代码
- Configuration conf = HBaseConfiguration.create();
- conf.set("hbase.zookeeper.quorum", "node.tracker1");
- conf.set("fs.default.name", "hdfs://node.tracker1:9000/hbase");
- conf.set("mapred.job.tracker", "node.tracker1:9001");
- Job job = new Job(conf, "Hbase_FreqCounter1");
- job.setJarByClass(FreqCounter1.class);
- Scan scan = new Scan();
- String columns = "details"; // comma seperated
- scan.addFamily(Bytes.toBytes(columns));
- scan.setFilter(new FirstKeyOnlyFilter());
- TableMapReduceUtil.initTableMapperJob("access_logs", scan, Mapper1.class, ImmutableBytesWritable.class,
- IntWritable.class, job);
- TableMapReduceUtil.initTableReducerJob("summary_user", Reducer1.class, job);
- / TableMapReduceUtil.addDependencyJars(job);
- DistributedCache.addFileToClassPath(new Path("hdfs://node.tracker1:9000/user/root/lib/stat-analysis-mapred-1.0-SNAPSHOT.jar"),job.getConfiguration());
- job.submit();
- Configuration conf = HBaseConfiguration.create();
- conf.set("hbase.zookeeper.quorum", "node.tracker1");
- conf.set("fs.default.name", "hdfs://node.tracker1:9000/hbase");
- conf.set("mapred.job.tracker", "node.tracker1:9001");
- Job job = new Job(conf, "Hbase_FreqCounter1");
- job.setJarByClass(FreqCounter1.class);
- Scan scan = new Scan();
- String columns = "details"; // comma seperated
- scan.addFamily(Bytes.toBytes(columns));
- scan.setFilter(new FirstKeyOnlyFilter());
- TableMapReduceUtil.initTableMapperJob("access_logs", scan, Mapper1.class, ImmutableBytesWritable.class,
- IntWritable.class, job);
- TableMapReduceUtil.initTableReducerJob("summary_user", Reducer1.class, job);
- / TableMapReduceUtil.addDependencyJars(job);
- DistributedCache.addFileToClassPath(new Path("hdfs://node.tracker1:9000/user/root/lib/stat-analysis-mapred-1.0-SNAPSHOT.jar"),job.getConfiguration());
- job.submit();
4.运行java application,登陆node的MR管理页面,可以看到
- 远程提交Map/Reduce任务
- 远程提交Map/Reduce任务
- 【hadoop】map reduce任务提交后系统干嘛了
- storm 远程提交任务
- 详谈JobInProgress中Map/Reduce任务分配
- Hadoop Map Reduce Task默认任务数调优
- 详谈JobInProgress中Map/Reduce任务分配
- 详谈JobInProgress中Map/Reduce任务分配
- map-reduce任务的执行流程
- map任务split切片 reduce个数 partition
- map和reduce任务各参数参考
- 远程调用执行Hadoop Map/Reduce
- 远程调用执行Hadoop Map/Reduce
- Map/Reduce Task 远程调试详解
- Hadoop的map任务和reduce任务的数量
- map任务和reduce任务个数如何计算
- Git提交远程分支任务
- 实现每个Map或Reduce任务只执行一次map或reduce方法
- kvm镜像创建
- eclipseJEE+tomcat+mysql配置
- 【转】字体图标
- Linux系统IO复用接口(select、poll、epoll)
- OCP-1Z0-053-V12.02-97题
- 远程提交Map/Reduce任务
- Mac系统升级至OS X Mavericks后Genymotion出现的问题及解决方法
- UVALive - 3938 "Ray, Pass me the dishes!"
- linux内核移植(6410)
- Web开发相关收藏
- java邮件开发(二)--构建复杂邮件
- 关于surfaceview画背景格的几个问题
- OCP-1Z0-053-V12.02-102题
- php 8个常用功能