远程提交Map/Reduce任务
来源:互联网 发布:繁体字转换软件 编辑:程序博客网 时间:2024/05/22 16:30
1. 将开发好MR代码打包成jar。添加到distributed cache中。
- bin/hadoop fs -copyFromLocal /root/stat-analysis-mapred-1.0-SNAPSHOT.jar /user/root/lib
bin/hadoop fs -copyFromLocal /root/stat-analysis-mapred-1.0-SNAPSHOT.jar /user/root/lib
2. 在服务器端创建和你客户端一模一样的用户。创建目录 /tmp/hadoop-root/stagging/用户
3. 客户端提交job的代码
- Configuration conf = HBaseConfiguration.create();
- conf.set("hbase.zookeeper.quorum", "node.tracker1");
- conf.set("fs.default.name", "hdfs://node.tracker1:9000/hbase");
- conf.set("mapred.job.tracker", "node.tracker1:9001");
- Job job = new Job(conf, "Hbase_FreqCounter1");
- job.setJarByClass(FreqCounter1.class);
- Scan scan = new Scan();
- String columns = "details"; // comma seperated
- scan.addFamily(Bytes.toBytes(columns));
- scan.setFilter(new FirstKeyOnlyFilter());
- TableMapReduceUtil.initTableMapperJob("access_logs", scan, Mapper1.class, ImmutableBytesWritable.class,
- IntWritable.class, job);
- TableMapReduceUtil.initTableReducerJob("summary_user", Reducer1.class, job);
- / TableMapReduceUtil.addDependencyJars(job);
- DistributedCache.addFileToClassPath(new Path("hdfs://node.tracker1:9000/user/root/lib/stat-analysis-mapred-1.0-SNAPSHOT.jar"),job.getConfiguration());
- job.submit();
Configuration conf = HBaseConfiguration.create(); conf.set("hbase.zookeeper.quorum", "node.tracker1"); conf.set("fs.default.name", "hdfs://node.tracker1:9000/hbase"); conf.set("mapred.job.tracker", "node.tracker1:9001"); Job job = new Job(conf, "Hbase_FreqCounter1"); job.setJarByClass(FreqCounter1.class); Scan scan = new Scan(); String columns = "details"; // comma seperated scan.addFamily(Bytes.toBytes(columns)); scan.setFilter(new FirstKeyOnlyFilter()); TableMapReduceUtil.initTableMapperJob("access_logs", scan, Mapper1.class, ImmutableBytesWritable.class, IntWritable.class, job); TableMapReduceUtil.initTableReducerJob("summary_user", Reducer1.class, job);// TableMapReduceUtil.addDependencyJars(job); DistributedCache.addFileToClassPath(new Path("hdfs://node.tracker1:9000/user/root/lib/stat-analysis-mapred-1.0-SNAPSHOT.jar"),job.getConfiguration()); job.submit();
4.运行java application,登陆node的MR管理页面,可以看到
- 远程提交Map/Reduce任务
- 远程提交Map/Reduce任务
- 【hadoop】map reduce任务提交后系统干嘛了
- storm 远程提交任务
- 详谈JobInProgress中Map/Reduce任务分配
- Hadoop Map Reduce Task默认任务数调优
- 详谈JobInProgress中Map/Reduce任务分配
- 详谈JobInProgress中Map/Reduce任务分配
- map-reduce任务的执行流程
- map任务split切片 reduce个数 partition
- map和reduce任务各参数参考
- 远程调用执行Hadoop Map/Reduce
- 远程调用执行Hadoop Map/Reduce
- Map/Reduce Task 远程调试详解
- Hadoop的map任务和reduce任务的数量
- map任务和reduce任务个数如何计算
- Git提交远程分支任务
- 实现每个Map或Reduce任务只执行一次map或reduce方法
- HDU1969---Pie 二分
- 字符串hash函数
- c++: the hint of pointer
- poj 1226 Substrings(二分 + kmp)
- Problem 1056 - 函数问题 ---三分法
- 远程提交Map/Reduce任务
- poj 2185 Milking Grid(最小矩阵覆盖,二维 kmp 匹配)
- mysql show columns等show的用法
- 关于斯特方码
- linux c使用互斥锁实现同步
- HTML5做手机站页面字体显示很小的解决方法
- MVC3 上传文件
- HDU1035Robot Motion(模拟&&深搜)
- uva 439 Knight Moves