oozie 中使用hbase
来源:互联网 发布:stm8数据手册 编辑:程序博客网 时间:2024/05/24 04:30
I want to share a little hack I used to be able to consume TableMapper with or without filters in Oozie Workflow. The first think to understand is how TableMapReduceUtil.initTableMapperJob works.
TableMapReduceUtil.initTableMapperJob(tableName, scan, MyTableMapper.class, Writable.class, Writable.class, job);
tableName goes to : hbase.mapreduce.inputtable property
scan goes to : hbase.mapreduce.scan property (its converted to string)
TableMapper class goes to : mapreduce.map.class property (if you are not using new-api use mapred.mapper.class)
Mapper Key Output Class : mapred.mapoutput.key.class (it works for both new and old api)
Mapper Value Output Class : mapred.mapoutput.value.class
So basicly we are going to set these values in oozie workflow.xml and we would be able to run our TableMapper via oozie. The only tricky here is to pass scan to oozie in which I used a java job to get the string value.
package com.ozbuyucusu.hbase.helper;
import java.io.ByteArrayOutputStream;
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.util.Properties;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.util.Base64;
import org.apache.hadoop.hbase.util.Bytes;
public class ScanStringGenerator {
}
You can modify the Scan to add filters and giving a start and an end (which I did in most cases). The scan parameters can be passed to this helper class via arguments easily. And finally we are gonna put this helper class and in workflow.
<workflow-app xmlns='uri:oozie:workflow:0.2' name='sample-job-wf'>
</workflow-app>
I guess there would be better ways to use hbase tablemapper inside oozie workflow (which I couldn't find) but this small hack (it looks ugly indeed) works like a charm for me.
- oozie 中使用hbase
- oozie使用中常见问题及解决方法
- hue中使用oozie的workflow执行mr
- Hue中使用Oozie的workflow执行MR过程
- HBase中Disruptor使用
- oozie使用的注意事项
- 使用oozie的注意事项
- Oozie的使用
- 让oozie优雅的支持hbase
- hbase使用中一个问题
- HBase shell中使用fliter
- Hbase中Filter的使用
- Oozie
- Oozie
- OOZIE
- oozie
- oozie
- oozie
- UITextField 只能输入字母、数字的方法小结
- 39个超实用jQuery实例应用特效
- 11级_Java_曹建波 9.10 JDB处理大数据&大文本&二进制数据&批处理&事务
- 友善之臂Micro2440下的Embedded Linux的定时reboot
- ubuntu共享打印机
- oozie 中使用hbase
- oracle存储过程中无法调不同用户表的问题
- Inside Qt Series (八):Meta Object Class overview
- HDU 2055
- java数据库
- HDU 1150
- 条款21:必须返回对象时,别妄想返回其reference。
- 在C语言有一定基础之后,怎么样能够更快的成长?
- Inside Qt Series (九):QMetaObject class data members