flume自定义sink source
来源:互联网 发布:社交软件发展前景 编辑:程序博客网 时间:2024/05/23 22:59
flume
#定义source、channel、sink
a1.sources = s1
a1.channels = c1a1.sinks = k1
#分别配置sources,channels,sinks
a1.sources.s1.type = spooldir
a1.sources.s1.spoolDir = /opt/performance/timeData/1476962511a1.sources.s1.logPattern = *.log
a1.sinks.k1.type = file_roll
a1.sinks.k1.sink.directory = /opt/performance/timeData/flume
a1.channels.c1.type = memory
a1.channels.c1.capacity = 100000
a1.channels.c1.transactionCapacity = 10000#配置source和sink 的channel
a1.sources.s1.channels = c1a1.sinks.k1.channel = c1 #没有s 应为一个sink只能对一个channel
jar包依赖:
自定义jar包处理:
1、自定义java文件打包后,直接放在flume的lib包下
2、在flume-conf.properties中配置自定义type(包全称,否则找不到type包)
自定义sink:
extends AbstractSink implentments Configurable
实现public Status process() throws EventDeliveryException() //实现event处理的核心程序
public void configure(Contect context) //从配置文件中读取配置参数
小demo:
@Override
public Status process() throws EventDeliveryException {
Status status = null;
Channel ch = getChannel();
Transaction txn = ch.getTransaction(); //flume的source+sink是在同一个事物中进行的,为了稳定性考虑
txn.begin();
try {
Event event = ch.take();
System.out.println(pro.get("ip")+" "+pro.getProperty("port")+" "+event);
txn.commit();
status = Status.READY;
} catch (Throwable t) {
txn.rollback();
status = status.BACKOFF;
if(t instanceof Error){
throw t;
}
}finally{
txn.close();
}
return status;
}
@Override
public void configure(Context context) {
//读取配置信息
String ip = context.getString("ip", "192.168.184.46");
String port = context.getString("port", "8086");
pro.setProperty("ip", ip);
pro.setProperty("port", port);
}
自定义source:
extends AbstractSource implements Configurable EventDrivenSource(后者其他的处理event接口)
关键是在核心进程中处理event :getChannelProcessor().processEvent(EventBuilder.withBody(readDataBytes,headers))
- flume自定义sink source
- flume 自定义source,sink,channel,拦截器
- Flume自定义Source、Sink和Interceptor(简单功能实现)
- flume自定义sink
- flume开发--自定义Sink
- flume开发--自定义Sink
- flume自定义sink
- flume source、sink、Channels测试
- flume之source,channel,sink
- flume 自定义 hbase sink 类
- flume-自定义Sink基本框架
- flume之自定义sink组件
- Flume Source--spooldir Sink--file_roll channels--file
- flume的source, channel, sink 列表
- flume组件汇总 (source, channel, sink)
- Flume笔记二之source,channel,sink
- flume安装及source、sink、Channels测试
- Docker内flume http source & hdfs sink
- iOS开发Debug之CocoaPods安装可能遇到的问题
- Mac os 本地化localizations
- django使用多数据库,以及admin管理使用的方法。
- CRF++安装
- linux下mysql安装
- flume自定义sink source
- java判断文件后缀与前缀是否是设定的值
- 五步精听法
- spring加载properties文件,并将属性值注入到指定类的指定成员变量
- 大姐大经济的
- ssh整合问题
- Android:圆环进度控件
- 哈希的拓展——位图
- Page Object Model (POM) & Page Factory in Selenium: Ultimate Guide