flume-ng-extends
来源:互联网 发布:mysql 联合唯一键用法 编辑:程序博客网 时间:2024/06/02 05:00
这个可以taildir文件,监听文件目录写入事件,只要往这个目录写入文件就能触发
首先需要flume-ng-extends:从获取源代码 https://github.com/jinoos/flume-ng-extends,maven编译打包
将flume-ng-extends-0.0.1-SNAPSHOT.jar,另外还要下载commons-vfs2-2.0.jar这个jar包(这个不然可能会报错)一起放到flume的lib下
2,agent配置
hadoop@stormspark:~/bigdata/flume-1.4.0-bin/dirTailTest$ cat ../conf/flume-ng-extend.properties a.sources = sourcesa.sinks = sinksa.channels = c#configure sourcesa.sources.sources.type = com.jinoos.flume.DirectoryTailSourcea.sources.sources.dirs = s0 s1a.sources.sources.dirs.s0.path = /home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s0ina.sources.sources.dirs.s1.path = /home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in#congfigure sinksa.sinks.sinks.type = logger#configure channalsa.channels.c.type = memory#bind channela.sources.sources.channels = c启动这个agent
bin/flume-ng agent --conf conf --conf-file conf/flume-ng-extend.properties --name a -Dflume.root.logger=INFO,console然后往s0in,s1in目录丢进去一些文件
cp ../conf/*.properties ./s1in/
可以看到已经触发了
2014-01-17 20:29:35,774 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: SOURCE, name: sources, registered successfully.2014-01-17 20:29:35,774 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: SOURCE, name: sources started2014-01-17 20:29:53,314 (pool-3-thread-2) [WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileChanged(DirectoryTailSource.java:417)] /home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in/flume-ng-extend.propertiesis not in monitoring list.2014-01-17 20:29:53,812 (pool-3-thread-5) [WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in2014-01-17 20:29:53,812 (pool-3-thread-4) [WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in2014-01-17 20:29:53,812 (pool-3-thread-6) [WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in2014-01-17 20:29:53,812 (pool-3-thread-3) [WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in2014-01-17 20:29:53,812 (pool-3-thread-7) [WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in
1 0
- flume-ng-extends
- Flume-ng-extends的应用
- flume-ng
- Flume NG
- flume ng
- Flume-ng
- Flume OG & Flume NG
- flume OG VS flume NG
- FLume NG 开发环境
- Flume NG configuration sample
- Flume NG 配置详解
- flume-ng 使用系列
- Flume-ng使用指南
- flume ng编译问题解决
- Flume-ng配置
- Flume NG 配置
- flume-ng 整体介绍
- Flume-ng 监控介绍
- 寒假第四天--排序--排序问题
- - RPATH RUNPATH 和 LD_LIBRARY_PATH
- 一位有着20年经验的企业家告诉创业者的5件事
- Java 的 BufferReader 分析
- Makefile中include、-include、sinclude的区别
- flume-ng-extends
- GDB 高级使用
- 寒假第四天--排序--排序
- 三种方法遍历Map集合 +遍历Set集合
- 寒假第四天--排序--实况球员综合能力大比拼
- hadoop完全分布式
- 十款最实用的Android UI设计工具
- MySQL压缩包版的配置
- 寒假第四天--排序--选夫婿1