flume-02-flume案例集锦

1.一个单点avro例子

配置文件 testAvro.conf

#test avro sourcesa1.sources=r1a1.channels=c1a1.sinks=k1a1.sources.r1.type = avroa1.sources.r1.channels=c1a1.sources.r1.bind=192.168.1.102a1.sources.r1.port=55555#Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity=1000a1.channels.c1.transactionCapacity = 100#sink配置a1.sinks.k1.type=loggera1.sinks.k1.channel=c1

这里的a1.sources.r1.bind=192.168.1.102需要修改成自己的
启动flume服务

$ bin/flume-ng agent -c conf -f conf/testAvro.conf  -n a1   -Dflume.root.logger=INFO,console

生成测试数据

# echo "hello world" > $FLUME_HOME/log.00

测试

$ bin/flume-ng avro-client -c conf -H 192.168.1.102 -p 55555  -F $FLUME_HOME/log.00

测试结果

[root@biluos conf]# ../bin/flume-ng agent --conf conf --conf-file avro.conf --name a1 -Dflume.root.logger=INFO,console17/09/04 15:17:10 INFO ipc.NettyServer: [id: 0xab0c38b9, /192.168.10.173:51021 => /192.168.10.173:55555] OPEN17/09/04 15:17:10 INFO ipc.NettyServer: [id: 0xab0c38b9, /192.168.10.173:51021 => /192.168.10.173:55555] BOUND: /192.168.10.173:5555517/09/04 15:17:10 INFO ipc.NettyServer: [id: 0xab0c38b9, /192.168.10.173:51021 => /192.168.10.173:55555] CONNECTED: /192.168.10.173:5102117/09/04 15:17:10 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64                hello world }17/09/04 15:17:10 INFO ipc.NettyServer: [id: 0xab0c38b9, /192.168.10.173:51021 :> /192.168.10.173:55555] DISCONNECTED17/09/04 15:17:10 INFO ipc.NettyServer: [id: 0xab0c38b9, /192.168.10.173:51021 :> /192.168.10.173:55555] UNBOUND17/09/04 15:17:10 INFO ipc.NettyServer: [id: 0xab0c38b9, /192.168.10.173:51021 :> /192.168.10.173:55555] CLOSED17/09/04 15:17:10 INFO ipc.NettyServer: Connection to /192.168.10.173:51021 disconnected.

2.案例2：Spool

Spool监测配置的目录下新增的文件，并将文件中的数据读取出来。需要注意两点：
　　　　1) 拷贝到spool目录下的文件不可以再打开编辑。
　　　　2) spool目录下不可包含相应的子目录

创建配置文件vi spool.confa1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = spooldira1.sources.r1.channels = c1a1.sources.r1.spoolDir = /opt/moudles/apache-flume-1.7.0-bin/logsa1.sources.r1.fileHeader = true# Describe the sinka1.sinks.k1.type = logger# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1启动[root@biluos conf]# ../bin/flume-ng agent --conf conf --conf-file spool.conf  --name a1 -Dflume.root.logger=INFO,console创建测试数据[root@biluos apache-flume-1.7.0-bin]# echo "spool test1" > $FLUME_HOME/logs/spool_text1.log输出结果17/09/04 15:25:12 INFO avro.ReliableSpoolingFileEventReader: Last read took us just up to a file boundary. Rolling to the next file, if there is one.17/09/04 15:25:12 INFO avro.ReliableSpoolingFileEventReader: Preparing to move file /opt/moudles/apache-flume-1.7.0-bin/logs/spool_text2.log to /opt/moudles/apache-flume-1.7.0-bin/logs/spool_text2.log.COMPLETED17/09/04 15:25:13 INFO sink.LoggerSink: Event: { headers:{file=/opt/moudles/apache-flume-1.7.0-bin/logs/spool_text2.log} body: 73 70 6F 6F 6C 20 74 65 73 74 31                spool test1 }再次创建同一个文件[root@biluos apache-flume-1.7.0-bin]# echo "spool test1" > $FLUME_HOME/logs/spool_text2.log输出结果17/09/04 15:25:19 INFO sink.LoggerSink: Event: { headers:{file=/opt/moudles/apache-flume-1.7.0-bin/logs/spool_text2.log} body: 73 70 6F 6F 6C 20 74 65 73 74 31                spool test1 }17/09/04 15:25:19 INFO avro.ReliableSpoolingFileEventReader: Last read took us just up to a file boundary. Rolling to the next file, if there is one.17/09/04 15:25:19 INFO avro.ReliableSpoolingFileEventReader: Preparing to move file /opt/moudles/apache-flume-1.7.0-bin/logs/spool_text2.log to /opt/moudles/apache-flume-1.7.0-bin/logs/spool_text2.log.COMPLETED17/09/04 15:25:19 ERROR source.SpoolDirectorySource: FATAL: Spool Directory source r1: { spoolDir: /opt/moudles/apache-flume-1.7.0-bin/logs }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.java.lang.IllegalStateException: File name has been re-used with different files. Spooling assumptions violated for /opt/moudles/apache-flume-1.7.0-bin/logs/spool_text2.log.COMPLETED        at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.rollCurrentFile(ReliableSpoolingFileEventReader.java:463)追加数据到同一个文件[root@biluos apache-flume-1.7.0-bin]# echo "spool test1" > $FLUME_HOME/logs/spool_text2.log输出结果    什么都没输出

案例3：Exec

这个列子在flume-01-flume的安装配置理解http://blog.csdn.net/qq_21383435/article/details/77835711已经讲解过

案例4：Syslogtcp

Syslogtcp监听TCP的端口做为数据源

vi syslog_tcp.confa1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = syslogtcpa1.sources.r1.port = 5140a1.sources.r1.host = localhosta1.sources.r1.channels = c1# Describe the sinka1.sinks.k1.type = logger# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1启动[root@biluos conf]# ../bin/flume-ng agent --conf conf --conf-file syslog_tcp.conf  --name a1 -Dflume.root.logger=INFO,console       生成测试数据[root@biluos apache-flume-1.7.0-bin]#  echo "hello idoall.org syslog" | nc localhost 5140如果提示没有这个命令，需要[root@biluos apache-flume-1.7.0-bin]# yum install nc结果17/09/04 15:44:23 INFO sink.LoggerSink: Event: { headers:{Severity=0, Facility=0, flume.syslog.status=Invalid} body: 68 65 6C 6C 6F 20 69 64 6F 61 6C 6C 2E 6F 72 67 hello idoall.org }

案例5：JSONHandler

vi post_json.confa1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = org.apache.flume.source.http.HTTPSourcea1.sources.r1.port = 8888a1.sources.r1.channels = c1# Describe the sinka1.sinks.k1.type = logger# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1启动[root@biluos conf]# ../bin/flume-ng agent --conf conf --conf-file post_json.conf  --name a1 -Dflume.root.logger=INFO,console       生成测试数据[root@biluos apache-flume-1.7.0-bin]# curl -X POST -d '[{ "headers" :{"a" : "a1","b" : "b1"},"body" : "idoall.org_body"}]' http://localhost:8888结果17/09/04 16:37:30 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: r1 started17/09/04 16:38:55 INFO sink.LoggerSink: Event: { headers:{a=a1, b=b1} body: 69 64 6F 61 6C 6C 2E 6F 72 67 5F 62 6F 64 79    idoall.org_body }

案例6：Hadoop sink

http://blog.csdn.net/qq_21383435/article/details/77835711
这里有案例

案例7：File Roll Sink

vim file_roll.confa1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = syslogtcpa1.sources.r1.port = 5555a1.sources.r1.host = localhosta1.sources.r1.channels = c1# Describe the sinka1.sinks.k1.type = file_rolla1.sinks.k1.sink.directory = /opt/moudles/apache-flume-1.7.0-bin/logs# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1启动[root@biluos conf]# ../bin/flume-ng agent --conf conf --conf-file file_roll.conf  --name a1 -Dflume.root.logger=INFO,console       生成测试数据# echo "hello idoall.org syslog" | nc localhost 5555# echo "hello idoall.org syslog 2" | nc localhost 5555结果[root@biluos conf]# ../bin/flume-ng agent --conf conf --conf-file file_roll.conf  --name a1 -Dflume.root.logger=INFO,console  这个命令下貌似没什么反应17/09/04 16:45:15 WARN source.SyslogUtils: Event created from Invalid Syslog data.17/09/04 16:45:41 WARN source.SyslogUtils: Event created from Invalid Syslog data.但是[root@biluos apache-flume-1.7.0-bin]# pwd/opt/moudles/apache-flume-1.7.0-bin[root@biluos apache-flume-1.7.0-bin]# ll logs/total 32-rw-r--r-- 1 root root   24 Sep  4 16:45 1504514695724-1-rw-r--r-- 1 root root   26 Sep  4 16:45 1504514695724-2-rw-r--r-- 1 root root    0 Sep  4 16:45 1504514695724-3-rw-r--r-- 1 root root    0 Sep  4 16:46 1504514695724-4-rw-r--r-- 1 root root    0 Sep  4 16:47 1504514695724-5-rw-r--r-- 1 root root    0 Sep  4 16:47 1504514695724-6-rw-r--r-- 1 root root    0 Sep  4 16:47 1504514695724-7查看/opt/moudles/apache-flume-1.7.0-bin/logs下是否生成文件,默认每30秒生成一个新文件[root@biluos logs]# cat 1504514695724-1hello idoall.org syslog[root@biluos logs]# cat 1504514695724-2hello idoall.org syslog 2[root@biluos logs]# cat 1504514695724-3[root@biluos logs]# cat 1504514695724-4[root@biluos logs]# cat 1504514695724-5查看得到除了前两个有数据，其他的都没有数据，但是，文件在持续生成，运用不好可能会导致大量的空文件生产

案例8：Replicating Channel Selector

　Flume支持Fan out流从一个源到多个通道。有两种模式的Fan out，分别是复制和复用。在复制的情况下，流的事件被发送到所有的配置通道。在复用的情况下，事件被发送到可用的渠道中的一个子集。Fan out流需要指定源和Fan out通道的规则。
　　　　这次我们需要用到m1,m2两台机器

参考文章：http://www.aboutyun.com/thread-8917-1-1.html
https://my.oschina.net/leejun2005/blog/288136