flume 的source 、channel和sink 多种组合
来源:互联网 发布:二值化算法 编辑:程序博客网 时间:2024/06/03 13:05
乐高积木flume
flume 有三大组件source 、channel和sink,各个组件之间都可以相互组合使用,各组件间耦合度低。使用灵活,方便。
1.多sink
channel 的内容只输出一次,同一个event 如果sink1 输出,sink2 不输出;如果sink1 输出,sink1 不输出。 最终 sink1+sink2=channel 中的数据。
配置文件如下:
a1.sources = r1a1.sinks = k1 k2a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = execa1.sources.r1.shell = /bin/bash -ca1.sources.r1.channels = c1a1.sources.r1.command = tail -F /opt/apps/logs/tail4.log# channela1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100#sink1a1.sinks.k1.channel = c1a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.kafka.topic = mytopica1.sinks.k1.kafka.bootstrap.servers = localhost:9092a1.sinks.k1.kafka.flumeBatchSize = 20a1.sinks.k1.kafka.producer.acks = 1a1.sinks.k1.kafka.producer.linger.ms = 1a1.sinks.ki.kafka.producer.compression.type = snappy#sink2a1.sinks.k2.type = file_rolla1.sinks.k2.channel = c1#a1.sinks.k2.sink.rollInterval=0a1.sinks.k2.sink.directory = /opt/apps/tmp
2.多 channel 多sink ,每个sink 输出内容一致
(memory channel 用于kafka操作,实时性高,file channel 用于 sink file 数据安全性高)
(多channel 单 sink 的情况没有举例,个人感觉用处不广泛。)
配置文件如下:
a1.sources = r1a1.sinks = k1 k2a1.channels = c1 c2# Describe/configure the sourcea1.sources.r1.type = execa1.sources.r1.shell = /bin/bash -ca1.sources.r1.channels = c1 c2a1.sources.r1.command = tail -F /opt/apps/logs/tail4.log#多个channel 的数据相同a1.sources.r1.selector.type=replicating# channel1a1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100#channel2a1.channels.c2.type = filea1.channels.c2.checkpointDir = /opt/apps/flume-1.7.0/checkpointa1.channels.c2.dataDirs = /opt/apps/flume-1.7.0/data#sink1a1.sinks.k1.channel = c1a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.kafka.topic = mytopica1.sinks.k1.kafka.bootstrap.servers = localhost:9092a1.sinks.k1.kafka.flumeBatchSize = 20a1.sinks.k1.kafka.producer.acks = 1a1.sinks.k1.kafka.producer.linger.ms = 1a1.sinks.ki.kafka.producer.compression.type = snappy#sink2a1.sinks.k2.type = file_rolla1.sinks.k2.channel = c2#a1.sinks.k2.sink.rollInterval=0a1.sinks.k2.sink.directory = /opt/apps/tmp
3. 多source 单 channel 单 sink
多个source 可以读取多种信息放在一个channel 然后输出到同一个地方
配置文件如下:
a1.sources = r1 r2a1.sinks = k1a1.channels = c1# source1a1.sources.r1.type = execa1.sources.r1.shell = /bin/bash -ca1.sources.r1.channels = c1a1.sources.r1.command = tail -F /opt/apps/logs/tail4.log# source2a1.sources.r2.type = execa1.sources.r2.shell = /bin/bash -ca1.sources.r2.channels = c1a1.sources.r2.command = tail -F /opt/apps/logs/tail2.log# channel1 in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100#sink1a1.sinks.k1.channel = c1a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.kafka.topic = mytopica1.sinks.k1.kafka.bootstrap.servers = localhost:9092a1.sinks.k1.kafka.flumeBatchSize = 20a1.sinks.k1.kafka.producer.acks = 1a1.sinks.k1.kafka.producer.linger.ms = 1a1.sinks.ki.kafka.producer.compression.type = snappy
flume 像乐高积木一样可以自己随心所欲将不同的组件进行搭配使用,耦合度低。
0 0
- flume 的source 、channel和sink 多种组合
- flume 的source 、channel和sink 多种组合
- flume的source, channel, sink 列表
- flume的source, channel, sink 列表
- flume中几种常见的source、channel、sink
- flume之source,channel,sink
- 从无到有系列之flume的source-channel-sink汇总03
- flume组件汇总 (source, channel, sink)
- Flume笔记二之source,channel,sink
- Flume组件汇总 source、sink、channel
- flume 自定义source,sink,channel,拦截器
- 【Flume】【源码分析】深入flume-ng的三大组件——source,channel,sink
- 【Flume】【源码分析】深入flume-ng的三大组件——source,channel,sink
- 【Flume】【源码分析】深入flume-ng的三大组件——source,channel,sink
- Flume学习笔记 --- Flume内置source,channel, sink介绍
- Flume NG 学习笔记(十) Transaction、Sink、Source和Channel开发
- Flume内置channel,source,sink三组件介绍
- Flume内置channel,source,sink三组件介绍
- 1622-5 孔富晨 总结《2016年12月31日》 【连续第92天天总结】
- 算法概论习题8.8——证明精确的4SAT是NP-完全问题
- android studio - freeLine安装
- Three.JS学习 2:Threejs定义点和面
- CentOS7.2内核升至最新
- flume 的source 、channel和sink 多种组合
- Spring MVC 参数自动绑定List的解决方案
- BNU X-Plosives(并查集) @
- xml操作(一):创建xml文档
- ACM篇:UVa 1599 -- Ideal Path
- 1622-5 孔富晨 总结《2017年1月1日》 【连续第93天天总结】
- Android-安全-签名验证让二次打包变的更难
- C语言中常见的字符串处理函数
- 计算机概论