flume 的source 、channel和sink 多种组合

原文链接： http://blog.csdn.net/u012373815/article/details/54351323

乐高积木flume

flume 有三大组件source 、channel和sink，各个组件之间都可以相互组合使用，各组件间耦合度低。使用灵活，方便。

1.多sink

channel 的内容只输出一次，同一个event 如果sink1 输出，sink2 不输出；如果sink1 输出，sink1 不输出。 最终 sink1+sink2=channel 中的数据。

配置文件如下：

a1.sources = r1a1.sinks = k1 k2a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = execa1.sources.r1.shell = /bin/bash -ca1.sources.r1.channels = c1a1.sources.r1.command = tail -F /opt/apps/logs/tail4.log# channela1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100#sink1a1.sinks.k1.channel = c1a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.kafka.topic = mytopica1.sinks.k1.kafka.bootstrap.servers = localhost:9092a1.sinks.k1.kafka.flumeBatchSize = 20a1.sinks.k1.kafka.producer.acks = 1a1.sinks.k1.kafka.producer.linger.ms = 1a1.sinks.ki.kafka.producer.compression.type = snappy#sink2a1.sinks.k2.type = file_rolla1.sinks.k2.channel = c1#a1.sinks.k2.sink.rollInterval=0a1.sinks.k2.sink.directory = /opt/apps/tmp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

2.多 channel 多sink ，每个sink 输出内容一致

（memory channel 用于kafka操作，实时性高，file channel 用于 sink file 数据安全性高） 
（多channel 单 sink 的情况没有举例，个人感觉用处不广泛。）

配置文件如下：

a1.sources = r1a1.sinks = k1 k2a1.channels = c1 c2# Describe/configure the sourcea1.sources.r1.type = execa1.sources.r1.shell = /bin/bash -ca1.sources.r1.channels = c1 c2a1.sources.r1.command = tail -F /opt/apps/logs/tail4.log#多个channel 的数据相同a1.sources.r1.selector.type=replicating# channel1a1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100#channel2a1.channels.c2.type = filea1.channels.c2.checkpointDir = /opt/apps/flume-1.7.0/checkpointa1.channels.c2.dataDirs = /opt/apps/flume-1.7.0/data#sink1a1.sinks.k1.channel = c1a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.kafka.topic = mytopica1.sinks.k1.kafka.bootstrap.servers = localhost:9092a1.sinks.k1.kafka.flumeBatchSize = 20a1.sinks.k1.kafka.producer.acks = 1a1.sinks.k1.kafka.producer.linger.ms = 1a1.sinks.ki.kafka.producer.compression.type = snappy#sink2a1.sinks.k2.type = file_rolla1.sinks.k2.channel = c2#a1.sinks.k2.sink.rollInterval=0a1.sinks.k2.sink.directory = /opt/apps/tmp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

3. 多source 单 channel 单 sink

多个source 可以读取多种信息放在一个channel 然后输出到同一个地方 
配置文件如下：

a1.sources = r1 r2a1.sinks = k1a1.channels = c1# source1a1.sources.r1.type = execa1.sources.r1.shell = /bin/bash -ca1.sources.r1.channels = c1a1.sources.r1.command = tail -F /opt/apps/logs/tail4.log# source2a1.sources.r2.type = execa1.sources.r2.shell = /bin/bash -ca1.sources.r2.channels = c1a1.sources.r2.command = tail -F /opt/apps/logs/tail2.log# channel1  in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100#sink1a1.sinks.k1.channel = c1a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.kafka.topic = mytopica1.sinks.k1.kafka.bootstrap.servers = localhost:9092a1.sinks.k1.kafka.flumeBatchSize = 20a1.sinks.k1.kafka.producer.acks = 1a1.sinks.k1.kafka.producer.linger.ms = 1a1.sinks.ki.kafka.producer.compression.type = snappy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

flume 像乐高积木一样可以自己随心所欲将不同的组件进行搭配使用，耦合度低。