flume-ng负载均衡load-balance、failover集群搭建

转自：http://blog.csdn.net/morning_pig/article/details/9093149

集群采用3台机器：

host1 load-balance设置  host2 机器1  host3 机器2

其中，host1 机器配置：

#Define a memory channel called c1 on a1  a1.channels = c1  a1.sources = r1  a1.sinks = k1 k2  a1.sinkgroups = g1  a1.sinkgroups.g1.sinks = k1 k2  a1.sinkgroups.g1.processor.type = load_balance  a1.sinkgroups.g1.processor.selector = round_robin  a1.sinkgroups.g1.processor.backoff = true    a1.channels.c1.type = file   a1.channels.c1.checkpointDir = /tmp/flume/loadcheckpoint  a1.channels.c1.dataDirs = /tmp/flume/loaddata    a1.sources.r1.channels = c1  a1.sources.r1.type = avro  a1.sources.r1.bind = 0.0.0.0  a1.sources.r1.port = 41415    a1.sinks.k1.channel = c1  a1.sinks.k1.type = avro  a1.sinks.k1.hostname = host2  a1.sinks.k1.port = 41414  a1.sinks.k2.channel = c1  a1.sinks.k2.type = AVRO  a1.sinks.k2.hostname = host3  a1.sinks.k2.port = 41414  

host2 机器配置：

a2.channels = c1  a2.sources = r1  a2.sinks = k1    a2.channels.c1.type = FILE  a1.channels.c1.checkpointDir = /tmp/flume/checkpoint  a1.channels.c1.dataDirs = /tmp/flume/data    a2.sources.r1.channels = c1  a2.sources.r1.type = AVRO  a2.sources.r1.bind = 0.0.0.0  a2.sources.r1.port = 41414    a2.sinks.k1.channel = c1  a2.sinks.k1.type = file_roll  a2.sinks.k1.sink.directory = /tmp/load/  a2.sinks.k1.sink.rollInterval = 0  

host3 机器配置：

a2.channels = c1  a2.sources = r1  a2.sinks = k1    a2.channels.c1.type = FILE  a1.channels.c1.checkpointDir = /tmp/flume/checkpoint  a1.channels.c1.dataDirs = /tmp/flume/data    a2.sources.r1.channels = c1  a2.sources.r1.type = AVRO  a2.sources.r1.bind = 0.0.0.0  a2.sources.r1.port = 41414    a2.sinks.k1.channel = c1  a2.sinks.k1.type = file_roll  a2.sinks.k1.sink.directory = /tmp/load/  a2.sinks.k1.sink.rollInterval = 0  

客户端机器暂时使用flume-ng的agent发送，配置如下：

# Define a memory channel called c1 on a1  a1.channels.c1.type = file   a1.channels.c1.checkpointDir = /tmp/flume/checkpoint  a1.channels.c1.dataDirs = /tmp/flume/data      a1.sources.r1.channels = c1  a1.sources.r1.type = exec  a1.sources.r1.command = cat /tmp/linux.log    a1.sinks.k1.type = avro   a1.sinks.k1.channel = c1  a1.sinks.k1.hostname = host1  a1.sinks.k1.port = 41415    a1.channels = c1  a1.sources = r1  a1.sinks = k1  

注意：启动方式最好是从下往上启动。即：先启动host2和host3，然后启动host1，最后启动client。

启动：

host2：

bin/flume-ng agent -c conf -f conf/load-sink1.conf -n a2  

host3：

bin/flume-ng agent -c conf -f conf/load-sink2.conf -n a2  

host1：

bin/flume-ng agent -c conf -f conf/load-balance.conf -n a1