flume-ng安装，应用，几个简单测试例子!

官网文档：http://flume.apache.org/FlumeUserGuide.html

Flume是Cloudera提供的一个高可用的，高可靠的，分布式的海量日志采集、聚合和传输的系统，Flume支持在日志系统中定制各类数据发送方，用于收集数据；同时，Flume提供对数据进行简单处理，并写到各种数据接受方（可定制）的能力。

当前Flume有两个版本Flume 0.9X版本的统称Flume-og，Flume1.X版本的统称Flume-ng。由于Flume-ng经过重大重构，与Flume-og有很大不同，使用时请注意区分。

日志收集

Flume最早是Cloudera提供的日志收集系统，目前是Apache下的一个孵化项目，Flume支持在日志系统中定制各类数据发送方，用于收集数据。

数据处理

Flume提供对数据进行简单处理，并写到各种数据接受方（可定制）的能力 Flume提供了从console（控制台）、RPC（Thrift-RPC）、text（文件）、tail（UNIX tail）、syslog（syslog日志系统，支持TCP和UDP等2种模式），exec（命令执行）等数据源上收集数据的能力。

省去长篇介绍........

下载安装：
wget http://mirror.bit.edu.cn/apache/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz
tar -zxvf apache-flume-1.7.0-bin.tar.gz

测试例子：
cd conf/
touch example.conf
vim  example.conf

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444


# Describe the sink
a1.sinks.k1.type = logger


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1


启动:
$ bin/flume-ng agent --conf conf --conf-file conf/example.conf --name a1 -Dflume.root.logger=INFO,console


 
yum install telnet


telnet localhost 44444


helloworld 测试(a->b传送数据)


安装avro （cmark）
wget https://cmake.org/files/v3.8/cmake-3.8.1.tar.gz

1: wget http://mirrors.hust.edu.cn/apache/avro/avro-1.7.7/c/avro-c-1.7.7.tar.gz
2:tar -zxvf avro-src-1.7.7.tar.gz
3:cd avro-c-1.7.7
4:mkdir build
5:cd build
6:cmake .. -DCMAKE_INSTALL_PREFIX=$PREFIX -DCMAKE_BUILD_TYPE=RelWithDebinfo
7:make
8:make install


关闭防火墙
service iptables stop

-------------测试 HTTP Source--------------------------
conf：httppost.conf


# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = org.apache.flume.source.http.HTTPSource
a1.sources.r1.port = 5140
a1.sources.r1.channels = c1


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


启动flume agent al
bin/flume-ng agent --conf conf --conf-file conf/httppost.conf --name a1 -Dflume.root.logger=INFO,console


发送一个请求
curl -XPOST  -d '[{"headers" : {"timestamp" : "434324343","host" : "random_host.example.com"},"body" : "random_body"},{"headers" : {"namenode" : "namenode.example.com","datanode" : "random_datanode.example.com"},"body" : "really_random_body" }]' http://localhost:5140


结果：控制台会打印出来   OK-----


-------------测试 avro Source--------------------------
conf：avrotest.conf


# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
a1.sources.r1.channels = c1


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


启动flume agent al
bin/flume-ng agent --conf conf --conf-file conf/avrotest.conf --name a1 -Dflume.root.logger=INFO,console




echo "hello world" > test.log


发送文件到flume
bin/flume-ng avro-client -c conf -H 192.168.254.129 -p 44444 -F /home/rui/log/test.log


-------------测试 Spooling Directory Source--------------------------
conf：directorytest.conf


# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /home/rui/log/flumespool
a1.sources.r1.fileHeader = true
a1.sources.r1.channels = c1


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


启动flume agent al
bin/flume-ng agent --conf conf --conf-file conf/directorytest.conf --name a1 -Dflume.root.logger=INFO,console


测试:
cd /home/rui/log/flumespool
echo "hello world" > test.log

-------------测试 avro sink-----------a1服务--send-->a2服---------------
接收端：touch receive.conf
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44445
a1.sources.r1.channels = c1


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


启动flume agent al
bin/flume-ng agent --conf conf --conf-file conf/receive.conf --name a1 -Dflume.root.logger=INFO,console


发送端：send.conf

# Name the components on this agent
a2.sources = r1
a2.sinks = k1
a2.channels = c1


# Describe/configure the source
#或者目录监听方式
a2.sources.r1.type = syslogtcp
a2.sources.r1.bind = localhost
a2.sources.r1.port = 44446
a2.sources.r1.channels = c1


# Describe the sink !!!!!!!!!!!!!!!!!!!!The point is here
a2.sinks.k1.type = avro  
a2.sinks.k1.channel = c1
a2.sinks.k1.hostname = 127.0.0.1
a2.sinks.k1.port = 44445


# Use a channel which buffers events in memory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100


启动flume agent al
bin/flume-ng agent --conf conf --conf-file conf/send.conf --name a2 -Dflume.root.logger=INFO,console


tcp方式测试：
echo "<37>hello via avro sink" | nc localhost 44446


如果没有nc 就 yum install nc


44445端口控制台打印现来：<37>hello via avro sink, 就成功了！