Flume的安装部署

1.上传并解压

apache-flume-1.6.0-bin.tar.gz 

tar -zxvf apache-flume-1.6.0-bin.tar.gz -C apps/

2.配置文件

cd    /home/hadoop/apps/apache-flume-1.6.0-bin/conf

vi spool-logger.conf   

内容为：

# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the source#监听目录,spoolDir指定目录, fileHeader要不要给文件夹前坠名a1.sources.r1.type = spooldira1.sources.r1.spoolDir = /home/hadoop/flumespoola1.sources.r1.fileHeader = true# Describe the sinka1.sinks.k1.type = logger# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1


3.新建目录
mkdir /home/hadoop/flumespool
4启动
cd   /home/hadoop/apps/apache-flume-1.6.0-bin
 然后  bin/flume-ng agent -c ./conf -f ./conf/spool-logger.conf -n a1 -Dflume.root.logger=INFO,console
结果截图：


5测试
测试： 往/home/hadoop/flumeSpool放文件（mv ././xxxFile /home/hadoop/flumeSpool），但是不要在里面生成文件


 当你对一个文件往flumespool放二次的时候就会报错

6.另外一种配置方式：
vi  tail-hdfs.conf
文件内容

# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = execa1.sources.r1.command = tail -F /home/hadoop/log/test.loga1.sources.r1.channels = c1# Describe the sinka1.sinks.k1.type = hdfsa1.sinks.k1.channel = c1a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/a1.sinks.k1.hdfs.filePrefix = events-a1.sinks.k1.hdfs.round = truea1.sinks.k1.hdfs.roundValue = 10a1.sinks.k1.hdfs.roundUnit = minutea1.sinks.k1.hdfs.rollInterval = 3a1.sinks.k1.hdfs.rollSize = 20a1.sinks.k1.hdfs.rollCount = 5a1.sinks.k1.hdfs.batchSize = 1a1.sinks.k1.hdfs.useLocalTimeStamp = true#生成的文件类型，默认是Sequencefile，可用DataStream，则为普通文本a1.sinks.k1.hdfs.fileType = DataStream# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1


新建文件：
cd  ~
mkdir /home/hadoop/log
touch /home/hadoop/log/test.log
启动：
用tail命令获取数据，下沉到hdfs
启动命令：
bin/flume-ng agent -c conf -f conf/tail-hdfs.conf -n a1
测试是否成功：
在开一下新的窗口
 hadoop fs -ls /flume