Flume之——采集Nginx日志到HDFS

来源：互联网发布：微信加粉软件编辑：程序博客网时间：2024/05/22 13:49

转载请注明出处：http://blog.csdn.net/l1028386804/article/details/78070111

今天，就给大家带来一篇Flume采集Nginx日志到HDFS的文章，好了，不多说了，直接进入主题。

一、下载

大家可以到Apache官网下载：http://flume.apache.org/

比如我下载的是apache-flume-1.7.0-bin.tar.gz

二、配置

1、解压

输入如下命令进行解压

tar -zxvf apache-flume-1.7.0-bin.tar.gz

2、配置/etc/profile

export FLUME_HOME=/opt/apache-flume-1.7.0-binexport PATH=$PATH:$FLUME_HOME/bin

3、修改$FLUME_HOME/conf/下的两个文件

在flume-env.sh中增加JAVA_HOME：

JAVA_HOME=/opt/jdk1.8.0_121

4、修改修改flume-conf.properties文件

# 配置Agenta1.sources = r1a1.sinks = k1a1.channels = c1# 配置Sourcea1.sources.r1.type = execa1.sources.r1.channels = c1a1.sources.r1.deserializer.outputCharset = UTF-8# 配置需要监控的日志输出目录a1.sources.r1.command = tail -F /usr/local/nginx/log/access.log# 配置Sinka1.sinks.k1.type = hdfsa1.sinks.k1.channel = c1a1.sinks.k1.hdfs.useLocalTimeStamp = truea1.sinks.k1.hdfs.path = hdfs://master:9000/flume/events/%Y-%ma1.sinks.k1.hdfs.filePrefix = %Y-%m-%d-%Ha1.sinks.k1.hdfs.fileSuffix = .loga1.sinks.k1.hdfs.minBlockReplicas = 1a1.sinks.k1.hdfs.fileType = DataStreama1.sinks.k1.hdfs.writeFormat = Texta1.sinks.k1.hdfs.rollInterval = 86400a1.sinks.k1.hdfs.rollSize = 1000000a1.sinks.k1.hdfs.rollCount = 10000a1.sinks.k1.hdfs.idleTimeout = 0# 配置Channela1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# 将三者连接a1.sources.r1.channel = c1a1.sinks.k1.channel = c1

以上文件设置了Source、Channel和Sink，将Nginx日志中的记录采集到HDFS，运行
flume-ng agent -n a1 -c conf -f $FLUME_HOME/conf/flume-conf.properties
如果没有报错，则安装设置成功了，Nginx中新增加的记录都会被Flume采集，并且存储到HDFS。

阅读全文

0 0