flume安装配置-采集日志到hadoop存储

来源：互联网发布：h5 banner轮播js源码编辑：程序博客网时间：2024/05/16 12:38

一、整体架构

flume其实就是一个日志采集agent，在每台应用服务器安装一个flume agent，然后事实采集日志到HDFS集群环境存储，以便后续使用hive或者pig等大数据分析日志，然后可转存到mysql供运维查询或分析用户行为等。

二、 fume与hadoop集群规划，hadoop集群安装参照：http://blog.csdn.net/liangjianyong007/article/details/52893234

集群规划：Hive只在一个节点（hadoop3）上安装即可主机名IP安装的软件运行的进程hadoop1192.168.31.10jdk、hadoop        NameNode、DFSZKFailoverController、hadoop2192.168.31.20jdk、hadoop、hive、mysql        NameNode、DFSZKFailoverController、hive、mysqlhadoop3192.168.31.30jdk、hadoop、flume        ResourceManager、fulmehadoop4192.168.31.40jdk、hadoop、zookeeperDataNode、NodeManager、JournalNode、QuorumPeerMainhadoop5192.168.31.50jdk、hadoop、zookeeperDataNode、NodeManager、JournalNode、QuorumPeerMainhadoop6192.168.31.60jdk、hadoop、zookeeper        DataNode、NodeManager、JournalNode、QuorumPeerMain

三、flume下载安装

1. 下载flume：http://archive.apache.org/dist/flume/

2. 安装

 tar -zxvf apache-flume-1.5.0-bin.tar.gz -C /usr/cloud/flume

3.配置环境变量

  vim /etc/profile    export JAVA_HOME=/usr/cloud/java/jdk1.6.0_24  export HADOOP_HOME=/usr/cloud/hadoop/hadoop-2.2.0  export HBASE_HOME=/usr/cloud/hbase/hbase-0.96.2  export FLUME_HOME=/usr/cloud/flume/apache-flume-1.5.0  export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$FLUME_HOME/bin    source /etc/profile

4.要把hadoop集群配置好的hdfs-site.xml和core-site.xml 放到hbase/conf下

        #执行hdfs的配置scp $HADOOP_HOME/etc/hadoop/core-site.xml  $FLUME_HOME/confscp $HADOOP_HOME/etc/hadoop/hdfs-site.xml  $FLUME_HOME/conf

5. 修改flume-env.sh

sudo vim $FLUME_HOME/conf/flume-env.sh#设置JAVA_HOMEJAVA_HOME=/usr/cloud/java/jdk1.6.0_24

6. 拷贝flume依赖的hadoopjar包到$FLUME_HOME/lib：

     scp $HADOOP_HOME/share/hadoop/common/hadoop-common-2.2.0.jar                  $FLUME_HOME/lib/
   scp $HADOOP_HOME/share/hadoop/common/lib/hadoop-auth-2.2.0.jar                 $FLUME_HOME/lib/
   scp $HADOOP_HOME/share/hadoop/common/lib/commons-configuration-1.6.jar       $FLUME_HOME/lib/
三、配置flume source，channel，sink ,在conf下新建文件flume.conf

vim /FLUME_HOME/conf/flume.conf #定义agent名， source、channel、sink的名称a4.sources = r1a4.channels = c1a4.sinks = k1#具体定义sourcea4.sources.r1.type = spooldira4.sources.r1.spoolDir = /usr/cloud/flume/log#具体定义channela4.channels.c1.type = memorya4.channels.c1.capacity = 10000a4.channels.c1.transactionCapacity = 100#定义拦截器，为消息添加时间戳a4.sources.r1.interceptors = i1a4.sources.r1.interceptors.i1.type = org.apache.flume.interceptor.TimestampInterceptor$Builder#具体定义sinka4.sinks.k1.type = hdfsa4.sinks.k1.hdfs.path = hdfs://ns1/flume/%Y%m%da4.sinks.k1.hdfs.filePrefix = events-a4.sinks.k1.hdfs.fileType = DataStream#不按照条数生成文件a4.sinks.k1.hdfs.rollCount = 0#HDFS上的文件达到128M时生成一个文件a4.sinks.k1.hdfs.rollSize = 134217728#HDFS上的文件达到60秒生成一个文件a4.sinks.k1.hdfs.rollInterval = 60#组装source、channel、sinka4.sources.r1.channels = c1a4.sinks.k1.channel = c1

三、启动flume，保证hdfs已经启动

        #编写脚本 start-flume.shvim $FLUME_HOME/bin/start-flume.sh$FLUME_HOME/bin/flume-ng agent -n a4 -c conf -f $FLUME_HOME/conf/a4.conf -Dflume.root.logger=INFO,console #启动flume$FLUME_HOME/bin/start-flume.sh

检查配置的文件是否上传到hdfs

附录：

在

/usr/cloud/flume/log在存储文件biz.log,内容：   ERROR [2016-10-27 14:44:02,482] com.alibaba.scm.common.monitor.log.ScmBizLogger:30 - |<>|traceId==>UUID:3ff0f75f-0ea8-466b-9c15-b96747b8d636<||>bizType==>3000<||>T==>107<||>SUCC==>true<||>bizName==>createInboundOrder_begain<||>IN==>{"appointmentDate":"Oct 27, 2016 2:43:52 PM","sellId":3693710345,"sellName":"天猫国际012","storeCode":"STORE_230095","customsInfoNo":"ASN00105244708","purchaseOrderNo":"PO160822222226","supplierId":300000000001402,"consignOrderNo":"CO160323143157012226","abroad":false,"demandDate":"Oct 27, 2016 2:43:52 PM","saleType":2,"shipType":0,"consignOrderItemDTOList":[{"scItemId":2100771461843,"itemId":2100770298741,"quantity":100,"gmtExpired":"Oct 10, 2016 12:00:00 AM","rejectDays":10,"managerType":3,"rowVersion":0}],"bizType":3000,"rowVersion":0}  <||>bizCode==>CO160323143157012226<||>CM==>InboundOrderClientImpl.createInboundOrder<||>OUT==>{"success":true,"isRetry":false}<||> ERROR [2016-10-27 14:44:03,716] com.alibaba.scm.common.monitor.log.ScmBizLogger:30 - |<>|traceId==>UUID:eb51567f-f83a-4dfb-8d38-9b6b58d84021<||>bizType==>3000<||>T==>1359<||>SUCC==>true<||>bizName==>createInboundOrder_end<||>IN==>{"appointmentDate":"Oct 27, 2016 2:43:52 PM","sellId":3693710345,"sellName":"天猫国际012","storeCode":"STORE_230095","customsInfoNo":"ASN00105244708","purchaseOrderNo":"PO160822222226","supplierId":300000000001402,"consignOrderNo":"CO160323143157012226","abroad":false,"demandDate":"Oct 27, 2016 2:43:52 PM","saleType":2,"shipType":0,"consignOrderItemDTOList":[{"scItemId":2100771461843,"itemId":2100770298741,"quantity":100,"gmtExpired":"Oct 10, 2016 12:00:00 AM","rejectDays":10,"managerType":3,"rowVersion":0}],"bizType":3000,"rowVersion":0}  <||>bizCode==>CO160323143157012226<||>CM==>InboundOrderClientImpl.createInboundOrder<||>OUT==>{"success":true,"isRetry":false,"model":{"purchaseOrderNo":"PO160822222226","inboundNo":"IO16102714440300373001","consignOrderNo":"CO160323143157012226","storeCode":"STORE_230095","supplierId":300000000001402,"sendQuantity":100,"preArrival":"Oct 27, 2016 2:43:52 PM","status":10,"demandDate":"Oct 27, 2016 2:43:52 PM","inboundItemDOList":[{"inboundNo":"IO16102714440300373001","inboundId":373001,"scItemId":2100771461843,"itemId":2100770298741,"sendQuantity":100,"status":1,"rejectDays":10,"managerType":3,"gmtExpired":"Oct 10, 2016 12:00:00 AM","storeCode":"STORE_230095","purchaseOrderNo":"PO160822222226","bizType":3000,"attribute":"MXSDUEDATE:20161010","attributeMap":{"MXSDUEDATE":"20161010"}}],"id":373001,"bizType":3000}}<||>

0 0