Flume概述

一、Flume是什么

　　Flume是一个分布式、可靠、和高可用的海量日志聚合的系统，支持在系统中定制各类数据发送方，用于收集数据；同时，Flume提供对数据进行简单处理，并写到各种数据接受方（可定制）的能力。

二 、Fulme 有什么特点

Fulme 特点如下：
1、可靠性

　　当节点出现故障时，日志能够被传送到其他节点上而不会丢失。Flume提供了三种级别的可靠性保障，从强到弱依次分别为：end-to-end（收到数据agent首先将event写到磁盘上，当数据传送成功后，再删除；如果数据发送失败，可以重新发送。），Store on failure（这也是scribe采用的策略，当数据接收方crash时，将数据写到本地，待恢复后，继续发送），Best effort（数据发送到接收方后，不会进行确认）。

2、可扩展性
　
　　Flume采用了三层架构，分别为agent，collector和storage，每一层均可以水平扩展。其中，所有agent和collector由master统一管理，这使得系统容易监控和维护，且master允许有多个（使用ZooKeeper进行管理和负载均衡），这就避免了单点故障问题。

3、可管理性

　　所有agent和colletor由master统一管理，这使得系统便于维护。多master情况，Flume利用ZooKeeper和gossip，保证动态配置数据的一致性。用户可以在master上查看各个数据源或者数据流执行情况，且可以对各个数据源配置和动态加载。Flume提供了web 和shell script command两种形式对数据流进行管理。

4、功能可扩展性

　　用户可以根据需要添加自己的agent，collector或者storage。此外，Flume自带了很多组件，包括各种agent（file， syslog等），collector和storage（file，HDFS等）。

5、文档丰富，社区活跃

　　Flume 已经成为 Hadoop 生态系统的标配，它的文档比较丰富，社区比较活跃，方便我们学习。

三、Flume NG基本架构

　　Flume NG是一个分布式、可靠、可用的系统，它能够将不同数据源的海量日志数据进行高效收集、聚合、移动，最后存储到一个中心化数据存储系统中。由原来的Flume OG到现在的Flume NG，进行了架构重构，并且现在NG版本完全不兼容原来的OG版本。经过架构重构后，Flume NG更像是一个轻量的小工具，非常简单，容易适应各种方式日志收集，并支持failover和负载均衡。

　　Flume NG 的架构图如下所示。
　　　　　
根据上图我们可以了解到Flume 的一些核心概念：

1、Event：一个数据单元，带有一个可选的消息头。

2、Flow：Event从源点到达目的点的迁移的抽象。

3、Client：操作位于源点处的Event，将其发送到Flume Agent。

4、Agent：一个独立的Flume进程，包含组件Source、Channel、Sink。

5、Source：用来消费传递到该组件的Event。

6、Channel：中转Event的一个临时存储，保存有Source组件传递过来的Event。

7、Sink：从Channel中读取并移除Event，将Event传递到Flow Pipeline中的下一个Agent（如果有的话）

在了解了Flume的核心概念之后，下面我们对各个核心概念进行深入剖析：
Event

1、Event 是Flume数据传输的基本单元。

2、Flume 以事件的形式将数据从源头传输到最终的目的。

3、Event 由可选的header和载有数据的一个byte array构成。

1）载有的数据对Flume是不透明的。

2）Header 是容纳了key-value字符串对的无序集合，key在集合内是唯一的。

3）Header 可以在上下文路由中使用扩展。
Client

1、Client 是一个将原始log包装成events并且发送它们到一个或者多个agent的实体。

2、Client 在Flume的拓扑结构中不是必须的，它的目的是从数据源系统中解耦Flume
Agent

1、一个Agent包含Source、Channel、Sink和其他组件。

2、它利用这些组件将events从一个节点传输到另一个节点或最终目的地。

3、agent是Flume流的基础部分。

4、Flume 为这些组件提供了配置、生命周期管理、监控支持。

Agent之Source

1、Source负责接收event或通过特殊机制产生event，并将events批量的放到一个或多个Channel。

2、Source包含event驱动和轮询两种类型。

3、Source 有不同的类型。

1）与系统集成的Source：Syslog，NetCat。

2）自动生成事件的Source：Exec

3）用于Agent和Agent之间的通信的IPC Source：Avro、Thrift。

4、Source必须至少和一个Channel关联。

Agent之Channel与Sink

Agent之Channel与Sink 的结构图如下所示：

Agent之Channel

1、Channel位于Source和Sink之间，用于缓存进来的event。

2、当Sink成功的将event发送到吓一跳的Channel或最终目的地，event才Channel中移除。

3、不同的Channel提供的持久化水平也是不一样的：

1）Memory Channel：volatile。

2）File Channel：基于WAL实现。

3）JDBC Channel：基于嵌入Database实现。

4、Channel支持事物，提供较弱的顺序保证。

5、Channel可以和任何数量的Source和Sink工作。

Agent之Sink

1、Sink负责将event传输到吓一跳或最终目的，成功完成后将event从Channel移除。

2、有不同类型的Sink：

1）存储event到最终目的的终端Sink。比如HDFS，HBase。

2）自动消耗的Sink。比如：Null Sink。

3）用于Agent间通信的IPC sink：Avro。

3、Sink必须作用于一个确切的Channel。

四、Flume NG安装部署

　　下面我们来安装部署 Flume NG，这里我们安装配置两个节点cloud003、cloud004，以Avro Source+Memory Channel+HDFS Sink结合方式为例：
　　1、下载flume安装包。
　　apache-flume-1.6.0-bin.tar.gz安装包分别解压到cloud003、cloud004节点上的/usr/Java/hadoop/app/目录下。这里我们以cloud003为例，cloud004同样操作。

<code class="hljs java has-numbering">[hadoop<span class="hljs-annotation">@cloud</span>003 app]$ tar -zxvf apache-flume-<span class="hljs-number">1.6</span><span class="hljs-number">.0</span>-bin.tar.gz <span class="hljs-comment">//解压</span>[hadoop<span class="hljs-annotation">@cloud</span>003 app]$ mv apache-flume-<span class="hljs-number">1.6</span><span class="hljs-number">.0</span>-bin flume <span class="hljs-comment">//修改文件名称</span></code><ul class="pre-numbering"><li>1</li><li>2</li></ul><ul class="pre-numbering"><li>1</li><li>2</li></ul>

2、在cloud003节点上，进入flume/conf目录。

<code class="hljs avrasm has-numbering">[hadoop@cloud003 app]$ cd flume/conf[hadoop@cloud003 conf]$ lsflume-conf<span class="hljs-preprocessor">.properties</span><span class="hljs-preprocessor">.template</span>  flume-env<span class="hljs-preprocessor">.ps</span>1<span class="hljs-preprocessor">.template</span>  flume-env<span class="hljs-preprocessor">.sh</span><span class="hljs-preprocessor">.template</span>  log4j<span class="hljs-preprocessor">.properties</span></code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li></ul><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li></ul>

需要通过flume-conf.properties.template复制一个flume-conf.properties配置文件。

<code class="hljs avrasm has-numbering">[hadoop@cloud003 conf]$ <span class="hljs-keyword">cp</span> flume-conf<span class="hljs-preprocessor">.properties</span><span class="hljs-preprocessor">.template</span> flume-conf<span class="hljs-preprocessor">.properties</span>[hadoop@cloud003 conf]$ lsflume-conf<span class="hljs-preprocessor">.properties</span>   flume-conf<span class="hljs-preprocessor">.properties</span><span class="hljs-preprocessor">.template</span>  flume-env<span class="hljs-preprocessor">.ps</span>1<span class="hljs-preprocessor">.template</span>  flume-env<span class="hljs-preprocessor">.sh</span><span class="hljs-preprocessor">.template</span>  log4j<span class="hljs-preprocessor">.properties</span></code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li></ul><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li></ul>

修改cloud003节点上的flume-conf.properties配置文件。这里收集日志文件到收集端。配置参数的详细说明可以参考官方文档。

<code class="hljs avrasm has-numbering">[hadoop@cloud003 conf]$ vi flume-conf<span class="hljs-preprocessor">.properties</span> <span class="hljs-preprocessor">#定义source、channel、sink 名称</span> a1<span class="hljs-preprocessor">.sources</span>  = <span class="hljs-built_in">r1</span>       //这里的a1命名可以自定义，但需要跟后面启动配置名称一致就可以 a1<span class="hljs-preprocessor">.sinks</span> =  k1 a1<span class="hljs-preprocessor">.channels</span>  = c1 <span class="hljs-preprocessor">#定义并配置r1</span> a1<span class="hljs-preprocessor">.sources</span><span class="hljs-preprocessor">.r</span>1<span class="hljs-preprocessor">.channels</span> = c1     a1<span class="hljs-preprocessor">.sources</span><span class="hljs-preprocessor">.r</span>1<span class="hljs-preprocessor">.type</span> = avro  //source类型 a1<span class="hljs-preprocessor">.sources</span><span class="hljs-preprocessor">.r</span>1<span class="hljs-preprocessor">.bind</span> = <span class="hljs-number">0.0</span><span class="hljs-number">.0</span><span class="hljs-number">.0</span>   //默认绑定本机 a1<span class="hljs-preprocessor">.sources</span><span class="hljs-preprocessor">.r</span>1<span class="hljs-preprocessor">.port</span>=<span class="hljs-number">41414</span>   //默认端口 <span class="hljs-preprocessor"># 定义并配置k1</span> a1<span class="hljs-preprocessor">.sinks</span><span class="hljs-preprocessor">.k</span>1<span class="hljs-preprocessor">.channel</span> = c1 a1<span class="hljs-preprocessor">.sinks</span><span class="hljs-preprocessor">.k</span>1<span class="hljs-preprocessor">.type</span> = avro    //sink类型 a1<span class="hljs-preprocessor">.sinks</span><span class="hljs-preprocessor">.k</span>1<span class="hljs-preprocessor">.hostname</span> = cloud004    //将数据传递给cloud004 a1<span class="hljs-preprocessor">.sinks</span><span class="hljs-preprocessor">.k</span>1<span class="hljs-preprocessor">.port</span> = <span class="hljs-number">41414</span>   //默认端口号 <span class="hljs-preprocessor">#定义并配置c1</span> a1<span class="hljs-preprocessor">.channels</span><span class="hljs-preprocessor">.c</span>1<span class="hljs-preprocessor">.type</span>=FILE   //channel类型</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li></ul><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li></ul>

3、在cloud004节点上，进入flume/conf目录。

<code class="hljs avrasm has-numbering">[hadoop@cloud004 app]$ cd flume/conf[hadoop@cloud004 conf]$ lsflume-conf<span class="hljs-preprocessor">.properties</span><span class="hljs-preprocessor">.template</span>  flume-env<span class="hljs-preprocessor">.ps</span>1<span class="hljs-preprocessor">.template</span>  flume-env<span class="hljs-preprocessor">.sh</span><span class="hljs-preprocessor">.template</span>  log4j<span class="hljs-preprocessor">.properties</span></code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li></ul><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li></ul>

需要通过flume-conf.properties.template复制一个flume-conf.properties配置文件。

<code class="hljs avrasm has-numbering">[hadoop@cloud004 conf]$ <span class="hljs-keyword">cp</span> flume-conf<span class="hljs-preprocessor">.properties</span><span class="hljs-preprocessor">.template</span> flume-conf<span class="hljs-preprocessor">.properties</span>[hadoop@cloud004 conf]$ lsflume-conf<span class="hljs-preprocessor">.properties</span>   flume-conf<span class="hljs-preprocessor">.properties</span><span class="hljs-preprocessor">.template</span>  flume-env<span class="hljs-preprocessor">.ps</span>1<span class="hljs-preprocessor">.template</span>  flume-env<span class="hljs-preprocessor">.sh</span><span class="hljs-preprocessor">.template</span>  log4j<span class="hljs-preprocessor">.properties</span></code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li></ul><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li></ul>

修改cloud004节点上的flume-conf.properties配置文件。从cloud003端接收数据，然后写入到HDFS文件系统中。配置参数的详细说明可以参考官方文档。

<code class="hljs avrasm has-numbering">[hadoop@cloud004 conf]$ vi flume-conf<span class="hljs-preprocessor">.properties</span><span class="hljs-preprocessor"># 定义source、channel、sink 名称</span>a1<span class="hljs-preprocessor">.sources</span> = <span class="hljs-built_in">r1</span>a1<span class="hljs-preprocessor">.sinks</span> = k1a1<span class="hljs-preprocessor">.channels</span> = c1<span class="hljs-preprocessor">#定义并配置 r1</span>a1<span class="hljs-preprocessor">.sources</span><span class="hljs-preprocessor">.r</span>1<span class="hljs-preprocessor">.type</span>  = avro  //这里要跟cloud003端的sink类型一致a1<span class="hljs-preprocessor">.sources</span><span class="hljs-preprocessor">.r</span>1<span class="hljs-preprocessor">.bind</span>  = <span class="hljs-number">0.0</span><span class="hljs-number">.0</span><span class="hljs-number">.0</span>a1<span class="hljs-preprocessor">.sources</span><span class="hljs-preprocessor">.r</span>1<span class="hljs-preprocessor">.port</span>  = <span class="hljs-number">41414</span>a1<span class="hljs-preprocessor">.sources</span><span class="hljs-preprocessor">.r</span>1<span class="hljs-preprocessor">.channels</span> = c1<span class="hljs-preprocessor">#定义并配置k1</span>a1<span class="hljs-preprocessor">.sinks</span><span class="hljs-preprocessor">.k</span>1<span class="hljs-preprocessor">.channel</span> = c1a1<span class="hljs-preprocessor">.sinks</span><span class="hljs-preprocessor">.k</span>1<span class="hljs-preprocessor">.type</span>=hdfs   //sink的输出类型为hdfsa1<span class="hljs-preprocessor">.sinks</span><span class="hljs-preprocessor">.k</span>1<span class="hljs-preprocessor">.hdfs</span><span class="hljs-preprocessor">.path</span>=hdfs://cloud001:<span class="hljs-number">9000</span>/data/flume   //hdfs上传文件路径a1<span class="hljs-preprocessor">.sinks</span><span class="hljs-preprocessor">.k</span>1<span class="hljs-preprocessor">.hdfs</span><span class="hljs-preprocessor">.fileType</span>=DataStream<span class="hljs-preprocessor">#定义并配置c1</span>a1<span class="hljs-preprocessor">.channels</span><span class="hljs-preprocessor">.c</span>1<span class="hljs-preprocessor">.type</span>=File</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li></ul><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li></ul>

4、首先要保证 Hadoop 集群正常运行，这里cloud001是Namenode。

<code class="hljs ruby has-numbering">[hadoop<span class="hljs-variable">@cloud001</span> hadoop]<span class="hljs-variable">$ </span>jps<span class="hljs-number">2625</span> <span class="hljs-constant">JournalNode</span><span class="hljs-number">1563</span> <span class="hljs-constant">QuorumPeerMain</span><span class="hljs-number">18808</span> <span class="hljs-constant">NameNode</span><span class="hljs-number">26146</span> <span class="hljs-constant">Jps</span><span class="hljs-number">3583</span> <span class="hljs-constant">ResourceManager</span></code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li></ul><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li></ul>

5、首先在cloud004节点上启动Agent，保证能接受传过来的数据，然后传递给hdfs。

<code class="hljs avrasm has-numbering">[hadoop@cloud003 flume]$ bin/flume-ng agent -c ./conf/ -f conf/flume-conf<span class="hljs-preprocessor">.properties</span> -Dflume<span class="hljs-preprocessor">.root</span><span class="hljs-preprocessor">.logger</span>=INFO,console -n a1<span class="hljs-label">Info:</span> Including Hadoop libraries found via (/usr/java/hadoop/bin/hadoop) for HDFS access<span class="hljs-label">Info:</span> Excluding /usr/java/hadoop/share/hadoop/common/lib/slf4j-api-<span class="hljs-number">1.7</span><span class="hljs-number">.5</span><span class="hljs-preprocessor">.jar</span> from classpath<span class="hljs-label">Info:</span> Excluding /usr/java/hadoop/share/hadoop/common/lib/slf4j-log4j12-<span class="hljs-number">1.7</span><span class="hljs-number">.5</span><span class="hljs-preprocessor">.jar</span> from classpath<span class="hljs-label">Info:</span> Including Hive libraries found via () for Hive access<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">08</span>:<span class="hljs-number">54</span>:<span class="hljs-number">22</span>,<span class="hljs-number">018</span> (conf-file-poller-<span class="hljs-number">0</span>) [INFO - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.node</span><span class="hljs-preprocessor">.PollingPropertiesFileConfigurationProvider</span>$FileWatcherRunnable<span class="hljs-preprocessor">.run</span>(PollingPropertiesFileConfigurationProvider<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">133</span>)] Reloading configuration file:conf/flume-conf<span class="hljs-preprocessor">.properties</span><span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">08</span>:<span class="hljs-number">54</span>:<span class="hljs-number">22</span>,<span class="hljs-number">033</span> (conf-file-poller-<span class="hljs-number">0</span>) [INFO - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.conf</span><span class="hljs-preprocessor">.FlumeConfiguration</span>$AgentConfiguration<span class="hljs-preprocessor">.addProperty</span>(FlumeConfiguration<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">1017</span>)] Processing:k1<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">08</span>:<span class="hljs-number">54</span>:<span class="hljs-number">22</span>,<span class="hljs-number">829</span> (lifecycleSupervisor-<span class="hljs-number">1</span>-<span class="hljs-number">0</span>) [INFO - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.instrumentation</span><span class="hljs-preprocessor">.MonitoredCounterGroup</span><span class="hljs-preprocessor">.start</span>(MonitoredCounterGroup<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">96</span>)] Component type: CHANNEL, name: c1 started<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">08</span>:<span class="hljs-number">54</span>:<span class="hljs-number">22</span>,<span class="hljs-number">830</span> (conf-file-poller-<span class="hljs-number">0</span>) [INFO - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.node</span><span class="hljs-preprocessor">.Application</span><span class="hljs-preprocessor">.startAllComponents</span>(Application<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">173</span>)] Starting Sink k1<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">08</span>:<span class="hljs-number">54</span>:<span class="hljs-number">22</span>,<span class="hljs-number">830</span> (conf-file-poller-<span class="hljs-number">0</span>) [INFO - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.node</span><span class="hljs-preprocessor">.Application</span><span class="hljs-preprocessor">.startAllComponents</span>(Application<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">184</span>)] Starting Source <span class="hljs-built_in">r1</span><span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">08</span>:<span class="hljs-number">54</span>:<span class="hljs-number">22</span>,<span class="hljs-number">832</span> (lifecycleSupervisor-<span class="hljs-number">1</span>-<span class="hljs-number">4</span>) [INFO - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.source</span><span class="hljs-preprocessor">.AvroSource</span><span class="hljs-preprocessor">.start</span>(AvroSource<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">228</span>)] Starting Avro source <span class="hljs-built_in">r1</span>: { bindAddress: <span class="hljs-number">0.0</span><span class="hljs-number">.0</span><span class="hljs-number">.0</span>, port: <span class="hljs-number">41414</span> }...<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">08</span>:<span class="hljs-number">54</span>:<span class="hljs-number">22</span>,<span class="hljs-number">835</span> (lifecycleSupervisor-<span class="hljs-number">1</span>-<span class="hljs-number">1</span>) [INFO - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.instrumentation</span><span class="hljs-preprocessor">.MonitoredCounterGroup</span><span class="hljs-preprocessor">.register</span>(MonitoredCounterGroup<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">120</span>)] Monitored counter group for type: SINK, name: k1: Successfully registered new MBean.<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">08</span>:<span class="hljs-number">54</span>:<span class="hljs-number">22</span>,<span class="hljs-number">835</span> (lifecycleSupervisor-<span class="hljs-number">1</span>-<span class="hljs-number">1</span>) [INFO - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.instrumentation</span><span class="hljs-preprocessor">.MonitoredCounterGroup</span><span class="hljs-preprocessor">.start</span>(MonitoredCounterGroup<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">96</span>)] Component type: SINK, name: k1 started<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">08</span>:<span class="hljs-number">54</span>:<span class="hljs-number">23</span>,<span class="hljs-number">326</span> (lifecycleSupervisor-<span class="hljs-number">1</span>-<span class="hljs-number">4</span>) [INFO - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.instrumentation</span><span class="hljs-preprocessor">.MonitoredCounterGroup</span><span class="hljs-preprocessor">.register</span>(MonitoredCounterGroup<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">120</span>)] Monitored counter group for type: SOURCE, name: <span class="hljs-built_in">r1</span>: Successfully registered new MBean.<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">08</span>:<span class="hljs-number">54</span>:<span class="hljs-number">23</span>,<span class="hljs-number">327</span> (lifecycleSupervisor-<span class="hljs-number">1</span>-<span class="hljs-number">4</span>) [INFO - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.instrumentation</span><span class="hljs-preprocessor">.MonitoredCounterGroup</span><span class="hljs-preprocessor">.start</span>(MonitoredCounterGroup<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">96</span>)] Component type: SOURCE, name: <span class="hljs-built_in">r1</span> started<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">08</span>:<span class="hljs-number">54</span>:<span class="hljs-number">23</span>,<span class="hljs-number">328</span> (lifecycleSupervisor-<span class="hljs-number">1</span>-<span class="hljs-number">4</span>) [INFO - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.source</span><span class="hljs-preprocessor">.AvroSource</span><span class="hljs-preprocessor">.start</span>(AvroSource<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">253</span>)] Avro source <span class="hljs-built_in">r1</span> started.</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li></ul><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li></ul>

需要注意的是：-n a1中的参数值a1必须与flume-conf.properties配置文件的a1名称一致。

6、在cloud003节点上，启动Avro Client，发送数据给cloud004节点的agent。

<code class="hljs avrasm has-numbering">[hadoop@cloud003 flume]$ bin/flume-ng avro-client -c ./conf/ -H cloud004 -p <span class="hljs-number">41414</span> -F /usr/java/hadoop/app/flume/mydata/<span class="hljs-number">2.</span>log -Dflume<span class="hljs-preprocessor">.root</span><span class="hljs-preprocessor">.logger</span>=DEBUG,console<span class="hljs-label">Info:</span> Including Hadoop libraries found via (/usr/java/hadoop/bin/hadoop) for HDFS access<span class="hljs-label">Info:</span> Excluding /usr/java/hadoop/share/hadoop/common/lib/slf4j-api-<span class="hljs-number">1.7</span><span class="hljs-number">.5</span><span class="hljs-preprocessor">.jar</span> from classpath<span class="hljs-label">Info:</span> Excluding /usr/java/hadoop/share/hadoop/common/lib/slf4j-log4j12-<span class="hljs-number">1.7</span><span class="hljs-number">.5</span><span class="hljs-preprocessor">.jar</span> from classpath<span class="hljs-label">Info:</span> Including Hive libraries found via () for Hive access<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">09</span>:<span class="hljs-number">10</span>:<span class="hljs-number">42</span>,<span class="hljs-number">629</span> (main) [DEBUG - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.api</span><span class="hljs-preprocessor">.NettyAvroRpcClient</span><span class="hljs-preprocessor">.configure</span>(NettyAvroRpcClient<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">499</span>)] Batch size string = <span class="hljs-number">5</span><span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">09</span>:<span class="hljs-number">10</span>:<span class="hljs-number">42</span>,<span class="hljs-number">672</span> (main) [WARN - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.api</span><span class="hljs-preprocessor">.NettyAvroRpcClient</span><span class="hljs-preprocessor">.configure</span>(NettyAvroRpcClient<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">634</span>)] Using default maxIOWorkers<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">09</span>:<span class="hljs-number">10</span>:<span class="hljs-number">43</span>,<span class="hljs-number">548</span> (main) [DEBUG - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.client</span><span class="hljs-preprocessor">.avro</span><span class="hljs-preprocessor">.AvroCLIClient</span><span class="hljs-preprocessor">.run</span>(AvroCLIClient<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">234</span>)] Finished<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">09</span>:<span class="hljs-number">10</span>:<span class="hljs-number">43</span>,<span class="hljs-number">548</span> (main) [DEBUG - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.client</span><span class="hljs-preprocessor">.avro</span><span class="hljs-preprocessor">.AvroCLIClient</span><span class="hljs-preprocessor">.run</span>(AvroCLIClient<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">237</span>)] Closing reader<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">09</span>:<span class="hljs-number">10</span>:<span class="hljs-number">43</span>,<span class="hljs-number">550</span> (main) [DEBUG - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.client</span><span class="hljs-preprocessor">.avro</span><span class="hljs-preprocessor">.AvroCLIClient</span><span class="hljs-preprocessor">.run</span>(AvroCLIClient<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">241</span>)] Closing RPC client<span class="hljs-number">2015</span>-<span class="hljs-number">08</span>-<span class="hljs-number">25</span> <span class="hljs-number">09</span>:<span class="hljs-number">10</span>:<span class="hljs-number">43</span>,<span class="hljs-number">567</span> (main) [DEBUG - org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.flume</span><span class="hljs-preprocessor">.client</span><span class="hljs-preprocessor">.avro</span><span class="hljs-preprocessor">.AvroCLIClient</span><span class="hljs-preprocessor">.main</span>(AvroCLIClient<span class="hljs-preprocessor">.java</span>:<span class="hljs-number">84</span>)] Exiting</code><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li></ul><ul class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li></ul>

需要注意：-H cloud004中的cloud004是agent节点地址，-F /usr/java/hadoop/app/flume/mydata/2.log 是发送的日志文件内容。

7、查看HDFS上同步过来的数据。

<code class="hljs haskell has-numbering">[hadoop@cloud001 hadoop]$ hadoop fs -ls /<span class="hljs-typedef"><span class="hljs-keyword">data</span>/flume/</span><span class="hljs-type">Found</span> <span class="hljs-number">1</span> items-rw-r<span class="hljs-comment">--r--   3 hadoop supergroup         21 2015-08-25 09:11 /data/flume/FlumeData.1440465043429</span></code>