flume搭建调试

来源：互联网发布：php软件开发要学什么编辑：程序博客网时间：2024/05/20 04:32

<Date: 2012-03-06><Author:medcl><Category:日志分析>查看评论

flume搭建调试

Installing CDH3

https://ccp.cloudera.com/display/CDHDOC/CDH3+Installation

流水账，备忘。

wget http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo -O /etc/yum.repos.d/cloudera.repo yum search hadoopyum -y install hadoop-0.20  yum -y install hadoop-0.20-namenodeyum -y install hadoop-0.20-datanode#yum -y install hadoop-0.20-secondarynamenodeyum -y install hadoop-0.20-jobtrackeryum -y install hadoop-0.20-tasktracker

Installing CDH3 Components

https://ccp.cloudera.com/display/CDHDOC/CDH3+Installation#CDH3Installation-InstallingCDH3Components

yum install

install/Use
---------------------------------
Flume flume
Sqoop sqoop
Hue hue
Pig hadoop-pig
Hive hadoop-hive
HBase hadoop-hbase
ZooKeeper hadoop-zookeeper
Oozie server oozie
Oozie client oozie-client
Whirr whirr
Snappy hadoop-0.20-native
Mahout mahout

flume分为：

flume 核心
flume.node 作为节点的服务自启动脚本
flume.master 作为maaster的服务自启动脚本

yum install flume*

[root@flume-hadoop-node-1 ~]# flumeusage: flume command [args...]commands include:   dump            Takes a specified source and dumps to console  source          Takes a specified source and dumps to console  node            Start a Flume node/agent (with watchdog)  master          Start a Flume Master server (with watchdog)  version         Dump flume build version information   node_nowatch    Start a flume node/agent (no watchdog)  master_nowatch  Start a Flume Master server (no watchdog)  class <class>   Run specified fully qualified class using Flume environment (no watchdog)                   ex: flume com.cloudera.flume.agent.FlumeNode   classpath       Dump the classpath used by the java executables  shell           Start the flume shell  killmaster      Kill a running master  dumplog         Takes a specified WAL/DFO log file and dumps to console  sink            Start a one-shot flume node with console source and specified sink    cd /etc/flume/confmv flume-site.xml.template flume-site.xmlvi flume-site.xml #修改masterhost为你的host  /etc/init.d/flume-master  start/etc/init.d/flume-node start

flume文档

http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html

flume总的来说，是面向流的设计，“source“和”sink"分别代表产生和消费，push、pull都支持，可以扩展支持各种数据源，及数据的处理，非常灵活。

先停掉服务，以前台模式运行，方便查看各种输出，直观的了解一把

/etc/init.d/flume-master stop && /etc/init.d/flume-node stop

启动flume

flume dump console

启动之后，你可以在输入任何字符，然后会有来自flume的回显，因为我们参数指定了console，这个其实是配置flume的source为console的输入，默认sink也是console

source为文件的情况
flume dump 'text("/etc/services")'

tail文件末尾信息的方法
flume dump 'tail("testfile")'

testfile可以不存在，没有问题，我们在另外的console里面创建这个文件，并添加些内容

[root@flume-hadoop-node-1 tmp]# echo "test flume">testfile[root@flume-hadoop-node-1 tmp]# echo "test flume 123">testfile[root@flume-hadoop-node-1 tmp]# echo "test flume 123">>testfile[root@flume-hadoop-node-1 tmp]# echo "test flume 1234">>testfile[root@flume-hadoop-node-1 tmp]# echo "test flume 12345\r\n123456">>testfile

在flume这边，就可以实时的看到反馈

2012-01-06 20:42:55,818 [main] INFO agent.LogicalNodeManager: Loading node name with FlumeConfigData: {srcVer:'Thu Jan 01 08:00:00 CST 1970' snkVer:'Thu Jan 01 08:00:00 CST 1970'  ts='Thu Jan 01 08:00:00 CST 1970' flowId:'null' source:'tail( "testfile" )' sink:'console' }2012-01-06 20:42:55,836 [main] INFO agent.LogicalNode: Node config successfully set to FlumeConfigData: {srcVer:'Thu Jan 01 08:00:00 CST 1970' snkVer:'Thu Jan 01 08:00:00 CST 1970'  ts='Thu Jan 01 08:00:00 CST 1970' flowId:'null' source:'tail( "testfile" )' sink:'console' }2012-01-06 20:42:55,920 [logicalNode dump-10] INFO debug.ConsoleEventSink: ConsoleEventSink( debug ) opened2012-01-06 20:42:55,973 [main] INFO agent.FlumeNode: Hadoop Security enabled: falseflume-hadoop-node-1 [INFO Fri Jan 06 20:43:21 CST 2012] { tailSrcFile : (long)8387236824819002469  (string) 'testfile' (double)4.914663849160389E252 } test flumeflume-hadoop-node-1 [INFO Fri Jan 06 20:43:36 CST 2012] { tailSrcFile : (long)8387236824819002469  (string) 'testfile' (double)4.914663849160389E252 } 123flume-hadoop-node-1 [INFO Fri Jan 06 20:43:48 CST 2012] { tailSrcFile : (long)8387236824819002469  (string) 'testfile' (double)4.914663849160389E252 } test flume 123flume-hadoop-node-1 [INFO Fri Jan 06 20:43:56 CST 2012] { tailSrcFile : (long)8387236824819002469  (string) 'testfile' (double)4.914663849160389E252 } test flume 1234flume-hadoop-node-1 [INFO Fri Jan 06 20:44:11 CST 2012] { tailSrcFile : (long)8387236824819002469  (string) 'testfile' (double)4.914663849160389E252 } test flume 12345\\r\\n123456

多个文件，也是可以的

flume dump 'multitail("test1", "test2")'

默认情况下，tail会处理文件的每一行，并分别生成event，默认分隔符是“\n”，并且不会排除分隔符本身，如果你需要自定义分隔符（采用正则表达式），也是可以的，支持
”prev":分隔符属于前一个event
"next"：分隔符属于下一个event
"exclude"：分隔符丢弃

tail("file", delim="\n\n+", delimMode="exclude")tail("file", delim="</a>", delimMode="prev")

开启一个UDP服务，并监听5140端口

 flume dump 'syslogUdp(5140)'

flume web console

http://10.129.8.125:35871/flumemaster.jsp

Cloudera Manager Free Edition

https://ccp.cloudera.com/display/express37/Cloudera+Manager+Free+Edition+Documentation

wget http://archive.cloudera.com/cloudera-manager/installer/latest/cloudera-manager-installer.binchmod a+x cloudera-manager-installer.bin ./cloudera-manager-installer.bin

安装之前，先禁用Selinux

vi /etc/selinux/config --SELINUX=disabled--  setenforce 0

./cloudera-manager-installer.bin
安装失败，查看日志，发现安装包下载不下来，只能手动下载安装了。

手动安装JDK

wget http://archive.cloudera.com/cloudera-manager/redhat/5/x86_64/cloudera-manager/3/RPMS/jdk-6u21-linux-amd64.rpmrpm -Uhv jdk-6u21-linux-amd64.rpm

http://archive.cloudera.com/cloudera-manager/redhat/5/x86_64/cloudera-manager/3/RPMS/cloudera-manager-daemons-3.7.2.143-1.noarch.rpm

----------------------------华丽的不行了的分割线-----------------------------------------

2台机器：125 126
125上配置：

 vi /etc/flume/conf/flume-site.xml <property><name>flume.collector.event.host</name><value>collector</value><description>This is the host name of the default "remote"     collector.</description></property>  <property><name>flume.collector.port</name><value>35853</value><description>This default tcp port that the collector listens to     in order to receive events it is collecting.</description></property>

启动flume各节点

 flume node_watch -n collector

HDFS服务器设置（新配）

hdfs://10.129.8.126/

 cp  /usr/lib/hadoop/example-confs/conf.pseudo/*  /etc/hadoop/conf/  mkdir /var/lib/hadoop-0.20/cache/hadoop/dfs/name -pchmod 777 -R /var/lib/hadoop-0.20/ sudo -u hdfs hadoop namenode -format  （注意大写的：Y）   [root@cloudera-node-1 logs]# hadoop fs -ls hdfs://127.0.0.1/ls: Wrong FS: hdfs://127.0.0.1/, expected: hdfs://cloudera-node-1Usage: java FsShell [-ls <path>][root@cloudera-node-1 logs]# hadoop fs -ls hdfs://cloudera-node-1ls: Pathname  from hdfs://cloudera-node-1 is not a valid DFS filename.Usage: java FsShell [-ls <path>][root@cloudera-node-1 logs]# hadoop fs -ls hdfs://cloudera-node-1/[root@cloudera-node-1 logs]# hadoop fs -mkdir  hdfs://cloudera-node-1/test[root@cloudera-node-1 logs]# hadoop fs -ls hdfs://cloudera-node-1/Found 1 itemsdrwxr-xr-x   - root supergroup          0 2012-02-03 00:54 /test[root@cloudera-node-1 logs]#

修改hadoop配置，使用外部ip

 vi /etc/hadoop/conf/core-site.xml <property>    <name>fs.default.name</name>    <value>hdfs://10.129.8.126:8020</value>  </property> /etc/init.d/hadoop-0.20-namenode restart [root@cloudera-node-1 logs]# hadoop fs -ls hdfs://10.129.8.126/Found 1 itemsdrwxr-xr-x   - root supergroup          0 2012-02-03 00:54 /test

设置访问权限：

 hadoop dfs -chmod 777  hdfs://10.129.8.126/flume/hadoop dfs -chmod 777  hdfs://10.129.8.126/flume/*

126节点，启动flume

flume node_nowatch

打开flume master

http://10.129.8.125:35871/flumemaster.jsp

cloudera-node-1 : text("/etc/services") | agentSink("10.129.8.125",35853);collector : collectorSource(35853) | collectorSink("hdfs://10.129.8.126/flume/","srcdata");

？？
Flume’s Tiered Event Sources

collectorSource[(port)]
Collector source. Listens for data from agentSinks forwarding to port port. If port is not specified, the node default collector TCP port, 35853.
!!

hadoop dfs -ls hdfs://10.129.8.126/flume/

125上报错：

 org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /flume/srcdata20120203-013616957+0800.2438481505068540.00000021.tmp could only be replicated to 0 nodes, instead of 1at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1520)at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:665)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:597)at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428) at org.apache.hadoop.ipc.Client.call(Client.java:1107)at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)at $Proxy6.addBlock(Unknown Source)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:597)at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)at $Proxy6.addBlock(Unknown Source)at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3178)at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3047)at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1900(DFSClient.java:2305)at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2500)

vi /etc/hadoop/conf/hdfs-site.xml
设置replica为0.也不行

 <delete>设置 vi /etc/hadoop/conf/hdfs-site.xml <property>    <name>dfs.thrift.address</name>    <value>10.129.8.126:10090</value></property></delete>

vi /etc/hadoop/conf/masters
替换localhost为ip：10.129.8.126

还是不行，在125上手动执行upload操作

 vi a.txthadoop dfs -put a.txt hdfs://10.129.8.126/flume/srcdata20120203-014405668+0800.2438950215947540.00000019.tmp.1

报一样的错误，

在126上执行如上操作，报同样错误，MD

看来是datanode挂了，但是服务显示启动，重启试试。

 [root@cloudera-node-1 ~]# /etc/init.d/hadoop-0.20-datanode statusdatanode (pid  4866) is running...[root@cloudera-node-1 ~]# /etc/init.d/hadoop-0.20-datanode restartStopping Hadoop datanode daemon (hadoop-datanode): stopping datanodedatanode is stopped                                        [  OK  ]Starting Hadoop datanode daemon (hadoop-datanode): starting datanode, logging to /usr/lib/hadoop-0.20/logs/hadoop-hadoop-datanode-cloudera-node-1.outdatanode (pid  8570) is running...                         [  OK  ][root@cloudera-node-1 ~]# vi /usr/lib/hadoop/logs/hadoop-hadoop-datanode-cloudera-node-1.log [root@cloudera-node-1 ~]# vi /usr/lib/hadoop/logs/hadoop-hadoop-datanode-cloudera-node-1.log [root@cloudera-node-1 ~]# vi /usr/lib/hadoop/logs/hadoop-hadoop-datanode-cloudera-node-1.log [root@cloudera-node-1 ~]# vi /usr/lib/hadoop/logs/hadoop-hadoop-datanode-cloudera-node-1.log [root@cloudera-node-1 ~]# hadoop dfs -put a.txt hdfs://10.129.8.126/flume/srcdata20120203-014405668+0800.2438950215947540.00000019.tmp.12put: Target hdfs://10.129.8.126/flume/srcdata20120203-014405668+0800.2438950215947540.00000019.tmp.12 already exists[root@cloudera-node-1 ~]# hadoop dfs -put a.txt hdfs://10.129.8.126/flume/srcdata20120203-014405668+0800.2438950215947540.00000019.tmp.123[root@cloudera-node-1 ~]#

ok了。

如果报safemode了

 2012-02-03 01:42:17,467 [logicalNode collector-19] INFO rolling.RollSink: closing RollSink 'escapedCustomDfs("hdfs://10.129.8.126/flume/","srcdata%{rolltag}" )'2012-02-03 01:42:17,467 [logicalNode collector-19] INFO rolling.RollSink: opening RollSink  'escapedCustomDfs("hdfs://10.129.8.126/flume/","srcdata%{rolltag}" )'2012-02-03 01:42:17,468 [logicalNode collector-19] INFO debug.InsistentOpenDecorator: Opened MaskDecorator on try 02012-02-03 01:42:17,469 [pool-7-thread-1] INFO hdfs.EscapedCustomDfsSink: Opening hdfs://10.129.8.126/flume/srcdata20120203-014217467+0800.2438842015436540.000000192012-02-03 01:42:17,476 [logicalNode collector-19] INFO debug.InsistentAppendDecorator: append attempt 3 failed, backoff (8000ms): org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create file/flume/srcdata20120203-014217467+0800.2438842015436540.00000019.tmp. Name node is in safe mode.The number of live datanodes 0 needs an additional 1 live datanodes to reach the minimum number 1. Safe mode will be turned off automatically.at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1182)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1150)at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:597)at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:576)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:597)at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)

执行
hadoop dfsadmin -safemode leave

ok，再来一遍

125上面；
flume node_nowatch -n collector

126上面：
flume node_nowatch

ok搞定

[root@flume-hadoop-node-1 log]# hadoop fs -ls hdfs://10.129.8.126/flume/Found 2 items-rw-r--r--   3 root supergroup   11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021-rw-r--r--   3 root supergroup          0 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023.tmp[root@flume-hadoop-node-1 log]# hadoop fs -ls hdfs://10.129.8.126/flume/Found 2 items-rw-r--r--   3 root supergroup   11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021-rw-r--r--   3 root supergroup    7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023[root@flume-hadoop-node-1 log]# hadoop fs -ls hdfs://10.129.8.126/flume/Found 2 items-rw-r--r--   3 root supergroup   11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021-rw-r--r--   3 root supergroup    7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023[root@flume-hadoop-node-1 log]# hadoop fs -tail hdfs://10.129.8.126/flume/srcdata20120203-021531413+0800.2440835961446540.00000021\t\t3881/udp\t\t\t# Data Acquisition and Control","timestamp":1328205987177,"pri":"INFO","nanos":100778763829457,"host":"cloudera-node-1","fields":{"AckTag":"20120203-020626329+0800.100777916519457.00000019","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000陋赂\u0010娄","rolltag":"20120203-021531413+0800.2440835961446540.00000021"}} {"body":"msdts1\t\t3882/tcp\t\t\t# DTS Service Port","timestamp":1328205987177,"pri":"INFO","nanos":100778763863457,"host":"cloudera-node-1","fields":{"AckTag":"20120203-020626329+0800.100777916519457.00000019","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000?隆w?","rolltag":"20120203-021531413+0800.2440835961446540.00000021"}} {"body":"msdts1\t\t3882/udp\t\t\t# DTS Service Port","timestamp":1328205987177,"pri":"INFO","nanos":100778763897457,"host":"cloudera-node-1","fields":{"AckTag":"20120203-020626329+0800.100777916519457.00000019","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u00005=?\u0002","rolltag":"20120203-021531413+0800.2440835961446540.00000021"}}

新加flume node
126上面：

flume node_nowatch -n agentAB

flume-master页面上面添加配置
agentAB : text("/var/log/dmesg") | agentSink("10.129.8.125",35853);

OK，没有问题，下面试试默认配置

flume node_nowatch -n agentABC
agentABC : text("/tmp/medcl") | agentSink("10.129.8.125");

这个时候，
node status里面
agentABC agentABC flume-hadoop-node-1 OPENING Fri Feb 03 02:31:11 CST 2012 3 Fri Feb 03 02:32:49 CST 2012

console端报错：

 2012-02-03 02:31:14,823 [logicalNode agentABC-22] INFO connector.DirectDriver: Connector logicalNode agentABC-22 exited with error: /tmp/medcl (No such file or directory)java.io.FileNotFoundException: /tmp/medcl (No such file or directory)at java.io.RandomAccessFile.open(Native Method)at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)at com.cloudera.flume.handlers.debug.TextFileSource.open(TextFileSource.java:75)at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:87)Exception in thread "logicalNode agentABC-22" java.lang.NullPointerExceptionat com.cloudera.flume.handlers.debug.TextFileSource.close(TextFileSource.java:69)at com.cloudera.flume.core.connector.DirectDriver$PumperThread.ensureClosed(DirectDriver.java:183)at com.cloudera.flume.core.connector.DirectDriver$PumperThread.errorCleanup(DirectDriver.java:204)at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:92)

创建文件
echo "hello world" > /tmp/medcl

继续失败着，不能自动恢复，只能重启node

 [root@cloudera-node-1 log]# hadoop dfs -tail hdfs://10.129.8.126/flume/srcdata20120203-023644240+0800.2442108787815540.00000021{"body":"hello world","timestamp":1328207806233,"pri":"INFO","nanos":2442110780978540,"host":"flume-hadoop-node-1","fields":{"AckTag":"20120203-023646196+0800.2442110743929540.00000022","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000\rJ\u0011?","rolltag":"20120203-023644240+0800.2442108787815540.00000021"}}

 flume node_nowatch -n agentABCDagentABCD : text("/tmp/medcl") | agentSink("10.129.8.125");

text sink只能执行一次，后续文件有变化，并不处理

tail就可以实现监听

flume node_nowatch -n collector #如果collector已经关闭，需要重新打开，配置文件在前面flume node_nowatch -n agentABCDEagentABCDE : tail("/tmp/medcl") | agentSink("10.129.8.125");

collector每30秒写一次hadoop，hadoop文件每次新建一个

[root@flume-hadoop-node-1 tmp]# echo "happy new year">>medcl[root@flume-hadoop-node-1 tmp]# hadoop fs -ls hdfs://10.129.8.126/flume/Found 7 items-rw-r--r--   3 root supergroup   11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021-rw-r--r--   3 root supergroup    7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023-rw-r--r--   3 root supergroup     197377 2012-02-03 02:25 /flume/srcdata20120203-022338788+0800.2441323335749540.00000021-rw-r--r--   3 root supergroup        318 2012-02-03 02:38 /flume/srcdata20120203-023644240+0800.2442108787815540.00000021-rw-r--r--   3 root supergroup     761621 2012-02-07 19:00 /flume/srcdata20120207-185754757+0800.2846579304755540.00000021-rw-r--r--   3 root supergroup        336 2012-02-07 19:02 /flume/srcdata20120207-185954947+0800.2846699494856540.00000021-rw-r--r--   3 root supergroup        329 2012-02-07 19:09 /flume/srcdata20120207-190658071+0800.2847122618653540.00000021[root@flume-hadoop-node-1 tmp]# [root@flume-hadoop-node-1 tmp]# hadoop fs -ls hdfs://10.129.8.126/flume/Found 8 items-rw-r--r--   3 root supergroup   11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021-rw-r--r--   3 root supergroup    7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023-rw-r--r--   3 root supergroup     197377 2012-02-03 02:25 /flume/srcdata20120203-022338788+0800.2441323335749540.00000021-rw-r--r--   3 root supergroup        318 2012-02-03 02:38 /flume/srcdata20120203-023644240+0800.2442108787815540.00000021-rw-r--r--   3 root supergroup     761621 2012-02-07 19:00 /flume/srcdata20120207-185754757+0800.2846579304755540.00000021-rw-r--r--   3 root supergroup        336 2012-02-07 19:02 /flume/srcdata20120207-185954947+0800.2846699494856540.00000021-rw-r--r--   3 root supergroup        329 2012-02-07 19:09 /flume/srcdata20120207-190658071+0800.2847122618653540.00000021-rw-r--r--   3 root supergroup        337 2012-02-07 19:12 /flume/srcdata20120207-190929343+0800.2847273890577540.00000021[root@flume-hadoop-node-1 tmp]# hadoop fs -get hdfs://10.129.8.126/flume/srcdata20120207-190929343+0800.2847273890577540.00000021 /tmp/lo2

如果是替换文件内容，不是追加，第一条记录会造成丢失，此处应该特别注意（bug？)

[root@flume-hadoop-node-1 tmp]# echo "who is your daddy?">medcl[root@flume-hadoop-node-1 tmp]# hadoop fs -ls hdfs://10.129.8.126/flume/Found 8 items-rw-r--r--   3 root supergroup   11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021-rw-r--r--   3 root supergroup    7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023-rw-r--r--   3 root supergroup     197377 2012-02-03 02:25 /flume/srcdata20120203-022338788+0800.2441323335749540.00000021-rw-r--r--   3 root supergroup        318 2012-02-03 02:38 /flume/srcdata20120203-023644240+0800.2442108787815540.00000021-rw-r--r--   3 root supergroup     761621 2012-02-07 19:00 /flume/srcdata20120207-185754757+0800.2846579304755540.00000021-rw-r--r--   3 root supergroup        336 2012-02-07 19:02 /flume/srcdata20120207-185954947+0800.2846699494856540.00000021-rw-r--r--   3 root supergroup        329 2012-02-07 19:09 /flume/srcdata20120207-190658071+0800.2847122618653540.00000021-rw-r--r--   3 root supergroup        337 2012-02-07 19:12 /flume/srcdata20120207-190929343+0800.2847273890577540.00000021

再追加一条数据

[root@flume-hadoop-node-1 tmp]# echo "here is a new line">>medcl[root@flume-hadoop-node-1 tmp]# hadoop fs -ls hdfs://10.129.8.126/flume/Found 9 items-rw-r--r--   3 root supergroup   11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021-rw-r--r--   3 root supergroup    7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023-rw-r--r--   3 root supergroup     197377 2012-02-03 02:25 /flume/srcdata20120203-022338788+0800.2441323335749540.00000021-rw-r--r--   3 root supergroup        318 2012-02-03 02:38 /flume/srcdata20120203-023644240+0800.2442108787815540.00000021-rw-r--r--   3 root supergroup     761621 2012-02-07 19:00 /flume/srcdata20120207-185754757+0800.2846579304755540.00000021-rw-r--r--   3 root supergroup        336 2012-02-07 19:02 /flume/srcdata20120207-185954947+0800.2846699494856540.00000021-rw-r--r--   3 root supergroup        329 2012-02-07 19:09 /flume/srcdata20120207-190658071+0800.2847122618653540.00000021-rw-r--r--   3 root supergroup        337 2012-02-07 19:12 /flume/srcdata20120207-190929343+0800.2847273890577540.00000021-rw-r--r--   3 root supergroup          0 2012-02-07 19:19 /flume/srcdata20120207-191702865+0800.2847727413000540.00000021.tmp [root@flume-hadoop-node-1 tmp]# hadoop fs -ls hdfs://10.129.8.126/flume/Found 9 items-rw-r--r--   3 root supergroup   11829304 2012-02-03 02:17 /flume/srcdata20120203-021531413+0800.2440835961446540.00000021-rw-r--r--   3 root supergroup    7080210 2012-02-03 02:18 /flume/srcdata20120203-021605232+0800.2440869780410540.00000023-rw-r--r--   3 root supergroup     197377 2012-02-03 02:25 /flume/srcdata20120203-022338788+0800.2441323335749540.00000021-rw-r--r--   3 root supergroup        318 2012-02-03 02:38 /flume/srcdata20120203-023644240+0800.2442108787815540.00000021-rw-r--r--   3 root supergroup     761621 2012-02-07 19:00 /flume/srcdata20120207-185754757+0800.2846579304755540.00000021-rw-r--r--   3 root supergroup        336 2012-02-07 19:02 /flume/srcdata20120207-185954947+0800.2846699494856540.00000021-rw-r--r--   3 root supergroup        329 2012-02-07 19:09 /flume/srcdata20120207-190658071+0800.2847122618653540.00000021-rw-r--r--   3 root supergroup        337 2012-02-07 19:12 /flume/srcdata20120207-190929343+0800.2847273890577540.00000021-rw-r--r--   3 root supergroup        341 2012-02-07 19:19 /flume/srcdata20120207-191702865+0800.2847727413000540.00000021[root@flume-hadoop-node-1 tmp]# hadoop fs -tail hdfs://10.129.8.126/flume/srcdata20120207-191702865+0800.2847727413000540.00000021{"body":"here is a new line","timestamp":1328613446703,"pri":"INFO","nanos":2847751251273540,"host":"flume-hadoop-node-1","fields":{"AckTag":"20120207-191720960+0800.2847745508415540.00000025","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000/rN?","tailSrcFile":"medcl","rolltag":"20120207-191702865+0800.2847727413000540.00000021"}}

果然，数据丢了一条了。

ok，前面提到了flume使用3种工作模式来保证数据的可靠性与可用性：
1.End2End，2端确认，失败会自动重试（重试次数多少，重试失败之后怎样处理，还要继续研究）
agentE2ESink[("machine"[,port])]

2.DiskFailover，失败写本地磁盘，周期性检查，collector可用的时候，自动重做任务。
agentDFOSink[("machine"[,port])]

3.高效模式，collector失败就丢弃日志，够狠够绝
agentBESink[("machine"[,port])]

前面使用到的agentSink，是第一种End2End的别名，效果和End2End一样。

多收集器的配置

多个collector能够提高吞吐量，因为日志收集都是平行，前面提到过，为保证可靠性，如果collector挂了，agent需要写本地磁盘，然后周期性的去重新连接collector，另外，日志收集停止了，后面的日志处理与分析也歇菜了，这个可不行的。
多个collector就可以解决这个问题，汗！

另外多个collector中，如果其中一个挂了，agent应该是能够自动切换的，怎么配呢？

使用failover chains，

agentA : src | agentE2EChain("collectorA:35853","collectorB:35853");agentB : src | agentE2EChain("collectorA:35853","collectorC:35853");agentC : src | agentE2EChain("collectorB:35853","collectorA:35853");agentD : src | agentE2EChain("collectorB:35853","collectorC:35853");agentE : src | agentE2EChain("collectorC:35853","collectorA:35853");agentF : src | agentE2EChain("collectorC:35853","collectorB:35853");collectorA : collectorSource(35853) | collectorSink("hdfs://...","src");collectorB : collectorSource(35853) | collectorSink("hdfs://...","src");collectorC : collectorSource(35853) | collectorSink("hdfs://...","src");

如上配置，chain指定了2个，第一个collector失败了之后，自动切换使用第二个。

自动FailoverChain，主要是通过使用特殊的source和sink名字（多master下不适用）

source使用：
autoCollectorSource

sink使用：
autoE2EChain, autoDFOChain, or autoBEChain

Logical Configurations
一个physical node包含若干个logical node，logical node又分为：logical sources 和logical sinks ，使用flow来隔离nodes和分组

logical node允许一个JVM实例包含多个logical nodes，实现在一个JVM上跑多个Source和Sink的线程。

每个logical node的名称必须唯一，包括physical node 名称或者 host名称都不能相同

logical定义分两步，

1.定义node类型
agent1 : _source_ | autoBEChain ;
collector1 : autoCollectorSource | collectorSink("hdfs://....") ;

2.mapping logical node和 physical node
map host1 agent1
map host2 collector1

3.解除一个logical节点
decommission agent1

试试

125上 1004  cd /tmp/ 1005  ls 1006  rm -rif flume-* 1007  /etc/init.d/flume-master restart 1008  /etc/init.d/flume-node star

126上

/etc/init.d/flume-node star

flume master页面
config：

agent1 : tail("/tmp/medcl") | autoBEChain ;collector1 : autoCollectorSource | collectorSink("hdfs://10.129.8.126/flume/","medcl") ;

注：主机名-ip
cloudera-node-1：10.129.8.126
flume-hadoop-node-1：10.129.8.125

raw command:

command: maparguments:10.129.8.125 agent1#flume-hadoop-node-1 agent1 command: map arguments: 10.129.8.126 collector1#cloudera-node-1 collector1试试解除map 10.129.8.125 agent2 decommission  agent2

（注意空格，decommission两端不能有空格）

或者unmap和map操作来移动logicalnode

unmap host2 collector1map host3 collector1

（
抓包得到请求为：
curl -XPOST http://10.129.8.125:35871/mastersubmit.jsp -d'cmd=unmap&args=10.129.8.125+agent1'
）

注：logical sources和logical sinks在多master下不适用

通过logical source和logical sink可以在不知道具体物理节点的时候就进行流程的配置，flume有一种翻译的机制，会自动将logical节点名称替换成实际的主机名和端口
事实上，autoSinks和auto-Chain也是这样来实现的。

Flow 隔离，（注，多master下也不适用，悲催啊）

假设你需要收集一个物理机的多种数据，并存放到不同的地方，一种方式是对所有的数据打上tag，通过同一个管道来传数据，然后通过后处理来分离数据

另一种是在整个传输过程中通过将两两种数据隔离，避免后处理的产生

Flume两种都支持，并且延时很低，通过引入flow的概念，将节点进行分组，配置方式如下:
flume master页面：
raw commands

命令：config
参数：[logincal node] [flow name] fooSrc autoBEChain

实际例子：

config AgentC myflow tail("/tmp/medcl") autoBEChainconfig CollectorC myflow autoCollectorSource collectorSink("hdfs://10.129.8.126/flume/","medcl_flow") map 10.129.8.125 AgentC map 10.129.8.126 CollectorC

!!!!

------------
1.问题：
fail( "logical node not mapped to physical node yet" )

1.使用主机名来做map，node status显示的是什么名称，map的时候就用什么名称
2.先map好logical node，然后再更新config配置

正常工作的配置，

map cloudera-node-1  agent1map flume-hadoop-node-1 collector1 agent1 : tail("/tmp/medcl") | agentSink("10.129.8.125",35853);collector1 : collectorSource(35853) | collectorSink("hdfs://10.129.8.126/flume/","medcl");

!!!!

多master配置
多master之间自动同步，一个master挂了，其下node会自动转移到其他master上去。

flume master有两种工作模式：standalone和distributed
如何配置呢？

<property><name>flume.master.servers</name><value>hostA,hostB</value></property>

一个Host则是standalone模式，多个host即distributed模式【分布式模式下，每个master的配置文件必须一样】
另外，每个master必须要配置不同的serverid，如下：

MaserA：<property><name>flume.master.serverid</name><value>0</value></property>MasterB：<property><name>flume.master.serverid</name><value>1</value></property>

【数字和前面配置的服务器列表的下标保持一致即可】
分布式环境下，至少需要3台服务器来保证允许一台失败，如果要允许同时两台挂掉，则至少需要5台服务器
，如果master节点存活率不能超过总数的一半，整个flume master 集群就会block住，无法读写配置信息

flume master存放配置信息的地方叫做：configuration store，支持插拔，本身支持两种实现：
基于内存的：MBCS和基于ZooKeeper的：ZBCS
默认ZBCS，flume内置zookeeper，支持配置到现有的zookeeper集群去

<property><name>flume.master.store</name><value>zookeeper</value></property>

【value值可选：zookeeper或者memory】

ZBCS配置

flume.master.zk.logdir：存储配置文件信息，更新日志，失败信息等
flume.master.zk.server.quorum.port：默认3182，zookeeper server本地监听
flume.master.zk.server.election.port：默认3183，zookeeper server用来寻找其它节点
flume.master.zk.client.port：默认3181，用来与zookeeper server通讯

FlumeMaster的gossip协议支持：

<property><name>flume.master.gossip.port</name><value>57890</value></property>

分布式模型下，flume node的配置也需要调整，从连一个改成连接多个master

<property><name>flume.master.servers</name><value>masterA,masterB,masterC</value></property>

flume node通过定期与master的端口做心跳检测，一旦master 连接失败，自动随机切换到剩下的可以连上的master上去。【master节点通过配置flume.master.heartbeat.port来配置心跳端口】

如果要使用外部的zookeeper，配置如下
conf/flume-site.xml.

<property>  <name>flume.master.zk.use.external</name>  <value>true</value></property> <property>  <name>flume.master.zk.servers</name>  <value>zkServerA:2181,zkServerB:2181,zkServerC:2181</value></property>

Flume与数据源集成
Flume强大就在于灵活，支持各种数据源，结构化的，非结构化的,半结构化等等
三种方式：
pushing、polling、embedding（嵌入flume组件到你的应用程序中）

Push Sources:
syslogTcp,syslogUdp:syslog,syslog-ng日志协议
scribe：scribe日志系统的协议

Polling：
tail,mulitail:监视文件内容的追加信息
exec：适合从现有系统抽取数据
poller：收集来着flume node本身的信息

Flume Event的数据模型
6个主要的字段;
Unix timestamp
Nanosecond timestamp 【纳秒级别的时间戳】
Priority
Source host
Body
Metadata table with an arbitrary number of attribute value pairs.

所有的event都有这几个字段，不过body长度可能为0，metadata表可能为空。

priority :TRACE, DEBUG, INFO, WARN, ERROR, or FATAL，这几种
body：raw格式，默认最大32KB，多余的截掉，通过参数flume.event.max.size.bytes来进行配置

使用event的字段来自定义输出位置
collectorSink("hdfs://namenode/flume/webdata/%H00/", "%{host}-")
%H 为时间timestamp字段里的小时，host为field里面的主机名

快速参考：
[horizontal] %{host}
host
%{nanos}
nanos
%{priority}
priority string
%{body}
body
%%
a % character.
%t
Unix time in millis

时间比较特殊，直接使用，不需要{}
collectorSink("hdfs://namenode/flume/webdata/%Y-%m-%d/%H00/", "web-")

快速参考：
%a

locale’s short weekday name (Mon, Tue, …)

locale’s full weekday name (Monday, Tuesday, …)

locale’s short month name (Jan, Feb,…)

locale’s long month name (January, February,…)

locale’s date and time (Thu Mar 3 23:05:25 2005)

day of month (01)

date; same as %m/%d/%y

hour (00..23)

hour (01..12)

day of year (001..366)

hour ( 0..23)

hour ( 1..12)

month (01..12)

minute (00..59)

locale’s equivalent of am or pm

seconds since 1970-01-01 00:00:00 UTC

second (00..60)

last two digits of year (00..99)

year (2010)

+hhmm numeric timezone (for example, -0400)

输出文件格式

两种方式：
一直是在 flume-site.xml里面设置默认值，另外是由特定的sink来决定

1.flume-site.xml
flume.collector.output.format

格式快速参考

avro

Avro Native file format. Default currently is uncompressed.

avrodata

Binary encoded data written in the avro binary format.

avrojson

JSON encoded data generated by avro.

default

a debugging format.

json

JSON encoded data.

log4j

a log4j pattern similar to that used by CDH output pattern.

raw

Event body only. This is most similar to copying a file but does not preserve any uniqifying metadata like host/timestamp/nanos.

syslog

a syslog like text output format.

seqfile

the binary hadoop Sequence file format with WritableEventKeys keys, and WritableEvent as values.

2.分别配置

collectorSink( "dfsdir","prefix"[, rollmillis[, format]])text("file"[,format])formatDfs("hdfs://nn/file" [, format])escapedFormatDfs("hdfs://nn/file" [, format])

压缩seqfile
formatDfs("hdfs://nn/dir/file", seqfile("bzip2"))

HDFS大量小文件与高延迟的处理
Flume两种策略来处理
1.合并小文件到大的文件
2.使用CombinedFileInputFormat

<property>    <name>flume.collector.dfs.compress.codec</name>    <value>None</value>    <description>Writes formatted data compressed in specified codec to    dfs. Value is None, GzipCodec, DefaultCodec (deflate), BZip2Codec,    or any other Codec Hadoop is aware of </description>  </property>

seqfile和avrodata支持内部的压缩，具体再研究

DataFlow定义语言

Fan out，往所有sinks写：
[ console, collectorSink ]

Fail over，当前失败，转移到下一个，尝试候选sink：

< logicalSink("collector1") ? logicalSink("collector2") >

配置样例：

agent1 : source | < logicalSink("collector1") ? logicalSink("collector2") > ;

Roll sink，每隔一段时间，关闭当前实例，创建新的实例，每次会创建新的独立的文件：
roll(millis) sink
配置样例：

roll(1000) [ console, escapedCustomDfs("hdfs://namenode/flume/file-%{rolltag}") ]

Sink Decorators，sink装饰器
Fan out和Failover影响messages去哪里，但不修改数据，如果要过滤数据什么的，使用sink decorator

sink decorator可以做很多事情，如可以给数据流添加属性，可以通过写ahead 日志来确保可靠性，或者通过批量、压缩来提供网络吞吐，抽样甚至轻量级的分析

flumenode: source | intervalSampler(10) sink;
flumenode: source | batch(100) sink;
flumenode: source | batch(100) gzip sink;
collector(15000) { escapedCustomDfs("xxx","yyy-%{rolltag}") }
collector(15000) { [ escapedCustomDfs("xxx","yyy-%{rolltag}"), hbase("aaa", "bbb-%{rolltag}"), elasticSearch("eeee","ffff") ] } 【同时往3个sink里面写数据，可能有些是持久化的，有些是瞬时的，都成功之后，才会确认成功】

node1 : tail("foo") | ackedWriteAhead batch(100) gzip lazyOpen stubbornAppend logicalSink("bar");【write ahead，批量100，gzip压缩】

Metadata支持正则来进行抽取
支持类似select语法来筛选

thriftSink and thriftSource

扩展与插件

http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_semantics_of_flume_extensions

附录真是好啊

http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_flume_source_catalog

 map cloudera-node-1 agent2agent2 : syslogTcp(2012) | agentSink("10.129.8.125",35853); flume node_nowatch -n medclagent2 : syslogTcp(2012) | agentSink("10.129.8.125",35853);

测试syslog信息

1.NC连接nc 10.129.8.126 2012 2.输入syslog消息（遵照格式：http://blog.csdn.net/xcj0535/article/details/4158624）<165>Aug 24 05:34:00 CST 1987 mymachine myproc[10]: %% It'stime to make the do-nuts. %% Ingredients: Mix=OK, Jelly=OK #Devices: Mixer=OK, Jelly_Injector=OK, Frier=OK # Transport:Conveyer1=OK, Conveyer2=OK # %% <1> medcl is back  syslog的格式  下面是一个syslog消息：<30>Oct 9 22:33:20 hlfedora auditd[1787]: The audit daemon is exiting.其中“<30>”是PRI部分，“Oct 9 22:33:20 hlfedora”是HEADER部分，“auditd[1787]: The audit daemon is exiting.”是MSG部分。   [root@cloudera-node-1 ~]# hadoop fs -cat /flume/medcl20120209-221925655+0800.3031470203471540.00000026{"body":"medcl is back","timestamp":1328797314800,"pri":"INFO","nanos":692106386851457,"host":"cloudera-node-1","fields":{"AckTag":"20120209-222148285+0800.692099872659457.00000037","syslogfacility":"\u0001","AckType":"msg","AckChecksum":"\u0000\u0000\u0000\u0000qu锚茫","syslogseverity":"\u0003","rolltag":"20120209-221925655+0800.3031470203471540.00000026"}}  upload到HDFS的文件包含了太多内容raw下 collector2 : syslogTcp( 2013) | collectorSink( "hdfs://10.129.8.126/flume/", "medcl_raw",3000,raw  );  C:\Windows\system32>nc 10.129.8.125 2013<1> i will be back<1> i will be back2<1> i will be back3<1> i will be back4 [root@cloudera-node-1 ~]# hadoop fs -cat /flume/medcl_raw20120209-235000888+0800.3036905435701540.00000069 i will be back i will be back2 i will be back3 i will be back4

.NET Agent 25个线程，结果压趴下了[另外后续测试发现经常无原因socket断开，服务端socket直接挂掉，flume显示error]。
2012-02-10 21:29:44,154 ERROR com.cloudera.flume.core.connector.DirectDriver: Exiting driver logicalNode collector2-20 in error state SyslogTcpSourceThreads | Collector because null

syslogTcp不稳定，果断换thriftRpc作为Source，经测果然很稳定

 thrift-0.6.0.exe -r -gen csharp flume.thrift 2012-02-13 23:36:30,574 [pool-4-thread-1] ERROR server.TSaneThreadPoolServer: Thrift error occurred during processing of message.org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:213)at com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor.process(ThriftFlumeEventServer.java:224)at org.apache.thrift.server.TSaneThreadPoolServer$WorkerProcess.run(TSaneThreadPoolServer.java:280)at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)at java.lang.Thread.run(Thread.java:619)

此异常可能是因为服务端和客户端使用了不相同的transport，如framed和buffered不匹配

 collector3 : thriftSource( 2014 )| collectorSink( "hdfs://10.129.8.126/flume/", "medcl_thrift",60000,raw  );collector4 : thriftSource( 2015 )| collectorSink( "hdfs://10.129.8.126/flume/", "medcl_thrift",60000,raw  );collector5 : thriftSource( 2016 )| collectorSink( "hdfs://10.129.8.126/flume/", "medcl_thrift",60000,raw  );collector6 : thriftSource( 2017 )| collectorSink( "hdfs://10.129.8.126/flume/", "medcl_thrift",60000,raw  );collector7 : thriftSource( 2018 )| collectorSink( "hdfs://10.129.8.126/flume/", "medcl_thrift2",30000); map cloudera-node-1 collector7

vi flume-site.xml,添加压缩和默认roll时间

<property>    <name>flume.collector.dfs.compress.gzip</name>    <value>true</value>    <description>Writes compressed output in gzip format to dfs. value is     boolean type, i.e. true/false</description>  </property> <property>    <name>flume.collector.roll.millis</name>    <value>60000</value>    <description>The time (in milliseconds)    between when hdfs files are closed and a new file is opened    (rolled).    </description>  </property>

测试文件模板

collector8 : thriftSource( 2019 )| collectorSink("hdfs://10.129.8.126/flume/app/%{host}/%Y-%m-%d/", "%H%M%S-test1-%t",5000);map cloudera-node-1 collector8 [root@flume-hadoop-node-1 ~]# hadoop fs -lsr hdfs://10.129.8.126/flume/appdrwxr-xr-x   - flume supergroup          0 2012-02-17 00:49 /flume/app/MEDCL-THINKdrwxr-xr-x   - flume supergroup          0 2012-02-17 00:49 /flume/app/MEDCL-THINK/4113221-02-12-rw-r--r--   1 flume supergroup        219 2012-02-17 00:49 /flume/app/MEDCL-THINK/4113221-02-12/203942-test1-12973855419598268720120217-004946767+0800.1305778353827457.00006891

更新

 collector8 : thriftSource( 2019 )| collectorSink("hdfs://10.129.8.126/flume/%{catalog}/2012-%m/%d/", "%a-%{host}-",5000,raw());

结果：

/flume/FileTemplateRaw/2012-11/19/Fri-MEDCL-THINK-20120217-013005302+0800.1308196889416457.00007109

collector8 : thriftSource( 2019 )| collectorSink("hdfs://10.129.8.126/flume/%{catalog}/2012", "",5000,raw());