Flume Hello World！

来源：互联网发布：mysql官方编辑：程序博客网时间：2024/05/01 21:53

Flume 是 Cloudera 公司开源出来的一套日志收集系统。模型如下所示：

图中Source，Sink分别代表数据源和数据目的地，channel表示Source和Sink之间的通道。配置文件为/path/to/flume/conf/flume.conf

# Define a memory channel called ch1 on agent1agent1.channels.ch1.type = memory # Define an Avro source called avro-source1 on agent1 and tell it# to bind to 0.0.0.0:41414. Connect it to channel ch1.agent1.sources.avro-source1.channels = ch1agent1.sources.avro-source1.type = avroagent1.sources.avro-source1.bind = 0.0.0.0agent1.sources.avro-source1.port = 41414 # Define a logger sink that simply logs all events it receives# and connect it to the other end of the same channel.agent1.sinks.log-sink1.channel = ch1agent1.sinks.log-sink1.type = hdfsagent1.sinks.log-sink1.hdfs.path = hdfs://qaserver:9001/hello/helloworld  # Finally, now that we've defined all of our components, tell# agent1 which ones we want to activate.agent1.channels = ch1agent1.sources = avro-source1agent1.sinks = log-sink1

启动flume

/path/to/flume/bin/flume-ng agent --conf /path/to/flume/conf/ -f /path/to/flume/conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent1

执行以下命令将./hello.txt内容发送到avro-source1，即本地端口41414。

/path/to/flume/bin/flume-ng avro-client -H localhost -p 41414 -F ./hello.txt

./hello.txt内容如下

Hello World!

查看HDFS文件内容hadoop fs -cat /hello/helloworld/FlumeData.1394013090403ƒšî‘ÓD‘§´Ùache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable…=>ûù7ög           Hello World!

0 0