flume spark streaming配置详解

来源：互联网发布：人机界面简单编程实例编辑：程序博客网时间：2024/05/01 05:47

Apache Flume是一个用来有效地收集，聚集和移动大量日志数据的分布式的，有效的服务。这里我们解释一下怎样配置Flume和Spark Streaming来从Flume获取数据，我们让flume直接将收集到的数据发送到spark streaming去处理

package com.pinganfu.flumesparkimport org.apache.spark.SparkConfimport org.apache.spark.storage.StorageLevelimport org.apache.spark.streaming.flume.FlumeUtilsimport org.apache.spark.streaming.{Milliseconds, StreamingContext}object SparkFlumeEvent{  def main(args: Array[String]) {    val batchInterval = Milliseconds(5000)    val sparkConf = new SparkConf().setAppName("flumetospark").setMaster("local[2]")    val ssc = new StreamingContext(sparkConf, batchInterval)    val stream = FlumeUtils.createStream(ssc,"localhost",33333, StorageLevel.MEMORY_AND_DISK)    stream.count().map(cnt => "Received " + cnt + " flume events." ).print()    ssc.start()    ssc.awaitTermination()  }}

Flume配置

a1.channels = c1a1.sinks = k1a1.sources = r1a1.sinks.k1.type = avroa1.sinks.k1.channel = c1a1.sinks.k1.hostname = localhosta1.sinks.k1.port = 33333 a1.sources.r1.type = netcata1.sources.r1.bind = localhosta1.sources.r1.port = 44444a1.sources.r1.channels = c1 a1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100

启动三个窗口

先启动spark-shell（引入两个依赖包）

bin/spark-shell  --jars lib/spark-streaming-flume_2.10-1.0.0.jar,lib/flume-ng-sdk-1.6.0.jar

然后将上面代码贴进去

再启动flume

bin/flume-ng agent --conf conf --conf-file conf/flume.conf --name a1 -Dflume.root.logger=INFO,console

再启动telnet localhost 44444

发数据

截图如下：

0 0