Spark Streaming整合kafak
来源:互联网 发布:建筑施工图设计软件 编辑:程序博客网 时间:2024/06/05 22:40
package day05.dimport org.apache.spark.{HashPartitioner, SparkConf}import org.apache.spark.storage.StorageLevelimport org.apache.spark.streaming.kafka.KafkaUtilsimport org.apache.spark.streaming.{Seconds, StreamingContext}object KafkaWordCount { val updateFunc = (iter: Iterator[(String, Seq[Int], Option[Int])]) => { //iter.flatMap(it=>Some(it._2.sum + it._3.getOrElse(0)).map(x=>(it._1,x))) iter.flatMap { case (x, y, z) => Some(y.sum + z.getOrElse(0)).map(i => (x, i)) } } def main(args: Array[String]): Unit = { LoggerLevels.setStreamingLogLevels() val Array(zkQuorum, group, topics, numThreads) = args val sparkConf = new SparkConf().setAppName("KafkaWordCount").setMaster("local[2]") val ssc = new StreamingContext(sparkConf, Seconds(5)) ssc.checkpoint("c://ck2") //"alog-2016-04-16,alog-2016-04-17,alog-2016-04-18" //"Array((alog-2016-04-16, 2), (alog-2016-04-17, 2), (alog-2016-04-18, 2))" val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap val data = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap, StorageLevel.MEMORY_AND_DISK_SER) val words = data.map(_._2).flatMap(_.split(" ")) val wordCounts = words.map((_, 1)).updateStateByKey(updateFunc, new HashPartitioner(ssc.sparkContext.defaultParallelism), true) ssc.start() ssc.awaitTermination() }}
阅读全文
0 0
- Spark Streaming整合kafak
- Spark Structured Streaming、Kafak整合
- Spark Streaming+ FlumeNG整合
- Spark Streaming整合Kafka
- spark streaming整合sparksql
- spark streaming 整合kafka
- 整合Kafka到Spark Streaming
- Spark Streaming + Kafka整合指南
- 整合Kafka到Spark Streaming
- spark Streaming 和drools整合
- Spark Streaming + Kafka整合实例
- Spark-Streaming与Kafka整合
- Spark Streaming 和 Flume-NG的整合
- Spark Streaming和Kafka整合开发指南
- Spark Streaming和Kafka整合开发指南
- Kafka + Spark Streaming+Hive(HBase) 项目整合
- SparkStream:5)Spark streaming+kafka整合实战
- Zookeeper+Kafka+Spark streaming单机整合开发
- netty demo学习文章汇总
- 下拉框问题
- CF C. The Values You Can Make 0-1背包 好题
- 06:浮点数向零舍入
- 【转载】知道创宇研发技能表3.1
- Spark Streaming整合kafak
- Python搭建插件式框架(基于组件开发)【转】
- Git创建远程分支步骤
- STL(三)C++ I/O流技术
- Git详解之七 自定义Git
- ajax请求内部实现
- iOS内购流程文档-Lion
- 设置excel 满足条件的单元格变色
- python之os模块