spark源码解析1

来源：互联网发布：著名搞笑网络歌曲编辑：程序博客网时间：2024/09/21 09:28

part1 - KafkaUtils .createDirectStream

最近的工作中，需要对kafka中的数据做一系列的聚合计算，但数据量又比较大，processtime 大于每个batch的duration，造成每个处理任务不断的被延迟。为解决这个问题，想到将多个计算任务启动多个yarn application，但并不确定启动多个application是否会每个读到相同的一份数据，还是多个共享一份数据，因此，查看了一下源码。

createDirectStream的一个实现如下：

 def createDirectStream[    K: ClassTag,    V: ClassTag,    KD <: Decoder[K]: ClassTag,    VD <: Decoder[V]: ClassTag] (      ssc: StreamingContext,      kafkaParams: Map[String, String],      topics: Set[String]  ): InputDStream[(K, V)] = {    val messageHandler = (mmd: MessageAndMetadata[K, V]) => (mmd.key, mmd.message)    val kc = new KafkaCluster(kafkaParams)    val fromOffsets = getFromOffsets(kc, kafkaParams, topics)    new DirectKafkaInputDStream[K, V, KD, VD, (K, V)](      ssc, kafkaParams, fromOffsets, messageHandler)  }

可以看到。这个方法返回一个DirectKafkaInputDStream

再查看DirectKafkaInputDStream的代码，

class DirectKafkaInputDStream[  K: ClassTag,  V: ClassTag,  U <: Decoder[K]: ClassTag,  T <: Decoder[V]: ClassTag,  R: ClassTag](    ssc_ : StreamingContext,    val kafkaParams: Map[String, String],    val fromOffsets: Map[TopicAndPartition, Long],    messageHandler: MessageAndMetadata[K, V] => R  ) extends InputDStream[R](ssc_) with Logging {  val maxRetries = context.sparkContext.getConf.getInt(    "spark.streaming.kafka.maxRetries", 1)

可见DirectKafkaInputDStream继承了InputDStream，而InputDStream继承自DStream，DStream有个compute方法，这个方法是为指定时间产生RDD的，DirectKafkaInputDStream重写了这个方法，代码如下：

override def compute(validTime: Time): Option[KafkaRDD[K, V, U, T, R]] = {    val untilOffsets = clamp(latestLeaderOffsets(maxRetries))    val rdd = KafkaRDD[K, V, U, T, R](      context.sparkContext, kafkaParams, currentOffsets, untilOffsets, messageHandler)    // Report the record number and metadata of this batch interval to InputInfoTracker.    val offsetRanges = currentOffsets.map { case (tp, fo) =>      val uo = untilOffsets(tp)      OffsetRange(tp.topic, tp.partition, fo, uo.offset)    }    val description = offsetRanges.filter { offsetRange =>      // Don't display empty ranges.      offsetRange.fromOffset != offsetRange.untilOffset    }.map { offsetRange =>      s"topic: ${offsetRange.topic}\tpartition: ${offsetRange.partition}\t" +        s"offsets: ${offsetRange.fromOffset} to ${offsetRange.untilOffset}"    }.mkString("\n")    // Copy offsetRanges to immutable.List to prevent from being modified by the user    val metadata = Map(      "offsets" -> offsetRanges.toList,      StreamInputInfo.METADATA_KEY_DESCRIPTION -> description)    val inputInfo = StreamInputInfo(id, rdd.count, metadata)    ssc.scheduler.inputInfoTracker.reportInfo(validTime, inputInfo)    currentOffsets = untilOffsets.map(kv => kv._1 -> kv._2.offset)    Some(rdd)  }

val rdd = KafkaRDD[K, V, U, T, R]( context.sparkContext, kafkaParams, currentOffsets, untilOffsets, messageHandler) 可见与currentOffsets有关，currentOffsets = fromOffset，fromOffset由Map[TopicAndPartition, Long]这个决定，默认是最新的offset，所以多个application不会共享一个topic

0 0