SparkStreaming之DStream operations

Transformations on DStreams

           和RDDs一样，各种转换允许数据从inputDstream得到之后进行各种改造。DStreams支持各种转换，他们是基于Spark的RDD的，一些常规的转换如下：

但是和和RDD有点区别，那就是DStream是内部含有多个RDD，它是用HashMap接受流进来的RDD

private[streaming] var generatedRDDs = new HashMap[Time, RDD[T]]()

下面是一个RDD内部数据（环境等变量没表示）集合的情况

而DStream是如下：

大部分的操作在RDD中是有的，下面讲解transform和updateStateByKey

(1)transform函数

先看一下源码

/**   * Return a new DStream in which each RDD is generated by applying a function   * on each RDD of 'this' DStream.   */  def transform[U: ClassTag](transformFunc: (RDD[T], Time) => RDD[U]): DStream[U] = ssc.withScope {    // because the DStream is reachable from the outer object here, and because    // DStreams can't be serialized with closures, we can't proactively check    // it for serializability and so we pass the optional false to SparkContext.clean    val cleanedF = context.sparkContext.clean(transformFunc, false)    val realTransformFunc = (rdds: Seq[RDD[_]], time: Time) => {      assert(rdds.length == 1)      cleanedF(rdds.head.asInstanceOf[RDD[T]], time)    }    new TransformedDStream[U](Seq(this), realTransformFunc)  }

transform函数就是把DStream"打散"，然后让每个RDD进入transformFunc中，然后再组合，下面看看transform

函数的作用，程序如下：

//use the transform to sort the top N    hottestDStream.transform(hottestItemRDD => {      val top3 = hottestItemRDD.map(pair => (pair._2, pair._1)).sortByKey(false).        map(pair => (pair._2, pair._1)).        take(3) //选出前3个      for (item <- top3) {        println(item)      }      hottestItemRDD    }).print

可以看出，我们在transformFunc中进行一些过滤，选择一些我们需要的特征。已达到我们取DStream内部的数据

（2）updateStateByKey

当新的信息流进来的时候，更新状态时，updateStateByKey运算允许你保留任何状态时候的数据。为了用它，你必须要有两个操作
1、定义状态，這个状态可以是任何类型的
2、定义状态更新函数，指定一个函数，這个函数可以用之前状态和新输入的流的数据
对于每个批次，Spark会对所有存在的keys，进行应用状态更新函数。在实时计算的时候，有时候我们需要对全部

的数据进行统计，不管是否有新的数据进入。如果状态更新函数返回None,那么就会消除key-value pair.源码如下：

 /**   * Return a new "state" DStream where the state for each key is updated by applying   * the given function on the previous state of the key and the new values of each key.   * org.apache.spark.Partitioner is used to control the partitioning of each RDD.   * @param updateFunc State update function. Note, that this function may generate a different   *                   tuple with a different key than the input key. Therefore keys may be removed   *                   or added in this way. It is up to the developer to decide whether to   *                   remember the  partitioner despite the key being changed.   * @param partitioner Partitioner for controlling the partitioning of each RDD in the new   *                    DStream   * @param rememberPartitioner Whether to remember the partitioner object in the generated RDDs.   * @param initialRDD initial state value of each key.   * @tparam S State type   */  def updateStateByKey[S: ClassTag](      updateFunc: (Iterator[(K, Seq[V], Option[S])]) => Iterator[(K, S)],      partitioner: Partitioner,      rememberPartitioner: Boolean,      initialRDD: RDD[(K, S)]    ): DStream[(K, S)] = ssc.withScope {     new StateDStream(self, ssc.sc.clean(updateFunc), partitioner,       rememberPartitioner, Some(initialRDD))  }

updateStateByKey输入的是四个函数
 （1）updateFunc: (Iterator[(K, Seq[V], Option[S])]) => Iterator[(K, S)]
 更新函数是 输入(Iterator[(K, Seq[V], Option[S])])类型，返回Iterator[(K, S)]类型
 (Iterator[(K, Seq[V], Option[S])])是一个三元Iterator,分别表示
 K: key
 Seq[v]:一个时间间隔产生Key对应的Value集合
 Option[S]是上个函数的累加值。
 Iterator[(K, S)]是一个二元的Iterator,分别表示：
 K:key
 S:是当前时间结束后得到的累加值
 （2） partitioner
 abstract class Partitioner extends Serializable {
  def numPartitions: Int
  def getPartition(key: Any): Int
}
分区进行hash分区算法，默认是sparkContext.defaultParallelism也就是不设置就跟你的core有关
 （3）rememberPartitioner： Whether to remember the partitioner object in the generated RDDs.
 （4）initialRDD: RDD[(K, S)]
 记录的是 每个key对应的状态（State），S是State type在StreamingContext中初始值为：private var state: StreamingContextState = INITIALIZED
 继续找下去 发现是用Option进行了封装。也它可以是Int,String這样的类型
在把 new StateDStream分析下去，内部是有computeUsingPreviousRDD（进行 cogroup）和

compute（得到之前状态的RDD）

updateStateByKey的效果：

1：在socket端每秒产生一个单词，单词如下：

spark
Streaming
better
than
storm
you
need
it
yes
do
it

2、SparkStreaming程序：

import org.apache.log4j.{Level, Logger}import org.apache.spark.SparkConfimport org.apache.spark.streaming._object statefulNetworkWordCount {  def main(args: Array[String]) {    if (args.length < 2) {      System.err.println("Usage: StatefulNetworkWordCount <hostname> <port>")      System.exit(1)    }    Logger.getLogger("org.apache.spark").setLevel(Level.ERROR)    Logger.getLogger("org.eclipse.jetty.server").setLevel(Level.OFF)    val sparkConf = new SparkConf().setAppName("StatefulNetworkWordCount").setMaster("local[2]")    // Create the context with a 1 second batch size    val ssc = new StreamingContext(sparkConf, Seconds(1))    ssc.checkpoint(".")    // Initial state RDD for mapWithState operation    val initialRDD = ssc.sparkContext.parallelize(List(("hello", 1), ("world", 1)))    // Create a ReceiverInputDStream on target ip:port and count the    // words in input stream of \n delimited test (eg. generated by 'nc')    val lines = ssc.socketTextStream(args(0), args(1).toInt)    val words = lines.flatMap(_.split(" "))    val wordDstream = words.map(x => (x, 1))    // Update the cumulative count using mapWithState    // This will give a DStream made of state (which is the cumulative count of the words)    val mappingFunc = (word: String, one: Option[Int], state: State[Int]) => {      val sum = one.getOrElse(0) + state.getOption.getOrElse(0)      val output = (word, sum)      state.update(sum)      output    }    val stateDstream = wordDstream.mapWithState(      StateSpec.function(mappingFunc).initialState(initialRDD))    stateDstream.print()    ssc.start()    ssc.awaitTermination()  }}

结果如下：

 总结：我们在进行实时监控的时候，我们可以看利用updateStateByKey，来整个监控时间段的流过的每个item。

从而来进行全局的分析。