spark updateStateByKey用法更新状态

因为本人刚开始接触大数据开发，在使用spark做开发过程遇到了一些问题，所以写下来作为笔记。
先把代码贴出来吧。（网上找的一段代码示例）

关于updateStateByKey ：
1.重点：首先会以DStream中的数据进行按key做reduce操作，然后再对各个批次的数据进行累加
2.updateStateByKey 方法中
updateFunc就要传入的参数，他是一个函数。Seq[V]表示当前key对应的所有值，Option[S] 是当前key的历史状态，返回的是新的

object UpdateStateByKeyDemo {  def main(args: Array[String]) {    val conf = new SparkConf().setAppName("UpdateStateByKeyDemo")    val ssc = new StreamingContext(conf,Seconds(20))    //要使用updateStateByKey方法，必须设置Checkpoint。    ssc.checkpoint("/checkpoint/")    val socketLines = ssc.socketTextStream("localhost",9999)    socketLines.flatMap(_.split(",")).map(word=>(word,1)).updateStateByKey( (currValues:Seq[Int],preValue:Option[Int]) =>{           //将目前值相加           val currValueSum = 0           for(currValue <- currValues){               currValueSum += currValue           }           //上面其实可以这样：val currValueSum = currValues.sum，我是为了让读者更直观。           //上面的Int类型都可以用对象类型替换           Some(currValue + preValue.getOrElse(0)) //目前值的和加上历史值    }).print()    ssc.start()    ssc.awaitTermination()    ssc.stop()  }}

源码：

/**   * Return a new "state" DStream where the state for each key is updated by applying   * the given function on the previous state of the key and the new values of each key.   * Hash partitioning is used to generate the RDDs with Spark's default number of partitions.   * @param updateFunc State update function. If `this` function returns None, then   *                   corresponding state key-value pair will be eliminated.   * @tparam S State type   */  def updateStateByKey[S: ClassTag](      updateFunc: (Seq[V], Option[S]) => Option[S]    ): DStream[(K, S)] = ssc.withScope {    updateStateByKey(updateFunc, defaultPartitioner())  }

最终调用的这个：

/**   * Return a new "state" DStream where the state for each key is updated by applying   * the given function on the previous state of the key and the new values of the key.   * org.apache.spark.Partitioner is used to control the partitioning of each RDD.   * @param updateFunc State update function. If `this` function returns None, then   *                   corresponding state key-value pair will be eliminated.   * @param partitioner Partitioner for controlling the partitioning of each RDD in the new   *                    DStream.   * @tparam S State type   */  def updateStateByKey[S: ClassTag](      updateFunc: (Seq[V], Option[S]) => Option[S],      partitioner: Partitioner    ): DStream[(K, S)] = ssc.withScope {    val cleanedUpdateF = sparkContext.clean(updateFunc)    val newUpdateFunc = (iterator: Iterator[(K, Seq[V], Option[S])]) => {      iterator.flatMap(t => cleanedUpdateF(t._2, t._3).map(s => (t._1, s)))    }    updateStateByKey(newUpdateFunc, partitioner, true)  }

其中defaultPartitioner():

private[streaming] def defaultPartitioner(numPartitions: Int = self.ssc.sc.defaultParallelism) = {    new HashPartitioner(numPartitions)  }

我目前项目中的spark版本是1.5的，据说1.6版本中的mapWithState 性能较updateStateByKey提升10倍。有机会了解了解