浅析Spark的RangePartitioner

 Spark版本推进到2.0，其在shuffle时提供的partitioner主要有两种，HashPartitioner与RangePartitioner。本文主要对RangePartitioner的实现作一简单说明。
 partitioner主要实现两个方法

  def numPartitions: Int  def getPartition(key: Any): Int

RangePartitioner实现这两方法主要依赖于数组变量rangeBounds: Array[K] ，数组存放的是排序好的（K类型进行的排序）一序列K值，根据这些值来确定RDD中每一个元素shuffle后的存放的partition，下面来看一下rangeBounds获取的代码

  private var rangeBounds: Array[K] = {    if (partitions <= 1) {      Array.empty    } else {      // This is the sample size we need to have roughly balanced output partitions, capped at 1M.      val sampleSize = math.min(20.0 * partitions, 1e6)      // Assume the input partitions are roughly balanced and over-sample a little bit.      val sampleSizePerPartition = math.ceil(3.0 * sampleSize / rdd.partitions.size).toInt      val (numItems, sketched) = RangePartitioner.sketch(rdd.map(_._1), sampleSizePerPartition)      if (numItems == 0L) {        Array.empty      } else {        // If a partition contains much more than the average number of items, we re-sample from it        // to ensure that enough items are collected from that partition.        val fraction = math.min(sampleSize / math.max(numItems, 1L), 1.0)        val candidates = ArrayBuffer.empty[(K, Float)]        val imbalancedPartitions = mutable.Set.empty[Int]        sketched.foreach { case (idx, n, sample) =>          if (fraction * n > sampleSizePerPartition) {            imbalancedPartitions += idx          } else {            // The weight is 1 over the sampling probability.            val weight = (n.toDouble / sample.size).toFloat            for (key <- sample) {              candidates += ((key, weight))            }          }        }        if (imbalancedPartitions.nonEmpty) {          // Re-sample imbalanced partitions with the desired sampling probability.          val imbalanced = new PartitionPruningRDD(rdd.map(_._1), imbalancedPartitions.contains)          val seed = byteswap32(-rdd.id - 1)          val reSampled = imbalanced.sample(withReplacement = false, fraction, seed).collect()          val weight = (1.0 / fraction).toFloat          candidates ++= reSampled.map(x => (x, weight))        }        RangePartitioner.determineBounds(candidates, partitions)      }    }  }

 排序界线是通过抽样的方式来确定的，也就是在RDD中抽取一下样本，然后根据该样本来确定rangeBounds。具体步骤如下：
1、先确定每个partition中要抽取的样本数量，方式如下：

      val sampleSize = math.min(20.0 * partitions, 1e6)      val sampleSizePerPartition = math.ceil(3.0 * sampleSize / rdd.partitions.size).toInt

该值由shuffle后的partitions与当前rdd的partitions数量共同决定，由于样本都要抽取到driver中进行计算，1e6可以确保不会发生OOM。
2、对于抽取过来的样本，(Long, Array[(Int, Long, Array[K])]) 会以该种格式返回，信息包括RDD元素的个数，partitionId，partition中元素的个数及抽取的样本数组。
3、由个各partition中抽取上来的样本，会计算一下val fraction = math.min(sampleSize / math.max(numItems, 1L), 1.0)因子，该因子主要描述每个partition的大小，及如果一个partition中元素过多，该因子的值就会偏大，通过该因子判断，如果一个partition中的元素过多，会对其进行重新的抽样。最后每个样本都会有一个权重值，该值与partition相关，及同一个partition抽取的样本的权重是一样的。

val imbalanced = new PartitionPruningRDD(rdd.map(_._1), imbalancedPartitions.contains)val seed = byteswap32(-rdd.id - 1)val reSampled = imbalanced.sample(withReplacement = false, fraction, seed).collect()

4、把所有的样本进行排序并计算出一个平均权重，通过不断递增的权重与样本的权重做比较，来筛选出rangeBounds数组中的值。该做法主要是要保证数组中的值多数是来源于权重大的partition。

通过以上步骤来筛选出想要的rangeBounds，并通过rangeBounds完成numPartitions与getPartition(key: Any)方法。