【Spark Java API】Transformation(4)—coalesce、repartition

coalesce

---

官方文档描述：

Return a new RDD that is reduced into `numPartitions` partitions.

函数原型：

def coalesce(numPartitions: Int): JavaRDD[T]def coalesce(numPartitions: Int, shuffle: Boolean): JavaRDD[T]

源码分析：

def coalesce(numPartitions: Int, shuffle: Boolean = false)(implicit ord: Ordering[T] = null)    : RDD[T] = withScope {  if (shuffle) {    /** Distributes elements evenly across output partitions, starting from a random partition. */    val distributePartition = (index: Int, items: Iterator[T]) => {        var position = (new Random(index)).nextInt(numPartitions)        items.map { t =>          // Note that the hash code of the key will just be the key itself. The HashPartitioner          // will mod it with the number of total partitions.          position = position + 1          (position, t)       }    } : Iterator[(Int, T)]    // include a shuffle step so that our upstream tasks are still distributed    new CoalescedRDD(  new ShuffledRDD[Int, T, T](mapPartitionsWithIndex(distributePartition),        new HashPartitioner(numPartitions)),        numPartitions).values    } else {        new CoalescedRDD(this, numPartitions)   }}

从源码中可以看出，当shuffle=false时，由于不进行shuffle，问题就变成parent RDD中哪些partition可以合并在一起，合并的过程依据设置的numPartitons中的元素个数进行合并处理。当shuffle=true时，进行shuffle操作，原理很简单，先是对partition中record进行k-v转换，其中key是由 (new Random(index)).nextInt(numPartitions)+1计算得到，value为record，index 是该 partition 的索引，numPartitions 是 CoalescedRDD 中的 partition 个数，然后 shuffle 后得到 ShuffledRDD， 可以得到均分的 records，再经过复杂算法来建立 ShuffledRDD 和 CoalescedRDD 之间的数据联系，最后过滤掉 key，得到 coalesce 后的结果 MappedRDD。

实例：

List<Integer> data = Arrays.asList(1, 2, 4, 3, 5, 6, 7);JavaRDD<Integer> javaRDD = javaSparkContext.parallelize(data);// shuffle默认是falseJavaRDD<Integer> coalesceRDD = javaRDD.coalesce(2);   System.out.println(coalesceRDD);JavaRDD<Integer> coalesceRDD1 = javaRDD.coalesce(2,true);System.out.println(coalesceRDD1);

注意：

coalesce() 可以将 parent RDD 的 partition 个数进行调整，比如从 5 个减少到 3 个，或者从 5 个增加到 10 个。需要注意的是当 shuffle = false 的时候，是不能增加 partition 个数的（即不能从 5 个变为 10 个）。

repartition

---

官网文档描述：

Return a new RDD that has exactly numPartitions partitions.Can increase or decrease the level of parallelism in this RDD. Internally, this uses a shuffle to redistribute data.If you are decreasing the number of partitions in this RDD, consider using `coalesce`,which can avoid performing a shuffle.

特别需要说明的是，如果使用repartition对RDD的partition数目进行缩减操作，可以使用coalesce函数，将shuffle设置为false，避免shuffle过程，提高效率。

函数原型：

def repartition(numPartitions: Int): JavaRDD[T]

源码分析：

def repartition(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] = withScope {     coalesce(numPartitions, shuffle = true)}

从源码中可以看到repartition等价于 coalesce(numPartitions, shuffle = true)

实例：

List<Integer> data = Arrays.asList(1, 2, 4, 3, 5, 6, 7);JavaRDD<Integer> javaRDD = javaSparkContext.parallelize(data);//等价于 coalesce(numPartitions, shuffle = true)JavaRDD<Integer> repartitionRDD = javaRDD.repartition(2);System.out.println(repartitionRDD);