Spark Transformation —— randomSplit

来源：互联网发布：android 移动网络状态编辑：程序博客网时间：2024/06/03 17:51

def randomSplit(weights: Array[Double], seed: Long = Utils.random.nextLong): Array[RDD[T]]

该函数根据weights权重，将一个RDD切分成多个RDD。该权重参数为一个Double数组,第二个参数为random的种子，基本可忽略。

scala> var rdd = sc.makeRDD(1 to 10,10)rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[16] at makeRDD at :21scala> rdd.collectres6: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)  scala> var splitRDD = rdd.randomSplit(Array(0.1,0.2,0.3,0.4))splitRDD: Array[org.apache.spark.rdd.RDD[Int]] = Array(MapPartitionsRDD[17] at randomSplit at :23, MapPartitionsRDD[18] at randomSplit at :23, MapPartitionsRDD[19] at randomSplit at :23, MapPartitionsRDD[20] at randomSplit at :23)//这里注意：randomSplit的结果是一个RDD数组scala> splitRDD.sizeres8: Int = 4//由于randomSplit的第一个参数weights中传入的值有4个，因此，就会切分成4个RDD,//把原来的rdd按照权重0.1,0.2,0.3,0.4，随机划分到这4个RDD中，权重高的RDD，划分到//的几率就大一些。//注意，权重的总和加起来为1，否则会不正常scala> splitRDD(0).collectres10: Array[Int] = Array(1, 4)scala> splitRDD(1).collectres11: Array[Int] = Array(3)                                                    scala> splitRDD(2).collectres12: Array[Int] = Array(5, 9)scala> splitRDD(3).collectres13: Array[Int] = Array(2, 6, 7, 8, 10)

0 0