MultiBroadcastTest

来源:互联网 发布:知微数据华为 编辑:程序博客网 时间:2024/06/03 21:42

1. 发送多个广播,去数据用 barr1.value

 val barr1 = sc.broadcast(arr1)    val barr2 = sc.broadcast(arr2)


val sparkConf = new SparkConf().setAppName("Multi-Broadcast Test")    val sc = new SparkContext(sparkConf)    val slices = if (args.length > 0) args(0).toInt else 2    val num = if (args.length > 1) args(1).toInt else 1000000    val arr1 = new Array[Int](num)    for (i <- 0 until arr1.length) {      arr1(i) = i    }    val arr2 = new Array[Int](num)    for (i <- 0 until arr2.length) {      arr2(i) = i    }    val barr1 = sc.broadcast(arr1)    val barr2 = sc.broadcast(arr2)    val observedSizes: RDD[(Int, Int)] = sc.parallelize(1 to 10, slices).map { _ =>      (barr1.value.size, barr2.value.size)    }    // Collect the small RDD so we can print the observed sizes locally.    observedSizes.collect().foreach(i => println(i))    sc.stop()


0 0