MultiBroadcastTest
来源:互联网 发布:知微数据华为 编辑:程序博客网 时间:2024/06/03 21:42
1. 发送多个广播,去数据用 barr1.value
val barr1 = sc.broadcast(arr1) val barr2 = sc.broadcast(arr2)
val sparkConf = new SparkConf().setAppName("Multi-Broadcast Test") val sc = new SparkContext(sparkConf) val slices = if (args.length > 0) args(0).toInt else 2 val num = if (args.length > 1) args(1).toInt else 1000000 val arr1 = new Array[Int](num) for (i <- 0 until arr1.length) { arr1(i) = i } val arr2 = new Array[Int](num) for (i <- 0 until arr2.length) { arr2(i) = i } val barr1 = sc.broadcast(arr1) val barr2 = sc.broadcast(arr2) val observedSizes: RDD[(Int, Int)] = sc.parallelize(1 to 10, slices).map { _ => (barr1.value.size, barr2.value.size) } // Collect the small RDD so we can print the observed sizes locally. observedSizes.collect().foreach(i => println(i)) sc.stop()
0 0