scala并行化集合

scala底层已经实现并行化集合 (但瓶颈会限制在单机上====> 所以 分布式 spark)

reduce求和是采用reduceLeft (从左叠加)
缺点: 速度回随着元素的增加而增加!!!
且叠加时为单线程

优化:使用并行化集合 scala.collection.parallel
parallel

scala> val lst0 = List(1,2,8,5,6,3,0,7,9)lst0: List[Int] = List(1, 2, 8, 5, 6, 3, 0, 7, 9)scala> lst0.reduce(_+_)res17: Int = 41scala> lst0.parres18: scala.collection.parallel.immutable.ParSeq[Int] = ParVector(1, 2, 8, 5, 6, 3, 0, 7, 9)scala> lst0.par.reduce(_+_)res19: Int = 41

par 调用的不再是reduceLeft 而是 放在线程池里 多个线程去计算

并行化集合 每个初始值100都会在每个线程内初始一次
多核 每次分配的线程都不一样 所以结果每次不一定相等

scala> lst0.par.reduce(_+_)res19: Int = 41scala> lst0.par.fold(0)(_+_)res20: Int = 41scala> lst0.par.fold(100)(_+_)res21: Int = 741scala> lst0.par.fold(100)(_+_)res22: Int = 441scala> lst0.par.fold(100)(_+_)res23: Int = 841scala> lst0.par.fold(100)(_+_)res24: Int = 841scala> lst0.par.fold(100)(_+_)res25: Int = 841

scala> val arr = List(List(1,2,3), List(3,4,5), List(2), List(0))scala> arr.aggregate(100)(_+_.sum,_+_)res27: Int = 120scala> arr.aggregate(100)(_+_.sum,_+_)res28: Int = 120scala> arr.aggregate(100)(_+_.sum,_+_)res29: Int = 120//最大只会是420scala> arr.par.aggregate(100)(_+_.sum,_+_)res30: Int = 420//最大只会是420scala> arr.par.aggregate(100)(_+_.sum,_+_)res31: Int = 420scala> arr.par.aggregate(100)(_+_.sum,_+_)res45: Int = 420scala> arr.par.aggregate(100)(_+_.sum,_+_)res46: Int = 320scala> arr.par.aggregate(100)(_+_.sum,_+_)res47: Int = 420scala> arr.par.aggregate(100)(_+_.sum,_+_)