SimpleSkewedGroupByTest

1. spark 在做group可以做不对称的group

2. 设置倾斜率 ratio, numReducers reducer个数

3.用随机数补全byteArr

val ranGen = new Random

 val byteArr = new Array[Byte](valSize)        ranGen.nextBytes(byteArr)

4. 大小为numKVPairs，元素类型为（Int, Array[Byte]）的数组

var result = new Array[(Int, Array[Byte])](numKVPairs)

5.按倾斜比率补全result， ratio/ (numReducers + ratio -1)概率group时由reduce0 处理

val offset = ranGen.nextInt(1000) * numReducers        if (ranGen.nextDouble < ratio / (numReducers + ratio - 1)) {          // give ratio times higher chance of generating key 0 (for reducer 0)          result(i) = (offset, byteArr)        } else {          // generate a key for one of the other reducers          val key = 1 + ranGen.nextInt(numReducers-1) + offset          result(i) = (key, byteArr)        }

6. 下面为验证 落到 reduce(编号0~numReducers-1)的个数，落到reducer0的个数/ pairs1.count 约等于  ratio/ (numReducers + ratio -1)

pairs1.groupByKey(numReducers).map{case (k,v) => (k, v.size)}.collectAsMap)