spark性能调优三--shuffle调优

shuffle原理

一，合并map端输出文件

SparkConf conf = new SparkConf()                .setAppName(Constants.SPARK_APP_NAME_SESSION)                .setMaster("local")                .set("spark.default.paralelism", "500")                .set("spark.locality.wait","10")                .set("spark.storage.memoryFraction","0.5")                .set("spark.shuffle.consolidateFiles", "true")                .set("spark.serializer","org.apache.spark.serializer.KryoSerializer")                .registerKryoClasses(new Class[]{CategorySortKey.class});

二，调节map端缓存大小和reduce端缓存占比

三，HashShuffleManage和SortedShuffleManage的调节