spark性能调优二--JVM调优

来源：互联网发布：apache maven怎么安装编辑：程序博客网时间：2024/06/18 12:01

- 一降低cache占比
- 二调节exector堆外内存与连接等待时长

一，降低cache占比

001.PNG-88.9kB

002.PNG-60.5kB

003.PNG-67.4kB

004.PNG-112.5kB

SparkConf conf = new SparkConf()                .setAppName(Constants.SPARK_APP_NAME_SESSION)                .setMaster("local")                .set("spark.default.paralelism", "500")                .set("spark.locality.wait","10")                .set("spark.storage.memoryFraction","0.5")                .set("spark.serializer","org.apache.spark.serializer.KryoSerializer")                .registerKryoClasses(new Class[]{CategorySortKey.class});

二，调节exector堆外内存与连接等待时长

/usr/local/spark/bin/spark-submit \--class com.ibeifeng.sparkstudy.WordCount \--num-executors 80 \--driver-memory 6g \--executor-memory 6g \--executor-cores 3 \--master yarn-cluster \--queue root.default \--conf spark.yarn.executor.memoryOverhead=2048 \--conf spark.core.connection.ack.wait.timeout=300 \/usr/local/spark/spark.jar \

001.PNG-41.5kB

002.PNG-81.9kB

003.PNG-45.7kB

004.PNG-73.8kB

005.PNG-71.1kB

006.PNG-24.4kB

0 0