ganglia学习2之Spark编译带有ganglia的版本

来源：互联网发布：mysql可视化工具使用编辑：程序博客网时间：2024/06/02 00:16

更多代码请见：https://github.com/xubo245/SparkLearning

spark源码解读系列环境：spark-1.5.2、hadoop-2.6.0、scala-2.10.4，ganglia-3.6.1

1.理解

Spark除了自带的UI，还支持ganglia，编辑$SPARK_HOME/conf/metrics.properties文件（没有可以拷贝metrics.properties.template）

2.代码：

（1）设置maven编译时的jvm方法区大小

set MAVEN_OPTS=-Xmx2g -XX:MaxPermSize=5120M -XX:ReservedCodeCacheSize=512m

（2）编译

mvn -DskipTests -Pspark-ganglia-lgpl -Phadoop-2.6 -Dhadoop.version=2.6.0 clean package

（3）替换

将spark/lib下的jar包替换

3.结果：

（1）编译好jar包的位置：

（2）编译好的差别：

spark-assembly-1.5.2-hadoop2.6.0.jar\org\apache\spark\metrics\sink

参考

【1】http://spark.apache.org/【2】http://spark.apache.org/docs/1.5.2/programming-guide.html【3】https://github.com/xubo245/SparkLearning【4】http://ganglia.info/

附录：

    hadoop@Master:~/xubo/project/alignment/SparkSW/SparkSW20161026$ ./newSW.sh     (445,UniRef100_Q91G54)                                                              11/9/16 3:20:40 PM =============================================================    -- Gauges ----------------------------------------------------------------------    DAGScheduler.job.activeJobs                 value = 0    DAGScheduler.job.allJobs                 value = 2    DAGScheduler.stage.failedStages                 value = 0    DAGScheduler.stage.runningStages                 value = 0    DAGScheduler.stage.waitingStages                 value = 0    app-20161109152032-0000.driver.BlockManager.disk.diskSpaceUsed_MB                 value = 0    app-20161109152032-0000.driver.BlockManager.memory.maxMem_MB                 value = 33915    app-20161109152032-0000.driver.BlockManager.memory.memUsed_MB                 value = 0    app-20161109152032-0000.driver.BlockManager.memory.remainingMem_MB                 value = 33914    app-20161109152032-0000.driver.jvm.PS-MarkSweep.count                 value = 0    app-20161109152032-0000.driver.jvm.PS-MarkSweep.time                 value = 0    app-20161109152032-0000.driver.jvm.PS-Scavenge.count                 value = 1    app-20161109152032-0000.driver.jvm.PS-Scavenge.time                 value = 75    app-20161109152032-0000.driver.jvm.heap.committed                 value = 16464740352    app-20161109152032-0000.driver.jvm.heap.init                 value = 17179869184    app-20161109152032-0000.driver.jvm.heap.max                 value = 16464740352    app-20161109152032-0000.driver.jvm.heap.usage                 value = 0.061964455569204956    app-20161109152032-0000.driver.jvm.heap.used                 value = 1020228672    app-20161109152032-0000.driver.jvm.non-heap.committed                 value = 59179008    app-20161109152032-0000.driver.jvm.non-heap.init                 value = 24576000    app-20161109152032-0000.driver.jvm.non-heap.max                 value = 318767104    app-20161109152032-0000.driver.jvm.non-heap.usage                 value = 0.1825535673844187    app-20161109152032-0000.driver.jvm.non-heap.used                 value = 58192072    app-20161109152032-0000.driver.jvm.pools.Code-Cache.committed                 value = 2555904    app-20161109152032-0000.driver.jvm.pools.Code-Cache.init                 value = 2555904    app-20161109152032-0000.driver.jvm.pools.Code-Cache.max                 value = 50331648    app-20161109152032-0000.driver.jvm.pools.Code-Cache.usage                 value = 0.03363037109375    app-20161109152032-0000.driver.jvm.pools.Code-Cache.used                 value = 1692672    app-20161109152032-0000.driver.jvm.pools.PS-Eden-Space.committed                 value = 4295491584    app-20161109152032-0000.driver.jvm.pools.PS-Eden-Space.init                 value = 4295491584    app-20161109152032-0000.driver.jvm.pools.PS-Eden-Space.max                 value = 4295491584    app-20161109152032-0000.driver.jvm.pools.PS-Eden-Space.usage                 value = 0.21837350292897234    app-20161109152032-0000.driver.jvm.pools.PS-Eden-Space.used                 value = 938021544    app-20161109152032-0000.driver.jvm.pools.PS-Old-Gen.committed                 value = 11453595648    app-20161109152032-0000.driver.jvm.pools.PS-Old-Gen.init                 value = 11453595648    app-20161109152032-0000.driver.jvm.pools.PS-Old-Gen.max                 value = 11453595648    app-20161109152032-0000.driver.jvm.pools.PS-Old-Gen.usage                 value = 7.152339100979584E-7    app-20161109152032-0000.driver.jvm.pools.PS-Old-Gen.used                 value = 8192    app-20161109152032-0000.driver.jvm.pools.PS-Perm-Gen.committed                 value = 56623104    app-20161109152032-0000.driver.jvm.pools.PS-Perm-Gen.init                 value = 22020096    app-20161109152032-0000.driver.jvm.pools.PS-Perm-Gen.max                 value = 268435456    app-20161109152032-0000.driver.jvm.pools.PS-Perm-Gen.usage                 value = 0.21047666668891907    app-20161109152032-0000.driver.jvm.pools.PS-Perm-Gen.used                 value = 56499400    app-20161109152032-0000.driver.jvm.pools.PS-Survivor-Space.committed                 value = 715653120    app-20161109152032-0000.driver.jvm.pools.PS-Survivor-Space.init                 value = 715653120    app-20161109152032-0000.driver.jvm.pools.PS-Survivor-Space.max                 value = 715653120    app-20161109152032-0000.driver.jvm.pools.PS-Survivor-Space.usage                 value = 0.11485862871666094    app-20161109152032-0000.driver.jvm.pools.PS-Survivor-Space.used                 value = 82198936    app-20161109152032-0000.driver.jvm.total.committed                 value = 16523919360    app-20161109152032-0000.driver.jvm.total.init                 value = 17204445184    app-20161109152032-0000.driver.jvm.total.max                 value = 16783507456    app-20161109152032-0000.driver.jvm.total.used                 value = 1078421080    -- Timers ----------------------------------------------------------------------    DAGScheduler.messageProcessingTime                 count = 78             mean rate = 9.57 calls/second         1-minute rate = 2.20 calls/second         5-minute rate = 2.20 calls/second        15-minute rate = 2.20 calls/second                   min = 0.05 milliseconds                   max = 105.40 milliseconds                  mean = 2.24 milliseconds                stddev = 12.39 milliseconds                median = 0.17 milliseconds                  75% <= 0.24 milliseconds                  95% <= 3.32 milliseconds                  98% <= 39.28 milliseconds                  99% <= 105.40 milliseconds                99.9% <= 105.40 milliseconds    11/9/16 3:20:46 PM =============================================================    -- Gauges ----------------------------------------------------------------------    DAGScheduler.job.activeJobs                 value = 0    DAGScheduler.job.allJobs                 value = 1    DAGScheduler.stage.failedStages                 value = 0    DAGScheduler.stage.runningStages                 value = 0    DAGScheduler.stage.waitingStages                 value = 0    app-20161109152041-0001.driver.BlockManager.disk.diskSpaceUsed_MB                 value = 0    app-20161109152041-0001.driver.BlockManager.memory.maxMem_MB                 value = 33915    app-20161109152041-0001.driver.BlockManager.memory.memUsed_MB                 value = 0    app-20161109152041-0001.driver.BlockManager.memory.remainingMem_MB                 value = 33915    app-20161109152041-0001.driver.jvm.PS-MarkSweep.count                 value = 0    app-20161109152041-0001.driver.jvm.PS-MarkSweep.time                 value = 0    app-20161109152041-0001.driver.jvm.PS-Scavenge.count                 value = 1    app-20161109152041-0001.driver.jvm.PS-Scavenge.time                 value = 75    app-20161109152041-0001.driver.jvm.heap.committed                 value = 16464740352    app-20161109152041-0001.driver.jvm.heap.init                 value = 17179869184    app-20161109152041-0001.driver.jvm.heap.max                 value = 16464740352    app-20161109152041-0001.driver.jvm.heap.usage                 value = 0.15169611051270587    app-20161109152041-0001.driver.jvm.heap.used                 value = 2497637072    app-20161109152041-0001.driver.jvm.non-heap.committed                 value = 60751872    app-20161109152041-0001.driver.jvm.non-heap.init                 value = 24576000    app-20161109152041-0001.driver.jvm.non-heap.max                 value = 318767104    app-20161109152041-0001.driver.jvm.non-heap.usage                 value = 0.18816762221486946    app-20161109152041-0001.driver.jvm.non-heap.used                 value = 59981648    app-20161109152041-0001.driver.jvm.pools.Code-Cache.committed                 value = 2555904    app-20161109152041-0001.driver.jvm.pools.Code-Cache.init                 value = 2555904    app-20161109152041-0001.driver.jvm.pools.Code-Cache.max                 value = 50331648    app-20161109152041-0001.driver.jvm.pools.Code-Cache.usage                 value = 0.038546244303385414    app-20161109152041-0001.driver.jvm.pools.Code-Cache.used                 value = 1940096    app-20161109152041-0001.driver.jvm.pools.PS-Eden-Space.committed                 value = 4295491584    app-20161109152041-0001.driver.jvm.pools.PS-Eden-Space.init                 value = 4295491584    app-20161109152041-0001.driver.jvm.pools.PS-Eden-Space.max                 value = 4295491584    app-20161109152041-0001.driver.jvm.pools.PS-Eden-Space.usage                 value = 0.5623174662935156    app-20161109152041-0001.driver.jvm.pools.PS-Eden-Space.used                 value = 2415429944    app-20161109152041-0001.driver.jvm.pools.PS-Old-Gen.committed                 value = 11453595648    app-20161109152041-0001.driver.jvm.pools.PS-Old-Gen.init                 value = 11453595648    app-20161109152041-0001.driver.jvm.pools.PS-Old-Gen.max                 value = 11453595648    app-20161109152041-0001.driver.jvm.pools.PS-Old-Gen.usage                 value = 7.152339100979584E-7    app-20161109152041-0001.driver.jvm.pools.PS-Old-Gen.used                 value = 8192    app-20161109152041-0001.driver.jvm.pools.PS-Perm-Gen.committed                 value = 58195968    app-20161109152041-0001.driver.jvm.pools.PS-Perm-Gen.init                 value = 22020096    app-20161109152041-0001.driver.jvm.pools.PS-Perm-Gen.max                 value = 268435456    app-20161109152041-0001.driver.jvm.pools.PS-Perm-Gen.usage                 value = 0.2162216305732727    app-20161109152041-0001.driver.jvm.pools.PS-Perm-Gen.used                 value = 58041552    app-20161109152041-0001.driver.jvm.pools.PS-Survivor-Space.committed                 value = 715653120    app-20161109152041-0001.driver.jvm.pools.PS-Survivor-Space.init                 value = 715653120    app-20161109152041-0001.driver.jvm.pools.PS-Survivor-Space.max                 value = 715653120    app-20161109152041-0001.driver.jvm.pools.PS-Survivor-Space.usage                 value = 0.11485862871666094    app-20161109152041-0001.driver.jvm.pools.PS-Survivor-Space.used                 value = 82198936    app-20161109152041-0001.driver.jvm.total.committed                 value = 16525492224    app-20161109152041-0001.driver.jvm.total.init                 value = 17204445184    app-20161109152041-0001.driver.jvm.total.max                 value = 16783507456    app-20161109152041-0001.driver.jvm.total.used                 value = 2557618720    -- Timers ----------------------------------------------------------------------    DAGScheduler.messageProcessingTime                 count = 11             mean rate = 2.08 calls/second         1-minute rate = 2.00 calls/second         5-minute rate = 2.00 calls/second        15-minute rate = 2.00 calls/second                   min = 0.05 milliseconds                   max = 223.38 milliseconds                  mean = 19.91 milliseconds                stddev = 63.48 milliseconds                median = 0.06 milliseconds                  75% <= 0.08 milliseconds                  95% <= 223.38 milliseconds                  98% <= 223.38 milliseconds                  99% <= 223.38 milliseconds                99.9% <= 223.38 milliseconds    hadoop@Master:~/xubo/project/alignment/SparkSW/SparkSW20161026$

0 0