spark源码下载及编译

来源：互联网发布：富阳市网络维修招聘编辑：程序博客网时间：2024/05/21 21:35

从http://spark.apache.org/下载spark源码包。

然后

(1)解压

$tar -zxvf spark-x.y.z-bin-hadoopx.y.tgz

(2)修改pom.xml文件

该文件在解压目录中的conf文件夹当中，我的部分修改如下：

<java.version>1.8</java.version>
<hadoop.version>2.7.3</hadoop.version>
<protobuf.version>2.5.0</protobuf.version>
<hbase.version>0.98.9-hadoop2</hbase.version>
<zookeeper.version>3.4.6</zookeeper.version>
<derby.version>10.11.1.1</derby.version>

(3)如果要支持Scala 2.11，请运行以下脚本：-- 该版本太新了，目前还有些组件不支持，所以不建议修改

~spark-2.1.0$sh ./dev/change-version-to-2.11.sh

(4)修改maven环境变量配置

export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=1024M -XX:ReservedCodeCacheSize=1024m"

(5)maven编译

~spark-2.1.0$./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -DskipTests clean package

该过程需要很长时间，也可能会出现build failure，多执行几次。

(6)生成安装包

如果不需要hive，则执行

./dev/make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.7,parquet-provided"

如果需要hive，则执行

./dev/make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pyarn

我的没有要hive，最后编译成功信息如下：

main:[INFO] Executed tasks[INFO] ------------------------------------------------------------------------[INFO] Reactor Summary:[INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 19.393 s][INFO] Spark Project Tags ................................. SUCCESS [ 18.437 s][INFO] Spark Project Sketch ............................... SUCCESS [ 15.209 s][INFO] Spark Project Networking ........................... SUCCESS [ 19.325 s][INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 12.823 s][INFO] Spark Project Unsafe ............................... SUCCESS [ 24.586 s][INFO] Spark Project Launcher ............................. SUCCESS [ 23.378 s][INFO] Spark Project Core ................................. SUCCESS [31:01 min][INFO] Spark Project ML Local Library ..................... SUCCESS [ 39.471 s][INFO] Spark Project GraphX ............................... SUCCESS [32:14 min][INFO] Spark Project Streaming ............................ SUCCESS [33:25 min][INFO] Spark Project Catalyst ............................. SUCCESS [  01:54 h][INFO] Spark Project SQL .................................. SUCCESS [  04:23 h][INFO] Spark Project ML Library ........................... SUCCESS [  01:08 h][INFO] Spark Project Tools ................................ SUCCESS [ 10.568 s][INFO] Spark Project Hive ................................. SUCCESS [25:59 min][INFO] Spark Project REPL ................................. SUCCESS [18:52 min][INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 42.815 s][INFO] Spark Project YARN ................................. SUCCESS [  01:08 h][INFO] Spark Project Assembly ............................. SUCCESS [ 35.475 s][INFO] Spark Project External Flume Sink .................. SUCCESS [ 25.452 s][INFO] Spark Project External Flume ....................... SUCCESS [27:54 min][INFO] Spark Project External Flume Assembly .............. SUCCESS [ 40.121 s][INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [10:36 min][INFO] Spark Project Examples ............................. SUCCESS [06:14 min][INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 34.261 s][INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [40:25 min][INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 15.937 s][INFO] Kafka 0.10 Source for Structured Streaming ......... SUCCESS [18:20 min][INFO] Spark Project Java 8 Tests ......................... SUCCESS [02:35 min][INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 12:49 h[INFO] Finished at: 2017-04-26T21:07:50+08:00[INFO] Final Memory: 126M/904M[INFO] ------------------------------------------------------------------------

可以看到花的时间非常久。编译成功后的安装包是在最早的解压目录下，文件名是spark-2.1.0-bin-hadoop2-without-hive.tgz

0 0