搭建Spark源码阅读环境

1.安装JDK1.7、Maven3.2.5、Scala2.10.4 

2.下载解压Spark1.4.0 
下载地址： http://mirrors.cnnic.cn/apache/spark/spark-1.4.0/spark-1.4.0.tgz 

3.下载安装 IntelliJ IDEA 14.1.4，给IDEA安装 Scala 插件，然后导入Spark项目 
 
 

4.在IDEA中编译Spark 
若编译时遇到如下图所示的错误： 
 
在Maven视窗点击“Generate Sources and Update Folders For All Projects”，可解决此问题 
 
在“Generate Sources and Update Folders For All Projects”过程中可能会遇到“Unable to import maven project: See logs for details”错误；可点击Help-->Show Log in Exploere，查看错误日志信息如下： 

2015-08-29 22:59:08,909 [ 105808]   WARN -      #org.jetbrains.idea.maven - embedder class org.jetbrains.idea.maven.project.MavenEmbeddersManager.FOR_FOLDERS_RESOLVE is already used 2015-08-29 22:59:08,916 [ 105815]   WARN - ution.rmi.RemoteProcessSupport - Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space 2015-08-29 22:59:08,923 [ 105822]   WARN - ution.rmi.RemoteProcessSupport - Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space 2015-08-29 22:59:08,923 [ 105822]  ERROR -      #org.jetbrains.idea.maven - Cannot reconnect. java.lang.RuntimeException: Cannot reconnect.at org.jetbrains.idea.maven.server.RemoteObjectWrapper.perform(RemoteObjectWrapper.java:82)at org.jetbrains.idea.maven.server.MavenEmbedderWrapper.customizeForResolve(MavenEmbedderWrapper.java:64)at org.jetbrains.idea.maven.project.MavenProjectsTree.executeWithEmbedder(MavenProjectsTree.java:1366)at org.jetbrains.idea.maven.project.MavenProjectsTree.resolveFolders(MavenProjectsTree.java:1312)at org.jetbrains.idea.maven.project.MavenProjectsProcessorFoldersResolvingTask.perform(MavenProjectsProcessorFoldersResolvingTask.java:39)at org.jetbrains.idea.maven.project.MavenProjectsProcessor.doProcessPendingTasks(MavenProjectsProcessor.java:134)at org.jetbrains.idea.maven.project.MavenProjectsProcessor.access$100(MavenProjectsProcessor.java:30)at org.jetbrains.idea.maven.project.MavenProjectsProcessor$2.run(MavenProjectsProcessor.java:109)at org.jetbrains.idea.maven.utils.MavenUtil$6.run(MavenUtil.java:441)at com.intellij.openapi.application.impl.ApplicationImpl$8.run(ApplicationImpl.java:400)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)at java.util.concurrent.FutureTask.run(FutureTask.java:262)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at java.lang.Thread.run(Thread.java:745)at org.jetbrains.ide.PooledThreadExecutor$1$1.run(PooledThreadExecutor.java:56)Caused by: java.rmi.UnmarshalException: Error unmarshaling return; nested exception is: java.io.EOFExceptionat sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:246)at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)at com.sun.proxy.$Proxy91.set(Unknown Source)at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at com.intellij.execution.rmi.RemoteUtil.invokeRemote(RemoteUtil.java:124)at com.intellij.execution.rmi.RemoteUtil.access$100(RemoteUtil.java:36)at com.intellij.execution.rmi.RemoteUtil$2$1$1.compute(RemoteUtil.java:105)at com.intellij.execution.rmi.RemoteUtil.executeWithClassLoader(RemoteUtil.java:181)at com.intellij.execution.rmi.RemoteUtil$2$1.invoke(RemoteUtil.java:102)at com.sun.proxy.$Proxy91.set(Unknown Source)at org.jetbrains.idea.maven.server.MavenServerManager.create(MavenServerManager.java:175)at org.jetbrains.idea.maven.server.MavenServerManager.create(MavenServerManager.java:71)at org.jetbrains.idea.maven.server.RemoteObjectWrapper.getOrCreateWrappee(RemoteObjectWrapper.java:41)at org.jetbrains.idea.maven.server.MavenServerManager$5.create(MavenServerManager.java:481)at org.jetbrains.idea.maven.server.MavenServerManager$5.create(MavenServerManager.java:469)at org.jetbrains.idea.maven.server.RemoteObjectWrapper.getOrCreateWrappee(RemoteObjectWrapper.java:41)at org.jetbrains.idea.maven.server.MavenEmbedderWrapper.doCustomize(MavenEmbedderWrapper.java:87)at org.jetbrains.idea.maven.server.MavenEmbedderWrapper.access$000(MavenEmbedderWrapper.java:36)at org.jetbrains.idea.maven.server.MavenEmbedderWrapper$2.execute(MavenEmbedderWrapper.java:67)at org.jetbrains.idea.maven.server.RemoteObjectWrapper.perform(RemoteObjectWrapper.java:76)... 15 moreCaused by: java.io.EOFExceptionat java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2598)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:244)... 38 more2015-08-29 22:59:08,924 [ 105823]  ERROR -      #org.jetbrains.idea.maven - IntelliJ IDEA 14.1.4  Build #IU-141.1532.4 2015-08-29 22:59:08,924 [ 105823]  ERROR -      #org.jetbrains.idea.maven - JDK: 1.7.0_80 2015-08-29 22:59:08,924 [ 105823]  ERROR -      #org.jetbrains.idea.maven - VM: Java HotSpot(TM) 64-Bit Server VM 2015-08-29 22:59:08,924 [ 105823]  ERROR -      #org.jetbrains.idea.maven - Vendor: Oracle Corporation 2015-08-29 22:59:08,924 [ 105823]  ERROR -      #org.jetbrains.idea.maven - OS: Windows 8 2015-08-29 22:59:08,925 [ 105824]  ERROR -      #org.jetbrains.idea.maven - Last Action: Maven.UpdateFolders

这种情况应该是GFW的原因，换一个时间段或者使用代理就可以了

5.在IDE中运行master：直接运行类org.apache.spark.deploy.master.Master，若报如下图所示错误 
 
修改项目根路径下的pom.xml，将guava的版本为范围由provided修改为compile 

<dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>14.0.1</version> <scope>compile</scope> </dependency> 

运行成功时，控制台输出如下： 
 

6.在IDE中运行Worker：运行类org.apache.spark.deploy.worker.Worker，并指定参数 --webui-port 8081 spark://192.168.63.1:7077