[streamingpro]开发环境搭建

streamingpro

环境准备

csdn_common

git clone到本地,然后执行mvn install安装到本地仓库

➜  csdn_common git:(master) mvn install[INFO] Building jar: /Users/doctorq/Documents/Developer/git-project/csdn_common/target/csdn-common-1.5-SNAPSHOT-javadoc.jar[INFO][INFO] --- maven-install-plugin:2.4:install (default-install) @ csdn-common ---[INFO] Installing /Users/doctorq/Documents/Developer/git-project/csdn_common/target/csdn-common-1.5-SNAPSHOT.jar to /Users/doctorq/.m2/repository/net/csdn/csdn-common/1.5-SNAPSHOT/csdn-common-1.5-SNAPSHOT.jar[INFO] Installing /Users/doctorq/Documents/Developer/git-project/csdn_common/pom.xml to /Users/doctorq/.m2/repository/net/csdn/csdn-common/1.5-SNAPSHOT/csdn-common-1.5-SNAPSHOT.pom[INFO] Installing /Users/doctorq/Documents/Developer/git-project/csdn_common/target/csdn-common-1.5-SNAPSHOT-sources.jar to /Users/doctorq/.m2/repository/net/csdn/csdn-common/1.5-SNAPSHOT/csdn-common-1.5-SNAPSHOT-sources.jar[INFO] Installing /Users/doctorq/Documents/Developer/git-project/csdn_common/target/csdn-common-1.5-SNAPSHOT-javadoc.jar to /Users/doctorq/.m2/repository/net/csdn/csdn-common/1.5-SNAPSHOT/csdn-common-1.5-SNAPSHOT-javadoc.jar[INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 47.729 s[INFO] Finished at: 2016-10-20T18:37:23+08:00[INFO] Final Memory: 30M/310M[INFO] ------------------------------------------------------------------------

active_orm

git clone到本地,然后执行mvn install安装到本地仓库

ServiceframeworkDispatcher

git clone到本地,然后执行mvn install安装到本地仓库

ServiceFramework

git clone到本地，然后执行mvn install安装到本地仓库

streamingpro

mvn clean package看执行结果，解决相关错误。

比如测试代码中含有以下错误

[ERROR] /Users/doctorq/Documents/Developer/git-project/streamingpro/src/test/java/org/apache/spark/streaming/BasicStreamingOperation.scala:23: error: value clear is not a member of object serviceframework.dispatcher.StrategyDispatcher[INFO]         StrategyDispatcher.clear[INFO]                            ^[ERROR] /Users/doctorq/Documents/Developer/git-project/streamingpro/src/test/java/org/apache/spark/streaming/BasicStreamingOperation.scala:40: error: value clear is not a member of object serviceframework.dispatcher.StrategyDispatcher[INFO]         StrategyDispatcher.clear

测试代码，我直接注释了.如果mvn clean package执行成功的话会生成streamingpro-0.4.1-SNAPSHOT-online-1.6.1.jar

导入到ide

以maven项目方式导入到IDEA

指定不同的profile

mvn clean package -Pdev以dev配置打包

打全量包

mvn clean package -Pdev -Pbuild-distr

版本提升

因为我的环境spark为2.0.1，scala是2.11.8,所以我做了jar包的版本修改

pom.xml

运行

Spark Streaming + Spark SQL 实现配置化ETL流程

continus

下面就可以开始读源码+二次开发了。感谢[祝威廉]同学(https://github.com/allwefantasy)开源这个项目