Maven编译Flume-ng 1.5.0

确保可以上网，因为编译时要下载很多的jar 包

1、安装Maven

下载：apache-maven-3.0.5-bin.tar.gz

解压：[sparkadmin@hadoop4 ~]$ tar -zxvfapache-maven-3.0.5-bin.tar.gz

配置环境变量

[sparkadmin@hadoop4 ~]$ vim .bashrc

export MAVEN_HOME=/home/sparkadmin/apache-maven-3.0.5
export PATH=.:$PATH:$MAVEN_HOME/bin

刷新配置：

[sparkadmin@hadoop4 ~]$ source .bashrc

检查是否安装成功

[sparkadmin@hadoop4 ~]$ mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.102s
[INFO] Finished at: Wed Sep 28 11:49:16 CST 2016
[INFO] Final Memory: 10M/481M
[INFO] ------------------------------------------------------------------------
[ERROR] No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-site, site, post-site, site-deploy, pre-clean, clean, post-clean. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoGoalSpecifiedException

出现上面信息表示maven安装成功。

2、编译Flume

下载：apache-flume-1.5.0-src.tar.gz

[sparkadmin@hadoop4 ~]$ tar -zxvf apache-flume-1.5.0-src.tar.gz

Flume-ng 1.5.0 默认的Hadoop版本是1.0.1，Hbase是0.92.1，在编译时指定Hadoop2，默认的hadoop2是hadoop 2.4.0

加参数：-Phadoop-2

 <hadoop.version>1.0.1</hadoop.version>
 <hbase.version>0.92.1</hbase.version>

编译：

[sparkadmin@hadoop4 apache-flume-1.5.0-src]$ mvn install -Phadoop-2 -DskipTests -Dtar

报错解决办法：

第一个错误

[ERROR] Failed to execute goal on project flume-hdfs-sink: Could not resolve dependencies for project org.apache.flume.flume-ng-sinks:flume-hdfs-sink:jar:1.5.0: Failure to find org.apache.hadoop:hadoop-test:jar:2.4.0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of repo1.maven.org has elapsed or updates are forced -> [Help 1]

apache-flume-1.5.0-src/flume-ng-sinks/flume-hdfs-sink下面的pom.xml中注释掉：

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-test</artifactId>
 <version>test</version>
</dependency>

第二个错误：

[ERROR] Failed to execute goal on project flume-ng-hbase-sink: Could not resolve dependencies for project org.apache.flume.flume-ng-sinks:flume-ng-hbase-sink:jar:1.5.0: Failure to find org.apache.hadoop:hadoop-test:jar:2.4.0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of repo1.maven.org has elapsed or updates are forced -> [Help 1]

apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink下面的pom.xml中注释掉：

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-test</artifactId>
<scope>test</scope>
</dependency>

第三个错误：

[ERROR] Failed to execute goal on project flume-tools: Could not resolve dependencies for project org.apache.flume:flume-tools:jar:1.5.0: Failure to find org.apache.hadoop:hadoop-test:jar:2.4.0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of repo1.maven.org has elapsed or updates are forced -> [Help 1]

apache-flume-1.5.0-src/flume-tools下面的pom.xml中注释掉：

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-test</artifactId>
<scope>test</scope>
</dependency>

第四个错误：

[ERROR] Failed to execute goal on project flume-ng-morphline-solr-sink: Could not resolve dependencies for project org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.5.0: Failed to collect dependencies for [org.apache.flume:flume-ng-core:jar:1.5.0 (compile), org.slf4j:slf4j-api:jar:1.6.1 (compile), org.kitesdk:kite-morphlines-all:pom:0.12.0 (compile?), org.slf4j:jcl-over-slf4j:jar:1.6.1 (provided), org.apache.solr:solr-test-framework:jar:4.3.0 (test), org.kitesdk:kite-morphlines-solr-core:jar:tests:0.12.0 (test), junit:junit:jar:4.10 (test)]: Failed to read artifact descriptor for ua_parser:ua-parser:jar:1.3.0: Could not transfer artifact ua_parser:ua-parser:pom:1.3.0 from/to maven-twttr (http://maven.twttr.com): Failed to transfer file: http://maven.twttr.com/ua_parser/ua-parser/1.3.0/ua-parser-1.3.0.pom. Return code is: 504 , ReasonPhrase:Gateway Timeout. -> [Help 1]

这是因为：ua-parser-1.3.0下载失败

在根目录apache-flume-1.5.0-src 下面的pom.xml 中只需要加一个新的repository即可：

<repository>
  <id>p2.jfrog.org</id>
  <url>http://p2.jfrog.org/libs-releases</url>
</repository>

[sparkadmin@hadoop4 apache-flume-1.5.0-src]$ mvn install -Phadoop-2 -DskipTests -Dtar

............................................

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Flume ...................................... SUCCESS [1.398s]
[INFO] Flume NG SDK ...................................... SUCCESS [2.502s]
[INFO] Flume NG Configuration ............................ SUCCESS [0.308s]
[INFO] Flume NG Core ..................................... SUCCESS [2.131s]
[INFO] Flume NG Sinks .................................... SUCCESS [0.087s]
[INFO] Flume NG HDFS Sink ................................ SUCCESS [1.123s]
[INFO] Flume NG IRC Sink ................................. SUCCESS [0.206s]
[INFO] Flume NG Channels ................................. SUCCESS [0.064s]
[INFO] Flume NG JDBC channel ............................. SUCCESS [0.268s]
[INFO] Flume NG file-based channel ....................... SUCCESS [0.403s]
[INFO] Flume NG Spillable Memory channel ................. SUCCESS [0.204s]
[INFO] Flume NG Node ..................................... SUCCESS [0.272s]
[INFO] Flume NG Embedded Agent ........................... SUCCESS [0.578s]
[INFO] Flume NG HBase Sink ............................... SUCCESS [0.672s]
[INFO] Flume NG ElasticSearch Sink ....................... SUCCESS [0.368s]
[INFO] Flume NG Morphline Solr Sink ...................... SUCCESS [1.567s]
[INFO] Flume NG Kite Dataset Sink ........................ SUCCESS [0.775s]
[INFO] Flume Sources ..................................... SUCCESS [0.038s]
[INFO] Flume Scribe Source ............................... SUCCESS [0.195s]
[INFO] Flume JMS Source .................................. SUCCESS [0.274s]
[INFO] Flume Twitter Source .............................. SUCCESS [0.194s]
[INFO] Flume legacy Sources .............................. SUCCESS [0.036s]
[INFO] Flume legacy Avro source .......................... SUCCESS [0.702s]
[INFO] Flume legacy Thrift Source ........................ SUCCESS [0.183s]
[INFO] Flume NG Clients .................................. SUCCESS [0.029s]
[INFO] Flume NG Log4j Appender ........................... SUCCESS [2.799s]
[INFO] Flume NG Tools .................................... SUCCESS [1.268s]
[INFO] Flume NG distribution ............................. SUCCESS [21.332s]
[INFO] Flume NG Integration Tests ........................ SUCCESS [19.271s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 59.916s
[INFO] Finished at: Wed Sep 28 17:43:25 CST 2016
[INFO] Final Memory: 89M/2968M
[INFO] ------------------------------------------------------------------------

出现上面信息表示编译成功。

最后将会在flume-ng-dist/target目录下产生以下的文件：

[sparkadmin@hadoop4 target]$ ll
total 29184
drwxrwxr-x 3 sparkadmin sparkadmin     4096 Sep 28 17:42 apache-flume-1.5.0-bin
-rw-rw-r-- 1 sparkadmin sparkadmin 27315535 Sep 28 17:43 apache-flume-1.5.0-bin.tar.gz
drwxrwxr-x 3 sparkadmin sparkadmin     4096 Sep 28 17:43 apache-flume-1.5.0-src
-rw-rw-r-- 1 sparkadmin sparkadmin  2545097 Sep 28 17:43 apache-flume-1.5.0-src.tar.gz
drwxrwxr-x 2 sparkadmin sparkadmin     4096 Sep 28 17:42 archive-tmp
drwxrwxr-x 3 sparkadmin sparkadmin     4096 Sep 28 17:42 maven-shared-archive-resources
-rw-rw-r-- 1 sparkadmin sparkadmin     1014 Sep 28 17:43 rat.txt