Ambari源码编译

Ambari为Apache基金会的一个顶级项目。作为一个为Hadoop系统提供一站式维护的工具，ambari支持通过web页面/RestfulApi的方式进行对集群的一键部署，最重要的是可以自己编写部署脚本部署自己的分布式系统。这个是富有价值的，现在有的公司使用的是自研的分布式框架，想要自动部署大多也是自研一套分布式框架。现在Ambari的出现，缓解了这个问题，用户只需要使用python按照规则编写一套部署在自己系统上的安装/启动脚本，就可以将自己的系统使用Ambari进行一键式部署。

Ambari不仅提供集群组件的部署功能，还提供了集群的监控功能，不过现在的监控只支持Hadoop生态中的服务，如果想使用Ambari进行监控，可能需要二次开发，这里我没有使用Ambari进行过对自研计算框架或其他服务的监控。

这次，我也要系统的学习一下Ambari底层的实现，以及监控功能如何使用。所以，现在先从编译开始，一步一步进行总结。

首先，我的环境是Mac OS，不过其他Linux环境也可以。

1.安装maven 下面是Mac环境的 Linux/Windows可以百度，具体安装大致相似。

    (1) maven 3.0.4下载链接(注意版本一定要3.0.4 或 3.0.5) http://archive.apache.org/dist/maven/maven-3/3.0.4/binaries/apache-maven-3.0.4-bin.tar.gz

    (2) 解包并copy到对应目录

tar zxvf apache-maven-3.0.4-bin.tar.gzcp apache-maven-3.0.4 /opt/local/share/java/maven3.0.4

    (3) vi ~/.bash_profile 文件中依次配置 

M3_HOME=/opt/local/share/java/maven3.0.4PATH=$M3_HOME/bin:$PATHexport M3_HOMEexport PATH

    (4) source ~/.bash_profile 并通过mvn --version 查看版本是否正确

      maven的版本一定要是3.0.4或3.0.5，对于其他的三系版本，可能在Mac系统下会有bug，在编译的时候显示缺少对应的类。详情可以参见maven的jira: https://issues.apache.org/jira/browse/MNG-5787

      具体的错误可能如下:

[INFO] [INFO] >>> maven-eclipse-plugin:2.10:eclipse (default-cli) > generate-resources @ ambari-admin >>>[INFO] [INFO] --- build-helper-maven-plugin:1.8:regex-property (parse-package-version) @ ambari-admin ---[INFO] [INFO] --- build-helper-maven-plugin:1.8:regex-property (parse-package-release) @ ambari-admin ---[INFO] [INFO] --- build-helper-maven-plugin:1.8:parse-version (parse-version) @ ambari-admin ---[INFO] [INFO] --- build-helper-maven-plugin:1.8:regex-property (regex-property) @ ambari-admin ---[INFO] [INFO] --- frontend-maven-plugin:0.0.16:install-node-and-npm (install node and npm) @ ambari-admin ---Downloading: http://maven.oschina.net/content/groups/public/org/codehaus/plexus/plexus-classworlds/2.4/plexus-classworlds-2.4.jarDownloaded: http://maven.oschina.net/content/groups/public/org/codehaus/plexus/plexus-classworlds/2.4/plexus-classworlds-2.4.jar (46 KB at 165.3 KB/sec)[INFO] ------------------------------------------------------------------------[INFO] Reactor Summary:[INFO] [INFO] Ambari Main ........................................ SUCCESS [  1.323 s][INFO] Apache Ambari Project POM .......................... SUCCESS [  0.033 s][INFO] Ambari Web ......................................... SUCCESS [  0.029 s][INFO] Ambari Views ....................................... SUCCESS [  0.270 s][INFO] Ambari Admin View .................................. FAILURE [  3.559 s][INFO] ambari-metrics ..................................... SKIPPED[INFO] Ambari Metrics Common .............................. SKIPPED[INFO] Ambari Metrics Hadoop Sink ......................... SKIPPED[INFO] Ambari Metrics Flume Sink .......................... SKIPPED[INFO] Ambari Metrics Kafka Sink .......................... SKIPPED[INFO] Ambari Metrics Storm Sink .......................... SKIPPED[INFO] Ambari Metrics Collector ........................... SKIPPED[INFO] Ambari Metrics Monitor ............................. SKIPPED[INFO] Ambari Metrics Assembly ............................ SKIPPED[INFO] Ambari Server ...................................... SKIPPED[INFO] Ambari Agent ....................................... SKIPPED[INFO] Ambari Client ...................................... SKIPPED[INFO] Ambari Python Client ............................... SKIPPED[INFO] Ambari Groovy Client ............................... SKIPPED[INFO] Ambari Shell ....................................... SKIPPED[INFO] Ambari Python Shell ................................ SKIPPED[INFO] Ambari Groovy Shell ................................ SKIPPED[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 01:42 min[INFO] Finished at: 2016-01-26T22:15:04+08:00[INFO] Final Memory: 23M/165M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.16:install-node-and-npm (install node and npm) on project ambari-admin: Execution install node and npm of goal com.github.eirslett:frontend-maven-plugin:0.0.16:install-node-and-npm failed: A required class was missing while executing com.github.eirslett:frontend-maven-plugin:0.0.16:install-node-and-npm: org/slf4j/helpers/MarkerIgnoringBase[ERROR] -----------------------------------------------------[ERROR] realm =    plugin>com.github.eirslett:frontend-maven-plugin:0.0.16[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy[ERROR] urls[0] = file:/opt/local/mvnlib/m3/com/github/eirslett/frontend-maven-plugin/0.0.16/frontend-maven-plugin-0.0.16.jar[ERROR] urls[1] = file:/opt/local/mvnlib/m3/com/github/eirslett/frontend-plugin-core/0.0.16/frontend-plugin-core-0.0.16.jar[ERROR] urls[2] = file:/opt/local/mvnlib/m3/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar[ERROR] urls[3] = file:/opt/local/mvnlib/m3/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar[ERROR] urls[4] = file:/opt/local/mvnlib/m3/org/apache/commons/commons-compress/1.5/commons-compress-1.5.jar[ERROR] urls[5] = file:/opt/local/mvnlib/m3/org/tukaani/xz/1.2/xz-1.2.jar[ERROR] urls[6] = file:/opt/local/mvnlib/m3/commons-io/commons-io/1.3.2/commons-io-1.3.2.jar[ERROR] urls[7] = file:/opt/local/mvnlib/m3/org/apache/httpcomponents/httpclient/4.3.1/httpclient-4.3.1.jar[ERROR] urls[8] = file:/opt/local/mvnlib/m3/org/apache/httpcomponents/httpcore/4.3/httpcore-4.3.jar[ERROR] urls[9] = file:/opt/local/mvnlib/m3/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar[ERROR] urls[10] = file:/opt/local/mvnlib/m3/commons-codec/commons-codec/1.6/commons-codec-1.6.jar[ERROR] urls[11] = file:/opt/local/mvnlib/m3/org/codehaus/plexus/plexus-utils/3.0.10/plexus-utils-3.0.10.jar[ERROR] urls[12] = file:/opt/local/mvnlib/m3/javax/enterprise/cdi-api/1.0/cdi-api-1.0.jar[ERROR] urls[13] = file:/opt/local/mvnlib/m3/javax/annotation/jsr250-api/1.0/jsr250-api-1.0.jar[ERROR] urls[14] = file:/opt/local/mvnlib/m3/com/google/guava/guava/10.0.1/guava-10.0.1.jar[ERROR] urls[15] = file:/opt/local/mvnlib/m3/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar[ERROR] urls[16] = file:/opt/local/mvnlib/m3/org/sonatype/sisu/sisu-guice/3.1.0/sisu-guice-3.1.0-no_aop.jar[ERROR] urls[17] = file:/opt/local/mvnlib/m3/aopalliance/aopalliance/1.0/aopalliance-1.0.jar[ERROR] urls[18] = file:/opt/local/mvnlib/m3/org/eclipse/sisu/org.eclipse.sisu.inject/0.0.0.M2a/org.eclipse.sisu.inject-0.0.0.M2a.jar[ERROR] urls[19] = file:/opt/local/mvnlib/m3/asm/asm/3.3.1/asm-3.3.1.jar[ERROR] urls[20] = file:/opt/local/mvnlib/m3/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.jar[ERROR] urls[21] = file:/opt/local/mvnlib/m3/org/apache/maven/plugin-tools/maven-plugin-annotations/3.2/maven-plugin-annotations-3.2.jar[ERROR] urls[22] = file:/opt/local/mvnlib/m3/com/googlecode/slf4j-maven-plugin-log/slf4j-maven-plugin-log/1.0.0/slf4j-maven-plugin-log-1.0.0.jar[ERROR] Number of foreign imports: 1[ERROR] import: Entry[import  from realm ClassRealm[maven.api, parent: null]][ERROR] [ERROR] -----------------------------------------------------: org.slf4j.helpers.MarkerIgnoringBase[ERROR] -> [Help 1][ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginContainerException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR]   mvn <goals> -rf :ambari-admin

2.安装 python 具体过程请自行搜索。

3.Ambari官网下载源码 Ambari 2.2.0下载链接:http://mirrors.hust.edu.cn/apache/ambari/ambari-2.2.0/apache-ambari-2.2.0-src.tar.gz 使用华科的镜像比较快

4.解包源码后 使用mvn 进行编译，由于我是用eclipse进行源码阅读，所以直接使用下面命令生成ecilpse项目。

mvn eclipse:eclipse

由于Ambari运行环境不支持Mac如果需要编译成binary的需要参照官方文档：https://cwiki.apache.org/confluence/display/AMBARI/Build+and+install+Ambari+2.2.0+from+Source选择生成的是rpm 还是 jdeb格式

5.遇到问题如下：

    (1)下载phantomjs-1.9.7-macosx.zip失败 配置的链接被墙。

[INFO] > phantomjs@1.9.10 install /Users/prime7/dc/ambari-2.2.0/ambari-admin/src/main/resources/ui/admin-web/node_modules/phantomjs[INFO] > node install.js[INFO][INFO] Downloading https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-macosx.zip[INFO] Saving to /var/folders/5c/s73d64p57gq4572b4hxjvn040000gn/T/phantomjs/phantomjs-1.9.7-macosx.zip[INFO] Receiving...[INFO][INFO] Error making request.[INFO] Error: socket hang up[INFO]     at createHangUpError (http.js:1472:15)[INFO]     at CleartextStream.socketCloseListener (http.js:1522:23)[INFO]     at CleartextStream.EventEmitter.emit (events.js:117:20)[INFO]     at tls.js:696:10[INFO]     at process._tickCallback (node.js:415:13)[INFO][INFO] Please report this full log at https://github.com/Medium/phantomjs[INFO] npm ERR! phantomjs@1.9.10 install: `node install.js`[INFO] npm ERR! Exit status 1[INFO] npm ERR![INFO] npm ERR! Failed at the phantomjs@1.9.10 install script.[INFO] npm ERR! This is most likely a problem with the phantomjs package,[INFO] npm ERR! not with npm itself.[INFO] npm ERR! Tell the author that this fails on your system:[INFO] npm ERR!     node install.js[INFO] npm ERR! You can get their info via:[INFO] npm ERR!     npm owner ls phantomjs[INFO] npm ERR! There is likely additional logging output above.[INFO][INFO] npm ERR! System Darwin 15.0.0[INFO] npm ERR! command "/Users/prime7/dc/ambari-2.2.0/ambari-admin/src/main/resources/ui/admin-web/node/node" "/Users/prime7/dc/ambari-2.2.0/ambari-admin/src/main/resources/ui/admin-web/node/npm/bin/npm-cli.js" "install" "--unsafe-perm" "--color=false"[INFO] npm ERR! cwd /Users/prime7/dc/ambari-2.2.0/ambari-admin/src/main/resources/ui/admin-web[INFO] npm ERR! node -v v0.10.26[INFO] npm ERR! npm -v 1.4.3[INFO] npm ERR! code ELIFECYCLE[INFO] npm ERR![INFO] npm ERR! Additional logging details can be found in:[INFO] npm ERR!     /Users/prime7/dc/ambari-2.2.0/ambari-admin/src/main/resources/ui/admin-web/npm-debug.log[INFO] npm ERR! not ok code 0[INFO] ------------------------------------------------------------------------[INFO] Reactor Summary:[INFO][INFO] Ambari Main ....................................... SUCCESS [0.740s][INFO] Apache Ambari Project POM ......................... SUCCESS [0.491s][INFO] Ambari Web ........................................ SUCCESS [0.064s][INFO] Ambari Views ...................................... SUCCESS [0.208s][INFO] Ambari Admin View ................................. FAILURE [44.926s][INFO] ambari-metrics .................................... SKIPPED[INFO] Ambari Metrics Common ............................. SKIPPED[INFO] Ambari Metrics Hadoop Sink ........................ SKIPPED[INFO] Ambari Metrics Flume Sink ......................... SKIPPED[INFO] Ambari Metrics Kafka Sink ......................... SKIPPED[INFO] Ambari Metrics Storm Sink ......................... SKIPPED[INFO] Ambari Metrics Collector .......................... SKIPPED[INFO] Ambari Metrics Monitor ............................ SKIPPED[INFO] Ambari Metrics Assembly ........................... SKIPPED[INFO] Ambari Server ..................................... SKIPPED[INFO] Ambari Agent ...................................... SKIPPED[INFO] Ambari Client ..................................... SKIPPED[INFO] Ambari Python Client .............................. SKIPPED[INFO] Ambari Groovy Client .............................. SKIPPED[INFO] Ambari Shell ...................................... SKIPPED[INFO] Ambari Python Shell ............................... SKIPPED[INFO] Ambari Groovy Shell ............................... SKIPPED[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 47.645s[INFO] Finished at: Sat Jan 23 14:42:30 CST 2016[INFO] Final Memory: 23M/229M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.16:npm (npm install) on project ambari-admin: Failed to run task: 'npm install --unsafe-perm --color=false' failed. (error code 1) -> [Help 1][ERROR][ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR][ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException[ERROR][ERROR] After correcting the problems, you can resume the build with the command[ERROR]   mvn <goals> -rf :ambari-admin

解决过程：查看/Users/prime7/dc/ambari-2.2.0/ambari-admin/src/main/resources/ui/admin-web/npm-debug.log  发现下载的对应链接无法访问，上网查了下是被墙了，可以自行百度phantomjs-1.9.7-macosx.zip进行下载。下载后参照上面错误第一行的uri将对应下载后的zip拷贝到对应路径下。这个是国内镜像下载地址 http://npm.taobao.org/dist/phantomjs/

    (2).编译 Ambari Metrics Collector 缺少ambari-metrics-common jar包。参见 https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development

官方做法是

cd ambari-metricsmvn clean package -Dbuild-rpm -DskipTestsFor Ubuntu:cd ambari-metricsmvn clean package -Dbuild-deb -DskipTests

实际在Mac上可以不用生成对应rpm deb 

<pre name="code" class="plain">cd ambari-metrics

<span style="font-family: Arial, Helvetica, sans-serif;">mvn  package -DskipTests</span>

<span style="font-family: Arial, Helvetica, sans-serif;">mvn  install -DskipTests</span>

对应错误：

[INFO] >>> maven-eclipse-plugin:2.10:eclipse (default-cli) @ ambari-metrics-timelineservice >>>Downloading: http://repo.hortonworks.com/content/groups/public/org/apache/ambari/ambari-metrics-common/2.2.0.0.0/ambari-metrics-common-2.2.0.0.0.jarDownloading: http://repo.maven.apache.org/maven2/org/apache/ambari/ambari-metrics-common/2.2.0.0.0/ambari-metrics-common-2.2.0.0.0.jar[INFO] ------------------------------------------------------------------------[INFO] Reactor Summary:[INFO][INFO] Ambari Main ....................................... SUCCESS [0.714s][INFO] Apache Ambari Project POM ......................... SUCCESS [0.041s][INFO] Ambari Web ........................................ SUCCESS [0.047s][INFO] Ambari Views ...................................... SUCCESS [0.137s][INFO] Ambari Admin View ................................. SUCCESS [9.250s][INFO] ambari-metrics .................................... SUCCESS [0.454s][INFO] Ambari Metrics Common ............................. SUCCESS [0.189s][INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [0.496s][INFO] Ambari Metrics Flume Sink ......................... SUCCESS [0.290s][INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [0.093s][INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.376s][INFO] Ambari Metrics Collector .......................... FAILURE [6.079s][INFO] Ambari Metrics Monitor ............................ SKIPPED[INFO] Ambari Metrics Assembly ........................... SKIPPED[INFO] Ambari Server ..................................... SKIPPED[INFO] Ambari Agent ...................................... SKIPPED[INFO] Ambari Client ..................................... SKIPPED[INFO] Ambari Python Client .............................. SKIPPED[INFO] Ambari Groovy Client .............................. SKIPPED[INFO] Ambari Shell ...................................... SKIPPED[INFO] Ambari Python Shell ............................... SKIPPED[INFO] Ambari Groovy Shell ............................... SKIPPED[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 19.204s[INFO] Finished at: Sat Jan 23 15:16:46 CST 2016[INFO] Final Memory: 35M/241M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal on project ambari-metrics-timelineservice: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-timelineservice:jar:2.2.0.0.0: Could not find artifact org.apache.ambari:ambari-metrics-common:jar:2.2.0.0.0 in apache-hadoop (http://repo.hortonworks.com/content/groups/public/) -> [Help 1][ERROR][ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR][ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException[ERROR][ERROR] After correcting the problems, you can resume the build with the command[ERROR]   mvn <goals> -rf :ambari-metrics-timelineservice

(3) phantomjs-1.9.8-macosx.zip 下载失败 具体原因和(1)一致。