Hadoop源码编译，hadoop-2.4.1+Lubuntu（Ubuntu）

先从Hadoop官网上面下载最新的源码，我的是2.4.1版本。

1、编译环境：

按照build.txt要求，安装jdk, maven, protocbuff, cmake，基本的依赖包gcc,gcc-c++,make

jdk在官网上下载然后安装，注意在、/etc/profile添加路径，用java -version验证

maven我是直接用apt-geit install安装的只有3.0.5版本不过已经够用了，而且这样安装好像自己配置好了，可以用mvn -v验证安装

protocbuff这个切记不要用apt-gei install 安装，否则会出现找不到的情况，下载地址:https://code.google.com/p/protobuf/downloads/list，要下载2.5.0版本噢，安装参照说明书readme.txt

tar -xvf protobuf-2.5.0.tar.bz2  
cd protobuf-2.5.0  
./configure --prefix=/usr  //很重要,否则找不到protoc
make

make check

make install  

验证protoc --version

2、编译hadoop
a.官方下载hadoop源码
b.由于maven国外服务器可能连不上，先给maven配置一下国内镜像，在maven目录下，conf/settings.xml,
在<mirrors></mirros>里添加，原本的不要动
<mirror>  
  <id>nexus-osc</id>  
  <mirrorOf>*</mirrorOf>  
  <name>Nexusosc</name>  
  <url>http://maven.oschina.net/content/groups/public/</url>  
</mirror> 
c，在<profiles></profiles>内新添加
<profile>  
       <id>jdk-1.7</id>  
       <activation>  
         <jdk>1.4</jdk>  
       </activation>  
       <repositories>  
         <repository>  
           <id>nexus</id>  
           <name>local private nexus</name>  
           <url>http://maven.oschina.net/content/groups/public/</url>  
           <releases>  
             <enabled>true</enabled>  
           </releases>  
           <snapshots>  
             <enabled>false</enabled>  
           </snapshots>  
         </repository>  
       </repositories>  
       <pluginRepositories>  
         <pluginRepository>  
           <id>nexus</id>  
          <name>local private nexus</name>  
           <url>http://maven.oschina.net/content/groups/public/</url>  
           <releases>  
             <enabled>true</enabled>  
           </releases>  
           <snapshots>  
             <enabled>false</enabled>  
           </snapshots>  
         </pluginRepository>  
       </pluginRepositories>  
</profile>  
d.编译clean

进入Hadoop源码目录
cd /opt/hadoop-2.4.1-src
mvn clean install  

e、成功后就直接可以编译源码了

mvn package -Pdist,native -DskipTests -Dtar
f、 如果遇到下面的 错误

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on project hadoop-common: There are test failures.

[ERROR] 
[ERROR] Please refer to /opt/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common
一是命令行，
mvn clean package -Dmaven.test.skip=true
二是写入pom文件(我用的是这种)
<plugin>  
        <groupId>org.apache.maven.plugins</groupId>  
        <artifactId>maven-surefire-plugin</artifactId>  
        <version>2.4.2</version>  
        <configuration>  
          <skipTests>true</skipTests>  
        </configuration>  
</plugin>  

g、 如果再次编译遇到下面的 错误

遇到报错Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common:
这是没有安装zlib1g-dev的关系，这个可以 使用apt-get安装

h、 如果再次编译遇到下面的错误

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (dist) on project hadoop-hdfs-httpfs: An Ant BuildException has occured: exec returned: 2 -> [Help 1]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluen ... oExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-hdfs-httpfs

这是因为/home/pory/workplace/hadoop-2.4.1-src/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads目录下的文件没有下载完全，可以手动下一份匹配版本的文件放在下面，我的是在http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.36/bin/

i、 如果再次编译遇到下面的错误

 Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on

 project hadoop-pipes: An Ant BuildException has occured: exec returned: 1

[ERROR] around Ant part ...<exec dir="/home/pory/workplace/hadoop-2.4.1-src/hadoop-

tools/hadoop-pipes/target/native" executable="cmake" failonerror="true">... @ 5:131 in

 /home/pory/workplace/hadoop-2.4.1-src/hadoop-tools/hadoop-pipes/target/antrun/build-

main.xml

[ERROR] -> [Help 1]

这是没有装，openssl-devl, ncurses-devel, libssl-dev依次安装
j、 现在可以进行编译了
mvn package -Pdist,native -DskipTests -Dtar  

(时间很长，大概10-20分钟)

[INFO] Apache Hadoop Main ................................ SUCCESS [3.925s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [1.771s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [3.452s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.789s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [3.344s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [6.148s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [5.660s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [3.913s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [3.263s]
[INFO] Apache Hadoop Common .............................. SUCCESS [1:59.035s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [7.498s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.096s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [4:22.182s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [31.969s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [9.321s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [7.185s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.120s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.105s]
[INFO] hadoop-yarn-api ................................... SUCCESS [2:00.751s]
[INFO] hadoop-yarn-common ................................ SUCCESS [27.064s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.257s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [9.999s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [15.689s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [3.127s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [4.339s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [11.914s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [1.374s]
[INFO] hadoop-yarn-client ................................ SUCCESS [4.260s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.110s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [2.947s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [2.532s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.147s]
[INFO] hadoop-yarn-project ............................... SUCCESS [12.669s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.185s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [19.701s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [22.842s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [2.952s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [8.085s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [8.187s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [6.061s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [2.444s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [5.023s]
[INFO] hadoop-mapreduce .................................. SUCCESS [10.355s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [4.414s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [14.480s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [3.606s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [5.323s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [3.776s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [2.607s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [2.859s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [15.844s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [5.402s]
[INFO] Apache Hadoop Client .............................. SUCCESS [15.849s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.659s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [5.262s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [15.331s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.165s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [1:10.622s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15:42.745s
[INFO] Finished at: Mon Aug 11 19:55:29 CST 2014
[INFO] Final Memory: 67M/212M

编译后的路径在:hadoop-2.4.1-src/hadoop-dist/target/

总结：hadoop编译出问题一般是环境缺少相应的依赖库，首先严格按照说明安装相应的库版本号也必须符合要求。其次如果真是出问题了，可以用mvn -X查看调试信息，看是哪里出问题，一般都能找到解决方法