hadoop2.6.0的编译小结

来源:互联网 发布:换发型软件 编辑:程序博客网 时间:2024/06/01 07:54

网上针对hadoop2.x的编译教程很多:这里给几个链接

http://www.centoscn.com/image-text/install/2014/0401/2685.html

http://www.iyunv.com/thread-38384-1-1.html

这里摘抄一些:

编译hadoop2.6.0源码


1、准备工作


1.1、安装jdk、gcc、gcc-c++ 、make、cmake、openssl-devel、ncurses-devel
yum install gcc  
yum intall gcc-c++  
yum install make 
yum install cmake  
yum install openssl-devel  
yum install ncurses-devel


1.2、安装maven
http://mirror.bit.edu.cn/apache/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.zip
tar zxf apache-maven-3.0.5-bin.tar.gz(已编译版本)


mv apache-maven-3.0.5 /usr/local


vi /etc/profile
export M2_HOME=/usr/local/apache-maven-3.0.5
export M2=$M2_HOME/bin
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH="$M2:$PATH"


. /etc/profile


mvn -version


修改配置文件
vi /usr/local/apache-maven-3.0.5/conf/settings.xml


<mirrors></mirros>里添加,其他的不需改动,具体修改如下:


   <mirror>
        <id>nexus-osc</id>
         <mirrorOf>*</mirrorOf>
     <name>Nexusosc</name>
     <url>http://maven.oschina.net/content/groups/public/</url>
   </mirror>
   
   
在<profiles></profiles>内新添加


<profile>
       <id>jdk-1.7</id>
       <activation>
         <jdk>1.7</jdk>
       </activation>
       <repositories>
         <repository>
           <id>nexus</id>
           <name>local private nexus</name>
           <url>http://maven.oschina.net/content/groups/public/</url>
           <releases>
             <enabled>true</enabled>
           </releases>
           <snapshots>
             <enabled>false</enabled>
           </snapshots>
         </repository>
       </repositories>
       <pluginRepositories>
         <pluginRepository>
           <id>nexus</id>
          <name>local private nexus</name>
           <url>http://maven.oschina.net/content/groups/public/</url>
           <releases>
             <enabled>true</enabled>
           </releases>
           <snapshots>
             <enabled>false</enabled>
           </snapshots>
         </pluginRepository>
       </pluginRepositories>
     </profile>


1.3、安装protobuf
https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz


tar xvf protobuf-2.5.0.tar.gz 
cd protobuf-2.5.0
./configure 
make 
make install
ldconfig 


vi /etc/profile
export LD_LIBRARY_PATH=~/protobuf-2.5.0
. /etc/profile


protoc --version


1.4、安装ant
http://mirrors.cnnic.cn/apache//ant/binaries/apache-ant-1.9.4-bin.tar.gz
tar xvf  apache-ant-1.9.4-bin.tar.gz
mv apache-ant-1.9.4 /usr/local


vi /etc/profile
export ANT_HOME=/usr/local/apache-ant-1.9.4
export PATH="$JAVA_HOME/bin:$PATH:$HADOOP_PREFIX/bin:$PATH:$M2:$PATH:$ANT_HOME/bin"
. /etc/profile


ant -version


2、编译
进入源码文件夹
cd /hadoop/hadoop-2.6.0-src
mvn clean
mvn package -Pdist,native -DskipTests -Dtar  


......
[INFO] Apache Hadoop Pipes ............................... SUCCESS [11.372s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [7.403s]
[INFO] Apache Hadoop Amazon Web Services support ......... SUCCESS [2:22.913s]
[INFO] Apache Hadoop Client .............................. SUCCESS [11.900s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.660s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [10.072s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [14.665s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.038s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [1:14.706s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 36:42.709s
[INFO] Finished at: Wed Dec 10 00:54:26 CST 2014
[INFO] Final Memory: 135M/433M
[INFO] ------------------------------------------------------------------------

编译结果
/hadoop/hadoop-2.6.0-src/hadoop-dist/target
[root@lsn-linux target]# pwd
/hadoop/hadoop-2.6.0-src/hadoop-dist/target
[root@lsn-linux target]# ls
antrun                    hadoop-2.6.0.tar.gz            maven-archiver
dist-layout-stitching.sh  hadoop-dist-2.6.0.jar          test-dir
dist-tar-stitching.sh     hadoop-dist-2.6.0-javadoc.jar
hadoop-2.6.0              javadoc-bundle-options
[root@lsn-linux target]#

以上来自百度文库http://wenku.baidu.com/link?url=otisA_6BN9ilIGhQt4Rw6umemW9MQ-ZEjOEjHGYKmqrL5x1WLsMfbWZQgxS1nQKDRi3SyVP2VD9UV3SggXCEq4dQ4MmP9JalDKBWVU-kYyO

谢谢!

我在编译中出现的问题:

首先得让虚拟机连外网,我用的是VM_ware,在VM_ware软件的编辑栏里,有虚拟机设置选项,在里面设置为桥接

1、下载的软件版本不对

比如说hadoop2.6.0-src.tar.gz,下成hadoop2.6.0.tar.gz;

apache-maven-bin.tar.gz下成Apache-maven.tar.gz等

另外,我装的是jdk1.8的版本,maven是3.0.5的版本,好像这两个版本不兼容,总是会抛出找不到java 库类的问题。后来把maven升级到最新版3.2.x就可以了。

在编译的过程中,会出错。这里截个图:

1) unexpected end tag: </ul>
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-annotations: MavenReportException: Error while creating archive:
[ERROR] Exit code: 1 - /root/hadoop-2.4.0-src/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceStability.java:27: error: unexpected end tag: </ul>
[ERROR] * </ul>
[ERROR] ^
[ERROR] 
[ERROR] Command line was: /root/jdk1.8.0/jre/../bin/javadoc @options @packages
原因是InterfaceStability.java中的注释问题:



解决办法,将JDK换成1.7版本,使用JDK1.8编译就会遇到上述问题,将</ul>行删除可以解决问题,但后续还会遇到类似的问题,所以不要使用JDK1.8编译Hadoop 2.6.0。

2、编辑maven的settings.xml出问题

建议在Ultraedit上编辑,因为这样会将正文和注释很清楚的区别开。我漏了前后的<profile><\profile>;

3、网速有时候很不给力,需要多重启几下机子


0 0
原创粉丝点击