Hadoop2.6.4版本64位编译

Hadoop2.X版本的64位编译

本文以Hadoop2.6.4版本的64位编译为例，虚拟机系统环境为CentOS
为了解决执行hadoop命令时总有警告：
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

1. 安装过程中遇到的问题

先来说下安装过程中出现的问题
1. 由于系统环境安装的是jdk1.8版本，出现错误

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-annotations: MavenReportException: Error while creating archive:[ERROR] Exit code: 1 - /app/compile/hadoop-2.6.4-src/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceStability.java:27: error: unexpected end tag: </ul>[ERROR] * </ul>[ERROR] ^[ERROR] [ERROR] Command line was: /usr/local/jdk/jre/../bin/javadoc @options @packages[ERROR] [ERROR] Refer to the generated Javadoc files in '/app/compile/hadoop-2.6.4-src/hadoop-common-project/hadoop-annotations/target' dir.

本机尝试在jdk1.8环境下将org/apache/hadoop/classification/InterfaceStability.java:27文件中的</ul>去除掉进行编译，后续报错

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-nfs: MavenReportException: Error while creating archive:[ERROR] Exit code: 1 - /app/compile/hadoop-2.6.4-src/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java:51: warning: no description for @param[ERROR] * @param program[ERROR] ^[ERROR] /app/compile/hadoop-2.6.4-src/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountdBase.java:52: warning: no description for @throws[ERROR] * @throws IOException[ERROR] ^

需要更换jdk为1.7版本，编译Hadoop需要在jdk1.7环境下进行
2. 需要用root账号进行编译

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (create-testdirs) on project hadoop-project: Error executing ant tasks: /app/compile/hadoop-2.6.4-src/hadoop-project/target/antrun/build-main.xml (Permission denied) -> [Help 1][ERROR][ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.

2. 安装依赖包

2.1 安装jdk1.7版本

本机安装jdk1.7.0_79版本，参见Linux下安装JDK

2.2 安装配置maven

下载maven安装包（建议安装3.0以上版本）本机安装选择的是maven3.3.9的二进制包

tar -zxvf apache-maven-3.3.9-bin.tar.gz -C /usr/local/vi /etc/profileexport MAVEN_HOME=/usr/local/apache-maven-3.3.9export PATH=$PATH:$MAVEN_HOME/binsource /etc/profile[root@cyyun lib]# mvn -versionApache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)Maven home: /usr/local/apache-maven-3.3.9Java version: 1.7.0_79, vendor: Oracle CorporationJava home: /usr/local/jdk/jreDefault locale: en_US, platform encoding: UTF-8OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"

此时maven安装成功
设置maven的settings.xml
这里指定阿里云的maven仓库，会大大提高包的下载速度，毕竟中央仓库在国外

<mirror>    <id>nexus-aliyun</id>    <mirrorOf>*</mirrorOf>    <name>Nexusaliyun</name>    <url>http://maven.aliyun.com/nexus/content/groups/public</url></mirror>

2.3 安装Ant

下载Ant1.9.8版本，1.10.x版本需要jdk1.8的环境，暂不使用

tar -zxvf apache-ant-1.9.8-bin.tar.gz -C /usr/local/vi /etc/profileexport ANT_HOME=/usr/local/apache-ant-1.9.8export PATH=$PATH:$ANT_HOME/binsource /etc/profile[root@cyyun ~]# ant -versionApache Ant(TM) version 1.9.8 compiled on December 25 2016

此时Ant安装成功

2.4 安装protobuf

参见protobuf-2.5.0的下载与安装

Hadoop使用protocol buffer进行通信，需要下载和安装protobuf-2.5.0.tar.gz。由于现在 protobuf-2.5.0.tar.gz已经无法在官网中下载了，本人将protobuf-2.5.0.tar.gz上传到百度云盘供大家下载，地址： http://pan.baidu.com/s/1pJlZubT

tar -zxvf protobuf-2.5.0.tar.gz -C /usr/local/cd /usr/local/protobuf-2.5.0./configuremakemake checkmake install

验证是否安装成功

[root@cyyun ~]# protoc --versionlibprotoc 2.5.0

此时protobuf安装成功

2.5 安装依赖包

yum install autoconf automake libtool cmake ncurses-devel openssl-devel gcc gcc-c++ svn 

svn可能不需要，安装上也无影响

2.6 下载Hadoop源码

下载Hadoop2.6.4的源码包hadoop-2.6.4-src.tar.gz，将其解压

tar -zxvf hadoop-2.6.4-src.tar.gz -C /app/compile/

3. 编译Hadoop

3.1 编译Hadoop

在Hadoop源代码的根目录下执行

cd /app/compile/hadoop-2.6.4-src/mvn package -Pdist,native -DskipTests –Dtar

漫长的等待过后编译成功

[INFO] ------------------------------------------------------------------------[INFO] Reactor Summary:[INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [ 11.888 s][INFO] Apache Hadoop Project POM .......................... SUCCESS [  5.357 s][INFO] Apache Hadoop Annotations .......................... SUCCESS [  7.229 s][INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.324 s][INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  4.416 s][INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  7.496 s][INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  6.621 s][INFO] Apache Hadoop Auth ................................. SUCCESS [  7.012 s][INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  4.873 s][INFO] Apache Hadoop Common ............................... SUCCESS [03:20 min][INFO] Apache Hadoop NFS .................................. SUCCESS [ 27.053 s][INFO] Apache Hadoop KMS .................................. SUCCESS [ 18.815 s][INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.546 s][INFO] Apache Hadoop HDFS ................................. SUCCESS [08:35 min][INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 59.205 s][INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 23.495 s][INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 13.669 s][INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.240 s][INFO] hadoop-yarn ........................................ SUCCESS [  0.295 s][INFO] hadoop-yarn-api .................................... SUCCESS [01:59 min][INFO] hadoop-yarn-common ................................. SUCCESS [01:09 min][INFO] hadoop-yarn-server ................................. SUCCESS [  0.533 s][INFO] hadoop-yarn-server-common .......................... SUCCESS [ 31.774 s][INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 57.158 s][INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  8.371 s][INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  9.341 s][INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [01:13 min][INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 16.459 s][INFO] hadoop-yarn-client ................................. SUCCESS [ 12.375 s][INFO] hadoop-yarn-applications ........................... SUCCESS [  0.122 s][INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  3.845 s][INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  2.000 s][INFO] hadoop-yarn-site ................................... SUCCESS [  0.112 s][INFO] hadoop-yarn-registry ............................... SUCCESS [  9.664 s][INFO] hadoop-yarn-project ................................ SUCCESS [  8.961 s][INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.317 s][INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 39.935 s][INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 36.232 s][INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 11.957 s][INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 13.317 s][INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 13.429 s][INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 10.005 s][INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  2.039 s][INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 10.499 s][INFO] hadoop-mapreduce ................................... SUCCESS [ 13.785 s][INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 10.537 s][INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 52.092 s][INFO] Apache Hadoop Archives ............................. SUCCESS [  5.572 s][INFO] Apache Hadoop Rumen ................................ SUCCESS [  9.371 s][INFO] Apache Hadoop Gridmix .............................. SUCCESS [  6.367 s][INFO] Apache Hadoop Data Join ............................ SUCCESS [  4.211 s][INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  2.762 s][INFO] Apache Hadoop Extras ............................... SUCCESS [  3.921 s][INFO] Apache Hadoop Pipes ................................ SUCCESS [  4.534 s][INFO] Apache Hadoop OpenStack support .................... SUCCESS [  9.314 s][INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 14.693 s][INFO] Apache Hadoop Client ............................... SUCCESS [ 17.670 s][INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  6.587 s][INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 22.836 s][INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 14.913 s][INFO] Apache Hadoop Tools ................................ SUCCESS [  0.276 s][INFO] Apache Hadoop Distribution ......................... SUCCESS [01:31 min][INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 29:36 min[INFO] Finished at: 2017-01-11T02:04:55+08:00[INFO] Final Memory: 101M/237M[INFO] ------------------------------------------------------------------------

在/app/compile/hadoop-2.6.4-src/hadoop-dist目录下生成了target目录，编译好的hadoop-2.6.4.tar.gz安装包就在这个目录下

3.2 验证编译是否成功

cd /app/compile/hadoop-2.6.4-src/hadoop-dist/target/hadoop-2.6.4/lib/native[root@cyyun native]# file *libhadoop.a:        current ar archivelibhadooppipes.a:   current ar archivelibhadoop.so:       symbolic link to `libhadoop.so.1.0.0'libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not strippedlibhadooputils.a:   current ar archivelibhdfs.a:          current ar archivelibhdfs.so:         symbolic link to `libhdfs.so.0.0.0'libhdfs.so.0.0.0:   ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

目录中查看libhadoop.so.1.0.0属性，该文件为ELF 64-bit LSB则表示文件成功编译为64位

如果之前已经安装好没有编译过的Hadoop版本，将编译后的/app/compile/hadoop-2.6.4-src/hadoop-dist/target/hadoop-2.6.4/lib/native目录下的所有文件拷贝到Hadoop对应安装目录文件下即可。

[root@cyyun hadoop-2.6.4]# hadoop fs -ls /17/01/11 03:52:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableFound 9 items-rw-r--r--   1 root supergroup         25 2016-12-20 16:44 /1.txt-rw-r--r--   1 root supergroup       1366 2016-11-04 17:36 /README.txtdrwxr-xr-x   - root supergroup          0 2016-11-21 13:47 /flowdrwx------   - root supergroup          0 2016-11-21 11:53 /historydrwxr-xr-x   - root supergroup          0 2017-01-04 22:47 /inputdrwxr-xr-x   - root supergroup          0 2017-01-04 23:00 /outputdrwxr-xr-x   - root supergroup          0 2016-12-24 01:33 /testdrwxr-xr-x   - root supergroup          0 2017-01-09 16:50 /tmpdrwxr-xr-x   - root supergroup          0 2017-01-09 16:50 /user

拷贝替换后，警告消失

[root@cyyun native]# hadoop fs -ls /Found 9 items-rw-r--r--   1 root supergroup         25 2016-12-20 16:44 /1.txt-rw-r--r--   1 root supergroup       1366 2016-11-04 17:36 /README.txtdrwxr-xr-x   - root supergroup          0 2016-11-21 13:47 /flowdrwx------   - root supergroup          0 2016-11-21 11:53 /historydrwxr-xr-x   - root supergroup          0 2017-01-04 22:47 /inputdrwxr-xr-x   - root supergroup          0 2017-01-04 23:00 /outputdrwxr-xr-x   - root supergroup          0 2016-12-24 01:33 /testdrwxr-xr-x   - root supergroup          0 2017-01-09 16:50 /tmpdrwxr-xr-x   - root supergroup          0 2017-01-09 16:50 /user

本文参考:
Hadoop2.X 64位编译
hadoop2.6.4 安装和编译
hadoop 2.6.4 编译
CentOS6.4编译Hadoop2.2.0