Hadoop源码本机编译

在启动hadoop时，我看到有下面警告信息

[hadoop@Yarn-Master ~]$ hadoop/2.6.1/sbin/start-dfs.sh 15/12/14 10:19:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableStarting namenodes on [master]master: starting namenode, logging to /home/hadoop/hadoop/2.6.1/logs/hadoop-hadoop-namenode-Yarn-Master.outmaster: starting datanode, logging to /home/hadoop/hadoop/2.6.1/logs/hadoop-hadoop-datanode-Yarn-Master.outslave_2: ssh: connect to host slave_2 port 22: No route to hostslave_1: ssh: connect to host slave_1 port 22: No route to hostStarting secondary namenodes [0.0.0.0]0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop/2.6.1/logs/hadoop-hadoop-secondarynamenode-Yarn-Master.out15/12/14 10:19:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable[hadoop@Yarn-Master ~]$ 

Google出来可以参考http://hadoop.apache.org/docs/r2.6.1/hadoop-project-dist/hadoop-common/NativeLibraries.html这篇官方文档需要自己编译hadoop的本地库，我查看了一下自己的本地库，果然是64位的，但我的机器是32位的

[hadoop@Yarn-Master ~]$ file hadoop/2.6.1/lib/native/libhadoop.so.1.0.0 hadoop/2.6.1/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped[hadoop@Yarn-Master ~]$ 

自己从网上下载了2.6.1版本的源码包，直接编译，报各种错误，然后各种Google，找到两篇不错的博客：centos下http://blog.pengduncun.com/?p=1208，ubuntu下http://my.oschina.net/laigous/blog/356552，虽然两个系统需要的软件发布版本不同，但要装的软件是一样的，下面均以CentOS为例。

编译需要的软件

首先是官方文档上写明了的

The packages you need to install on the target platform are:

• C compiler (e.g. GNU C Compiler)
• GNU Autools Chain: autoconf, automake, libtool
• zlib-development package (stable version >= 1.2.0)
• openssl-development package(e.g. libssl-dev)

Once you installed the prerequisite packages use the standard hadoop pom.xml file and pass along the native flag to build the native hadoop library:

1、C相关的编译器，可以直接使用 yum groupinstall "Development Tools"来安装，有人说gcc需要4.4版本的，但是我的gcc默认版本就是4.4的所以没验证

[hadoop@Yarn-Master ~]$ gcc --versiongcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16)Copyright (C) 2010 Free Software Foundation, Inc.This is free software; see the source for copying conditions.  There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.[hadoop@Yarn-Master ~]$ 

[hadoop@Yarn-Master ~]$ libtool --versionltmain.sh (GNU libtool) 2.2.6bWritten by Gordon Matzigkeit <[e-mail]>, 1996Copyright (C) 2008 Free Software Foundation, Inc.This is free software; see the source for copying conditions.  There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.[hadoop@Yarn-Master ~]$ automake --versionautomake (GNU automake) 1.11.1Copyright (C) 2009 Free Software Foundation, Inc.License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl-2.0.html>This is free software: you are free to change and redistribute it.There is NO WARRANTY, to the extent permitted by law.Written by Tom Tromey <[e-mail]>       and Alexandre Duret-Lutz <[e-mail]>.[hadoop@Yarn-Master ~]$ autoconf --versionautoconf (GNU Autoconf) 2.63Copyright (C) 2008 Free Software Foundation, Inc.License GPLv2+: GNU GPL version 2 or later<http://gnu.org/licenses/old-licenses/gpl-2.0.html>This is free software: you are free to change and redistribute it.There is NO WARRANTY, to the extent permitted by law.Written by David J. MacKenzie and Akim Demaille.[hadoop@Yarn-Master ~]$ 

2、zlib zlib-devel openssl openssl-devel这四个库可以直接使用yum安装

[root@Yarn-Master hadoop]# yum install zlib zlib-devel openssl openssl-develLoaded plugins: fastestmirrorSetting up Install ProcessLoading mirror speeds from cached hostfile * base: mirrors.zju.edu.cn * extras: mirrors.zju.edu.cn * updates: mirrors.zju.edu.cnPackage zlib-1.2.3-29.el6.i686 already installed and latest versionPackage zlib-devel-1.2.3-29.el6.i686 already installed and latest versionPackage openssl-1.0.1e-42.el6.i686 already installed and latest versionPackage openssl-devel-1.0.1e-42.el6.i686 already installed and latest versionNothing to do[root@Yarn-Master hadoop]# 

常规的java编译需要的软件

maven：http://maven.apache.org/download.cgi，ant：http://ant.apache.org/bindownload.cgi下载软件包以后，解压到/usr/local文件夹下，

[root@Yarn-Master hadoop]# ls -al /usr/local/total 64drwxr-xr-x. 16 root root 4096 Dec 13 16:28 .drwxr-xr-x. 13 root root 4096 Oct 28 09:55 ..drwxr-xr-x.  6 root root 4096 Dec 13 16:15 apache-ant-1.9.6drwxr-xr-x.  6 root root 4096 Dec 11 17:49 apache-maven-3.3.9drwxr-xr-x.  2 root root 4096 Sep 23  2011 bindrwxr-xr-x.  2 root root 4096 Sep 23  2011 etcdrwxr-xr-x.  8 root root 4096 Dec 13 16:28 findbugs-3.0.1drwxr-xr-x.  2 root root 4096 Sep 23  2011 gamesdrwxr-xr-x.  2 root root 4096 Sep 23  2011 includedrwxr-xr-x.  2 root root 4096 Sep 23  2011 libdrwxr-xr-x.  2 root root 4096 Sep 23  2011 libexecdrwxr-xr-x.  5 root root 4096 Dec 11 19:56 protobuf-2.5.0drwxr-xr-x.  5 root root 4096 Dec 11 19:06 protobuf-2.6.1drwxr-xr-x.  2 root root 4096 Sep 23  2011 sbindrwxr-xr-x.  5 root root 4096 Oct 28 09:21 sharedrwxr-xr-x.  2 root root 4096 Sep 23  2011 src[root@Yarn-Master hadoop]# 

然后，再在/etc/profile.d/下建2个shell文件：ant.sh, maven.sh，其中MAVEN的shell文件如下：

[root@Yarn-Master hadoop]# cat /etc/profile.d/maven.sh ## set the maven environment#export MAVEN_HOME=/usr/local/apache-maven-3.3.9export PATH=$MAVEN_HOME/bin:$PATH[root@Yarn-Master hadoop]# 

然后，运行（该命令只在当前用户下生效，如果切换用户需要再执行下，但是重启后会自动执行的）

[root@Yarn-Master hadoop]# source /etc/profile

查看是否安装正确

[root@Yarn-Master hadoop]# which ant/usr/local/apache-ant-1.9.6/bin/ant[root@Yarn-Master hadoop]# which mvn/usr/local/apache-maven-3.3.9/bin/mvn[root@Yarn-Master hadoop]# 

其他hadoop需要的

1、cmake：可以直接使用yum命令安装

[root@Yarn-Master hadoop]# yum install cmakeLoaded plugins: fastestmirrorSetting up Install ProcessLoading mirror speeds from cached hostfile * base: mirrors.zju.edu.cn * extras: mirrors.zju.edu.cn * updates: mirrors.zju.edu.cnPackage cmake-2.8.12.2-4.el6.i686 already installed and latest versionNothing to do[root@Yarn-Master hadoop]# 

2、protobuf：需要先检查下自己的hadoop编译时的版本

[hadoop@Yarn-Master ~]$ hadoop/2.6.1/bin/hadoop versionHadoop 2.6.1Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r b4d876d837b830405ccdb6af94742f99d49f9c04Compiled by jenkins on 2015-09-16T21:07ZCompiled with protoc 2.5.0From source with checksum ba9a9397365e3ec2f1b3691b52627fThis command was run using /home/hadoop/hadoop/2.6.1/share/hadoop/common/hadoop-common-2.6.1.jar[hadoop@Yarn-Master ~]$ 

然后去https://github.com/google/protobuf/tags找到对应的版本，下载安装就好

3、findbugs：http://findbugs.sourceforge.net/downloads.html同上

[hadoop@Yarn-Master ~]$ which protoc/usr/local/protobuf-2.5.0/bin/protoc[hadoop@Yarn-Master ~]$ which fb/usr/local/findbugs-3.0.1/bin/fb[hadoop@Yarn-Master ~]$ 

编译

完成以上步骤后，就可以使用官方文档里的命令完成编译工作

Once you installed the prerequisite packages use the standard hadoop pom.xml file and pass along the native flag to build the native hadoop library:

   $ mvn package -Pdist,native -DskipTests -Dtar

You should see the newly-built library in:

   $ hadoop-dist/target/hadoop-2.6.1/lib/native

[INFO] ------------------------------------------------------------------------[INFO] Reactor Summary:[INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [  2.740 s][INFO] Apache Hadoop Project POM .......................... SUCCESS [  2.573 s]...[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.524 s][INFO] Apache Hadoop Distribution ......................... SUCCESS [01:16 min][INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 18:54 min[INFO] Finished at: 2015-12-13T18:52:17+08:00[INFO] Final Memory: 86M/247M[INFO] ------------------------------------------------------------------------[hadoop@Yarn-Master hadoop-2.6.1-src]$ 

检查是否编译正确

[hadoop@Yarn-Master ~]$ file downloads/hadoop-2.6.1-src/hadoop-dist/target/hadoop-2.6.1/lib/native/libhadoop.so.1.0.0 downloads/hadoop-2.6.1-src/hadoop-dist/target/hadoop-2.6.1/lib/native/libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped[hadoop@Yarn-Master ~]$ 

使用编译的本地库

先将编译后的库拷贝到hadoop目录中

[hadoop@Yarn-Master ~]$ cp -r downloads/hadoop-2.6.1-src/hadoop-dist/target/hadoop-2.6.1/lib hadoop/src_2.6.1/[hadoop@Yarn-Master ~]$ ls hadoop/src_2.6.1/lib[hadoop@Yarn-Master ~]$ ls hadoop/src_2.6.1/lib/native/libhadoop.a       libhadoop.so        libhadooputils.a  libhdfs.solibhadooppipes.a  libhadoop.so.1.0.0  libhdfs.a         libhdfs.so.0.0.0[hadoop@Yarn-Master ~]$ 

然后修改环境变量，使用编译的本地库

export HADOOP_SRC_HOME=$HOME/hadoop/src_2.6.1export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_SRC_HOME/lib/nativeexport HADOOP_OPTS="-Djava.library.path=$HADOOP_SRC_HOME/lib/native"

再启动就可以看到警告信息消失了

[hadoop@Yarn-Master ~]$ export HADOOP_ROOT_LOGGER=DEBUG,console[hadoop@Yarn-Master ~]$ hadoop/2.6.1/sbin/start-dfs.sh 15/12/14 11:25:44 DEBUG util.Shell: setsid exited with exit code 0...15/12/14 11:25:45 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics15/12/14 11:25:45 DEBUG security.Groups:  Creating new Groups object15/12/14 11:25:45 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...15/12/14 11:25:45 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library15/12/14 11:25:45 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution15/12/14 11:25:45 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping