编译hadoop 2.3的native library

用了hadoop 2.3 以后,每次启动都出现 

引用

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

而且hadoop的cache和短路读(Short-Circuit Local Reads)都需要native library的支持 

但是它自带的native library是32位的,需要自己编译适合自己的. 
编译方法是 
http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/NativeLibraries.html 

编译过程中可能出现 

引用

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.3.0:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version -> [Help 1]

这时需要在系统中安装protobuf 
从http://code.google.com/p/protobuf/downloads/list下载源码, 
./configure 
make 
make install 
即可. 
需要安装cmake,可以通过yum install cmake安装 

编译过程中还出现 

引用

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1 
[ERROR] around Ant part ...<exec dir="/usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:136 in /usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml 
[ERROR] -> [Help 1] 
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1 
around Ant part ...<exec dir="/usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:136 in /usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml 
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216) 
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) 
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) 
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108) 
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116) 
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361) 
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155) 
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584) 
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213) 
at org.apache.maven.cli.MavenCli.main(MavenCli.java:157) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) 
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) 
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) 
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) 
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: exec returned: 1 
around Ant part ...<exec dir="/usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:136 in /usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml 
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355) 
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133) 
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) 
... 19 more 
Caused by: /usr/local/odpf/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml:4: exec returned: 1 
at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:646) 
at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:672) 
at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:498) 
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291) 
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) 
at org.apache.tools.ant.Task.perform(Task.java:348) 
at org.apache.tools.ant.Target.execute(Target.java:390) 
at org.apache.tools.ant.Target.performTasks(Target.java:411) 
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399) 
at org.apache.tools.ant.Project.executeTarget(Project.java:1368) 
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:327) 
... 21 more 
[ERROR] 
[ERROR] Re-run Maven using the -X switch to enable full debug logging. 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles: 
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException 
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command 
[ERROR]   mvn <goals> -rf :hadoop-common

这样的问题 

发现是少了几个必须的包的开发包.比如装了zlib和openssl,但实际需要的是zlib-devel和openssl-devel 

最后运行mvn package -Pdist,native -Dskiptests -Dtar 
在test阶段各种失败 
这个应该是hadoop的bug,skiptests在native profile激活时没起作用 
https://issues.apache.org/jira/browse/HADOOP-8480 
最后走投无路灵机一动,运行mvn compile -Pdist,native -Dskiptests -Dtar 
在./hadoop-common-project/hadoop-common/target/native/target/usr/local/lib 

中找到了想要的libhadoop.so 

cp  libhadoop.so.1.0.0  $HADOOP_HOME/lib

ln -s libhadoop.so.1.0.0 libhadoop.so