Windows源码编译Hadoop 2.8.0

相关依赖

1.JDK 1.7+
2.maven 3.0+
3.findbugs
4.zlib headers 1.2.7+ (如果要编译得到native code)
5.cmake 2.6+
6.protocbuffer 2.5.0
7.windows SDK or Visual Studio 2010 SP1(如果是vs2010，最好升级为sp1，否则那个cvtres.exe会疯掉)
8.Cygwin
9.Hadoop源码

---

配置编译环境

• JDK
  1.安装JDK
  2.新建系统变量JAVA_HOME=D：\Java\jdk(JAVA_HOME路径不能有空格)
  3.配置环境变量PATH，追加路径%JAVA_HOME%\bin
• maven
  1.解压maven到D:\maven
  2.新建系统变量M2_HOME=D：\maven\
  3.配置环境变量PATH，追加路径%M2_HOME%\bin
• Findbugs
  1.解压Findbugs到D:\findbugs
  2.配置环境变量PATH，追加路径D：\findbugs\bin
• zlib
  1.解压zlib到D:\zlib
  2.配置环境变量PATH，追加路径D：\zlib\(路径一定要指向zlib.dll)
• cmake
  1.解压cmake到D:\cmake
  2.配置环境变量PATH，追加路径D：\cmake\bin
• protobuffer
  （我直接把protoc.exe放到C:\Windows\System32\ 目录下，好像也可以）
  1.解压protobuffer源码至D:\protobuf-2.5.0
  2.解压protobuf-2.5.0-win64.zip中的protoc.exe复制到D:\protobuf-2.5.0\src目录下
  3.命令行进入源码目录，执行mvn install
  4.配置环境变量PATH，追加路径指向protoc.exe
• Cygwin
  1.解压Cygwin到D:\Cygwin
  2.配置环境变量PATH，追加路径D：\Cygwin\bin
• VS2010

编译Hadoop源码

• 设置环境变量Platform=x64 (Platform的大小写一定不能写错啊)
• 编译Hadoop
  1.解压源码
  2.使用VS2010编译源码
  开始–>所有程序–>Microsoft Visual Studio 2010–>Visual Studio Tools–>Visual Studio x64 Win64 命令提示(2010)(我用这个有点问题，可以使用Visual Studio x64兼容命令提示)
  
  在该命令行中进入hadoop源码目录,执行mvn package -Pdist,native-win -DskipTests -Dtar，编译成功后的文件在hadoop-dist/target目录下

遇到的问题

1.hadoop-common编译失败

[INFO] --- hadoop-maven-plugins:2.6.0:version-info (version-info) @ hadoop-common --- [WARNING] [svn, info] failed: java.io.IOException: Cannot run program "svn": CreateProcess error=2, ?????????[WARNING] [git, branch] failed with error code 128[INFO] SCM: NONE......C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.Cpp.Platform.targets(55,5): error MSB8020: The build tools for Visual Studio 2010 (Platform Toolset = 'v100') cannot be found. To build using the v100 build tools, please install Visual Studio 2010 build tools. Alternatively, you may upgrade to the current Visual Studio tools by selecting the Project menu or right-click the solution, and then selecting "Retarget solution".[C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj]已完成生成项目“C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj”(默认目标)的操作 - 失败。已完成生成项目“C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj.metaproj”(默认目标)的操作 - 失败。 已完成生成项目“C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln”(默认目标)的操作 - 失败。生成失败。......[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. Process exited with an error: 1(Exit value: 1) -> [Help 1]......[ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hadoop-common

解决方法，Visual Studio版本问题，官方默认使用Visual Studio 2010 Professional，但本文采用Visual Studio 2015，因此对于生成失败的项目，需要用Visual Studio 2015重新打开，升级项目至Visual Studio 2015版本即可：

C:\WorkSpace\hadoop-2.8.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln

2、 hadoop-hdfs编译失败

main:[mkdir] Created dir: C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native[exec] -- The C compiler identification is unknown[exec] -- The CXX compiler identification is unknown[exec] CMake Error in CMakeLists.txt:[exec] No CMAKE_C_COMPILER could be found.[exec] CMake Error in CMakeLists.txt:[exec] No CMAKE_CXX_COMPILER could be found.[exec] -- Configuring incomplete, errors occurred![exec] See also "C:/WorkSpace/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeOutput.log".[exec] See also "C:/WorkSpace/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeError.log".......[ERROR] After correcting the problems, you can resume the build with the command[ERROR] mvn <goals> -rf :hadoop-hdfs

问题原因是Visual Studio版本问题，解决方法：打开文件C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\pom.xml，将-G 'Visual Studio 10 Win64'修改为：-G 'Visual Studio 14 Win64'

若仍无法编译，可查看日志C:/WorkSpace/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeError.log。若日志提示找不到Microsoft.Cpp.Default.props，可在C:\Program Files (x86)中查找Microsoft.Cpp.Default.props，例如最后找到所在路径C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140
用文本编辑器打开项目文件
C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native\CMakeFiles\3.6.1\CompilerIdC\CompilerIdC.vcxproj

查找Microsoft.Cpp.Default.props所在行，如下：

<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />

原因是没有系统变量VCTargetsPath，新建系统变量VCTargetsPath=C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140，然后重新编译：

1. mvn package -Pdist,native-win -DskipTests -Dtar -rf :hadoop-hdfs

注：若还提示找不到，可能是环境变量未生效，关闭命令行窗口，重新打开执行编译命令，或者重启电脑试试。

编译hadoop-hdfs的时候出错，出错信息如下：

    [exec]   LINK : fatal error LNK1123: 转换到 COFF 期间失败: 文件无效或损坏     [exec]   [D:\source\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native\CMakeFiles\CMakeTmp\cmTryCompileExec3777276464.vcxproj]     [exec]     [exec]     [exec]     [exec]   已完成生成项目“D:\source\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native\CMakeFiles\CMakeTmp\cmTryCompileExec3777276464.vcxproj”(默认目标)的操作     [exec]   - 失败。     [exec]     [exec]     [exec]     [exec]   生成失败。     [exec]     [exec]     [exec]     [exec]     [exec]   “D:\source\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native\CMakeFiles\CMakeTmp\cmTryCompileExec3777276464.vcxproj”(默认目标)     [exec]   (1) ->     [exec]     [exec]   (Link 目标) ->     [exec]     [exec]     LINK : fatal error LNK1123: 转换到 COFF 期间失败: 文件无效或损坏[D:\source\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native\CMakeFiles\CMakeTmp\cmTryCompileExec3777276464.vcxproj]     [exec]     [exec]     [exec]     [exec]       0 个警告     [exec]       1 个错误     [exec]     [exec]     [exec]     [exec]   已用时间 00:00:00.58     [exec]     [exec]     [exec]     [exec]     [exec]     [exec]   CMake will not be able to correctly generate this project.     [exec] Call Stack (m     [exec] ost recent call first):     [exec]     [exec]     [exec]     [exec] -- Configuring incomplete, errors occurred!     [exec] See also "D:/source/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeOutput.log".     [exec] See also "D:/source/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeError.log".[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------

原因：Visual studio 2010版本，升级到sp1。
不建议按照网上介绍的删除Visual studio中的cvtres.exe
删掉后，这个问题解决了，但是会报找不到cvtres.exe的错误。

1. cmake依赖库没有完全，导致出错

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (dist) on project hadoop-kms: An Ant BuildException has occured: exec returned: 2[ERROR] around Ant part ...<exec dir="/home/lch/hadoop-2.6.0-src/hadoop-common-project/hadoop-kms/target" executable="sh" failonerror="true">... @ 10:117 in /home/lch/hadoop-2.6.0-src/hadoop-common-project/hadoop-kms/target/antrun/build-main.xml[ERROR] -> [Help 1]org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (dist) on project hadoop-kms: An Ant BuildException has occured: exec returned: 2around Ant part ...<exec dir="/home/lch/hadoop-2.6.0-src/hadoop-common-project/hadoop-kms/target" executable="sh" failonerror="true">... @ 10:117 in /home/lch/hadoop-2.6.0-src/hadoop-common-project/hadoop-kms/target/antrun/build-main.xml        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)分析：在目录下面/home/lch/hadoop-2.6.0-src/hadoop-common-project/hadoop-kms/target/antrun下面执行命令：   ant -buildfile build-main.xml

可以看到错误原因：cmake差少zlib的符号，安装zlib，然后设置Path添加zlib路径

1. 
   编译hadoop-common时，报错protoc comile error
   原因：使用的protoc是用cygwin编译出来的，而使用cygwin编译的protoc无法识别windows的路径
   修改方式：使用Visual stutio将protoc重新编译。将得到的libprotobuf.lib，libprotobuf-lite.lib，libprotoc.lib和protoc.exe拷贝到c盘下，并在path中配置
   proto的vs编译方式，可以参考：http://blog.csdn.net/kaitiren/article/details/9192185

3.出错信息：

Project "D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln" on node 0 (default targets).  Building solution configuration "Release|x64".D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj(21,14): error MSB4066: 无法识别元素 <ItemGroup> 中的属性“Label”。Done Building Project "D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln" (default targets) -- FAILED.Build FAILED."D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln" (default target) (1) ->(libwinutils target) ->  D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj(21,14): error MSB4066: 无法识别元素 <ItemGroup> 中的属性“Label”。    0 Warning(s)    1 Error(s)Time Elapsed 00:00:00.41

原因：使用的MSBuild的.net版本不对。在.net 4.0中这个标签已经被加入
D:\source\hadoop-2.6.0-src\hadoop-common-project>msbuild.exe /toolsversion:versi
on
Microsoft (R) 生成引擎版本 3.5.30729.5420
[Microsoft .NET Framework 版本 2.0.50727.5448]
版权所有(C) Microsoft Corporation 2007。保留所有权利。
而我的环境是安装了.Net FrameWork 4.0的，但是cmd shell启动的时候，它会执行文件：D:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\SetEnv.cmd
修改SetEnv.cmd，即可,将.Net FrameWork 4.0的配置信息加入到Path中，如：

SET "FxTools=%windir%\Microsoft.NET\Framework64\v4.0.30319;%windir%\Microsoft.NET\Framework\v4.0.30319;%windir%\Microsoft.NET\Framework64\v3.5;%windir%\Microsoft.NET\Framework\v3.5;%windir%\Microsoft.NET\Framework64\v2.0.50727;%windir%\Microsoft.NET\Framework\v2.0.50727"

1. 出错信息：

    [exec] CMake Error at /usr/share/cmake-2.8.11.2/Modules/FindPackageHandleStandardArgs.cmake:108 (message):     [exec]   Could NOT find JNI (missing: JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY     [exec]   JAVA_INCLUDE_PATH JAVA_INCLUDE_PATH2 JAVA_AWT_INCLUDE_PATH)     [exec] Call Stack (most recent call first):     [exec]   /usr/share/cmake-2.8.11.2/Modules/FindPackageHandleStandardArgs.cmake:315 (_FPHSA_FAILURE_MESSAGE)     [exec]   /usr/share/cmake-2.8.11.2/Modules/FindJNI.cmake:249 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)     [exec]   D:/source/hadoop-2.6.0-src/hadoop-common-project/hadoop-common/src/JNIFlags.cmake:123 (find_package)     [exec]   D:/source/hadoop-2.6.0-src/hadoop-common-project/hadoop-common/src/CMakeLists.txt:24 (include)     [exec]     [exec]

修改：找到usr\share\cmake-2.8.11.2\Modules\FindJNI.cmake，然而后将open_jdk全部注释掉

1. 出错信息：

[INFO][INFO] --- exec-maven-plugin:1.2:exec (compile-ms-native-dll) @ hadoop-common ---Building the projects in this solution one at a time. To enable parallel build,please add the "/m" switch.生成启动时间为 2015/5/13 8:33:09。节点 1 上的项目“D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.sln”(默认目标)。ValidateSolutionConfiguration:  正在生成解决方案配置“Release|x64”。项目“D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.sln”(1)正在节点 1 上生成“D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj”(2) (默认目标)。InitializeBuildStatus:  正在对“..\..\..\target\native\Release\native.unsuccessfulbuild”执行 Touch 任务。ClCompile:  D:\Microsoft Visual Studio 10.0\VC\bin\AMD64\CL.exe /c /I..\winutils\include /I..\..\..\target\native\javah /I"C:\Program Files (x86)\Java\jdk1.7.0_67\include" /I"C:\Program Files (x86)\Java\jdk1.7.0_67\include\win32" /I.\src /Zi /nologo/W3 /WX- /O2 /Oi /GL /D WIN32 /D NDEBUG /D _WINDOWS /D _USRDLL /D NATIVE_EXPORTS/D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Fo"..\..\..\target\native\Release\\" /Fd"..\..\..\target\native\Release\vc100.pdb" /Gd /TC /wd4244 /errorReport:queue src\org\apache\hadoop\io\compress\zlib\ZlibCompressor.c src\org\apache\hadoop\io\compress\zlib\ZlibDecompressor.c  ZlibCompressor.cd:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36):fatal error C1083: 无法打开包括文件:“zlib.h”: No such file or directory [D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]  ZlibDecompressor.cd:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36):fatal error C1083: 无法打开包括文件:“zlib.h”: No such file or directory [D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]已完成生成项目“D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj”(默认目标)的操作 - 失败。已完成生成项目“D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.sln”(默认目标)的操作 - 失败。生成失败。“D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.sln”(默认目标) (1) ->“D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj”(默认目标) (2) ->(ClCompile 目标) ->  d:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36): fatal error C1083: 无法打开包括文件:“zlib.h”: No such file or directory [D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]  d:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36): fatal error C1083: 无法打开包括文件:“zlib.h”: No such file or directory [D:\source\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]    0 个警告    2 个错误

原因：因为zlib.h无法找到,解决方法比较挫，是将zlib的两个头文件放在C:\Program Files (x86)\Java\jdk1.7.0_67\include中

1. 编译hadoop-hdfs的时候，出错。
   出错信息：

main:    [mkdir] Created dir: D:\source\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native     [exec] CMake Error: Could not create named generator Visual Studio 10 Win64

原因：
cygwin中存在一个cmake，而这个cmake不能够被使用，需要使用windows版本的cmake.并在path设置中，将这个cmake设置在cygwin前面