Windows 10编译Hadoop 2.6.0源码
来源:互联网 发布:可靠性仿真软件 编辑:程序博客网 时间:2024/06/05 17:22
转载自:https://www.zybuluo.com/ncepuwanghui/note/343755
原作者:Sam Wong向原文作者的辛勤劳动致敬!
版本控制
@Title Windows 10编译Hadoop 2.6.0源码@Version v1.0@Timestamp 2016-08-22 16:21@Author Sam Wong@Mail ncepuwanghui@foxmail.com
系统环境
@OS Windows 10 Enterprise (64bit)@JDK JDK 1.7.0_67@Hadoop Hadoop 2.6.0
下载相关软件包
Building on Windows
Requirements:
Windows System JDK 1.6+ Maven 3.0 or later Findbugs 1.3.9 (if running
findbugs) ProtocolBuffer 2.5.0 CMake 2.6 or newer Windows SDK or
Visual Studio 2010 Professional Unix command-line tools from GnuWin32
or Cygwin: sh, mkdir, rm, cp, tar, gzip zlib headers (if building
native code bindings for zlib) Internet connection for first build (to
fetch all Maven and Hadoop dependencies)If using Visual Studio, it must be Visual Studio 2010 Professional
(not 2012). Do not use Visual Studio Express. It does not support
compiling for 64-bit, which is problematic if running a 64-bit system.
JDK
jdk-7u67-windows-x64.exe
下载地址:http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html
Maven
apache-maven-3.3.9-bin.zip
下载地址:https://maven.apache.org/download.cgi
Findbugs
findbugs-1.3.9.zip
下载地址:https://sourceforge.net/projects/findbugs/files/findbugs/1.3.9/
ProtocolBuffer
protobuf-2.5.0.zip
protoc-2.5.0-win32.zip
下载地址:https://github.com/google/protobuf/releases/tag/v2.5.0
注:除了下载protobuf源码外,还需要下载相应版本的编译过的用于Windows平台的protoc命令(protoc-2.5.0-win32.zip),该命令用于将.proto文件转化为Java或C++源文件。
CMake
cmake-3.6.1-win64-x64.zip
下载地址:https://cmake.org/download/
Visual Studio 2010 Professional
Cygwin
setup-x86_64.exe
下载地址:https://cygwin.com/install.html
Zlib headers
zlib128.zip
下载地址:http://www.zlib.net/
安装配置所需环境
JDK
① 安装JDK
② 新建系统变量JAVA_HOME=C:\Progra~1\Java\jdk1.7.0_67
注:JAVA_HOME路径不能有空格,如果安装路径名中有空格,要使用旧式的8.3名字格式,即C:\Program Files\要写成C:\Progra~1\。
③ 编辑系统变量Path,新建%JAVA_HOME%\bin和%JAVA_HOME%\jre\bin
Maven
① 解压Maven到指定目录
② 新建系统变量M2_HOME=C:\WorkSpace\apache-maven-3.3.9
② 编辑系统变量Path,新建%M2_HOME%\bin
Findbugs
① 解压Findbugs到指定目录
② 编辑系统变量Path,新建C:\WorkSpace\findbugs-1.3.9\bin
ProtocolBuffer
① 解压ProtocolBuffer到指定目录
② 解压protoc-2.5.0-win32.zip,将protoc.exe复制到C:\WorkSpace\protobuf-2.5.0\src目录下
③ 安装ProtocolBuffer,打开CMD命令行
cd C:\WorkSpace\protobuf-2.5.0\javamvn test mvn installprotoc --version
若没有protoc.exe或复制目录不正确,会提示如下错误:
[INFO] Executing tasks [mkdir] Created dir: C:\WorkSpace\protobuf-2.5.0\java\target\generated-sources[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 04:22 min[INFO] Finished at: 2016-08-22T17:56:19+08:00[INFO] Final Memory: 16M/439M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.3:run (generate-sources) on project protobuf-java: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "..\src\protoc": CreateProcess error=2, ????????? -> [Help 1][ERROR][ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR][ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
④ 将protoc.exe所在路径C:\WorkSpace\protobuf-2.5.0\src,添加到系统变量Path
CMake
① 解压CMake到指定目录
② 编辑系统变量Path,新建C:\WorkSpace\cmake-3.6.1-win64-x64\bin
Cygwin
① 安装Cygwin64
② 编辑系统变量Path,新建C:\WorkSpace\cygwin64\bin
Zlib
① 解压Zlib到指定目录
② 新建系统变量ZLIB_HOME=C:\WorkSpace\zlib-1.2.8
编译Hadoop2.6.0
设置Platform
新建系统变量Platform=x64
编译Hadoop2.6.0
用右键以管理员身份运行Visual Studio x64 Win64 命令提示(2010),开始菜单–>所有程序–>Microsoft Visual Studio 2010–>Visual Studio Tools–>Visual Studio x64 Win64 命令提示(2010),执行如下命令:
cd C:\WorkSpace\hadoop-2.6.0-srcmvn package -Pdist,native-win -DskipTests -Dtar
编译完成后,编译好的hadoo-2.6.0在目录C:\WorkSpace\hadoop-2.6.0-src\hadoop-dist\target\中。
注:本文采用Visual Studio 2015编译,开始–>所有应用–>Visual Studio 2015–>VS2015 x64
本机工具命令提示符
遇到的错误
① hadoop-common编译失败
[INFO] --- hadoop-maven-plugins:2.6.0:version-info (version-info) @ hadoop-common ---[WARNING] [svn, info] failed: java.io.IOException: Cannot run program "svn": CreateProcess error=2, ?????????[WARNING] [git, branch] failed with error code 128[INFO] SCM: NONE......C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.Cpp.Platform.targets(55,5): error MSB8020: The build tools for Visual Studio 2010 (Platform Toolset = 'v100') cannot be found. To build using the v100 build tools, please install Visual Studio 2010 build tools. Alternatively, you may upgrade to the current Visual Studio tools by selecting the Project menu or right-click the solution, and then selecting "Retarget solution".[C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj]已完成生成项目“C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj”(默认目标)的操作 - 失败。已完成生成项目“C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj.metaproj”(默认目标)的操作 - 失败。已完成生成项目“C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln”(默认目标)的操作 - 失败。生成失败。......[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. Process exited with an error: 1(Exit value: 1) -> [Help 1]org.ap......[ERROR] After correcting the problems, you can resume the build with the command[ERROR] mvn <goals> -rf :hadoop-common
解决方法,Visual Studio版本问题,官方默认使用Visual Studio 2010 Professional,但本文采用Visual Studio 2015,因此对于生成失败的项目,需要用Visual Studio 2015重新打开,升级项目至Visual Studio 2015版本即可:
C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.slnC:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.sln
② hadoop-hdfs编译失败
main: [mkdir] Created dir: C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native [exec] -- The C compiler identification is unknown [exec] -- The CXX compiler identification is unknown [exec] CMake Error in CMakeLists.txt: [exec] No CMAKE_C_COMPILER could be found. [exec] CMake Error in CMakeLists.txt: [exec] No CMAKE_CXX_COMPILER could be found. [exec] -- Configuring incomplete, errors occurred! [exec] See also "C:/WorkSpace/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeOutput.log". [exec] See also "C:/WorkSpace/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeError.log".......[ERROR] After correcting the problems, you can resume the build with the command[ERROR] mvn <goals> -rf :hadoop-hdfs
问题原因是Visual Studio版本问题,解决方法:打开文件C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\pom.xml`,将
-G 'Visual Studio 10 Win64'
修改为:
-G 'Visual Studio 14 Win64'
若仍无法编译,可查看日志C:/WorkSpace/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeError.log。若日志提示找不到Microsoft.Cpp.Default.props,可在C:\Program Files (x86)中查找Microsoft.Cpp.Default.props,例如最后找到所在路径C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140
用文本编辑器打开项目文件
C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native\CMakeFiles\3.6.1\CompilerIdC\CompilerIdC.vcxprojC:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native\CMakeFiles\3.6.1\CompilerIdCXX\CompilerIdCXX.vcxproj
查找Microsoft.Cpp.Default.props所在行,如下:
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
原因是没有系统变量VCTargetsPath,新建系统变量VCTargetsPath=C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140,然后重新编译:
mvn package -Pdist,native-win -DskipTests -Dtar -rf :hadoop-hdfs
注:若还提示找不到,可能是环境变量未生效,关闭命令行窗口,重新打开执行编译命令,或者重启电脑试试。
参考链接
Build and Install Hadoop 2.x or newer on Windows
Windows 10下编译安装Hadoop2.6
在windows 7 64bit下编译Hadoop 2.6.0源码
编译本地64位版本的hadoop-2.6.0
在Windows下编译Hadoop2.5
Protobuf 的安装使用总结
- Windows 10编译Hadoop 2.6.0源码
- Windows源码编译Hadoop 2.8.0
- hadoop-2.6.0源码编译
- 在windows 7 64bit下编译Hadoop 2.6.0源码
- 在Windows操作系统上编译Hadoop源码
- Windows Hadoop 2.6.0编译安装
- hadoop-2.2.0源码编译
- hadoop-2.2.0源码编译
- hadoop-1.0.0源码编译
- hadoop-2.5.0源码编译
- Win7 64bit hadoop-2.6.0源码编译部署包
- CentOS 64位上编译Hadoop 2.6.0源码包
- 64位centos 下编译 hadoop 2.6.0 源码
- linux环境下编译hadoop-2.6.0源码
- Spark 2.2源码编译 & 支持hadoop-2.6.0-cdh5.7.0
- Windows 源码编译Hadoop 2.7.4生成X64
- Ubuntu12.04 编译 Hadoop 2.6.4 源码
- Maven编译hadoop-2.6.4源码
- webpack ------ loaders
- 复化辛普森法
- 《Java 编程思想》-第4章 控制执行流程 笔记
- 学习笔记之建立lamp平台
- EasyLogging++学习笔记(1)—— 简要介绍
- Windows 10编译Hadoop 2.6.0源码
- 大一上c++上机实践
- tcpdump使用参考链接
- Linux图形界面知识(介绍X、X11、GNOME、Xorg、KDE等之间的关系)
- iPhone中的info.plist文件
- 多线程下载问题
- spring boot 整合 freemark(简单结构)
- Maven原理和Maven2新特性
- Binary search(1) -- First Bad Version, Arranging Coins