window 开发hadoop
来源:互联网 发布:js字符串true false 编辑:程序博客网 时间:2024/05/21 22:25
1、安装eclipse 插件
下载hadoop2x-eclipse-plugin-master 进行编译
ant jar -Dversion=2.8.0 -Dhadoop.version=2.8.0 -Declipse.home=D:\software\eclipse-jee-kepler-SR2-win32-x86_64\eclipse -Dhadoop.home=D:\bigdata\hadoop-2.8.0
其中会遇到各种问题大部分都是由于build.xml或者hadoop-common中的core-default.xml。
core-default.xml 增加:
<property>
<name>fs.hdfs.impl</name>
<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
<description>The FileSystem for hdfs: uris.</description>
</property>
build.xml
<copy file="${hadoop.home}/share/hadoop/common/lib/protobuf-java-${protobuf.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/log4j-${log4j.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/commons-configuration-${commons-configuration.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/commons-lang-${commons-lang.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/commons-collections-${commons-collections.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/jackson-core-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/jackson-mapper-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-log4j12-${slf4j-log4j12.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-api-${slf4j-api.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/guava-${guava.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/hadoop-auth-${hadoop.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/netty-${netty.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.home}/share/hadoop/common/lib/htrace-core4-${htrace.version}-incubating.jar" todir="${build.dir}/lib" verbose="true"/>
<jar
jarfile="${build.dir}/hadoop-${name}-${hadoop.version}.jar"
manifest="${root}/META-INF/MANIFEST.MF">
<manifest>
<attribute name="Bundle-ClassPath"
value="classes/,
lib/hadoop-mapreduce-client-core-${hadoop.version}.jar,
lib/hadoop-mapreduce-client-common-${hadoop.version}.jar,
lib/hadoop-mapreduce-client-jobclient-${hadoop.version}.jar,
lib/hadoop-auth-${hadoop.version}.jar,
lib/hadoop-common-${hadoop.version}.jar,
lib/hadoop-hdfs-${hadoop.version}.jar,
lib/hadoop-hdfs-client-${hadoop.version}.jar,
lib/protobuf-java-${protobuf.version}.jar,
lib/log4j-${log4j.version}.jar,
lib/commons-cli-${commons-cli.version}.jar,
lib/commons-configuration-${commons-configuration.version}.jar,
lib/commons-httpclient-${commons-httpclient.version}.jar,
lib/commons-lang-${commons-lang.version}.jar,
lib/commons-collections-${commons-collections.version}.jar,
lib/jackson-core-asl-${jackson.version}.jar,
lib/jackson-mapper-asl-${jackson.version}.jar,
lib/slf4j-log4j12-${slf4j-log4j12.version}.jar,
lib/slf4j-api-${slf4j-api.version}.jar,
lib/guava-${guava.version}.jar,
lib/netty-${netty.version}.jar,
lib/htrace-core4-${htrace.version}-incubating.jar"/>
2、下载hadoop-2.8.0-src编译winutils和native ,编译winuntils.中时修改下libwinuntils.c
//const WCHAR* wsceConfigRelativePath = WIDEN_STRING(STRINGIFY(WSCE_CONFIG_DIR)) L"\\" WIDEN_STRING(STRINGIFY(WSCE_CONFIG_FILE));
const WCHAR* wsceConfigRelativePath = STRINGIFY(WSCE_CONFIG_DIR) "\\" STRINGIFY(WSCE_CONFIG_FILE);
3复制文件hadoop.lib、winutils.exe 到hadoop 的安装目录的bin
4 复制hadoop.dll到syetm/win32下面
- window 开发hadoop
- window下hadoop-2.2.0应用程序开发_java
- [Hadoop]->[开发环境搭建] Window环境下配置Hadoop开发环境
- Window下配置Hadoop
- hadoop 开发
- Hadoop开发
- X Window开发编程
- window移动应用开发
- window mobile 开发常见问题
- Window Service 开发笔记
- 学习window 8开发
- window驱动开发环境
- window C++开发工具
- X Window开发编程
- nodejs开发环境(window)
- window service 开发步骤
- window平台mupdf开发
- X Window开发编程
- Leetcode 481(Java)
- MongoDB 更新文档
- 使用Kotlin优雅的开发Android应用
- 高通平台EFS的制作
- html页面引入公共部分,头部和尾部
- window 开发hadoop
- 使用pyinstaller把py文件打包为exe文件
- android 启动流程
- C/C++中的函数参数传递机制
- UVA10763解题报告
- shiro简单配置
- 语音识别概述
- Kotlin 官方参考文档 中文版_kotlin-reference-chinese.pdf
- 欢迎使用CSDN-markdown编辑器