Hadoop2.4.0 Eclipse插件制作
来源:互联网 发布:如何在淘宝加盟代理 编辑:程序博客网 时间:2024/05/01 03:16
跌跌撞撞两天装Hadoop2.4.0的环境,今天终于可以尝试写点程序了,想了一想,不得其法,但有一点是肯定的,那就是需要用eclipse,查了一上午的资料,怎么讲的都有,最后锁定了这个帖子,hadoop2x-eclipse-plugin。想用Github下载,但是最后下载失败,页面右侧有个Download ZIP,用这个下载成功。下边来说说我的整个制作过程吧:
- 想导入到eclipse中制作,但是导入进去之后觉得麻烦就算了,直接在win7 64位、JDK1.6、ANT1.8的环境下以命令行运行
- 依葫芦画瓢,设置eclipse的目录和hadoop的目录,在没有修改任何文件的情况下直接运行
- D:\SDK\hadoop2x-eclipse-plugin-master\src\contrib\eclipse-plugin>ant jar -Dversion=2.4.0 -Declipse.home=D:\WorkTools\MyEclipse10\Common -Dhadoop.home=D:\SDK\hadoop-2.4.0
- 在ivy-resolve-common处卡了大概138分钟,最后build失败,查看原因有两个,一个是找不到4个依赖包,一个是找不到1个核心包
- 依赖包分别是这些,可能是换路径了吧:
- https://oss.sonatype.org/content/groups/public/com/google/google/1/google-1.jar
- https://oss.sonatype.org/content/groups/public/net/java/jvnet-parent/1/jvnet-parent-1.jar
- https://oss.sonatype.org/content/groups/public/com/sun/jersey/contribs/jersey-contribs/1.9/jersey-contribs-1.9.jar
- https://oss.sonatype.org/content/groups/public/org/eclipse/jetty/jetty-parent/14/jetty-parent-14.jar
- 核心包是commons-lang-2.5.jar
- 想了一下,觉得那四个依赖包似乎没什么用,便想跳过那个步骤,想直接进行编译插件的步骤,便修改了“hadoop2x-eclipse-plugin-master\src\contrib\eclipse-plugin\build.xml”文件
- <target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib"> 去掉depends
- 核心包去hadoop2.4目录找,发现hadoop2.4用的commons-lang-2.6.jar,便在build.xml中找到相应位置改为2.6,这个需要说的是作者可能偷懒,在拷贝文件中使用的是变量,在配置信息中直接使用了2.5,不统一,变量配置信息在“hadoop2x-eclipse-plugin-master\ivy\libraries.properties”文件中,搜索“commons-lang.version”将对应的版本改为2.6,好了,在命令行下运行ant,成功在“hadoop2x-eclipse-plugin-master\build\contrib\eclipse-plugin”下生成插件
- 兴奋的拿着插件跟myeclipse10里试,失败(插件压根不出现),跟eclipse3.7里试,失败(插件能出现,但是无法正常使用,新增不了location)
- 拿着现成的2.2的插件均可以正常使用
- 后来经过排查原因myeclipse10里是因为插件放的位置不对,不能直接放到plugin里;eclipse3.7里失败是因为插件还缺少包
- 如何排查的呢?看eclipse的启动日志,这个日志在workplace中的.metadata文件下的.log文件,在eclipse3.7里报的错误是无法初始化hadoop,很郁闷,没太大帮助,于是就上eclilpse的官网下载了最新版的eclipse4.3.2,将ant语句改为
- D:\SDK\hadoop2x-eclipse-plugin-master\src\contrib\eclipse-plugin>ant jar -Dversion=2.4.0 -Declipse.home=D:\WorkTools\eclipse4.3.2 -Dhadoop.home=D:\SDK\hadoop-2.4.0
- 对,改为以eclipse4.3.2为eclipse目录(这个配置主要就是为生成eclipse插件提供jar包支持),生成插件后放到eclipse4.3.2的plugins里,启动eclipse并观察日志,抛出异常
- java.lang.NoClassDefFoundError: org/apache/commons/collections/map/UnmodifiableMap
- yes~ 折腾了又两个多小时,终于让我抓住了,找到这个包,在build.xml中添加上,别忘了在配置信息中也要添加,就是文件最下边的attribute,如果想以后方便,版本号也可以通过变量形式在libraries.properties中维护,搞定后,重新运行ant
- 终于行了,在myeclipse10下也可正常使用了(这么建立文件夹MyEclipse 10\dropins\hadoop\eclipse\plugins,把插件扔这个文件夹下边就行了)
------------------------------------------------最终的build.xml文件,修改的地方有注释---------------------------------------------------------------
- <?xml version="1.0" encoding="UTF-8" standalone="no"?>
- <project default="jar" name="eclipse-plugin">
- <import file="../build-contrib.xml"/>
- <path id="eclipse-sdk-jars">
- <fileset dir="${eclipse.home}/plugins/">
- <include name="org.eclipse.ui*.jar"/>
- <include name="org.eclipse.jdt*.jar"/>
- <include name="org.eclipse.core*.jar"/>
- <include name="org.eclipse.equinox*.jar"/>
- <include name="org.eclipse.debug*.jar"/>
- <include name="org.eclipse.osgi*.jar"/>
- <include name="org.eclipse.swt*.jar"/>
- <include name="org.eclipse.jface*.jar"/>
- <include name="org.eclipse.team.cvs.ssh2*.jar"/>
- <include name="com.jcraft.jsch*.jar"/>
- </fileset>
- </path>
- <path id="hadoop-sdk-jars">
- <fileset dir="${hadoop.home}/share/hadoop/mapreduce">
- <include name="hadoop*.jar"/>
- </fileset>
- <fileset dir="${hadoop.home}/share/hadoop/hdfs">
- <include name="hadoop*.jar"/>
- </fileset>
- <fileset dir="${hadoop.home}/share/hadoop/common">
- <include name="hadoop*.jar"/>
- </fileset>
- </path>
- <path id="classpath">
- <pathelement location="${build.classes}"/>
- <path refid="eclipse-sdk-jars"/>
- <path refid="hadoop-sdk-jars"/>
- </path>
- <target name="check-contrib" unless="eclipse.home">
- <property name="skip.contrib" value="yes"/>
- <echo message="eclipse.home unset: skipping eclipse plugin"/>
- </target>
- <!-- 这里我最后删除了depends="init, ivy-retrieve-common" -->
- <target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib">
- <echo message="contrib: ${name}"/>
- <javac
- encoding="${build.encoding}"
- srcdir="${src.dir}"
- includes="**/*.java"
- destdir="${build.classes}"
- debug="${javac.debug}"
- deprecation="${javac.deprecation}">
- <classpath refid="classpath"/>
- </javac>
- </target>
- <target name="jar" depends="compile" unless="skip.contrib">
- <mkdir dir="${build.dir}/lib"/>
- <copy todir="${build.dir}/lib/" verbose="true">
- <fileset dir="${hadoop.home}/share/hadoop/mapreduce">
- <include name="hadoop*.jar"/>
- </fileset>
- </copy>
- <copy todir="${build.dir}/lib/" verbose="true">
- <fileset dir="${hadoop.home}/share/hadoop/common">
- <include name="hadoop*.jar"/>
- </fileset>
- </copy>
- <copy todir="${build.dir}/lib/" verbose="true">
- <fileset dir="${hadoop.home}/share/hadoop/hdfs">
- <include name="hadoop*.jar"/>
- </fileset>
- </copy>
- <copy todir="${build.dir}/lib/" verbose="true">
- <fileset dir="${hadoop.home}/share/hadoop/yarn">
- <include name="hadoop*.jar"/>
- </fileset>
- </copy>
- <copy todir="${build.dir}/classes" verbose="true">
- <fileset dir="${root}/src/java">
- <include name="*.xml"/>
- </fileset>
- </copy>
- <copy file="${hadoop.home}/share/hadoop/common/lib/protobuf-java-${protobuf.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <copy file="${hadoop.home}/share/hadoop/common/lib/log4j-${log4j.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <copy file="${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <copy file="${hadoop.home}/share/hadoop/common/lib/commons-configuration-${commons-configuration.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <copy file="${hadoop.home}/share/hadoop/common/lib/commons-lang-${commons-lang.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <!-- 下边这行是新增的,原来没有 -->
- <copy file="${hadoop.home}/share/hadoop/common/lib/commons-collections-${commons-collections.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <copy file="${hadoop.home}/share/hadoop/common/lib/jackson-core-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <copy file="${hadoop.home}/share/hadoop/common/lib/jackson-mapper-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-log4j12-${slf4j-log4j12.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-api-${slf4j-api.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <copy file="${hadoop.home}/share/hadoop/common/lib/guava-${guava.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <copy file="${hadoop.home}/share/hadoop/common/lib/hadoop-auth-${hadoop.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <copy file="${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <copy file="${hadoop.home}/share/hadoop/common/lib/netty-${netty.version}.jar" todir="${build.dir}/lib" verbose="true"/>
- <jar
- jarfile="${build.dir}/hadoop-${name}-${version}.jar"
- manifest="${root}/META-INF/MANIFEST.MF">
- <manifest>
- <attribute name="Bundle-ClassPath"
- value="classes/,
- lib/hadoop-mapreduce-client-core-${hadoop.version}.jar,
- lib/hadoop-mapreduce-client-common-${hadoop.version}.jar,
- lib/hadoop-mapreduce-client-jobclient-${hadoop.version}.jar,
- lib/hadoop-auth-${hadoop.version}.jar,
- lib/hadoop-common-${hadoop.version}.jar,
- lib/hadoop-hdfs-${hadoop.version}.jar,
- lib/protobuf-java-${protobuf.version}.jar,
- lib/log4j-${log4j.version}.jar,
- lib/commons-cli-1.2.jar,
- lib/commons-configuration-1.6.jar,
- lib/commons-httpclient-3.1.jar,
- <!-- 下边这行原来写的是commons-lang-2.5 -->
- lib/commons-lang-${commons-lang.version}.jar,
- <!-- 下边这行是新增的,原来没有 -->
- lib/commons-collections-${commons-collections.version}.jar,
- lib/jackson-core-asl-1.8.8.jar,
- lib/jackson-mapper-asl-1.8.8.jar,
- lib/slf4j-log4j12-1.7.5.jar,
- lib/slf4j-api-1.7.5.jar,
- lib/guava-${guava.version}.jar,
- lib/netty-${netty.version}.jar"/>
- </manifest>
- <fileset dir="${build.dir}" includes="classes/ lib/"/>
- <fileset dir="${root}" includes="resources/ plugin.xml"/>
- </jar>
- </target>
- </project>
0 0
- Hadoop2.4.0 Eclipse插件制作
- Hadoop2.4.0 Eclipse插件制作
- Hadoop2.4.0 Eclipse插件制作及问题记录
- Hadoop2.6.0eclipse插件制作与安装
- hadoop 2.7.2 (hadoop2.x)使用ant制作eclipse插件hadoop-eclipse-plugin-2.7.2.jar
- Hadoop2.2.0的Eclipse插件
- Hadoop2.2.0的Eclipse插件
- hadoop2.2.0 eclipse插件编译
- hadoop2.5.2 eclipse 插件编译
- eclipse hadoop2.4.1插件开发
- hadoop2.5.2eclipse插件编译
- hadoop2.5.2 eclipse 插件编译
- hadoop2.x eclipse插件编译
- windows下 hadoop2.4.0 eclipse 插件安装及配置
- hadoop eclipse插件制作
- hadoop2.2的eclipse插件编译
- Hadoop2.2.0的eclipse插件的编译
- Hadoop2.2.0的eclipse插件的编译
- 《分析服务从入门到精通读书笔记》第一章、数据分析基础(1)
- 手机App客户端与服务器端的交互
- Alignment Rectangles
- 《分析服务从入门到精通读书笔记》第一章、数据分析层次结构(2)
- crontab定期执行命令详解,以及实例
- Hadoop2.4.0 Eclipse插件制作
- Spring框架简介
- linux Oracle11g R2的安装
- 《分析服务从入门到精通读书笔记》第一章、数据分析中的维度(3)
- mini2440的pwm驱动程序和测试程序详解
- windows 2008 server web服务器的搭建以及站点的简单部署
- 当前的系统和用户属性&long
- 《分析服务从入门到精通读书笔记》第一章、维度数据仓库(4)
- 我们一起来努力