Configure Eclipse for Hadoop 1.0.4 in Ubuntu

Studyed hadoop from http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/#configuring-ssh on Nov 24, 2012

and try to build the hadoop in my PC, this blog just for remarking

OS: Ubuntu 12.04.1 desktop

Eclipse IDE for Java EE Developers: Juno (4.2) SR1 Packages for Linux

Hadoop: 1.0.4

 

Prerequisites

Make sure your PC is connecting to the network.

Install autoreconf

# Install autoreconf by the following command, otherwise, then cannot run program autoreconf.

$ sudo apt-get install automake autoconf

Install Eclipse

# Install eclipse by using the command

$ sudo apt-get install eclipse eclipse-jdt automake autoconf libtool

# The above command will download the eclipse automatic for you,

# but it will get the old version of eclipse, so I choose download the last version of eclipse by myself.

 

If you choose install by command, the eclipse should be installed under "/usr/lib/eclipse", otherwise, you need to unzip the eclipse and move to "/usr/lib/eclipse"

Install libtool and ant

This step is for compiling eclipse hadoop plugin, without the plugin, the eclipse cannot connect to hadoop and run the hadoop programs.

# install ant

$ sudo apt-get install ant

# install libtool, without the libtool, the following error will occurs when compiling the plugin:

# task-controller:

# [exec] Can`t exec "libtoolize": No such file or directory at /usr/bin/autoreconf line 196.

$ sudo apt-get install libtool

 

Compile eclipse plugin

Step one:

Some documents need to be modified:

1. In $HADOOP_HOME/src/contrib/eclipse-plugin/build.properties

    Add this line at the head: eclipse.home = /usr/lib/eclipse

2. In $HADOOP_HOME/src/contrib/build-contrib.xml

    Add this line: <property name="version" value="1.0.4"/>

the value should be the version of hadopp which install in your PC

3. In $HADOOP_HOME/build.xml

    Add this line: <property name="version" value="1.0.4"/>

4. Make sure the following content is include in $HADOOP_HOME/src/contrib/eclipse-plugin/build.xml, otherwise, some important libs will not be included in the eclipse-hadoop-plugin.

<target name="jar" depends="compile" unless="skip.contrib">

<mkdir dir="${build.dir}/lib"/><copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>

<copy file="${hadoop.root}/lib/commons-cli-1.2.jar" todir="${build.dir}/lib" verbose="true"/>

<copy file="${hadoop.root}/lib/commons-lang-2.4.jar" todir="${build.dir}/lib" verbose="true"/>

<copy file="${hadoop.root}/lib/commons-configuration-1.6.jar" todir="${build.dir}/lib" verbose="true"/>

<copy file="${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar" todir="${build.dir}/lib" verbose="true"/>

<copy file="${hadoop.root}/lib/jackson-core-asl-1.8.8.jar" todir="${build.dir}/lib" verbose="true"/>

<copy file="${hadoop.root}/lib/commons-httpclient-3.0.1.jar" todir="${build.dir}/lib" verbose="true"/>

<copy file="${hadoop.root}/build/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>

<copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/>

<jar jarfile="${build.dir}/hadoop-${name}-${version}.jar" manifest="${root}/META-INF/MANIFEST.MF"><fileset dir="${build.dir}" includes="classes/ lib/"/>

<fileset dir="${root}" includes="resources/ plugin.xml"/>

</jar>

</target>

 

Step two:

Use ant to compile plugin. There several methods, here I will give you two.

Method one:

Type these command:

$ cd $HADOOP_HOME

$ ant

If build success, there were four jar generated in build/

• hadoop-minicluster-1.0.4.jar
• hadoop-client-1.0.4.jar
• hadoop-core-1.0.4.jar
• hadoop-tools-1.0.4.jar

Then 

$ cd $HADOOP_HOME/src/contrib/eclipse-plugin

$ ant

If build success, the hadoop-eclipse-plugin-1.0.4.jar will be generated in $HADOOP_HOME/build/contrib/eclipse-plugin.

 

Method two:

First, get sun`s jdk SE 5 (to build forrest) in http://www.oracle.com/technetwork/java/archive-139210.html

$ cd ~/Downloads

$ chmod +x jdk-1_5

$ sudo ./jdk-1_5

$ mv jdk1.5.0 /opt/sun-jdk-1.5

 

Then, download Apache forrest 0.8 (0.9 will crash). I unzip it to /opt/

$ tar -xvzf ~/Downloads/apache-forrest-0.8

$ mv apache-forrest-0.8 /opt/forrest

Now lets build the plugin, go to the hadoop directory:

$ ant clean package -Djava5.home=/opt/sun-jdk-1.5 -Dforrest.home=/opt/forrest

If build success, the hadoop-eclipse-plugin-1.0.4.jar will be generated in $HADOOP_HOME/build/contrib/eclipse-plugin.

 

Both two methods will generated a hadoop-eclipse-plugin-1.0.4.jar

Extract the jar, you may find seven lib in it.

• commons-cli-1.2.jar
• commons-configuration-1.6.jar
• commons-httpclient-3.0.1.jar
• commons-lang-2.4.jar
• hadoop-core-asl-1.8.8.jar
• jackson-core-asl-1.8.8.jar
• jackson-mapper-asl-1.8.8.jar

Extract all of them to classes, put them in the reasonable path, After that, package the whole directory to be a jar and replace the origin one.

 

 

Step three:

Put the hadoop-eclipse-plugin-1.0.4.jar into eclipse/dropins.

Normally, all the eclipse plugin should be put in eclipse/plugins, but it does not work, ones we put it into the dropins folder by chance, we find that it works.

 

Remark：

I have met some problems while I compiling the plugin, this part I will give the solution of the problems.

 

Problem one:

  [exec] * [15/35]   [0/0]     0.086s 0b      hdfs_user_guide.pdf
  [exec] Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/fop/messaging/MessageHandler
  [exec]     at org.apache.cocoon.serialization.FOPSerializer.configure(FOPSerializer.java:122)
  [exec]     at org.apache.avalon.framework.container.ContainerUtil.configure(ContainerUtil.java:201)
  [exec]     at org.apache.avalon.excalibur.component.DefaultComponentFactory.newInstance(DefaultComponentFactory.java:289)

Solution: $ ant clean

 

Problem two:

[javac] /home/idlemind/workspace/hadoop/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java:396: error: type argument ? extends T is not within bounds of type-variable E

This is a bug of hadoop.

Solution: Modify the line 396 of Gridmix.java

             Original: private <T> String getEnum<? extends T>[] e) {

             Modify:  private String getEnum<?>[] e) {

 

             Original: for (Enum<? extends T>v: e) {

             Modify:  for (Enum<?>v: e) {

 

Problem three:

[exec] /usr/include/fts.h:41:3: error: #error "<fts.h> cannot be used with -D_FILE_OFFSET_BITS==64"
[exec] make: *** [impl/task-controller.o] Error 1

Solution: Find the line "AC_SYS_LARGEFILE" in $HADOOP_HOME/src/c++/task-controller/configure.ac, this line is no use at all, so, add a ‘#’ before it.

you can see more details of this problem in https://issues.apache.org/jira/browse/MAPREDUCE-2178