hadoop（备忘录）

1.配置

mapred-site.xml:

<configuration><property>    <name>mapred.job.tracker</name>    <value>hdfs://localhost:9001</value>  </property></configuration>

core-site.xml:

<configuration>  <property>        <name>hadoop.tmp.dir</name>        <value>/opt/hadoop/tmp/hadoop-${user.name}</value>        <description>A base for other temporary directories.</description>    </property><property>    <name>fs.default.name</name>    <value>hdfs://localhost:9000</value>  </property></configuration>

hdfs-site.xml:

<configuration> <property>    <name>dfs.replication</name>    <value>1</value>  </property></configuration>

2命令：

1）格式化

 bin/hadoop namenode -format

2）启动Hadoop

bin/start-all.sh

3) MapReduce：

http://localhost:50030/

4）HDFS：

http://localhost:50070/

3.编译生成Eclipse插件

1)修改build.xml

a.

<!— 2.添加如下，否则提示"软件包 org.apache.hadoop.fs 不存在"—>
<path id="hadoop-jars">
  <fileset dir="${hadoop.root}/">
    <include name="hadoop-*.jar"/>
  </fileset>
</path>

b.

<path id="classpath">
  <pathelement location="${build.classes}"/>
  <pathelement location="${hadoop.root}/build/classes"/>
  <path refid="eclipse-sdk-jars"/>
  <path refid="hadoop-jars"/>
</path>

c.

<!— 3.修改并添加 jar 包拷贝—>
<target name="jar" depends="compile" unless="skip.contrib">
  <mkdir dir="${build.dir}/lib"/>
  <copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>
  <copy file="${hadoop.root}/lib/commons-cli-1.2.jar" todir="${build.dir}/lib" verbose="true"/>
  <copy file="${hadoop.root}/lib/commons-lang-2.4.jar" todir="${build.dir}/lib" verbose="true"/>
  <copy file="${hadoop.root}/lib/commons-configuration-1.6.jar" todir="${build.dir}/lib" verbose="true"/>
  <copy file="${hadoop.root}/lib/jackson-mapper-asl-1.0.1.jar" todir="${build.dir}/lib" verbose="true"/>
  <copy file="${hadoop.root}/lib/jackson-core-asl-1.0.1.jar" todir="${build.dir}/lib" verbose="true"/>
  <copy file="${hadoop.root}/lib/commons-httpclient-3.0.1.jar" todir="${build.dir}/lib" verbose="true"/>
  <jar jarfile="${build.dir}/hadoop-${name}-${version}.jar" manifest="${root}/META-INF/MANIFEST.MF">
    <fileset dir="${build.dir}" includes="classes/ lib/"/>
    <fileset dir="${root}" includes="resources/ plugin.xml"/>
  </jar>
</target>

d.

lib / hadoop-core.jar, lib/commons-cli-1.2.jar,lib/commons-httpclient-3.0.1.jar, lib/jackson-core-asl-1.0.1.jar , lib/jackson-mapper-asl-1.0.1.jar, lib/commons-configuration-1.6.jar, lib/commons-lang-2.4.jar

e.运行

ant -Declipse.home=/opt/eclipse -Dversion=1.0.0

{HADOOP_HOME}/build/contrib 下生成 hadoop-eclipse-plugin-1.0.0.jar文件。

4.调试

在 Hadoop 安装目录下运行内容如下的 Shell 脚本:
export HADOOP_JOBTRACKER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,address=8788,server=y,suspend=y"
bin/start-all.sh
如果脚本运行成功,则可以看到 Shell 命令行终端显示如下信息:
Listening for transport dt_socket at address: 8788