Hadoop-2.5.1集群环境搭建

1. 新建目录，在这个目录下安装自己的jdk和hadoop
2. JDK安装
   这个很简单，官网下载jdk到自己的用户目录下,解压配置环境变量，设为系统默认的jdk。
3. Hadoop-2.5.1的安装
   下载官方安装文件，解压到自己的安装目录，先按本地模式安装，在按伪分布模式建立和修改配置文件。全分布配置，需要的配置如下：
   core-site.xml

<configuration>    <property>        <name>fs.default.name</name>        <value>hdfs://master:9000</value>    </property>    <property>        <name>hadoop.tmp.dir</name>        <value>/usr/local/tangm/hadoop/hadoop-2.5.1/tmp</value>    </property>    <property>            <name>dfs.namenode.name.dir</name>            <value>/usr/local/tangm/hadoop/hadoop-2.5.1/hdfs/name</value>    </property>    <property>            <name>dfs.datanode.data.dir</name>            <value>/usr/local/tangm/hadoop/hadoop-2.5.1/hdfs/data</value>    </property></configuration>

hadoop-env.sh

export JAVA_HOME=/usr/local/jdk1.7.0_45export HADOOP_HOME=/usr/local/tangm/hadoop/hadoop-2.5.1export PATH=$PATH:${HADOOP_HOME}/bin

mapred-site.xml

<configuration>    <property>        <name>mapreduce.framework.name</name>        <value>Yarn</value>    </property>    <property>        <name>mapreduce.jobhistory.address</name>        <value>master:10020</value>    </property>    <property>        <name>mapreduce.jobhistory.webapp.address</name>        <value>master:19888</value>    </property></configuration>

hdfs-site.xml

<configuration>    <property>            <name>dfs.replication</name>            <value>3</value>    </property>    <property>            <name>dfs.namenode.name.dir</name>            <value>/usr/local/tangm/hadoop/hadoop-2.5.1/hdfs/name</value>    </property>    <property>            <name>dfs.namenode.secondary.http-address</name>            <value>master:9001</value>    </property>    <property>            <name>dfs.datanode.data.dir</name>            <value>/usr/local/tangm/hadoop/hadoop-2.5.1/hdfs/data</value>    </property>    <property>            <name>dfs.webhdfs.enabled</name>            <value>true</value>    </property></configuration>

yarn-env.sh

export JAVA_HOME=/usr/local/jdk1.7.0_45

yarn-site.xml

<configuration>    <!-- Site specific YARN configuration properties -->    <property>        <name>mapreduce.framework.name</name>        <value>yarn</value>    </property>    <property>        <name>yarn.nodemanager.aux-services</name>        <value>mapreduce_shuffle</value>    </property>    <property>        <description>The address of the applications manager interface in the RM.</description>        <name>yarn.resourcemanager.address</name>        <value>master:18040</value>    </property>    <property>        <description>The address of the scheduler interface.</description>        <name>yarn.resourcemanager.scheduler.address</name>        <value>master:18030</value>    </property>    <property>        <description>The address of the RM web application.</description>        <name>yarn.resourcemanager.webapp.address</name>        <value>master:18088</value>    </property>    <property>        <description>The address of the resource tracker interface.</description>        <name>yarn.resourcemanager.resource-tracker.address</name>        <value>master:8025</value>      </property>    <property>        <description>The address of the resource tracker interface.</description>        <name>yarn.resourcemanager.admin.address</name>        <value>master:8033</value>      </property></configuration>

slaves

slave1slave2slave3

4.遇到的问题
（1）etc/hadoop/slaves：没有这个文件或目录

[root@master hadoop-2.5.1]# sbin/start-dfs.sh Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.Starting namenodes on [2015-06-03 17:13:21,548 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable]-: usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]-:            [-D [bind_address:]port] [-e escape_char] [-F configfile]-:            [-i identity_file] [-L [bind_address:]port:host:hostport]-:            [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]-:            [-R [bind_address:]port:host:hostport] [-S ctl_path]-:            [-W host:port] [-w local_tun[:remote_tun]]-:            [user@]hostname [command]util.NativeCodeLoader: ssh: Could not resolve hostname util.NativeCodeLoader: Temporary failure in name resolutionUnable: ssh: Could not resolve hostname Unable: Temporary failure in name resolution2015-06-03: ssh: Could not resolve hostname 2015-06-03: Temporary failure in name resolutionWARN: ssh: Could not resolve hostname WARN: Temporary failure in name resolutionnative-hadoop: ssh: Could not resolve hostname native-hadoop: Temporary failure in name resolution[main]: ssh: Could not resolve hostname [main]: Temporary failure in name resolution(NativeCodeLoader.java:<clinit>(62)): ssh: Could not resolve hostname (NativeCodeLoader.java:<clinit>(62)): Temporary failure in name resolution17:13:21,548: ssh: Could not resolve hostname 17:13:21,548: Temporary failure in name resolutionlibrary: ssh: Could not resolve hostname library: Temporary failure in name resolutionplatform...: ssh: Could not resolve hostname platform...: Name or service not knownload: ssh: Could not resolve hostname load: Temporary failure in name resolutionusing: ssh: Could not resolve hostname using: Temporary failure in name resolutionfor: ssh: Could not resolve hostname for: Temporary failure in name resolutionapplicable: ssh: Could not resolve hostname applicable: Temporary failure in name resolutionwhere: ssh: Could not resolve hostname where: Temporary failure in name resolutionbuiltin-java: ssh: Could not resolve hostname builtin-java: Temporary failure in name resolutionclasses: ssh: Could not resolve hostname classes: Temporary failure in name resolutionyour: ssh: Could not resolve hostname your: Temporary failure in name resolutionto: ssh: Could not resolve hostname to: Temporary failure in name resolutioncat: /etc/hadoop/slaves: 没有那个文件或目录

就是这个slaves文件没有，或者不是纯文本文档格式的文件，重写一个就ok。
（2）Exception in thread “main” java.lang.UnsupportedClassVersionError

[root@master hadoop-2.5.1]# bin/hadoop jar test.jar Exception in thread "main" java.lang.UnsupportedClassVersionError: com/tangm/CGAMapRedDriver : Unsupported major.minor version 52.0    at java.lang.ClassLoader.defineClass1(Native Method)    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)    at java.security.AccessController.doPrivileged(Native Method)    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)    at java.lang.Class.forName0(Native Method)    at java.lang.Class.forName(Class.java:270)    at org.apache.hadoop.util.RunJar.main(RunJar.java:205)[root@master hadoop-2.5.1]# 

eclipse中编译的jdk版本（Java Compiler）与服务器jdk不一致，选择一个一直的版本作为compiler就ok。
（3）hdfs操作命令执行不成功

[root@master hadoop-2.5.1]# bin/hadoop dfs -mkdir inputDEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.

hadoop版本将hadoop 命令换成了hdfs命令，上面的命令如同下面的命令：

[root@master hadoop]$ hdfs dfs -ls