hadoop安装测试流程

hadoop下载地址： 
http://mirrors.hust.edu.cn/apache/hadoop/common/stable1/hadoop-1.2.1.tar.gz


jdk下载地址：
http://download.oracle.com/otn-pub/java/jdk/8u5-b13/jdk-8u5-linux-i586.rpm?AuthParam=1400373937_3662abd85d49620b01b89da44dec609c


2、安装jdk
将jdk移动到/opt下，以root账号安装：
rpm -ivh jdk-8u5-linux-i586.rpm
安装位置在/usr/java。
设置环境路径：
/etc/profile在最后面加上：
export JAVA_HOME=/usr/java/jdk1.8.0_05
export CLASSPATH=.:$JAVA_HOME/lib
export PATH=$JAVA_HOME/bin:$PATH
保存，然后source /etc/profile


3、安装hadoop
解压到/opt路径下，并chown hadoop:hadoop hadoop-1.2.1


4、单机环境
默认fs.default.name为file:///，表示本地文件系统。
所以单机版的，用hadoop fs -ls列出的是本地的文件系统。


5、伪分布式
1)编辑hadoop-env.sh
加上：export JAVA_HOME=/usr/java/jdk1.8.0_05


2）编辑conf/core-site.xml
设置临时目录：hadoop.tmp.dir
 <property>
   <name>hadoop.tmp.dir</name>
   <value>/home/hadoop/hadoop/tmp</value>
   <description>A base for other temporary directories.</description>
 </property>
  


设置fs.default.name为：
<property>
  <name>fs.default.name</name>
  <value>hdfs://hadoop1:9000</value>
  <description>The name of the default file system.  A URI whose
         scheme and authority determine the FileSystem implementation.  The
         uri's scheme determines the config property (fs.SCHEME.impl) namin    g
         the FileSystem implementation class.  The uri's authority is use    d                determine the host, port, etc. for a filesystem.</description>
  </property>


3）编辑conf/hdfs-site.xml
设置dfs.name.dir（可不设置，默认即可）：
<property>
     <name>dfs.name.dir</name>
     <value>${hadoop.tmp.dir}/dfs/name</value>
 </property>
设置dfs.data.dir（可不设置，默认即可）：
<property>
    <name>dfs.data.dir</name>
    <value>${hadoop.tmp.dir}/dfs/data</value>
 </property>
 
设置replication（可不设置，默认默认为3即可）：：
<name>dfs.replication</name>
      <value>1</value>
 </property>


4）编辑conf/mapred-site.xml
设置mapred.job.tracker：
<property>
  <name>mapred.job.tracker</name>
  <value>hadoop1:9001</value>
  <description>The host and port that the MapReduce job tracker runs
        at.  If "local", then jobs are run in-process as a single map
          and reduce task.
  </description>
</property>




5）编辑conf/master
改为hadoop1
6）编辑conf/slavers
改为hadoop1
7）ssh-keygen -t rsa


8）格式化操作，在第一次启动之前必须：
[hadoop@hadoop1 hadoop-1.2.1]$ bin/hadoop namenode -format
Warning: $HADOOP_HOME is deprecated.


14/05/18 14:49:42 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = hadoop1/192.168.253.135
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.2.1
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG:   java = 1.8.0_05
************************************************************/
14/05/18 14:49:42 INFO util.GSet: Computing capacity for map BlocksMap
14/05/18 14:49:42 INFO util.GSet: VM type       = 32-bit
14/05/18 14:49:42 INFO util.GSet: 2.0% max memory = 1013645312
14/05/18 14:49:42 INFO util.GSet: capacity      = 2^22 = 4194304 entries
14/05/18 14:49:42 INFO util.GSet: recommended=4194304, actual=4194304
14/05/18 14:49:43 INFO namenode.FSNamesystem: fsOwner=hadoop
14/05/18 14:49:43 INFO namenode.FSNamesystem: supergroup=supergroup
14/05/18 14:49:43 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/05/18 14:49:44 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/05/18 14:49:44 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/05/18 14:49:44 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
14/05/18 14:49:44 INFO namenode.NameNode: Caching file names occuring more than 10 times 
14/05/18 14:49:45 INFO common.Storage: Image file /home/hadoop/hadoop/tmp/dfs/name/current/fsimage of size 112 bytes saved in 0 seconds.
14/05/18 14:49:45 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/home/hadoop/hadoop/tmp/dfs/name/current/edits
14/05/18 14:49:45 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/home/hadoop/hadoop/tmp/dfs/name/current/edits
14/05/18 14:49:45 INFO common.Storage: Storage directory /home/hadoop/hadoop/tmp/dfs/name has been successfully formatted.
14/05/18 14:49:45 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop1/192.168.253.135
************************************************************/


[hadoop@hadoop1 hadoop-1.2.1]$ start-all.sh 
Warning: $HADOOP_HOME is deprecated.


starting namenode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-namenode-hadoop1.out
hadoop@hadoop1's password: 
hadoop1: starting datanode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-hadoop1.out
hadoop@hadoop1's password: 
hadoop1: starting secondarynamenode, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode-hadoop1.out
starting jobtracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-jobtracker-hadoop1.out
hadoop@hadoop1's password: 
hadoop1: starting tasktracker, logging to /opt/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-hadoop1.out
[hadoop@hadoop1 hadoop-1.2.1]$ jps
31123 DataNode
31333 JobTracker
31527 Jps
31482 TaskTracker
31003 NameNode
31260 SecondaryNameNode


6、编写简单程序
文件查询等操作
eclipse建立工程，将hadoop-1.2.1/lib下的jar包，以及hadoop-1.2.1/hadoop-client-1.2.1.jar,hadoop-core-1.2.1.jar加入工程，编写如下代码：
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;




public class ListHdfs {
public static void main(String[] args){
     System.out.println("in ListHdfs.java");
 try{
       Configuration conf=new Configuration();
 
       FileSystem fs=FileSystem.get(conf);
 
       String uri="hdfs://hadoop1:9000/input/core-site.xml";
       Path path=new Path(uri);
 
       boolean isFile=fs.isFile(path);
      System.out.println("is File :"+isFile);
 
      }catch(Exception e){
          System.err.println("error!");
          e.printStackTrace();
      }
}
}


export为jar包，放到linux上。
运行：
[hadoop@hadoop1 javaprog]$ hadoop jar ListHdfs.jar
Warning: $HADOOP_HOME is deprecated.


in ListHdfs.java
is File :true
[hadoop@hadoop1 javaprog]$