安装hadoop过程

来源：互联网发布：网络用语兔子编辑：程序博客网时间：2024/05/16 11:13

1.ssh command not found

解决 yum remove openssh

yum -y install openssh*

2. centos 网管设置

   设置文件 /etc/resolv.conf
   nameserver 8.8.8.8
   nameserver 8.8.4.4

3. 默认网关设置

   设置文件 /etc/sysconfig/network
    NETWORKING=yes
    HOSTNAME=localhost.localdomain
    GATEWAY=192.168.1.1
4. IP设置

   设置文件 /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
   HWADDR=00:0C:29:EF:89:9A
   TYPE=Ethernet
   UUID=765d7345-0ea4-4a0b-8aa7-fd744b769ab6
   ONBOOT=yes
   NM_CONTROLLED=yes
   BOOTPROTO=static
   IPADDR=192.168.1.224
   NETMASK=255.255.255.0
   GATEWAY=192.168.1.1

5.环境变量设置

设置文件 /etc/profile

export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_HOME=/usr/local/hadoop
export JAVA_HOME=/usr/local/jdk
export PATH=.:$HADOOP_HOME:$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools/jar

6.设置无密码登录

cd ~ 可以进入你的home目录

1.在A机中执行 cd ~/.ssh

2.ssh-keygen -t rsa --------- 然后一路回车就会按照默认的选项将生成的密钥保存在.ssh/id_rsa文件中

3. cp id_rsa.pub authorized_keys 这步完成后，正常情况下就可以无密码登陆本机了即ssh localhost 无需输入密码

4. scp authorized_keys root@192.168.1.224:/root/.ssh

5. 进入B机 chomd 600 authorized_keys

正常情况下上面几步执行完成后，从主机A所在机器向主机A、主机B所在机器发起ssh连接，只有在第一次登录时需要输入密码，以后则不需要。

hadoop中运行bin/hadoop fs -mkdir /test-hadoop报错

mkdir: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /test-hadoop. Name node is in safe mode.

解决方法： bin/hadoop dfsadmin -safemode leave #关闭safe mode

hadoop 的文件命令格式如下:
    hadoop fs -cmd <args>
    //建立目录
    [hadoop@hadoop1 hadoop]$ bin/hadoop fs -mkdir /test-hadoop
    //査看目录
    [hadoop@hadoop1 hadoop]$ bin/hadoop fs -ls /
    Found 2 items
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 13:32 /test-hadoop
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp
    //査看目录包括子目录
    [hadoop@hadoop1 hadoop]$ bin/hadoop fs -lsr /
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 13:32 /test-hadoop
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp/hadoop-hadoop
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp/hadoop-hadoop/mapred
    drwx------   - hadoop supergroup          0 2012-02-02 11:32 /tmp/hadoop-hadoop/mapred/system
    -rw-------   2 hadoop supergroup          4 2012-02-02 11:32 /tmp/hadoop-hadoop/mapred/system/jobtracker.info
    //添加文件
    [hadoop@hadoop1 hadoop]$ bin/hadoop fs -put /home/hadoop/hadoop-0.20.203.0rc1.tar.gz /test-hadoop
    [hadoop@hadoop1 hadoop]$ bin/hadoop fs -lsr /
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 13:34 /test-hadoop
    -rw-r--r--   2 hadoop supergroup   60569605 2012-02-02 13:34 /test-hadoop/hadoop-0.20.203.0rc1.tar.gz
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp/hadoop-hadoop
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp/hadoop-hadoop/mapred
    drwx------   - hadoop supergroup          0 2012-02-02 11:32 /tmp/hadoop-hadoop/mapred/system
    -rw-------   2 hadoop supergroup          4 2012-02-02 11:32 /tmp/hadoop-hadoop/mapred/system/jobtracker.info
    //获取文件
    [hadoop@hadoop1 hadoop]$ bin/hadoop fs -get /test-hadoop/hadoop-0.20.203.0rc1.tar.gz /tmp/
    [hadoop@hadoop1 hadoop]$ ls /tmp/*.tar.gz
    /tmp/1.tar.gz /tmp/hadoop-0.20.203.0rc1.tar.gz
    //删除文件
    [hadoop@hadoop1 hadoop]$ bin/hadoop fs -rm /test-hadoop/hadoop-0.20.203.0rc1.tar.gz
    Deleted hdfs://namenode:9000/test-hadoop/hadoop-0.20.203.0rc1.tar.gz
    [hadoop@hadoop1 hadoop]$ bin/hadoop fs -lsr /
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 13:57 /test-hadoop
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp/hadoop-hadoop
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp/hadoop-hadoop/mapred
    drwx------   - hadoop supergroup          0 2012-02-02 11:32 /tmp/hadoop-hadoop/mapred/system
    -rw-------   2 hadoop supergroup          4 2012-02-02 11:32 /tmp/hadoop-hadoop/mapred/system/jobtracker.info
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 13:36 /user
    -rw-r--r--   2 hadoop supergroup        321 2012-02-02 13:36 /user/hadoop
    //删除目录
    [hadoop@hadoop1 hadoop]$ bin/hadoop fs -rmr /test-hadoop
    Deleted hdfs://namenode:9000/test-hadoop
    [hadoop@hadoop1 hadoop]$ bin/hadoop fs -lsr /
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp/hadoop-hadoop
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 11:32 /tmp/hadoop-hadoop/mapred
    drwx------   - hadoop supergroup          0 2012-02-02 11:32 /tmp/hadoop-hadoop/mapred/system
    -rw-------   2 hadoop supergroup          4 2012-02-02 11:32 /tmp/hadoop-hadoop/mapred/system/jobtracker.info
    drwxr-xr-x   - hadoop supergroup          0 2012-02-02 13:36 /user
    -rw-r--r--   2 hadoop supergroup        321 2012-02-02 13:36 /user/hadoop

    //hadoop fs 帮助（部分）
    [hadoop@hadoop1 hadoop]$ bin/hadoop fs -help
    hadoop fs is the command to execute fs commands. The full syntax is:

    hadoop fs [-fs <local | file system URI>] [-conf <configuration file>]
        [-D <propertyproperty=value>] [-ls <path>] [-lsr <path>] [-du <path>]
        [-dus <path>] [-mv <src> <dst>] [-cp <src> <dst>] [-rm [-skipTrash] <src>]
        [-rmr [-skipTrash] <src>] [-put <localsrc> ... <dst>] [-copyFromLocal <localsrc> ... <dst>]
        [-moveFromLocal <localsrc> ... <dst>] [-get [-ignoreCrc] [-crc] <src> <localdst>
        [-getmerge <src> <localdst> [addnl]] [-cat <src>]
        [-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>] [-moveToLocal <src> <localdst>]
        [-mkdir <path>] [-report] [-setrep [-R] [-w] <rep> <path/file>]
        [-touchz <path>] [-test -[ezd] <path>] [-stat [format] <path>]
        [-tail [-f] <path>] [-text <path>]
        [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
        [-chown [-R] [OWNER][:[GROUP]] PATH...]
        [-chgrp [-R] GROUP PATH...]
        [-count[-q] <path>]
        [-help [cmd]]

在Hadoop添加文件中可能出现

WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /test-hadoop/hadoop-1.1.2.tar.gz could only be replicated to 0 nodes, instead of 1

解决方法:

先执行 sh stop-all.sh

重新启动namenode

# hadoop-daemon.sh start namenode

starting namenode, logging to /usr/hadoop-0.21.0/bin/../logs/hadoop-root-namenode-www.keli.com.out

重新启动datanode

# hadoop-daemon.sh start datanode

0 0