Linux中Hadoop的安装和配置方法

来源：互联网发布：获取httppost请求数据编辑：程序博客网时间：2024/06/02 02:51

1、下载hadoop-2.4.1.tar.gz

2、切换创建usr/java目录解压tar包
mkdir /home/hadoop/app
cd /home/hadoop/app
tar -zxvf /software/hadoop-2.4.1.tar.gz

3、切换创建etc/hadoop修改配置文件
cd /home/hadoop/app/hadoop-2.4.1/etc/hadoop
修改jdk路径
vi hadoop-env.sh
JAVA_HOME = /usr/java/jdk1.7.0_67

4、修改配置文件
   4.1)vi core-site.xml
   ----------------------------------start----------------------------------
   <property>
   <name>fs.defaultFS</name>
   <value>hdfs://weekend110:9000/</value>
   </property>

   <property>
   <name>hadoop.tmp.dir</name>
   <value>/home/hadoop/app/hadoop-2.4.1/data/</value>
   </property>
   ----------------------------------end----------------------------------

   4.2)vi httpfs-site.xml
   ----------------------------------start----------------------------------
   <property>
   <name>dfs.replication</name>
   <value>1</value>
   </property>
   ----------------------------------end----------------------------------

   4.3)mv mapred-site.xml.template mapred-site.xml
   vi mapred-site.xml
   ----------------------------------start----------------------------------
   <property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
   </property>
   ----------------------------------end----------------------------------

   4.4)vi yarn-site.xml
   ----------------------------------start----------------------------------
   <property>
   <name>yarn.resourcemanager.hostname</name>
   <value>weekend110</value>
   </property>

   <property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
   </property>
   ----------------------------------end----------------------------------

5、关闭防火墙
service iptables stop
chkconfig iptables --list
chkconfig iptables off

6、设置环境变量
export JAVA_HOME=/usr/java/jdk1.7.0_67
export HADOOP_HOME=/home/hadoop/app/hadoop-2.4.1
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

source /etc/profile

7、hadoop清理格式化namenode
hadoop namenode -format

8、启动
start-dfs.sh stop-dfs.sh

9、查看进程jps
30370 Jps
30268 SecondaryNameNode
30083 DataNode
29991 NameNode

http://weekend110:50070/dfshealth.html#tab-overview

异常
ssh: Could not resolve hostname you: Temporary failure in name resolution
解决办法：在 /etc/profile 中添加如下信息：
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
执行 source /etc/profile 使之立即生效。

0 0