hadoop安装

来源：互联网发布：凯立德导航端口怎么看编辑：程序博客网时间：2024/04/29 22:38

准备：hadoop-2.7.1.tar.gz文件，jdk1.7、linux环境等

1、创建hadoop用户用来，运行hadoop集群

useradd hadoop

passwd hadoop

2、所有机器之间做双机互信，master通过远程启动datanode进程和tasktracker进程，如果不做双机互信，会导致每次启动集群服务都会需要密码

分别执行：

vim /etc/hosts

172.19.34.185 hadoop185

172.19.34.109 hadoop109

su hadoop

ssh-keygen -t rsa # 设置ssh无密码登录

ssh-copy-id -i hadoop185

3、各机器做时间同步

su root

/usr/sbin/ntpdate ntp.api.bz

crontab -e

*/5 * * * * /usr/sbin/ntpdate ntp.api.bz &>/dev/null

4、配置hadoop-master

tar xf hadoop-2.7.1.tar.gz -C /usr/local/

chown -R hadoop:hadoop /usr/local/hadoop-2.7.1

mv /usr/local/hadoop-2.7.1 /usr/local/hadoop

su hadoop

vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh

# 配置：exportJAVA_HOME=${JAVA_HOME} ，可通过echo ${JAVA_HOME}查看

vim /usr/local/hadoop/etc/hadoop/core-site.xml

<name>hadoop.tmp.dir</name>

<value>/usr/local/hadoop/tmp</value>

</property>

<name>fs.default.name</name>

<value>hdfs://linux-3:9000</value>

</property>

</configuration>

# hadoop.tmp.dir属性用于定义Hadoop的临时目录，其默认为/tmp/hadoop-${username}。HDFS进程的许多目录默认都在此目录中，要保证运行Hadoop进程的用户对其具有全部访问权限。

# fs.default.name属性用于定义HDFS的名称节点和其默认的文件系统，其值是一个URI，即NameNode的RPC服务器监听的地址(可以是主机名)和端口(默认为8020)。其默认值为file:///，即本地文件系统。

vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<value>/usr/local/hadoop/data</value>

</property>

<name>dfs.replication</name>

</property>

</configuration>

vi/usr/local/hadoop/etc/hadoop/mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

</configuration>

vi/usr/local/hadoop/etc/hadoop/yarn-site.xml

<name>yarn.resourcemanager.hostname</name>

<value>ubuntu3</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

vi /usr/local/hadoop/etc/hadoop/slaves

hadoop185

hadoop109

5、将已配置好的hadoop，scp到其它节机器

scp -r usr/local/hadoop hadoop@172.19.34.109:/usr/local/

6、到bin目录下执行format检查是否报错

cd /usr/local/hadoop/bin

./hadoop namenode –format

# 输出检测结果

15/07/17 09:37:08 INFO metrics.TopMetrics:NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25

15/07/17 09:37:08INFO namenode.FSNamesystem: Retry cache on namenode is enabled

15/07/17 09:37:08 INFO namenode.FSNamesystem:Retry cache will use 0.03 of total heap and retry cache entry expiry time is600000 millis

15/07/17 09:37:08INFO util.GSet: Computing capacity for map NameNodeRetryCache

15/07/17 09:37:08 INFO util.GSet: VM type = 64-bit

15/07/17 09:37:08 INFO util.GSet: 0.029999999329447746%max memory 888.9 MB = 273.1 KB

15/07/17 09:37:08INFO util.GSet: capacity = 2^15 =32768 entries

15/07/17 09:37:08 INFO namenode.FSImage:Allocated new BlockPoolId: BP-660215815-172.19.34.185-1437125828943

15/07/17 09:37:09 INFO common.Storage: Storagedirectory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.

15/07/17 09:37:09 INFOnamenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

15/07/17 09:37:09 INFO util.ExitUtil: Exitingwith status 0

15/07/17 09:37:09 INFO namenode.NameNode:SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG:Shutting down NameNode at linux-3/172.19.34.185

************************************************************/

# 报错则根据具体报错情况调整

# 启动hadoop

# 主节点运行 , 执行成功后两个节点执行jps看服务状态

./start-all.sh

[hadoop@linux-3 sbin]$ jps

27709 NameNode

28628 Jps

28186 ResourceManager

28024 SecondaryNameNode

27845 DataNode

28297 NodeManager

[hadoop@xxfw-tomcat sbin]$ jps

3825 NodeManager

3677 DataNode

4087 Jps

http://NameNode:8088/ 查看yarn

http://NameNode:50070/ 查看hdfs

0 0