Hadoop2.7.4 MapReduce集群的Linux安装步骤
来源:互联网 发布:怎么运行js文件 编辑:程序博客网 时间:2024/06/06 07:37
一、 目标
Hadoop2.X MapReduce集群搭建
操作系统:
CentOS Linux release 7.3.1611
服务器集群:
Node-0: 192.168.2.200
Node-1: 192.168.2.201
Node-2: 192.168.2.202
Node-3: 192.168.2.203
Node-4: 192.168.2.204
Node-5: 192.168.2.205
节点:
NN: NameNode
DN: DataNode
ZK: Zookeeper
ZKFC: Zookeeper Failover Controller
JN: JournalNoe
RM: YARN ResourceManager
DM: DataManager
部署规划:
NN
DN
ZK
ZKFC
JN
RM
DM
Node-0
1
1
1
1
Node-1
1
1
1
Node-2
1
1
1
1
Node-3
1
1
1
1
Node-4
1
1
1
1
Node-5
1
1
二、 MapReduce配置
1. 修改etc/hadoop/mapred-site.xml,指定yarn为Hadoop的执行框架
/home/hadoop/hadoop2.7/etc/hadoop
cp mapred-site.xml.template mapred-site.xml
vi mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
2. 修改etc/hadoop/yarn-site.xml(设置RM主机,为MapReduce设置Shuffle服务)
cd /home/hadoop/hadoop2.7/etc/hadoop
vi yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>node0</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
3. 将之前HDFS服务停止
ssh node0
cd /home/hadoop/hadoop2.7/sbin/
./stop-dfs.sh
exit
4. 将配置文件拷贝至集群其他节点
cd /home/hadoop/hadoop2.7/etc/hadoop/
scp ./* root@node1:/home/hadoop/hadoop2.7/etc/hadoop/
scp ./* root@node2:/home/hadoop/hadoop2.7/etc/hadoop/
scp ./* root@node3:/home/hadoop/hadoop2.7/etc/hadoop/
scp ./* root@node4:/home/hadoop/hadoop2.7/etc/hadoop/
scp ./* root@node5:/home/hadoop/hadoop2.7/etc/hadoop/
5. 启动集群(HDFS与MapReduce均启动)
实际上是start-dfs.sh和start-yarn.sh
ssh node0
cd /home/hadoop/hadoop2.7/sbin/
./start-all.sh
exit
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [node0 node1]
node0: starting namenode, logging to /usr/local/hadoop-2.7.4/logs/hadoop-root-namenode-gyrr-centos-node-0.out
node1: starting namenode, logging to /usr/local/hadoop-2.7.4/logs/hadoop-root-namenode-gyrr-centos-node-1.out
node2: starting datanode, logging to /usr/local/hadoop-2.7.4/logs/hadoop-root-datanode-gyrr-centos-node-2.out
node4: starting datanode, logging to /usr/local/hadoop-2.7.4/logs/hadoop-root-datanode-gyrr-centos-node-4.out
node3: starting datanode, logging to /usr/local/hadoop-2.7.4/logs/hadoop-root-datanode-gyrr-centos-node-3.out
node5: starting datanode, logging to /usr/local/hadoop-2.7.4/logs/hadoop-root-datanode-GYRR-CentOS-Node-5.out
Starting journal nodes [node2 node3 node4]
node3: starting journalnode, logging to /usr/local/hadoop-2.7.4/logs/hadoop-root-journalnode-gyrr-centos-node-3.out
node4: starting journalnode, logging to /usr/local/hadoop-2.7.4/logs/hadoop-root-journalnode-gyrr-centos-node-4.out
node2: starting journalnode, logging to /usr/local/hadoop-2.7.4/logs/hadoop-root-journalnode-gyrr-centos-node-2.out
Starting ZK Failover Controllers on NN hosts [node0 node1]
node0: starting zkfc, logging to /usr/local/hadoop-2.7.4/logs/hadoop-root-zkfc-gyrr-centos-node-0.out
node1: starting zkfc, logging to /usr/local/hadoop-2.7.4/logs/hadoop-root-zkfc-gyrr-centos-node-1.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.7.4/logs/yarn-root-resourcemanager-gyrr-centos-node-0.out
node4: starting nodemanager, logging to /usr/local/hadoop-2.7.4/logs/yarn-root-nodemanager-gyrr-centos-node-4.out
node2: starting nodemanager, logging to /usr/local/hadoop-2.7.4/logs/yarn-root-nodemanager-gyrr-centos-node-2.out
node5: starting nodemanager, logging to /usr/local/hadoop-2.7.4/logs/yarn-root-nodemanager-GYRR-CentOS-Node-5.out
node3: starting nodemanager, logging to /usr/local/hadoop-2.7.4/logs/yarn-root-nodemanager-gyrr-centos-node-3.out
jps与节点规划表中的节点进行对比
部署规划:
NN
DN
ZK
ZKFC
JN
RM
DM
Node-0
1
1
1
1
Node-1
1
1
1
Node-2
1
1
1
1
Node-3
1
1
1
1
Node-4
1
1
1
1
Node-5
1
1
三、 测试
1. 浏览器访问RM节点http://node0:8088/
- Hadoop2.7.4 MapReduce集群的Linux安装步骤
- Hadoop2.7.4集群的Linux安装步骤
- Hadoop2.7.4 MapReduce开发部署步骤
- hadoop2.x集群的安装
- zookeeper集群的Linux安装步骤
- Kafka集群的Linux安装步骤
- hadoop2.6.5集群安装及mapreduce测试运行
- 最全的hadoop2.4.1版本分布式集群高可用模式安装步骤
- Linux集群Hadoop2.5.1完全分布式安装
- Linux CentOS7安装Hadoop2.7集群
- hadoop2.6.5 HDFS Mapreduce的高可用集群搭建
- hadoop2.X集群上Hbase的安装
- 【Hadoop入门】Hadoop2.8的安装步骤
- hbase安装及部署(hadoop2.7.4集群)
- Hadoop2.7.4分布式集群安装配置文件
- hadoop2.2.0集群安装
- Hadoop2.5.1集群安装
- 安装hadoop2集群
- 计算机视觉入门概念
- SpringMVC + Mybatis bug调试 SQL正确,查数据库却返回NULL
- 做了点JavaScript的题目
- fastJson解析报错com.alibaba.fastjson.JSONException: create instance error...
- WXPaySDKDemo报错
- Hadoop2.7.4 MapReduce集群的Linux安装步骤
- SQLite数据库创建临时表、增加列的方法
- 有名管道FIFO
- 事件分发
- STM32之寄存器开发
- Logstash output Kafka with Kerberos学习
- sql常用
- 笔试题
- 获取范围不重复随机数