Hadoop基础教程-第9章 HA高可用(9.4 YARN 高可用)(草稿)
来源:互联网 发布:ubuntu pyqt5 安装 编辑:程序博客网 时间:2024/05/02 06:12
第9章 HA高可用
9.4 YARN 高可用
9.4.1 RM单点故障
http://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html
ResourceManager(RM)负责跟踪群集中的资源,并调度应用程序(例如MapReduce作业)。在Hadoop 2.4之前,ResourceManager是YARN集群中的单点故障。高可用性功能以活动/待机资源管理器对的形式添加冗余,以消除此单一故障点。
YARN高可用,也就是ResourceManager高可用,规划如下
9.4.2 配置yarn-site.xml
[root@node1 hadoop]# vi yarn-site.xml[root@node1 hadoop]# cat yarn-site.xml <?xml version="1.0"?><configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> <description>NodeManager上运行的附属服务(运行MapReduce程序)</description> </property> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> <description>启用RM高可用性</description> </property> <property> <name>yarn.resourcemanager.cluster-id</name> <value>yarn1</value> <description>YARN集群ID</description> </property> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> <description>启用HA时,群集中的RM节点列表</description> </property> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>node1</value> <description>第1个resourcemanager</description> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>node3</value> <description>第2个resourcemanager</description> </property> <property> <name>yarn.resourcemanager.zk-address</name> <value>node1:2181,node2:2181,node3:2181</value> <description>Zookeeper列表</description> </property></configuration>[root@node1 hadoop]#
9.4.3 修改yarn-env.sh
编译yarn-env.sh文件,增加PID文件存放目录
[root@node1 hadoop]# vi yarn-env.sh
添加一行,内容如下
export YARN_PID_DIR=/var/run
9.4.4 分发配置文件
[root@node1 hadoop]# scp yarn-site.xml node2:/opt/hadoop-2.7.3/etc/hadoop/yarn-site.xml 100% 841 0.8KB/s 00:00 [root@node1 hadoop]# scp yarn-site.xml node3:/opt/hadoop-2.7.3/etc/hadoop/yarn-site.xml 100% 841 0.8KB/s 00:00 [root@node1 hadoop]# scp yarn-env.sh node2:/opt/hadoop-2.7.3/etc/hadoop/yarn-env.sh 100% 4595 4.5KB/s 00:00 [root@node1 hadoop]# scp yarn-env.sh node3:/opt/hadoop-2.7.3/etc/hadoop/yarn-env.sh 100% 4595 4.5KB/s 00:00 [root@node1 hadoop]#
9.4.5 启动YARN
[root@node1 hadoop]# start-yarn.shstarting yarn daemonsstarting resourcemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-resourcemanager-node1.outnode2: starting nodemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-nodemanager-node2.outnode3: starting nodemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-nodemanager-node3.outnode1: starting nodemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-nodemanager-node1.out[root@node1 hadoop]#
node1
[root@node1 hadoop]# jps4337 Jps4036 NodeManager3189 DataNode3529 NameNode3402 DFSZKFailoverController3931 ResourceManager3310 JournalNode[root@node1 hadoop]#
node2
[root@node2 ~]# jps4306 Jps3828 DFSZKFailoverController3670 JournalNode4200 NodeManager3449 DataNode3947 NameNode[root@node2 ~]#
node3
[root@node3 ~]# jps2643 JournalNode2771 NodeManager2549 DataNode2871 Jps
单独启用备用resourcemanager
[root@node3 ~]# yarn-daemon.sh start resourcemanagerstarting resourcemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-resourcemanager-node3.out[root@node3 ~]# jps2643 JournalNode2771 NodeManager2900 ResourceManager2932 Jps2549 DataNode[root@node3 ~]#
9.4.6 Web
通过浏览器打开
http://192.168.80.131:8088
通过浏览器打开
http://192.168.80.133:8088
跳转到http://node1:8088/
实际上http://node1:8088/就是http://192.168.80.133:8088,是因为物理机Windows系统的hosts文件没有配置node1而已。
9.4.7 测试
[root@node1 hadoop]# jps4337 Jps4036 NodeManager3189 DataNode3529 NameNode3402 DFSZKFailoverController3931 ResourceManager3310 JournalNode[root@node1 hadoop]# kill 3931[root@node1 hadoop]#
这时再次打开http://192.168.80.133:8088
9.4.8 Zookeeper
[root@node2 ~]# zkCli.sh Connecting to localhost:21812017-07-22 10:36:18,876 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT2017-07-22 10:36:18,881 [myid:] - INFO [main:Environment@100] - Client environment:host.name=node22017-07-22 10:36:18,881 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_1122017-07-22 10:36:18,884 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation2017-07-22 10:36:18,885 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/opt/jdk1.8.0_112/jre2017-07-22 10:36:18,885 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper-3.4.10/bin/../build/classes:/opt/zookeeper-3.4.10/bin/../build/lib/*.jar:/opt/zookeeper-3.4.10/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper-3.4.10/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper-3.4.10/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper-3.4.10/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper-3.4.10/bin/../lib/jline-0.9.94.jar:/opt/zookeeper-3.4.10/bin/../zookeeper-3.4.10.jar:/opt/zookeeper-3.4.10/bin/../src/java/lib/*.jar:/opt/zookeeper-3.4.10/bin/../conf:.::/opt/jdk1.8.0_112/lib2017-07-22 10:36:18,885 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib2017-07-22 10:36:18,885 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp2017-07-22 10:36:18,885 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>2017-07-22 10:36:18,885 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux2017-07-22 10:36:18,885 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd642017-07-22 10:36:18,886 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-514.el7.x86_642017-07-22 10:36:18,886 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root2017-07-22 10:36:18,886 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root2017-07-22 10:36:18,886 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/root2017-07-22 10:36:18,888 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589eWelcome to ZooKeeper!2017-07-22 10:36:19,007 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)JLine support is enabled2017-07-22 10:36:19,252 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@876] - Socket connection established to localhost/127.0.0.1:2181, initiating session2017-07-22 10:36:19,282 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x25d6a8d40030005, negotiated timeout = 30000WATCHER::WatchedEvent state:SyncConnected type:None path:null[zk: localhost:2181(CONNECTED) 0] ls /[zookeeper, test, yarn-leader-election, hadoop-ha][zk: localhost:2181(CONNECTED) 1] ls /yarn-leader-election[yarn1][zk: localhost:2181(CONNECTED) 2] ls /yarn-leader-election/yarn1[ActiveBreadCrumb, ActiveStandbyElectorLock][zk: localhost:2181(CONNECTED) 3]
阅读全文
0 0
- Hadoop基础教程-第9章 HA高可用(9.4 YARN 高可用)(草稿)
- Hadoop基础教程-第9章 HA高可用(9.1 HDFS 高可用介绍)(草稿)
- Hadoop基础教程-第9章 HA高可用(9.2 HDFS 高可用配置)(草稿)
- Hadoop基础教程-第9章 HA高可用(9.3 HDFS 高可用运行)(草稿)
- Hadoop HA(高可用)配置
- Hadoop HDFS高可用(HA)
- Hadoop高可用集群搭建(HA)
- Yarn HA高可用配置
- hadoop系列文档4-配置Yarn高可用HA
- Hadoop系列-YARN RM HA 高可用集群
- Hadoop HA高可用集群搭建(Hadoop+Zookeeper+HBase)
- Hadoop,HA高可用集群
- Hadoop HA高可用配置
- hadoop namenode 高可用(HA)搭建指南 QJM方式
- Hadoop HA(高可用)环境的搭建
- Hadoop高可用集群(HA+JournalNode+zookeeper)
- Hadoop HA高可用集群搭建(2.7.2)
- Hadoop 2.7.3 高可用(HA)集群部署
- 创建博客初心
- 深度学习Alexnet网络对图像进行分类/预测(迁移学习)
- Leetcode 100. Same Tree
- HTML入门笔记13-HTML中font标签
- 实现一颗二叉树的层序遍历
- Hadoop基础教程-第9章 HA高可用(9.4 YARN 高可用)(草稿)
- 17_7_22 求二叉树叶子节点的个数/求二叉树第k层的节点个数
- 让你不再俱怕Fragment State Loss
- Spring AOP术语解释
- 如何使用 hive建立自己的函数在CLI界面上并使用
- 深入了解Java虚拟机
- mybatis和spring 整合
- leetcode[Number of Boomerangs]
- java.lang.IllegalThreadStateException