Hadoop MapReduce2 -单节点集群配置
来源:互联网 发布:知乎自考可以跨专业吗 编辑:程序博客网 时间:2024/05/16 18:47
Mapreduce 压缩包
你可以从release下载一个MapReduce压缩包。如果没有你可以自己创建一个压缩包。
$ mvn clean install -DskipTests$ cd hadoop-mapreduce-project$ mvn clean install assembly:assembly -Pnative
注意:你需要安装protoc2.5
忽略本地的MapReduce你可以忽略maven的 —Pnative参数。这个压缩包应该在target/directory下。
设置环境变量
假设你已经安装了hadoop-common/hadoop-hdfs,并设置了 $HADOOP_COMMON_HOME/$HADOOP_HDFS_HOME
解压hadoop-mapreduce压缩包。设置环境变量 $HADOOP_MAPRED_HOME 设置$HADOOP_YARN_HOME的环境变量和 $HADOOP_MAPRED_HOME 一样。
搭建配置文件
要启动ResourceManger和NodeManager,需要设置配置信息。假设$HADOOP_CONF_DIR 是配置HDFS和yarn-site.xml的配置目录。有两个文件你必须进行设置:mapred-site.xml and yarn-site.xml
配置mapred-site.xml:
添加以下配置到你的mapred-site.xml
<property>
<name>mapreduce.cluster.temp.dir</name> <value></value> <description>No description</description> <final>true</final> </property> <property> <name>mapreduce.cluster.local.dir</name> <value></value> <description>No description</description> <final>true</final> </property>
设置yarn-site.xml
添加以下的配置到你的yarn-site.xml中
<property>
<name>yarn.resourcemanager.resource-tracker.address</name> <value>host:port</value> <description>host is the hostname of the resource manager and port is the port on which the NodeManagers contact the Resource Manager. </description> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>host:port</value> <description>host is the hostname of the resourcemanager and port is the port on which the Applications in the cluster talk to the Resource Manager. </description> </property> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value> <description>In case you do not want to use the default scheduler</description> </property> <property> <name>yarn.resourcemanager.address</name> <value>host:port</value> <description>the host is the hostname of the ResourceManager and the port is the port on which the clients can talk to the Resource Manager. </description> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value></value> <description>the local directories used by the nodemanager</description> </property> <property> <name>yarn.nodemanager.address</name> <value>0.0.0.0:port</value> <description>the nodemanagers bind to this port</description> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>10240</value> <description>the amount of memory on the NodeManager in GB</description> </property> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> <description>directory on hdfs where the application logs are moved to </description> </property> <property> <name>yarn.nodemanager.log-dirs</name> <value></value> <description>the directories used by Nodemanagers as log directories</description> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> <description>shuffle service that needs to be set for Map Reduce to run </description> </property>
接下来配置capacity-scheduler.xml
添加以下配置到capacity-scheduler.xml
<property>
<name>yarn.scheduler.capacity.root.queues</name> <value>unfunded,default</value> </property> <property> <name>yarn.scheduler.capacity.root.capacity</name> <value>100</value> </property> <property> <name>yarn.scheduler.capacity.root.unfunded.capacity</name> <value>50</value> </property> <property> <name>yarn.scheduler.capacity.root.default.capacity</name> <value>50</value> </property>
运行deamons
假设环境变量$HADOOP_COMMON_HOME, $HADOOP_HDFS_HOME, $HADOO_MAPRED_HOME, $HADOOP_YARN_HOME,$JAVA_HOME and $HADOOP_CONF_DIR已经正确配置。设置$$YARN_CONF_DIR和$HADOOP_CONF_DIR配置也一样。
运行ResourceManager和NodeManager需要以下命令:
$ cd $HADOOP_MAPRED_HOME
$ sbin/yarn-daemon.sh start resourcemanager $ sbin/yarn-daemon.sh start nodemanager
如果ResourceManager和NodeManager能启动运行,你可以运行randomwriter例子测试:
$HADOOP_COMMON_HOME/bin/hadoop jar hadoop-examples.jar randomwriter out
0 0
- Hadoop MapReduce2 -单节点集群配置
- Hadoop 单节点集群配置
- Hadoop 单节点集群配置
- Hadoop单机单节点集群安装配置
- 【配置】Hadoop单节点集群搭建
- hadoop安装,并配置单节点hadoop集群
- Hadoop --MapReduce2 - 群集设置(多个节点以及大集群中数千个节点进行安装)
- Hadoop:设置单节点集群
- 创建单节点Hadoop集群
- Hadoop单节点配置
- Hadoop 单节点配置
- Hadoop多节点集群配置
- hadoop伪分布式集群(单节点与多节点)配置
- HDP 2.2 ( Hadoop 2.6 ) 集群的内存参数配置和参数调优 (Yarn/MapReduce2)
- Hadoop笔记之一:安装单节点集群
- 第一篇 Hadoop安装单节点集群
- Hadoop安装-单节点伪集群
- Hadoop-2.2.0中文文档——MapReduce 下一代 —配置一个单节点集群
- 基于PaaS平台租户部署及人员跨部门设计方案
- 学习标签更新20141206
- 03.Git 的origin和master分析
- Android桌面快捷方式
- ffmpeg 转码 原始视频
- Hadoop MapReduce2 -单节点集群配置
- poj 2029 Get Many Persimmon Trees 暴力枚举
- Android下屏保程序的开发
- JAVA字符串格式化-String.format()的使用
- 一个程序员的个人思考
- win8.1下codeblocks + MinGW 配置OpenCV2.4.10
- JSP中WebContent和WEB-INF下网页的路径问题和404报错
- 我的野蛮女友
- skyeye学习目录