配置hadoop 使用fair scheduler调度器
来源:互联网 发布:mac日历添加美国节日 编辑:程序博客网 时间:2024/05/17 23:35
hadoop版本为cloudera hadoop cdh3u3
配置步骤为
1. 将$HADOOP_HOME/contrib/fairscheduler/hadoop-fairscheduler-0.20.2-cdh3u3.jar拷贝到$HADOOP_HOME/lib文件夹中
2. 修改$HADOOP_HOME/conf/mapred-site.xml配置文件
- <property>
- <name>mapred.jobtracker.taskScheduler</name>
- <value>org.apache.hadoop.mapred.FairScheduler</value>
- </property>
- <property>
- <name>mapred.fairscheduler.allocation.file</name>
- <value>/home/hadoop/hadoop-0.20.2-cdh3u3/conf/fair-scheduler.xml</value>
- </property>
- <property>
- <name>mapred.fairscheduler.preemption</name>
- <value>true</value>
- </property>
- <property>
- <name>mapred.fairscheduler.assignmultiple</name>
- <value>true</value>
- </property>
- <property>
- <name>mapred.fairscheduler.poolnameproperty</name>
- <value>mapred.job.queue.name</value>
- <description>job.set("mapred.job.queue.name",pool); </description>
- </property>
- <property>
- <name>mapred.fairscheduler.preemption.only.log</name>
- <value>true</value>
- </property>
- <property>
- <name>mapred.fairscheduler.preemption.interval</name>
- <value>15000</value>
- </property>
- <property>
- <name>mapred.queue.names</name>
- <value>default,hadoop,hive</value>
- </property>
3. 在$HADOOP_HOME/conf/新建配置文件fair-scheduler.xml
- <?xml version="1.0"?>
- <allocations>
- <pool name="hive">
- <minMaps>90</minMaps>
- <minReduces>20</minReduces>
- <maxRunningJobs>20</maxRunningJobs>
- <weight>2.0</weight>
- <minSharePreemptionTimeout>30</minSharePreemptionTimeout>
- </pool>
- <pool name="hadoop">
- <minMaps>9</minMaps>
- <minReduces>2</minReduces>
- <maxRunningJobs>20</maxRunningJobs>
- <weight>1.0</weight>
- <minSharePreemptionTimeout>30</minSharePreemptionTimeout>
- </pool>
- <user name="hadoop">
- <maxRunningJobs>6</maxRunningJobs>
- </user>
- <poolMaxJobsDefault>10</poolMaxJobsDefault>
- <userMaxJobsDefault>8</userMaxJobsDefault>
- <defaultMinSharePreemptionTimeout>600</defaultMinSharePreemptionTimeout>
- <fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
- </allocations>
4. 在集群的各个节点执行以上步骤,然后重启集群,在http://namenode:50030/scheduler 即可查看到调度器运行状态,如果修改调度器配置的话,只需要修改文件fair-scheduler.xml ,不需重启配置即可生效。
5. 在执行hive任务时,设置hive属于的队列set mapred.job.queue.name=hive;
##########
另外,如果在执行MR JOB的时候出现XX用户访问不了YY队列的话,就需要在mapred-queue-acls.xml里配置相应的属性,来对访问权限进行控制,比如:
- <property>
- <name>mapred.queue.default.acl-submit-job</name>
- <value>*</value>
- <description> Comma separated list of user and group names that are allowed
- to submit jobs to the 'default' queue. The user list and the group list
- are separated by a blank. For e.g. user1,user2 group1,group2.
- If set to the special value '*', it means all users are allowed to
- submit jobs. If set to ' '(i.e. space), no user will be allowed to submit
- jobs.
- It is only used if authorization is enabled in Map/Reduce by setting the
- configuration property mapred.acls.enabled to true.
- Irrespective of this ACL configuration, the user who started the cluster and
- cluster administrators configured via
- mapreduce.cluster.administrators can submit jobs.
- </description>
- </property>
- <property>
- <name>mapred.queue.default.acl-administer-jobs</name>
- <value>*</value>
- <description> Comma separated list of user and group names that are allowed
- to view job details, kill jobs or modify job's priority for all the jobs
- in the 'default' queue. The user list and the group list
- are separated by a blank. For e.g. user1,user2 group1,group2.
- If set to the special value '*', it means all users are allowed to do
- this operation. If set to ' '(i.e. space), no user will be allowed to do
- this operation.
- It is only used if authorization is enabled in Map/Reduce by setting the
- configuration property mapred.acls.enabled to true.
- Irrespective of this ACL configuration, the user who started the cluster and
- cluster administrators configured via
- mapreduce.cluster.administrators can do the above operations on all the jobs
- in all the queues. The job owner can do all the above operations on his/her
- job irrespective of this ACL configuration.
- </description>
- </property>
0 0
- 配置hadoop 使用fair scheduler调度器
- 配置hadoop 使用fair scheduler调度器
- 配置hadoop 使用fair scheduler调度器
- hadoop fair scheduler配置和使用
- hadoop2.0 公平调度器(fair-scheduler)配置
- hadoop2.0 公平调度器(fair-scheduler)配置
- hadoop配置fair-scheduler的方法
- hadoop公平调度配置详解(二)fair-scheduler.xml篇
- hadoop公平调度其Fair Scheduler运行错误
- Hadoop学习之--Fair Scheduler作业调度分析
- Hadoop Fair Scheduler
- hadoop Capacity Scheduler调度器使用体验
- YARN Fair Scheduler 配置
- 配置Hadoop M/R 采用Fair Scheduler算法代替FIFO
- Hadoop多用户资源管理–Fair Scheduler介绍与配置
- Hadoop YARN配置参数—Fair Scheduler相关参数
- Hadoop多用户资源管理–Fair Scheduler介绍与配置
- Hadoop Yarn多用户资源管理–Fair Scheduler介绍与配置
- 关于301重定向,以及做重定向中需要注意的
- IOS语言Object-c语言随笔(四)
- XAMPP与VMWARE端口占用冲突
- [HTTP]_[C/C++]_[获取URL里的域名主体]
- 翻译
- 配置hadoop 使用fair scheduler调度器
- 第二次作业
- 给定能随机生成整数1到5的函数,写出能随机生成整数1到7的函数
- Android 截屏问题总结
- Xcode 5真机调试
- 求逆转一个整数的二进制表示
- PopupWindow listview setOnItemClickListener ExpandableListView
- 黑马程序员_Java集合框架(上)
- MIC编译器icc(Intel Cluster Studio XE for Linux)的安装过程