Hadoop集群安装 (5) 配置job queue_conf/mapred-queues.xml
来源:互联网 发布:淘宝买家秀福利网站 编辑:程序博客网 时间:2024/05/17 20:43
0.总体
(0)job queue的设置和scheduler密切相关
1) scheduler的配置见 conf/mapred-site.xml中的mapreduce.jobtracker.taskscheduler
2) Fairshare不支持queue
3)管理员应该认真选择 sheduler,然后选择配置对应的 queueFairshare scheduler, implement their own mechanisms for collections of jobs and do not rely on queues provided by the framework
(1) Job队列的作用
1)组织job
2)对作业队列的策略:调度策略,管理策略
(2)对作业队列的基本操作,例如:
job submission to a specific queue,
access control for queues,
queue states,
viewing configured queues and their properties
refresh of queue properties.
(3)作业队列的类型:
Single queue
The default configuration in Map/Reduce comprises of a single queue, as supported by the default scheduler. All jobs are submitted to this default queue which maintains jobs in a priority basedFIFO order.
Multiple single level queues
Multiple queues are defined, and jobs can be submitted to any of these queues. Different policies can be applied to these queues by schedulers that support this configuration to provide a better level of support. For example, the capacity scheduler provides ways of configuring different capacity and fairness guarantees on these queues.
Hierarchical queues:
Hierarchical queues are a configuration in which queues can contain other queues within them recursively. The queues that contain other queues are referred to as container queues. Queues that do not contain other queues are referred as leaf or job queues. Jobs can only be submitted to leaf queues. Hierarchical queues can potentially offer a higher level of control to administrators, as schedulers can now build a hierarchy of policies where policies applicable to a container queue can provide context for policies applicable to queues it contains. It also opens up possibilities for delegating queue administration where administration of queues in a container queue can be turned over to a different set of administrators, within the context provided by the container queue. For example, the capacity scheduler uses hierarchical queues to partition capacity of a cluster among container queues, and allowing queues they contain to divide that capacity in more ways.
(4)文件
hadoop-0.21.0/mapred/src/java/mapred-queues-default.xml,
正式配置文件在 conf/mapred-queues.xml
1. 层级作业队列(举例)
<queues aclsEnabled="$aclsEnabled"><queue><name>$queue-name</name><state>$state</state><queue><name>$child-queue1</name><properties><property key="$key" value="$value"/></properties><queue><name>$grand-child-queue1</name>...</queue></queue> <queue><name>$child-queue2</name>...</queue><queue><name>$leaf-queue</name><acl-submit-job>$acls</acl-submit-job><acl-administer-jobs>$acls</acl-administer-jobs><properties><property key="$key" value="$value"/>...</properties></queue></queue></queues>
2. 定义完作业队列之后,可以在conf/mapred-site.xml中设置
(1)声明作业提交给哪个队列
3. 查看作业队列<property><name>mapreduce.job.queuename</name><value>Queue-A:Queue-B:Queue-C</value></property>(2) 指明调度器和算法
<property> <name>mapreduce.jobtracker.taskscheduler</name> <value>org.apache.hadoop.mapred.JobQueueTaskScheduler</value> <description>The class responsible for scheduling the tasks.</description></property>
(1) 通过命令行查看
hadoop queue [-list] | [-info <job-queue-name> [-showJobs]] | [-showacls]
(2)通过网页查看
queueinfo.jsp
- Hadoop集群安装 (5) 配置job queue_conf/mapred-queues.xml
- Hadoop集群安装 (4) 配置JobTracker_conf/mapred-site.xml
- Hadoop配置项(mapred-site.xml)
- Hadoop配置项整理(mapred-site.xml)
- Hadoop配置项整理(mapred-site.xml)
- Hadoop配置项整理(mapred-site.xml)
- Hadoop配置项整理(mapred-site.xml)
- Hadoop学习5_Hadoop:hdfs-site.xml,mapred-site.xml,core-site.xml 配置项清单
- hadoop配置 - 1.x mapred-site.xml 参数设定
- hadoop 2.2.X 配置参数说明:mapred-site.xml
- hadoop配置文件mapred-site.xml
- Hadoop集群安装配置
- Hadoop集群安装配置
- Hadoop集群安装配置
- Hadoop集群安装配置
- HADOOP 集群安装配置
- Hadoop集群安装配置
- hadoop集群安装配置
- 数 组
- MPICH&PBS使用指南
- AnimationListener
- 黑客教父透泄密危机:2012年爆发更大安全事故
- rvct3.1和ads1.2编译器兼容问题
- Hadoop集群安装 (5) 配置job queue_conf/mapred-queues.xml
- 展讯2012年产品、市场及业绩分析
- 双指针函数参数什么时候需要加 const,什么时候不能加?
- JSP操作MySQL基础
- Python中的 new-style class 及其实例详解
- 值得关注的开源Web应用项目
- 块设备体系结构分析
- Linux内存管理之slab机制(初始化)
- ZOJ2229 POJ1922 HDU1445 Ride to School