spark 2.1 TaskSchedulerImpl is Simlar between local mode and yarn mode
来源:互联网 发布:大数据时代已经来临 编辑:程序博客网 时间:2024/06/07 10:00
- In local mode , task scheuler is instance of TaskSchedulerImpl.
master match { case "local" => val scheduler = new TaskSchedulerImpl(sc, MAX_LOCAL_TASK_FAILURES, isLocal = true)
YarnClusterManager use YarnScheduler
private[spark] class YarnClusterManager extends ExternalClusterManager { override def createTaskScheduler(sc: SparkContext, masterURL: String): TaskScheduler = { sc.deployMode match { case "cluster" => new YarnClusterScheduler(sc) case "client" => new YarnScheduler(sc) case _ => throw new SparkException(s"Unknown deploy mode '${sc.deployMode}' for Yarn") } }
YarnScheduler extends TaskSchedulerImpl, only override getRackForHost method.
private[spark] class YarnScheduler(sc: SparkContext) extends TaskSchedulerImpl(sc) { // RackResolver logs an INFO message whenever it resolves a rack, which is way too often. if (Logger.getLogger(classOf[RackResolver]).getLevel == null) { Logger.getLogger(classOf[RackResolver]).setLevel(Level.WARN) } // By default, rack is unknown override def getRackForHost(hostPort: String): Option[String] = { val host = Utils.parseHostPort(hostPort)._1 Option(RackResolver.resolve(sc.hadoopConfiguration, host).getNetworkLocation) }}
0 0
- spark 2.1 TaskSchedulerImpl is Simlar between local mode and yarn mode
- Spark 2.1 backend implementation vary greatly from local mode to yarn mode
- What is the difference between user mode and kernel mode, and real mode and protected mode?
- Difference between Monitor mode and Promiscuous mode
- Difference between Pager mode and session mode
- What is the difference between XHTML MP, XHTML Basic, WML, i-mode, and HDML?
- Storm Local Mode
- real mode and protected mode
- pager mode and session mode
- 关于master mode与local mode
- Spark Cluster Mode Overview
- 安装spark standalone mode
- Spark Standalone Mode
- Spark Standalone Mode 安装
- spark standalone mode
- spark's deploy mode
- Spark Standalone Mode
- mode
- CentOS rpm方式安装jenkins
- 前端基础进阶(九):详解面向对象、构造函数、原型与原型链
- POJ 1743 Musical Theme (后缀数组 + 二分)
- C#中dll调用方法
- android简单的分享功能
- spark 2.1 TaskSchedulerImpl is Simlar between local mode and yarn mode
- Application Cache 无法加载其他静态资源
- 前端基础
- ADB Connection Error
- Java连接数据库 <JDBC>
- Linux进程调度和切换过程分析
- 南阳oj 366 全排列 D的小L
- mysql查询今天、昨天、7天、近30天、本月、上一月 数据
- mac/linux solr 安装测试 one