spark 2.1 TaskSchedulerImpl is Simlar between local mode and yarn mode

来源:互联网 发布:大数据时代已经来临 编辑:程序博客网 时间:2024/06/07 10:00
  1. In local mode , task scheuler is instance of TaskSchedulerImpl.
master match {      case "local" =>        val scheduler = new TaskSchedulerImpl(sc, MAX_LOCAL_TASK_FAILURES, isLocal = true)

YarnClusterManager use YarnScheduler

private[spark] class YarnClusterManager extends ExternalClusterManager {  override def createTaskScheduler(sc: SparkContext, masterURL: String): TaskScheduler = {    sc.deployMode match {      case "cluster" => new YarnClusterScheduler(sc)      case "client" => new YarnScheduler(sc)      case _ => throw new SparkException(s"Unknown deploy mode '${sc.deployMode}' for Yarn")    }  }

YarnScheduler extends TaskSchedulerImpl, only override getRackForHost method.

private[spark] class YarnScheduler(sc: SparkContext) extends TaskSchedulerImpl(sc) {  // RackResolver logs an INFO message whenever it resolves a rack, which is way too often.  if (Logger.getLogger(classOf[RackResolver]).getLevel == null) {    Logger.getLogger(classOf[RackResolver]).setLevel(Level.WARN)  }  // By default, rack is unknown  override def getRackForHost(hostPort: String): Option[String] = {    val host = Utils.parseHostPort(hostPort)._1    Option(RackResolver.resolve(sc.hadoopConfiguration, host).getNetworkLocation)  }}
0 0
原创粉丝点击