spark调度系列------2. Spark Executor的创建和启动过程

    上一讲主要降到了spark executor资源在Master的分配原理。今天来讲Spark Executor的创建和启动过程。创建的过程可以功过如下时序图表示：

如果时序图看不清楚，可以ctrl+鼠标滚轮 放大看，这个时序图对理解Executor启动非常重要。

在Standalone模式下，Backend.start()方法最终调用了SparkDeploySchedulerBackend.start()，这个方法的作用是：

1. 调用父类的CoarseGrainedSchedulerBackend.start方法将配置参数复制给它的properties成员，并且创建driverEndPoint

2. 创建ApplicationDescription对象，这个对象是应用的描述，包括executor内存大小、executor core的个数、spark应用core的最大分配个数

3. 创建并启动AppClient

代码如下：

override def start() {  super.start()//调用父类的CoarseGrainedSchedulerBackend.start方法将配置参数复制给它的properties成员，并且创建driverEndPoint  // The endpoint for executors to talk to us  val driverUrl = rpcEnv.uriOf(SparkEnv.driverActorSystemName,    RpcAddress(sc.conf.get("spark.driver.host"), sc.conf.get("spark.driver.port").toInt),    CoarseGrainedSchedulerBackend.ENDPOINT_NAME)  val args = Seq(    "--driver-url", driverUrl,    "--executor-id", "{{EXECUTOR_ID}}",    "--hostname", "{{HOSTNAME}}",    "--cores", "{{CORES}}",    "--app-id", "{{APP_ID}}",    "--worker-url", "{{WORKER_URL}}")  val extraJavaOpts = sc.conf.getOption("spark.executor.extraJavaOptions")    .map(Utils.splitCommandString).getOrElse(Seq.empty)  val classPathEntries = sc.conf.getOption("spark.executor.extraClassPath")    .map(_.split(java.io.File.pathSeparator).toSeq).getOrElse(Nil)  val libraryPathEntries = sc.conf.getOption("spark.executor.extraLibraryPath")    .map(_.split(java.io.File.pathSeparator).toSeq).getOrElse(Nil)  // When testing, expose the parent class path to the child. This is processed by  // compute-classpath.{cmd,sh} and makes all needed jars available to child processes  // when the assembly is built with the "*-provided" profiles enabled.  val testingClassPath =    if (sys.props.contains("spark.testing")) {      sys.props("java.class.path").split(java.io.File.pathSeparator).toSeq    } else {      Nil    }  // Start executors with a few necessary configs for registering with the scheduler  val sparkJavaOpts = Utils.sparkJavaOpts(conf, SparkConf.isExecutorStartupConf)  val javaOpts = sparkJavaOpts ++ extraJavaOpts  val command = Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",    args, sc.executorEnvs, classPathEntries ++ testingClassPath, libraryPathEntries, javaOpts)  val appUIAddress = sc.ui.map(_.appUIAddress).getOrElse("")  val coresPerExecutor = conf.getOption("spark.executor.cores").map(_.toInt)  val appDesc = new ApplicationDescription(sc.appName, maxCores, sc.executorMemory,    command, appUIAddress, sc.eventLogDir, sc.eventLogCodec, coresPerExecutor)//创建ApplicationDescription对象，这个对象是应用的描述，包括executor内存大小、executor core的个数、spark应用core的最大分配个数  client = new AppClient(sc.env.rpcEnv, masters, appDesc, this, conf)  client.start()//创建并启动AppClient  waitForRegistration()}

AppClient.start()方法的作用是将创建AppClient的endpoint，这个endpoint负责向master发送创建Application、创建Executor、更新Executor状态等与Spark Master之间的交互，它是driver与Spark Master通信的终端。

代码如下：

def start() {  // Just launch an actor; it will call back into the listener.  endpoint = rpcEnv.setupEndpoint("AppClient", new ClientEndpoint(rpcEnv))}

AppCliet的endpoint的实际类型是ClientEndpoint，它的onStart方法在创建ClientEndpoint后接收和发送消息之前执行，相关代码如下：

override def onStart(): Unit = {      try {        registerWithMaster(1)//向Master发送注册Application消息      } catch {        case e: Exception =>          logWarning("Failed to connect to master", e)          markDisconnected()          stop()      }    }

Spark Master和Worker在执行sbin/start-all.sh的时候就已经启动了，它的其中一个功能是提供Application注册服务。

Master在接收到RegisterApplication消息之后会分配各个Executor资源，关于Executor资源如何分配，请参考上一篇文章：spark调度系列----1. spark stanalone模式下worker上executor资源的分配 。最终启动各个Executor，相关代码如下：

Master类的startExecutorsOnWorkers方法

private def startExecutorsOnWorkers(): Unit = {  // Right now this is a very simple FIFO scheduler. We keep trying to fit in the first app  // in the queue, then the second app, etc.  if (spreadOutApps) {    // Try to spread out each app among all the workers, until it has all its cores    for (app <- waitingApps if app.coresLeft > 0) {      val usableWorkers = workers.toArray.filter(_.state == WorkerState.ALIVE)//活着的节点        .filter(worker => worker.memoryFree >= app.desc.memoryPerExecutorMB && //节点的剩余内存大于executor内存          worker.coresFree >= app.desc.coresPerExecutor.getOrElse(1))//节点的空闲core个数大于一个executor需要的core个数        .sortBy(_.coresFree).reverse      val numUsable = usableWorkers.length      val assigned = new Array[Int](numUsable) // Number of cores to give on each node      var toAssign = math.min(app.coresLeft, usableWorkers.map(_.coresFree).sum)      var pos = 0      while (toAssign > 0) {        if (usableWorkers(pos).coresFree - assigned(pos) > 0) {//这个worker还有空闲core，则为app在这个worker分配一个core，轮询尽可能多的worker          toAssign -= 1          assigned(pos) += 1        }        pos = (pos + 1) % numUsable//选择下一个worker节点      }      // Now that we've decided how many cores to give on each node, let's actually give them      for (pos <- 0 until numUsable if assigned(pos) > 0) {        allocateWorkerResourceToExecutors(app, assigned(pos), usableWorkers(pos))//在选定的worker上分配executor,一个worker可能分配多个executor      }    }  } else {    // Pack each app into as few workers as possible until we've assigned all its cores    for (worker <- workers if worker.coresFree > 0 && worker.state == WorkerState.ALIVE) {      for (app <- waitingApps if app.coresLeft > 0) {        allocateWorkerResourceToExecutors(app, app.coresLeft, worker)      }    }  }}

Master类的allocateWorkerResourceToExecutors方法：

private def allocateWorkerResourceToExecutors(    app: ApplicationInfo,    coresToAllocate: Int,    worker: WorkerInfo): Unit = {  val memoryPerExecutor = app.desc.memoryPerExecutorMB  val coresPerExecutor = app.desc.coresPerExecutor.getOrElse(coresToAllocate)//如果没有明确指定一个executor core的个数，则把这个worker上分配的所有core分配给一个executor  var coresLeft = coresToAllocate  while (coresLeft >= coresPerExecutor && worker.memoryFree >= memoryPerExecutor) {//每次为一个executor分配的core个数至少为明确指定的core个数    val exec = app.addExecutor(worker, coresPerExecutor)    coresLeft -= coresPerExecutor    launchExecutor(worker, exec)//发送启动Executor消息    app.state = ApplicationState.RUNNING  }}

在这个方法里面，Master向worker发送启动Executor的消息。worker在接收到LauchExecutor消息之后，会创建ExecutorRunner对象，之后执行ExecutorRunner.start，代码如下：

 <pre name="code" class="java">case LaunchExecutor(masterUrl, appId, execId, appDesc, cores_, memory_) =>      if (masterUrl != activeMasterUrl) {        logWarning("Invalid Master (" + masterUrl + ") attempted to launch executor.")      } else {        try {          logInfo("Asked to launch executor %s/%d for %s".format(appId, execId, appDesc.name))          // Create the executor's working directory          val executorDir = new File(workDir, appId + "/" + execId)//日志目录          if (!executorDir.mkdirs()) {            throw new IOException("Failed to create directory " + executorDir)          }          // Create local dirs for the executor. These are passed to the executor via the          // SPARK_EXECUTOR_DIRS environment variable, and deleted by the Worker when the          // application finishes.          val appLocalDirs = appDirectories.get(appId).getOrElse {            Utils.getOrCreateLocalRootDirs(conf).map { dir =>              Utils.createDirectory(dir, namePrefix = "executor").getAbsolutePath()            }.toSeq          }          appDirectories(appId) = appLocalDirs          val manager = new ExecutorRunner(            appId,            execId,            appDesc.copy(command = Worker.maybeUpdateSSLSettings(appDesc.command, conf)),            cores_,            memory_,            self,            workerId,            host,            webUi.boundPort,            publicAddress,            sparkHome,            executorDir,            workerUri,            conf,            appLocalDirs, ExecutorState.LOADING)          executors(appId + "/" + execId) = manager          manager.start()//执行ExecutorRunner          coresUsed += cores_          memoryUsed += memory_          sendToMaster(ExecutorStateChanged(appId, execId, manager.state, None, None))        } catch {          case e: Exception => {            logError(s"Failed to launch executor $appId/$execId for ${appDesc.name}.", e)            if (executors.contains(appId + "/" + execId)) {              executors(appId + "/" + execId).kill()              executors -= appId + "/" + execId            }            sendToMaster(ExecutorStateChanged(appId, execId, ExecutorState.FAILED,              Some(e.toString), None))          }        }      }

ExecutorRunner.start最终会执行到ExecutorRunner.fetchAndRunExecutor，在这个方法里面启动了Executor进程，代码如下：

private def fetchAndRunExecutor() {    try {      // Launch the process      val builder = CommandUtils.buildProcessBuilder(appDesc.command, new SecurityManager(conf),        memory, sparkHome.getAbsolutePath, substituteVariables)      val command = builder.command()      logInfo("Launch command: " + command.mkString("\"", "\" \"", "\""))      builder.directory(executorDir)      builder.environment.put("SPARK_EXECUTOR_DIRS", appLocalDirs.mkString(File.pathSeparator))      // In case we are running this from within the Spark Shell, avoid creating a "scala"      // parent process for the executor command      builder.environment.put("SPARK_LAUNCH_WITH_SCALA", "0")      // Add webUI log urls      val baseUrl =        s"http://$publicAddress:$webUiPort/logPage/?appId=$appId&executorId=$execId&logType="      builder.environment.put("SPARK_LOG_URL_STDERR", s"${baseUrl}stderr")      builder.environment.put("SPARK_LOG_URL_STDOUT", s"${baseUrl}stdout")      process = builder.start()//启动Executor      val header = "Spark Executor Command: %s\n%s\n\n".format(        command.mkString("\"", "\" \"", "\""), "=" * 40)      // Redirect its stdout and stderr to files      val stdout = new File(executorDir, "stdout")      stdoutAppender = FileAppender(process.getInputStream, stdout, conf)

我的spark application提交命令为：

./spark-submit --class spark_security.login_users.Sockpuppet  --driver-memory 3g--executor-memory 3g--executor-cores 5  --total-executor-cores 15 --name Logintest --master spark://ddos12:7077    --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8888" --conf "spark.executor.extraJavaOptions=-Xdebug -Xrunjdwp:transport=dt_socket,address=9999,server=y,suspend=n" --conf spark.ui.port=4048 /home/wangbaogang/nocache_onewin.jar hdfs://ddos12:9000/prop/logindealer.properties

Executor启动命令在日志里面记录了启动命令，上面application提交命令对应的Executor启动命令为：

15/09/15 13:50:59 INFO ExecutorRunner: Launch command: "/export/servers/jdk1.7.0_79/bin/java" "-cp" "/export/servers/spark-1.4.1-bin-hadoop2.6/sbin/../conf/:/export/servers
/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar:/export/servers/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/export/servers/spark-1.4.1-bi
n-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/export/servers/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/export/servers/hadoop2.6.0/etc/hadoop/""-Xms3072M"
"-Xmx3072M" "-Dspark.ui.port=4048" "-Dspark.driver.port=55048" "-Xdebug" "-Xrunjdwp:transport=dt_socket,address=9999,server=y,suspend=n" "-XX:MaxPermSize=256m""org.apache
.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "akka.tcp://sparkDriver@192.168.185.12:55048/user/CoarseGrainedScheduler" "--executor-id" "1" "--hostname" "192

.168.185.12" "--cores" "5" "--app-id" "app-20150915135059-0019" "--worker-url" "akka.tcp://sparkWorker@192.168.185.12:29402/user/Worker"

启动后Executor对应的进程为：

12661 CoarseGrainedExecutorBackend --driver-url akka.tcp://sparkDriver@192.168.185.12:37004/user/CoarseGrainedScheduler --executor-id 1 --hostname 192.168.185.12--cores 5 --app-id app-20150915144135-0020 --worker-url akka.tcp://sparkWorker@192.168.185.12:29402/user/Worker

可见Executor是一个独立的Java进程，它的heap空间大小是3072M，有启动时候的--executor-memory 3g 设定，这个进程的启动入口类是org.apache
.spark.executor.CoarseGrainedExecutorBackend

Executor进程的启动是在CoarseGrainedExecutorBackend类里面，在这个object里面有一个main方法，这个main方法调用CoarseGrainedExecutorBackend.run启动Executor的执行逻辑

private def run(    driverUrl: String,    executorId: String,    hostname: String,    cores: Int,    appId: String,    workerUrl: Option[String],    userClassPath: Seq[URL]) {  SignalLogger.register(log)  SparkHadoopUtil.get.runAsSparkUser { () =>    // Debug code    Utils.checkHost(hostname)    // Bootstrap to fetch the driver's Spark properties.    val executorConf = new SparkConf    val port = executorConf.getInt("spark.executor.port", 0)    val fetcher = RpcEnv.create(      "driverPropsFetcher",      hostname,      port,      executorConf,      new SecurityManager(executorConf))    val driver = fetcher.setupEndpointRefByURI(driverUrl)//获取driver信息    val props = driver.askWithRetry[Seq[(String, String)]](RetrieveSparkProps) ++      Seq[(String, String)](("spark.app.id", appId))//请求driver的属性信息    fetcher.shutdown()    // Create SparkEnv using properties we fetched from the driver.    val driverConf = new SparkConf()    for ((key, value) <- props) {      // this is required for SSL in standalone mode      if (SparkConf.isExecutorStartupConf(key)) {        driverConf.setIfMissing(key, value)      } else {        driverConf.set(key, value)      }    }    if (driverConf.contains("spark.yarn.credentials.file")) {      logInfo("Will periodically update credentials from: " +        driverConf.get("spark.yarn.credentials.file"))      SparkHadoopUtil.get.startExecutorDelegationTokenRenewer(driverConf)    }    val env = SparkEnv.createExecutorEnv(      driverConf, executorId, hostname, port, cores, isLocal = false)//创建executor的SparkEnv信息    // SparkEnv sets spark.driver.port so it shouldn't be 0 anymore.    val boundPort = env.conf.getInt("spark.executor.port", 0)    assert(boundPort != 0)    // Start the CoarseGrainedExecutorBackend endpoint.    val sparkHostPort = hostname + ":" + boundPort    env.rpcEnv.setupEndpoint("Executor", new CoarseGrainedExecutorBackend(      env.rpcEnv, driverUrl, executorId, sparkHostPort, cores, userClassPath, env))//创建executor的endpoint，用于和driver的endpoint通信    workerUrl.foreach { url =>      env.rpcEnv.setupEndpoint("WorkerWatcher", new WorkerWatcher(env.rpcEnv, url))    }    env.rpcEnv.awaitTermination()    SparkHadoopUtil.get.stopExecutorDelegationTokenRenewer()  }}

在这个方法里面，主要是为Executor设置初始信息，获得driver的属性信息，创建Executor的SparkEnv，创建用户和driver通信的Executor endpoint

创建Executor的endpoint之后，会等待RegisteredExecutor事件，当接收到这个事件之后会创建真正的Executor。在CoarseGrainedExecutorBackend.receive进行事件的侦听

RegisteredExecutor事件的发送在CoarseGrainedSchedulerBackend.receiveAndReply方法里面，这个方法会侦听RegisterExecutor事件，侦听到这个事件时候会发送RegisteredExecutor事件，代码如下：

override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {  case RegisterExecutor(executorId, executorRef, hostPort, cores, logUrls) =>    Utils.checkHostPort(hostPort, "Host port expected " + hostPort)//侦听RegisterExecutor事件，并且发送Executor注册完成事件    if (executorDataMap.contains(executorId)) {      context.reply(RegisterExecutorFailed("Duplicate executor ID: " + executorId))    } else {      logInfo("Registered executor: " + executorRef + " with ID " + executorId)      context.reply(RegisteredExecutor)//发送RegisteredExecutor注册完成事件      addressToExecutorId(executorRef.address) = executorId      totalCoreCount.addAndGet(cores)      totalRegisteredExecutors.addAndGet(1)      val (host, _) = Utils.parseHostPort(hostPort)      val data = new ExecutorData(executorRef, executorRef.address, host, cores, cores, logUrls)      // This must be synchronized because variables mutated      // in this block are read when requesting executors      CoarseGrainedSchedulerBackend.this.synchronized {        executorDataMap.put(executorId, data)        if (numPendingExecutors > 0) {          numPendingExecutors -= 1          logDebug(s"Decremented number of pending executors ($numPendingExecutors left)")        }      }      listenerBus.post(        SparkListenerExecutorAdded(System.currentTimeMillis(), executorId, data))      makeOffers()//发送这个application的资源请求和分配    }

结论：这个方法首先收到RegisterExecutor事件，然后回应RegisteredExecutor事件，最后发送资源请求和分配

CoarseGrainedExecutorBackend对象在创建之后，它的接收和发送消息之前onStart方法会首先调用执行，发送RegisterExecutor事件，这条事件的接收方必须为receiveAndReply事件接收方法，接收到RegisteredExecutor响应事件之后发送这条事件RegisterExecutor事件

override def onStart() {//这个方法在CoarseGrainedExecutorBackend处理任何消息之前首先调用执行，应该是在这个类的对象初始化之后就执行  logInfo("Connecting to driver: " + driverUrl)  rpcEnv.asyncSetupEndpointRefByURI(driverUrl).flatMap { ref =>    // This is a very fast action so we can use "ThreadUtils.sameThread"    driver = Some(ref)    ref.ask[RegisteredExecutor.type](      RegisterExecutor(executorId, self, hostPort, cores, extractLogUrls))//发送RegisterExecutor消息，并且等待回应  }(ThreadUtils.sameThread).onComplete {    // This is a very fast action so we can use "ThreadUtils.sameThread"    case Success(msg) => Utils.tryLogNonFatalError {      Option(self).foreach(_.send(msg)) // msg must be RegisteredExecutor  回应的消息是RegisterExecutored消息，并且发送这个消息，这个消息在这个类的receive方法接收    }    case Failure(e) => {      logError(s"Cannot register with driver: $driverUrl", e)      System.exit(1)    }  }(ThreadUtils.sameThread)}

至此Spark Standalone模式，Executor启动完成