Spark2.0源码之4_Worker

Spark中Worker启动，可以

 1. ${SPARK_HOME}/sbin/start-all.sh  2. ${SPARK_HOME}/sbin/start-slaves.sh

start-all.sh脚本内容：

…

#Start all spark daemons.#Starts the master on this node.#Starts a worker on each node specified in conf/slavesif [ -z "${SPARK_HOME}" ]; then  export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"fi#Load the Spark configuration. "${SPARK_HOME}/sbin/spark-config.sh"#Start Master"${SPARK_HOME}/sbin"/start-master.sh # 启动master#Start Workers"${SPARK_HOME}/sbin"/start-slaves.sh # 启动workers

start-slaves.sh脚本内容：

#Starts a slave instance on each machine specified in the conf/slaves file.if [ -z "${SPARK_HOME}" ]; then  export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"fi. "${SPARK_HOME}/sbin/spark-config.sh". "${SPARK_HOME}/bin/load-spark-env.sh"#Find the port number for the masterif [ "$SPARK_MASTER_PORT" = "" ]; then  SPARK_MASTER_PORT=7077fiif [ "$SPARK_MASTER_HOST" = "" ]; then  case `uname` in      (SunOS)      SPARK_MASTER_HOST="`/usr/sbin/check-hostname | awk '{print $NF}'`"      ;;      (*)      SPARK_MASTER_HOST="`hostname -f`"      ;;  esacfi

Launch the slaves

start-slave.sh脚本内容：

#"${SPARK_HOME}/sbin/slaves.sh" cd "${SPARK_HOME}" \; "${SPARK_HOME}/sbin/start-slave.sh" "spark://$SPARK_MASTER_HOST:$SPARK_MASTER_PORT"if [ -z "${SPARK_HOME}" ]; then  export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"fi#NOTE: This exact class name is matched downstream by SparkSubmit.#Any changes need to be reflected there.CLASS="org.apache.spark.deploy.worker.Worker"if [[ $# -lt 1 ]] || [[ "$@" = *--help ]] || [[ "$@" = *-h ]]; then  echo "Usage: ./sbin/start-slave.sh [options] <master>"  pattern="Usage:"  pattern+="\|Using Spark's default log4j profile:"  pattern+="\|Registered signal handlers for"  "${SPARK_HOME}"/bin/spark-class $CLASS --help 2>&1 | grep -v "$pattern" 1>&2  exit 1fi. "${SPARK_HOME}/sbin/spark-config.sh". "${SPARK_HOME}/bin/load-spark-env.sh"#First argument should be the master; we need to store it aside because we may#need to insert arguments between it and the other argumentsMASTER=$1shift#Determine desired worker portif [ "$SPARK_WORKER_WEBUI_PORT" = "" ]; then  SPARK_WORKER_WEBUI_PORT=8081fi#Start up the appropriate number of workers on this machine.#quick local function to start a workerfunction start_instance {  WORKER_NUM=$1  shift  if [ "$SPARK_WORKER_PORT" = "" ]; then    PORT_FLAG=    PORT_NUM=  else    PORT_FLAG="--port"    PORT_NUM=$(( $SPARK_WORKER_PORT + $WORKER_NUM - 1 ))  fi  WEBUI_PORT=$(( $SPARK_WORKER_WEBUI_PORT + $WORKER_NUM - 1 ))  "${SPARK_HOME}/sbin"/spark-daemon.sh start $CLASS $WORKER_NUM \     --webui-port "$WEBUI_PORT" $PORT_FLAG $PORT_NUM $MASTER "$@"}if [ "$SPARK_WORKER_INSTANCES" = "" ]; then  start_instance 1 "$@"else  for ((i=0; i< $SPARK_WORKER_INSTANCES; i++)); do    start_instance $(( 1 + $i )) "$@"  donefi

最终调用脚本spark-daemon.sh，在spark-daemon.sh中使用spark_submit –class启动JAVA进程。在SparkSubmit中，使用反射加载类文件，并运行main函数。

Worker对象（object）源码：

private[deploy] object Worker extends Logging {  val SYSTEM_NAME = "sparkWorker"  val ENDPOINT_NAME = "Worker"  def main(argStrings: Array[String]) {    Utils.initDaemon(log)    val conf = new SparkConf    val args = new WorkerArguments(argStrings, conf)    val rpcEnv = startRpcEnvAndEndpoint(args.host, args.port, args.webUiPort, args.cores,      args.memory, args.masters, args.workDir, conf = conf)    // With external shuffle service enabled, if we request to launch multiple workers on one host,    // we can only successfully launch the first worker and the rest fails, because with the port    // bound, we may launch no more than one external shuffle service on each host.    // When this happens, we should give explicit reason of failure instead of fail silently. For    // more detail see SPARK-20989.    val externalShuffleServiceEnabled = conf.getBoolean("spark.shuffle.service.enabled", false)    val sparkWorkerInstances = scala.sys.env.getOrElse("SPARK_WORKER_INSTANCES", "1").toInt    require(externalShuffleServiceEnabled == false || sparkWorkerInstances <= 1,      "Starting multiple workers on one host is failed because we may launch no more than one " +        "external shuffle service on each host, please set spark.shuffle.service.enabled to " +        "false or set SPARK_WORKER_INSTANCES to 1 to resolve the conflict.")    rpcEnv.awaitTermination()  }  def startRpcEnvAndEndpoint(      host: String,      port: Int,      webUiPort: Int,      cores: Int,      memory: Int,      masterUrls: Array[String],      workDir: String,      workerNumber: Option[Int] = None,      conf: SparkConf = new SparkConf): RpcEnv = {    // The LocalSparkCluster runs multiple local sparkWorkerX RPC Environments    val systemName = SYSTEM_NAME + workerNumber.map(_.toString).getOrElse("")    val securityMgr = new SecurityManager(conf)    val rpcEnv = RpcEnv.create(systemName, host, port, conf, securityMgr)    val masterAddresses = masterUrls.map(RpcAddress.fromSparkURL(_))    rpcEnv.setupEndpoint(ENDPOINT_NAME, new Worker(rpcEnv, webUiPort, cores, memory,      masterAddresses, ENDPOINT_NAME, workDir, conf, securityMgr))    rpcEnv  }  def isUseLocalNodeSSLConfig(cmd: Command): Boolean = {    val pattern = """\-Dspark\.ssl\.useNodeLocalConf\=(.+)""".r    val result = cmd.javaOpts.collectFirst {      case pattern(_result) => _result.toBoolean    }    result.getOrElse(false)  }  def maybeUpdateSSLSettings(cmd: Command, conf: SparkConf): Command = {    val prefix = "spark.ssl."    val useNLC = "spark.ssl.useNodeLocalConf"    if (isUseLocalNodeSSLConfig(cmd)) {      val newJavaOpts = cmd.javaOpts          .filter(opt => !opt.startsWith(s"-D$prefix")) ++          conf.getAll.collect { case (key, value) if key.startsWith(prefix) => s"-D$key=$value" } :+          s"-D$useNLC=true"      cmd.copy(javaOpts = newJavaOpts)    } else {      cmd    }  }}

这里的创建Worker容器的过程和Master是一样。在创建Worker过程中，创建Worker的工作目录，内置WebUI的启动，同时会通知Master并注册。注册代码，查看Master：

case RegisterWorker(      id, workerHost, workerPort, workerRef, cores, memory, workerWebUiUrl, masterAddress) =>      logInfo("Registering worker %s:%d with %d cores, %s RAM".format(        workerHost, workerPort, cores, Utils.megabytesToString(memory)))      if (state == RecoveryState.STANDBY) {        workerRef.send(MasterInStandby)      } else if (idToWorker.contains(id)) {        workerRef.send(RegisterWorkerFailed("Duplicate worker ID"))      } else {        val worker = new WorkerInfo(id, workerHost, workerPort, cores, memory,          workerRef, workerWebUiUrl)        if (registerWorker(worker)) {          persistenceEngine.addWorker(worker)          workerRef.send(RegisteredWorker(self, masterWebUiUrl, masterAddress))          schedule()        } else {          val workerAddress = worker.endpoint.address          logWarning("Worker registration failed. Attempted to re-register worker at same " +            "address: " + workerAddress)          workerRef.send(RegisterWorkerFailed("Attempted to re-register worker at same address: "            + workerAddress))        }      }

最终Master将Worker的信息持久化到硬盘上。如果Worker注册成功，并且应用中存在提交的应用，还会在Worker上启动Executor，分配计算任务。