Spark Executor on YARN

YARN executor launch context

  env:    CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*    SPARK_YARN_STAGING_DIR -> file:/Users/User/.sparkStaging/application_1479802900835_0001    SPARK_USER -> User    SPARK_YARN_MODE -> true  command:    {{JAVA_HOME}}/bin/java \      -server \      -Xmx2048m \      -Djava.io.tmpdir={{PWD}}/tmp \      -Dspark.yarn.app.container.log.dir=<LOG_DIR> \      -XX:OnOutOfMemoryError='kill %p' \      org.apache.spark.executor.CoarseGrainedExecutorBackend \      --driver-url \      spark://CoarseGrainedScheduler@172.16.61.169:53279 \      --executor-id \      <executorId> \      --hostname \      <hostname> \      --cores \      1 \      --app-id \      application_1479802900835_0001 \      --user-class-path \      file:$PWD/__app__.jar \      1><LOG_DIR>/stdout \      2><LOG_DIR>/stderr

Application Context

    if (isClusterMode) {        runDriver(securityMgr)      } else {        runExecutorLauncher(securityMgr)      }

• runDriver（Cluster模式)

  1. startUserApplication():
     在新线程中启动用户类
  2. runAMEndpoint:
     创建AM endpoint，返回driver endpoint的引用
  3. registerAM：
     
     • 在RM上注册AM，返回YarnAllocator
     • allocator.allocateResources()
  4. userClassThread.join()

• runExecutorLauncher(Client模式)

RPCEndpoint

• 生命周期：
  constructor -> onStart -> receive* -> onStop
  其中receive方法可以被并发调用。

DriverEndpoint

• onStart: send ReviveOffers，供后续工作执行
• receive:
  
  • StatusUpdate
  • ReviveOffers => Launch Task
  • KillTask

CoarseGrainedExecutorBackend

• onStart: 连接Driver
• receive：
  
  • RegisterExecutor: 在driver上注册executor
  • RegisterExecutorFailed：退出executor
  • LaunchTask：在executor中执行任务
  • KillTask: 在executor中杀死任务
  • StopExecutor：发送shutdown消息User
  • Shutdown：停止executor