spark core 2.0 LiveListenerBus

 在SparkContext中可以年到listenerBus的类型是LiveListenerBus

 // An asynchronous listener bus for Spark events  private[spark] val listenerBus = new LiveListenerBus(this)

LiveListenerBus以异步的方式把SparkListenerEvents传递给注册的SparkListeners。

在Start() 方法被调用之前，所有提交的事件仅仅放到缓冲区里。 直到这个Listenerbus启动之后，这些事件才会被真正的广播到所有注册的监听器中。 当stop方法被调用后，listener bus会停止运行，在此之后，会丢掉所有到来的事件。

/** * Asynchronously passes SparkListenerEvents to registered SparkListeners. * * Until `start()` is called, all posted events are only buffered. Only after this listener bus * has started will events be actually propagated to all attached listeners. This listener bus * is stopped when `stop()` is called, and it will drop further events after stopping. */

调用start的时候启动一个线程。

  /**   * Start sending events to attached listeners.   *   * This first sends out all buffered events posted before this listener bus has started, then   * listens for any additional events asynchronously while the listener bus is still running.   * This should only be called once.   *   */  def start(): Unit = {    if (started.compareAndSet(false, true)) {      listenerThread.start()    } else {      throw new IllegalStateException(s"$name already started!")    }  }

在此线程里，有断取出事件调用postToAll

private val listenerThread = new Thread(name) {    setDaemon(true)    override def run(): Unit = Utils.tryOrStopSparkContext(sparkContext) {      LiveListenerBus.withinListenerThread.withValue(true) {        while (true) {          eventLock.acquire()          self.synchronized {            processingEvent = true          }          try {            val event = eventQueue.poll            if (event == null) {              // Get out of the while loop and shutdown the daemon thread              if (!stopped.get) {                throw new IllegalStateException("Polling `null` from eventQueue means" +                  " the listener bus has been stopped. So `stopped` must be true")              }              return            }            postToAll(event)          } finally {            self.synchronized {              processingEvent = false            }          }        }      }    }  }

postToAll方法在ListenerBus里定义，在这个方法里,对于每个 listener，调用doPostEvent(listener,event )方法。

 /**   * Post the event to all registered listeners. The `postToAll` caller should guarantee calling   * `postToAll` in the same thread for all events.   */  final def postToAll(event: E): Unit = {    // JavaConverters can create a JIterableWrapper if we use asScala.    // However, this method will be called frequently. To avoid the wrapper cost, here we use    // Java Iterator directly.    val iter = listeners.iterator    while (iter.hasNext) {      val listener = iter.next()      try {        doPostEvent(listener, event)      } catch {        case NonFatal(e) =>          logError(s"Listener ${Utils.getFormattedClassName(listener)} threw an exception", e)      }    }  }

doPostEvent在SparkListenerBus类中定义

protected override def doPostEvent(      listener: SparkListenerInterface,      event: SparkListenerEvent): Unit = {    event match {      case stageSubmitted: SparkListenerStageSubmitted =>        listener.onStageSubmitted(stageSubmitted)      case stageCompleted: SparkListenerStageCompleted =>        listener.onStageCompleted(stageCompleted)      case jobStart: SparkListenerJobStart =>        listener.onJobStart(jobStart)      case jobEnd: SparkListenerJobEnd =>        listener.onJobEnd(jobEnd)      case taskStart: SparkListenerTaskStart =>        listener.onTaskStart(taskStart)      case taskGettingResult: SparkListenerTaskGettingResult =>        listener.onTaskGettingResult(taskGettingResult)      case taskEnd: SparkListenerTaskEnd =>        listener.onTaskEnd(taskEnd)      case environmentUpdate: SparkListenerEnvironmentUpdate =>        listener.onEnvironmentUpdate(environmentUpdate)      case blockManagerAdded: SparkListenerBlockManagerAdded =>        listener.onBlockManagerAdded(blockManagerAdded)      case blockManagerRemoved: SparkListenerBlockManagerRemoved =>        listener.onBlockManagerRemoved(blockManagerRemoved)      case unpersistRDD: SparkListenerUnpersistRDD =>        listener.onUnpersistRDD(unpersistRDD)      case applicationStart: SparkListenerApplicationStart =>        listener.onApplicationStart(applicationStart)      case applicationEnd: SparkListenerApplicationEnd =>        listener.onApplicationEnd(applicationEnd)      case metricsUpdate: SparkListenerExecutorMetricsUpdate =>        listener.onExecutorMetricsUpdate(metricsUpdate)      case executorAdded: SparkListenerExecutorAdded =>        listener.onExecutorAdded(executorAdded)      case executorRemoved: SparkListenerExecutorRemoved =>        listener.onExecutorRemoved(executorRemoved)      case blockUpdated: SparkListenerBlockUpdated =>        listener.onBlockUpdated(blockUpdated)      case logStart: SparkListenerLogStart => // ignore event log metadata      case _ => listener.onOtherEvent(event)    }