Spark源码解析SparkStreaming数据接收

在上一篇博文中，我们讲述了一个SparkStreaming应用程序启动后开始的准备工作，即在executors启动receiver
这里我们将讲述接收数据到存储数据的过程
首先接受数据是在receiver的onStart方法里，在这里我们还是以SocketReceiver为例,在SocketReceiver的OnStart方法中启动一个线程，在该线程中调用receive方法，进行接收数据的处理

def receive() {  var socket: Socket = null  try {    logInfo("Connecting to " + host + ":" + port)    socket = new Socket(host, port)    logInfo("Connected to " + host + ":" + port)    val iterator = bytesToObjects(socket.getInputStream())    while(!isStopped && iterator.hasNext) {      store(iterator.next)    }   ...  } catch {  ...  } finally {    if (socket != null) {      socket.close()      logInfo("Closed socket to " + host + ":" + port)    }  }}

我们可以看到在这里，接收到数据后，会调用store方法存储接收到的数据，store方法是一个重载的方法，其有很多的实现，大致如下：

def store(dataItem: T) {    supervisor.pushSingle(dataItem)}def store(dataBuffer: ArrayBuffer[T]) {  supervisor.pushArrayBuffer(dataBuffer, None, None)}def store(dataBuffer: ArrayBuffer[T], metadata: Any) {  supervisor.pushArrayBuffer(dataBuffer, Some(metadata), None)}def store(dataIterator: Iterator[T]) {  supervisor.pushIterator(dataIterator, None, None)}...

还有很多的重载方法，但都是去调用supervisor的pushXXXX方法，在ReceiverSupervisorImpl内的这些方法，都会最终调用ReceiverSupervisorImpl#pushAndReportBlock

def pushAndReportBlock(    receivedBlock: ReceivedBlock,    metadataOption: Option[Any],    blockIdOption: Option[StreamBlockId]  ) {  val blockId = blockIdOption.getOrElse(nextBlockId)  val time = System.currentTimeMillis  //存储block到BlockManager中，这里我们就可以看到预写日志的机制  val blockStoreResult = receivedBlockHandler.storeBlock(blockId, receivedBlock)  val numRecords = blockStoreResult.numRecords  //封装一个ReceivedBlockInfo对象，里面有一个streamId  val blockInfo = ReceivedBlockInfo(streamId, numRecords, metadataOption, blockStoreResult)  //向ReceiverTracker发送AddBlock的消息，这个样例类中包含了block的相关信息blockInfo  trackerEndpoint.askWithRetry[Boolean](AddBlock(blockInfo))  logDebug(s"Reported block $blockId")}

我们在这里需要注意一下receivedBlockHandler，这个对象的初始化如下：

private val receivedBlockHandler: ReceivedBlockHandler = { //如果开启了预写日志机制，默认为false //那么receivedBlockHandler就是WriteAheadLogBasedBlockHandler //如果没有开启预写日志机制，那么receivedBlockHandler就是BlockManagerBasedBlockHandler if (WriteAheadLogUtils.enableReceiverLog(env.conf)) {   if (checkpointDirOption.isEmpty) {     throw new SparkException(       "Cannot enable receiver write-ahead log without checkpoint directory set. " +         "Please use streamingContext.checkpoint() to set the checkpoint directory. " +         "See documentation for more details.")   }   new WriteAheadLogBasedBlockHandler(env.blockManager, receiver.streamId,     receiver.storageLevel, env.conf, hadoopConf, checkpointDirOption.get) } else {   new BlockManagerBasedBlockHandler(env.blockManager, receiver.storageLevel) }}

我们可以看到对于是否开启预写日志机制将会创建不同的子类对象，在这里我们以WriteAheadLogBasedBlockHandler的storeBlock方法为例：

def storeBlock(blockId: StreamBlockId, block: ReceivedBlock): ReceivedBlockStoreResult = {   var numRecords = None: Option[Long]   // Serialize the block so that it can be inserted into both   //先用BlockManager序列化数据   val serializedBlock = block match {     case ArrayBufferBlock(arrayBuffer) =>       numRecords = Some(arrayBuffer.size.toLong)       blockManager.dataSerialize(blockId, arrayBuffer.iterator)     case IteratorBlock(iterator) =>       val countIterator = new CountingIterator(iterator)       val serializedBlock = blockManager.dataSerialize(blockId, countIterator)       numRecords = countIterator.count       serializedBlock     case ByteBufferBlock(byteBuffer) =>       byteBuffer     case _ =>       throw new Exception(s"Could not push $blockId to block manager, unexpected block type")   }   // Store the block in block manager   //将数据保存到BlockManager，默认的持久化策略是_SER,_2的，会序列化，会复制一份副本到其他Executor上的BlockManager中   //以供容错需要   val storeInBlockManagerFuture = Future {     val putResult =       blockManager.putBytes(blockId, serializedBlock, effectiveStorageLevel, tellMaster = true)     if (!putResult.map { _._1 }.contains(blockId)) {       throw new SparkException(         s"Could not store $blockId to block manager with storage level $storageLevel")     }   }   // Store the block in write ahead log   //将block存入预写日志   val storeInWriteAheadLogFuture = Future {     writeAheadLog.write(serializedBlock, clock.getTimeMillis())   }   // Combine the futures, wait for both to complete, and return the write ahead log record handle   val combinedFuture = storeInBlockManagerFuture.zip(storeInWriteAheadLogFuture).map(_._2)   val walRecordHandle = Await.result(combinedFuture, blockStoreTimeout)   WriteAheadLogBasedStoreResult(blockId, numRecords, walRecordHandle)}

在这个方法中我们先将block进行序列化，然后将序列化后的数据写入BlockManager，最后将序列化后的数据写入日志

我们再回到上面的pushAndReportBlock方法，上面我们解析的是其push部分，现在我们需要解析的是其Report部分
我们可以看到封装了一个blockInfo对象，然后向ReceiverTracker发送了AddBlock消息，在ReceiverTracker中接收到这个消息后的处理：

case AddBlock(receivedBlockInfo) =>    context.reply(addBlock(receivedBlockInfo))private def addBlock(receivedBlockInfo: ReceivedBlockInfo): Boolean = {  receivedBlockTracker.addBlock(receivedBlockInfo)}def addBlock(receivedBlockInfo: ReceivedBlockInfo): Boolean = synchronized {     try {       writeToLog(BlockAdditionEvent(receivedBlockInfo))       getReceivedBlockQueue(receivedBlockInfo.streamId) += receivedBlockInfo       logDebug(s"Stream ${receivedBlockInfo.streamId} received " +         s"block ${receivedBlockInfo.blockStoreResult.blockId}")       true     } catch {       case e: Exception =>         logError(s"Error adding block $receivedBlockInfo", e)         false     }}

这段代码中最重要的无非就是getReceivedBlockQueue(receivedBlockInfo.streamId) += receivedBlockInfo即将接收到的blockInfo存放在一个队列中，然后再将这个队列放入一个以streamId为key的map中

综上我们可以发现Receiver在接受到数据后，主要就是做了两件事：
1. 将接收到的消息存入BlockManager中
2. 将接受到的消息的blockInfo作为AddBlock消息的参数发送给ReceiverTracker，ReceiverTrakcer中使用一个Map[streamId,BlockQuqueu[blockInfo]]的数据结构存储之