Shuffle的读写操作（一）

下面是我们的ShuffleMapTask当中的runTask的方法，在这个方法当中主要是调用了我们的HashShuffleWrite当中的write方法来进行具体的写出操作

  /**   *   */  override def runTask(context: TaskContext): MapStatus = {    // Deserialize the RDD using the broadcast variable.       //反序列化的起始时间      val deserializeStartTime = System.currentTimeMillis()    // 获得反序列化器closureSerializer      val ser = SparkEnv.get.closureSerializer.newInstance()    // 调用反序列化器closureSerializer的deserialize()进行RDD和ShuffleDependency的反序列化,数据来源于taskBinary     val (rdd, dep) = ser.deserialize[(RDD[_], ShuffleDependency[_, _, _])](      ByteBuffer.wrap(taskBinary.value), Thread.currentThread.getContextClassLoader)    //计算Executor进行反序列化的时间      _executorDeserializeTime = System.currentTimeMillis() - deserializeStartTime    metrics = Some(context.taskMetrics)    var writer: ShuffleWriter[Any, Any] = null    try {      //获得shuffleManager      val manager = SparkEnv.get.shuffleManager      //根据partition指定分区的Shufflea获取Shuffle Writer,shuffleHandle是shuffle ID      //partitionId表示的是当前RDD的某个partition,也就是说write操作作用于partition之上        writer = manager.getWriter[Any, Any](dep.shuffleHandle, partitionId, context)      //针对RDD中的分区partition,调用rdd的iterator()方法后,再调用writer的write()方法,写数据        writer.write(rdd.iterator(partition, context).asInstanceOf[Iterator[_ <: Product2[Any, Any]]])      //停止writer,并返回标志位       writer.stop(success = true).get    } catch {      case e: Exception =>        try {          if (writer != null) {            writer.stop(success = false)          }        } catch {          case e: Exception =>            log.debug("Could not stop writer", e)        }        throw e    }  }

下面这个代码是我们的HashShuffleWrite的写方法的代码如下：

  /**    * Write a bunch of records to this task's output    * 将一堆记录写入此任务的输出*/    /**     * 主要处理两件事:     * 1)判断是否需要进行聚合,比如<hello,1>和<hello,1>都要写入的话,那么先生成<hello,2>     *   然后再进行后续的写入工作     * 2)利用Partition函数来决定<k,val>写入哪一个文件中.     */  override def write(records: Iterator[Product2[K, V]]): Unit = {    //判断aggregator是否被定义,需要做Map端聚合操作    val iter = if (dep.aggregator.isDefined) {      if (dep.mapSideCombine) {//判断是否需要聚合,如果需要,聚合records执行map端的聚合        //汇聚工作,reducebyKey是一分为二的,一部在ShuffleMapTask中进行聚合        //另一部分在resultTask中聚合        dep.aggregator.get.combineValuesByKey(records, context)      } else {        records      }    } else {      require(!dep.mapSideCombine, "Map-side combine without Aggregator specified!")      records    }     //利用getPartition函数来决定<k,val>写入哪一个文件中.    for (elem <- iter) {     //elem是类似于<k,val>的键值对,以K为参数用partitioner计算其对应的值,      val bucketId = dep.partitioner.getPartition(elem._1)//获得该element需要写入的partitioner      //实际调用FileShuffleBlockManager.forMapTask进入数据写入      //bucketId文件名称,key elem._1,value elem._2      shuffle.writers(bucketId).write(elem._1, elem._2)    }  }

FileShuffleBlockResolver类的主要解析如下：

/** * Manages assigning disk-based block writers to shuffle tasks. Each shuffle task gets one file * per reducer (this set of files is called a ShuffleFileGroup).  * 管理分配基于磁盘的块写入器来随机播放任务,每个shuffle任务每个reducer获取一个文件(这组文件称为ShuffleFileGroup) * * As an optimization to reduce the number of physical shuffle files produced, multiple shuffle * blocks are aggregated into the same file. There is one "combined shuffle file" per reducer * per concurrently executing shuffle task. As soon as a task finishes writing to its shuffle * files, it releases them for another task.  *  * 作为减少生成的物理随机播放文件数量的优化,多个shuffle块被聚合到同一个文件中,每个并发执行随机播放任务,每个reducer有一个“组合shuffle文件”  * 一旦任务完成对其随机播放文件的写入,它将释放它们用于另一个任务。  * * Regarding the implementation of this feature, shuffle files are identified by a 3-tuple:  * 关于此功能的实现,随机播放文件由3元组标识： *   - shuffleId: The unique id given to the entire shuffle stage.给予整个洗牌阶段的唯一身份 *   - bucketId: The id of the output partition (i.e., reducer id)输出分区的id（即reducer id） *   - fileId: The unique id identifying a group of "combined shuffle files." Only one task at a *       time owns a particular fileId, and this id is returned to a pool when the task finishes.  *      识别一组“组合的shuffle文件”的唯一ID,一次只有一个任务拥有一个特定的fileId,当任务完成时,这个id返回给一个池 * Each shuffle file is then mapped to a FileSegment, which is a 3-tuple (file, offset, length) * that specifies where in a given file the actual block data is located.  * 然后将每个随机shuffle文件映射到FileSegment,FileSegment是一个3元组(文件,偏移量,长度),用于指定给定文件中实际块数据所在的位置 * * Shuffle file metadata is stored in a space-efficient manner. Rather than simply mapping * ShuffleBlockIds directly to FileSegments, each ShuffleFileGroup maintains a list of offsets for * each block stored in each file. In order to find the location of a shuffle block, we search the * files within a ShuffleFileGroups associated with the block's reducer.  *  *Shuffle文件元数据以节省空间的方式存储,而不是简单的映射ShuffleBlock直接转到FileSegments,  * 每个ShuffleFileGroup为每个文件中存储的每个块维护一个偏移量列表,为了找到混洗块的位置,  * 我们搜索与块的reducer相关联的ShuffleFileGroup中的文件。 */

上面这个类的 forMapTask方法如下

/**   *    * Get a ShuffleWriterGroup for the given map task, which will register it as complete   * when the writers are closed successfully    * 为给定的Map任务获取一个ShuffleWriterGroup,当写关闭成功时,它将注册为完整的    * mapId对应RDD的partionsID   *    */  def forMapTask(shuffleId: Int, mapId: Int, numBuckets: Int, serializer: Serializer,      writeMetrics: ShuffleWriteMetrics): ShuffleWriterGroup = {    new ShuffleWriterGroup {      shuffleStates.putIfAbsent(shuffleId, new ShuffleState(numBuckets))      private val shuffleState = shuffleStates(shuffleId)      private var fileGroup: ShuffleFileGroup = null      val openStartTime = System.nanoTime      val serializerInstance = serializer.newInstance()      //如果consolidateShuffleFiles为true,那么在一个Task中,有多少个输出的Partition就会有多少个中间文件,默认为false      val writers: Array[DiskBlockObjectWriter] = if (consolidateShuffleFiles) {        fileGroup = getUnusedFileGroup()//获取没有使用的FileGroup        Array.tabulate[DiskBlockObjectWriter](numBuckets) { bucketId =>          //mapId对应RDD的partionsID          val blockId = ShuffleBlockId(shuffleId, mapId, bucketId)          blockManager.getDiskWriter(blockId, fileGroup(bucketId), serializerInstance, bufferSize,            writeMetrics)        }      } else {        Array.tabulate[DiskBlockObjectWriter](numBuckets) { bucketId =>          //mapId对应RDD的partionsID          val blockId = ShuffleBlockId(shuffleId, mapId, bucketId)          //如果blockFile已经存在,那么删除它并打印日志          val blockFile = blockManager.diskBlockManager.getFile(blockId)                     val tmp = Utils.tempFileWith(blockFile)              //tmp也就是blockFile如果已经存在则,在后面追加数据          blockManager.getDiskWriter(blockId, tmp, serializerInstance, bufferSize, writeMetrics)        }      }      // Creating the file to write to and creating a disk writer both involve interacting with      // the disk, so should be included in the shuffle write time.      //创建要写入和创建磁盘刻录机的文件都涉及与磁盘交互,因此应该包含在shuffle写入的时间。      writeMetrics.incShuffleWriteTime(System.nanoTime - openStartTime)      override def releaseWriters(success: Boolean) {        if (consolidateShuffleFiles) {          if (success) {            val offsets = writers.map(_.fileSegment().offset)            val lengths = writers.map(_.fileSegment().length)            //mapId对应RDD的partionsID            fileGroup.recordMapOutput(mapId, offsets, lengths)          }          recycleFileGroup(fileGroup)        } else {          //mapId对应RDD的partionsID          shuffleState.completedMapTasks.add(mapId)        }      }