Hadoop MapReduce之ReduceTask任务执行（三）

在reduce端的文件拷贝阶段，会将数据放入内存或直接放入磁盘中，如果文件全部拷贝完再进行合并那样必然降低作业效率，所以在拷贝进行到一定阶段，数据的合并就开始了，负责该工作的有两个线程：InMemFSMergeThread和LocalFSMerger，分别针对内存和磁盘Segment的合并。
首先看内存合并线程InMemFSMergeThread的run函数

  public void run() {    LOG.info(reduceTask.getTaskID() + " Thread started: " + getName());    try {      boolean exit = false;      do {        exit = ramManager.waitForDataToMerge(); //检测是否需要合并        if (!exit) {          doInMemMerge();//执行合并        }      } while (!exit);    } catch (Exception e) {      LOG.warn(reduceTask.getTaskID() +               " Merge of the inmemory files threw an exception: "               + StringUtils.stringifyException(e));      ReduceCopier.this.mergeThrowable = e;    } catch (Throwable t) {      String msg = getTaskID() + " : Failed to merge in memory"                    + StringUtils.stringifyException(t);      reportFatalError(getTaskID(), t, msg);    }  }

  下面是内存合并的条件，注释写的已经很清楚了，这里需要注意的是内存的使用量、拷贝完毕的文件数、挂起线程数，线程挂起的判断条件是用于保留map端数据的内存超过阈值，可参考ShuffleRamManager.reserve()函数

  public boolean waitForDataToMerge() throws InterruptedException {    boolean done = false;    synchronized (dataAvailable) {             // Start in-memory merge if manager has been closed or...      while (!closed             &&             // In-memory threshold exceeded and at least two segments             // have been fetched             (getPercentUsed() < maxInMemCopyPer || numClosed < 2)             &&             // More than "mapred.inmem.merge.threshold" map outputs             // have been fetched into memory             (maxInMemOutputs <= 0 || numClosed < maxInMemOutputs)             &&              // More than MAX... threads are blocked on the RamManager             // or the blocked threads are the last map outputs to be             // fetched. If numRequiredMapOutputs is zero, either             // setNumCopiedMapOutputs has not been called (no map ouputs             // have been fetched, so there is nothing to merge) or the             // last map outputs being transferred without             // contention, so a merge would be premature.             (numPendingRequests <                   numCopiers*MAX_STALLED_SHUFFLE_THREADS_FRACTION &&               (0 == numRequiredMapOutputs ||               numPendingRequests < numRequiredMapOutputs))) {        dataAvailable.wait();      }      done = closed;    }    return done;  }  

  这里的合并可以和map端的合并对比来看，逻辑大同小异，确定文件名、构建写入器，将segment放入合并队列中，如果有本地合并函数则先合并否则直接写入文件。

  private void doInMemMerge() throws IOException{    if (mapOutputsFilesInMemory.size() == 0) {      return;    }        //name this output file same as the name of the first file that is     //there in the current list of inmem files (this is guaranteed to    //be absent on the disk currently. So we don't overwrite a prev.     //created spill). Also we need to create the output file now since    //it is not guaranteed that this file will be present after merge    //is called (we delete empty files as soon as we see them    //in the merge method)    //figure out the mapId     TaskID mapId = mapOutputsFilesInMemory.get(0).mapId;    List<Segment<K, V>> inMemorySegments = new ArrayList<Segment<K,V>>();    long mergeOutputSize = createInMemorySegments(inMemorySegments, 0);    int noInMemorySegments = inMemorySegments.size();    Path outputPath =        mapOutputFile.getInputFileForWrite(mapId, mergeOutputSize);    Writer writer =       new Writer(conf, rfs, outputPath,                 conf.getMapOutputKeyClass(),                 conf.getMapOutputValueClass(),                 codec, null);    RawKeyValueIterator rIter = null;    try {      LOG.info("Initiating in-memory merge with " + noInMemorySegments +                " segments...");            rIter = Merger.merge(conf, rfs,                           (Class<K>)conf.getMapOutputKeyClass(),                           (Class<V>)conf.getMapOutputValueClass(),                           inMemorySegments, inMemorySegments.size(),                           new Path(reduceTask.getTaskID().toString()),                           conf.getOutputKeyComparator(), reporter,                           spilledRecordsCounter, null);            if (combinerRunner == null) {        Merger.writeFile(rIter, writer, reporter, conf);      } else {        combineCollector.setWriter(writer);        combinerRunner.combine(rIter, combineCollector);      }      writer.close();      LOG.info(reduceTask.getTaskID() +           " Merge of the " + noInMemorySegments +          " files in-memory complete." +          " Local file is " + outputPath + " of size " +           localFileSys.getFileStatus(outputPath).getLen());    } catch (Exception e) {       //make sure that we delete the ondisk file that we created       //earlier when we invoked cloneFileAttributes      localFileSys.delete(outputPath, true);      throw (IOException)new IOException              ("Intermediate merge failed").initCause(e);    }    // Note the output of the merge    FileStatus status = localFileSys.getFileStatus(outputPath);    synchronized (mapOutputFilesOnDisk) {      addToMapOutputFilesOnDisk(status);    }  }}

磁盘文件的合并与此大致相同，可以具体细节可以查看org.apache.hadoop.mapred.ReduceTask.ReduceCopier.LocalFSMerger