hadoop自定义SdfInputFormat，文件按标记分片

由于要用hadoop streaming处理sdf文件，而sdf文件的文件格式为

1  -OEChem-12181003042D.....$$$$

以$$$$结尾的多行。

而hadoop默认的分片为：以分块为基础的分片

    for (FileStatus file: files) {      Path path = file.getPath();      FileSystem fs = path.getFileSystem(job.getConfiguration());      long length = file.getLen();      BlockLocation[] blkLocations = fs.getFileBlockLocations(file, 0, length);      if ((length != 0) && isSplitable(job, path)) {        long blockSize = file.getBlockSize();        long splitSize = computeSplitSize(blockSize, minSize, maxSize);        long bytesRemaining = length;        while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) {          int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);          splits.add(new FileSplit(path, length-bytesRemaining, splitSize,                                   blkLocations[blkIndex].getHosts()));          bytesRemaining -= splitSize;        }        if (bytesRemaining != 0) {          splits.add(new FileSplit(path, length-bytesRemaining, bytesRemaining,                     blkLocations[blkLocations.length-1].getHosts()));        }      } else if (length != 0) {        splits.add(new FileSplit(path, 0, length, blkLocations[0].getHosts()));      } else {        //Create empty hosts array for zero length files        splits.add(new FileSplit(path, 0, length, new String[0]));      }    }

本来想直接修改FileInputFormat.java的getSplits方法。可是本人刚刚接触java和hadoop，必须参考其他例子。于是找到NLineInputFormat的源码，其分片(getSplits方法)是按指定行划分的，有了点头绪。又想到TextInputFormat是按行读取的，想想应该在读取分片时做了些处理，果然是，它在已分片的基础上，跳过起始的最后一行，多读取分片的最后一行。

LineRecordReader.java：    FileSystem fs = file.getFileSystem(job);    FSDataInputStream fileIn = fs.open(split.getPath());    boolean skipFirstLine = false;    if (codec != null) {      in = new LineReader(codec.createInputStream(fileIn), job);      end = Long.MAX_VALUE;    } else {      if (start != 0) {        skipFirstLine = true;        --start;        fileIn.seek(start);      }      in = new LineReader(fileIn, job);    }    if (skipFirstLine) {  // skip first line and re-establish "start".      start += in.readLine(new Text(), 0,                           (int)Math.min((long)Integer.MAX_VALUE, end - start));    }    this.pos = start;  }

于是想到只要改写TextInputFormat就OK了。

下面是源代码的地址：源代码

运行环境是eclipse3.7 +hadoop-1.0.0+64位gentoo