spatialhadoop2.3源码阅读(九) ShapeLineInputFormat & ShapeLineRecordReader & SpatialRecordReader[FileMBR]

1.ShapeLineInputFormat

ShapeLineInputFormat的作用就是生成ShapeLineRecordReader，其具体实现和spatialhadoop2.1源码阅读(三) 自定义InputFormat（SpatialInputFormat & ShapeInputFormat）中介绍的ShapeInputFormat的实现基本相同，具体可看该文章。ShapeLineInputFormat的源码如下：

public class ShapeLineInputFormat extends SpatialInputFormat<Rectangle, Text> {    @Override  public RecordReader<Rectangle, Text> getRecordReader(InputSplit split,      JobConf job, Reporter reporter) throws IOException {    if (reporter != null)      reporter.setStatus(split.toString());    this.rrClass = ShapeLineRecordReader.class;    return super.getRecordReader(split, job, reporter);  }}

接下来重点介绍ShapeLineRecordReader和 SpatialRecordReader的实现。

2.ShapeLineRecordReader

ShapeLineRecordReader继承自SpatialRecordReader，ShapeLineRecordReader中最重要的三个方法为createKey，createValue和next。除此之外还有三个构造器。这六个方法的具体实现都是对父类相应方法的调用，所以实现重点在于SpatialRecordReader类。

public class ShapeLineRecordReader    extends SpatialRecordReader<Rectangle, Text> {  public ShapeLineRecordReader(Configuration job, FileSplit split)      throws IOException {    super(job, split);  }  public ShapeLineRecordReader(CombineFileSplit split, Configuration conf,      Reporter reporter, Integer index) throws IOException {    super(split, conf, reporter, index);  }    public ShapeLineRecordReader(InputStream in, long offset, long endOffset)      throws IOException {    super(in, offset, endOffset);  }  @Override  public boolean next(Rectangle key, Text shapeLine) throws IOException {    boolean read_line = nextLine(shapeLine);    key.set(cellMbr);    return read_line;  }  @Override  public Rectangle createKey() {    return new Rectangle();  }  @Override  public Text createValue() {    return new Text();  }}

3.SpatialRecordReader(按照FileMBR MapReduce进行介绍)

3.1 构造函数

public SpatialRecordReader(Configuration job, long s, long l, Path p) throws IOException {    this.start = s;    this.end = s + l;    this.path = p;    this.fs = this.path.getFileSystem(job);    this.directIn = fs.open(this.path);    this.blockSize = fs.getFileStatus(this.path).getBlockSize();    this.cellMbr = new Rectangle();        LOG.info("Open a SpatialRecordReader to file: "+this.path);    codec = new CompressionCodecFactory(job).getCodec(this.path);    if (isCompressedInput()) {      decompressor = CodecPool.getDecompressor(codec);      if (codec instanceof SplittableCompressionCodec) {        final SplitCompressionInputStream cIn =            ((SplittableCompressionCodec)codec).createInputStream(                directIn, decompressor, start, end,                SplittableCompressionCodec.READ_MODE.BYBLOCK);        in = cIn;        start = cIn.getAdjustedStart();        end = cIn.getAdjustedEnd();        filePosition = cIn; // take pos from compressed stream      } else {        in = codec.createInputStream(directIn, decompressor);        filePosition = directIn;      }    } else {      directIn.seek(start);      in = directIn;      filePosition = directIn;    }    this.pos = start;    this.maxShapesInOneRead = job.getInt(SpatialSite.MaxShapesInOneRead, 1000000);    this.maxBytesInOneRead = job.getInt(SpatialSite.MaxBytesInOneRead, 32*1024*1024);    initializeReader();  }

2-4：初始化该输入分片所在文件名，起始位置和终止位置

6：打开输入文件，如果输入文件是压缩文件，则该输入流表示压缩输入流

12-14：判断输入文件是否是压缩文件

15-29：若输入文件是压缩文件，则in为解压缩后的输入流，同时根据压缩类型，更新参数。

30-32：若输入文件不是压缩文件，则in和directIn相等

34：设置偏移量，一次读入最多读入的记录数和字节数

38：首先判断输入文件是否有全局索引，如果存在则设置cellMbr，否则cellMbr无效；然后判断输入文件是否为R-tree indexed，根据不同情况初始化不同。

3.2 nextLine函数

protected boolean nextLine(Text value) throws IOException {    if (blockType == BlockType.RTREE && pos == 8) {      // File is positioned at the RTree header      // Skip the header and go to first data object in file      pos += RTree.skipHeader(in);      LOG.info("Skipped R-tree to position: "+pos);      // Reinitialize record reader at the new position      lineReader = new LineReader(in);    }    while (getFilePosition() <= end) {      value.clear();      int b = 0;      if (buffer != null) {        // Read the first line encountered in buffer        int eol = RTree.skipToEOL(buffer, 0);        b += eol;        value.append(buffer, 0, eol);        if (eol < buffer.length) {          // There are still some bytes remaining in buffer          byte[] tmp = new byte[buffer.length - eol];          System.arraycopy(buffer, eol, tmp, 0, tmp.length);          buffer = tmp;        } else {          buffer = null;        }        // Check if a complete line has been read from the buffer        byte last_byte = value.getBytes()[value.getLength()-1];        if (last_byte == '\n' || last_byte == '\r')          return true;      }            // Read the first line from stream      Text temp = new Text();      b += lineReader.readLine(temp);      if (b == 0) {        // Indicates an end of stream        return false;      }      pos += b;            // Append the part read from stream to the part extracted from buffer      value.append(temp.getBytes(), 0, temp.getLength());            if (value.getLength() > 1) {        // Read a non-empty line. Note that end-of-line character is included        return true;      }    }    // Reached end of file    return false;  }

2-9：对R-tree indexed进行特殊处理

13-30：在上面所讲的initializeReader函数中，会预先读取八个字节判断文件索引类型，buffer不为空，在这里对这种情况进行处理

33-38：每次读取一行数据，并判断是否到输入流的末尾

39：更新pos

42：对value进行赋值