spatialhadoop2.3源码阅读(十二) GridOutputFormat & GridRecordWriter[Grid Index MapReuce]
来源:互联网 发布:excad激光软件 编辑:程序博客网 时间:2024/05/17 09:39
1. GridOutputFormat
GridOutputFormat的作用是产生RecordWriter,其生成了GridRecordWriter,代码如下:
public class GridOutputFormat<S extends Shape> extends FileOutputFormat<IntWritable, S> { @Override public RecordWriter<IntWritable, S> getRecordWriter(FileSystem ignored, JobConf job, String name, Progressable progress) throws IOException { // Get grid info CellInfo[] cellsInfo = SpatialSite.getCells(job); GridRecordWriter<S> writer = new GridRecordWriter<S>(job, name, cellsInfo); return writer; } }
2.edu.umn.cs.spatialHadoop.mapred.GridRecordWriter
该类实现了MapReduce框架的RecordWriter接口,并继承了edu.umn.cs.spatialHadoop.core.GridRecordWriter类。
edu.umn.cs.spatialHadoop.mapred.GridRecordWriter的具体实现就是对edu.umn.cs.spatialHadoop.core.GridRecordWriter的调用,所以实现重点在于edu.umn.cs.spatialHadoop.core.GridRecordWriter类。
edu.umn.cs.spatialHadoop.mapred.GridRecordWriter的代码如下:
public class GridRecordWriter<S extends Shape>extends edu.umn.cs.spatialHadoop.core.GridRecordWriter<S> implements RecordWriter<IntWritable, S> { public GridRecordWriter(JobConf job, String name, CellInfo[] cells) throws IOException { super(null, job, name, cells); } @Override public void write(IntWritable key, S value) throws IOException { super.write(key.get(), value); } @Override public void close(Reporter reporter) throws IOException { super.close(reporter); }
3.edu.umn.cs.spatialHadoop.core.GridRecordWriter
3.1 构造函数
public GridRecordWriter(Path outDir, JobConf job, String prefix, CellInfo[] cells) throws IOException { if (job != null) { this.sindex = job.get("sindex", "heap"); this.pack = PackedIndexes.contains(sindex); this.expand = ExpandedIndexes.contains(sindex); } this.prefix = prefix; this.fileSystem = outDir == null ? FileOutputFormat.getOutputPath(job).getFileSystem(job): outDir.getFileSystem(job != null? job : new Configuration()); this.outDir = outDir; this.jobConf = job; if (cells != null) { // Make sure cellIndex maps to array index. This is necessary for calls that // call directly write(int, Text) int highest_index = 0; for (CellInfo cell : cells) { if (cell.cellId > highest_index) highest_index = (int) cell.cellId; } // Create a master file that contains meta information about partitions masterFile = fileSystem.create(getMasterFilePath()); this.cells = new CellInfo[highest_index + 1]; for (CellInfo cell : cells) this.cells[(int) cell.cellId] = cell; // Prepare arrays that hold cells information intermediateCellStreams = new OutputStream[this.cells.length]; intermediateCellPath = new Path[this.cells.length]; cellsMbr = new Rectangle[this.cells.length]; // Initialize the counters for each cell intermediateCellRecordCount = new int[this.cells.length]; intermediateCellSize = new int[this.cells.length]; } else { intermediateCellStreams = new OutputStream[1]; intermediateCellPath = new Path[1]; cellsMbr = new Rectangle[1]; intermediateCellSize = new int[1]; intermediateCellRecordCount = new int[1]; } for (int i = 0; i < cellsMbr.length; i++) { cellsMbr[i] = new Rectangle(Double.MAX_VALUE, Double.MAX_VALUE, -Double.MAX_VALUE, -Double.MAX_VALUE); } this.blockSize = fileSystem.getDefaultBlockSize(outDir); closingThreads = new ArrayList<Thread>(); text = new Text(); }
3-13:初始化参数;
18-23:判断输入的CellInfo数组中最大的ceilID
26:在输出目录中创建master文件
28-30:根据highest_index,为cells成员变量赋值
33-38:初始化一些与网格信息相关的数组
41-45:不执行
47-50:初始化cellsMbr
3.2. writeInternal函数
调用顺序为edu.umn.cs.spatialHadoop.mapred.GridRecordWriter.write->edu.umn.cs.spatialHadoop.core.GridRecordWriter.write->edu.umn.cs.spatialHadoop.core.GridRecordWriter.writeInternal,所以重点在于writeInternal函数的实现。具体代码如下:
protected synchronized void writeInternal(int cellIndex, S shape) throws IOException { if (cellIndex < 0) { // A special marker to close a cell closeCell(-cellIndex); return; } try { cellsMbr[cellIndex].expand(shape.getMBR()); } catch (NullPointerException e) { e.printStackTrace(); } // Convert shape to text text.clear(); shape.toText(text); // Write text representation to the file OutputStream cellStream = getIntermediateCellStream(cellIndex); cellStream.write(text.getBytes(), 0, text.getLength()); cellStream.write(NEW_LINE); intermediateCellSize[cellIndex] += text.getLength() + NEW_LINE.length; intermediateCellRecordCount[cellIndex]++; }
2-6:在Grid Index MapReuce的reduce实现中,对于每一个cell,当处理完该cell的所有数据后,总会输出一个key为-cellID的key-value对,用来标识该cell已经处理完毕,该处 代码就是对某一cell的最终处理。
8:计算当前cellIndex对应的cell已经写入文件的数据的最小包围矩形
13-14:将该shape转换为Text
16:获得该cellIndex对应的cell文件的输出流,在getIntermediateCellStream函数中,先判断输出流是否存在,若不存在则先获得输出路径,再构建输出流,若存在直接返 回
17-20:将数据写入输出流,同时更新已写入文件的数据大小和数据记录数。
3.3 closeCell函数
在上面讲述中,2-6行代码中涉及到closeCell函数,接下来主要介绍该函数。
protected void closeCell(int cellIndex) throws IOException { CellInfo cell = cells != null? cells[cellIndex] : new CellInfo(cellIndex+1, cellsMbr[cellIndex]); if (expand) cell.expand(cellsMbr[cellIndex]); if (pack) cell = new CellInfo(cell.cellId, cell.getIntersection(cellsMbr[cellIndex])); closeCellBackground(intermediateCellPath[cellIndex], getFinalCellPath(cellIndex), intermediateCellStreams[cellIndex], masterFile, cell, intermediateCellRecordCount[cellIndex], intermediateCellSize[cellIndex]); cellsMbr[cellIndex] = new Rectangle(Double.MAX_VALUE, Double.MAX_VALUE, -Double.MAX_VALUE, -Double.MAX_VALUE); intermediateCellPath[cellIndex] = null; intermediateCellStreams[cellIndex] = null; intermediateCellRecordCount[cellIndex] = 0; intermediateCellSize[cellIndex] = 0; }
2:获得cellIndex对应的CellInfo信息
3-4:判断当前的索引类型是否需要将网格信息和已写入的数据的最小包围矩形进行合并,expand在构造函数中确定
5-6:判断当前的索引类型是否需要将网格信息和已写入的数据的最小包围矩形取交集,pack在构造函数中确定
8:调用closeCellBackground进行处理。在该函数中,会单独开辟一个线程来进行最终处理,并将该线程加入成员变量closingThreads中。包括刷新并关闭该输出流,更新master文件。同时遍历closingThreads,若有已经结束的线程,则销毁,同时启动当前新建的线程。
11-16:销毁与该cell相关的成员变量。
3.4 close函数
调用顺序为edu.umn.cs.spatialHadoop.mapred.GridRecordWriter.close->edu.umn.cs.spatialHadoop.core.GridRecordWriter.close,所以重点在于close函数的实现。具体代码如下:
public synchronized void close(Progressable progressable) throws IOException { // Close all output files for (int cellIndex = 0; cellIndex < intermediateCellStreams.length; cellIndex++) { if (intermediateCellStreams[cellIndex] != null) { closeCell(cellIndex); } // Indicate progress. Useful if closing a single cell takes a long time if (progressable != null) progressable.progress(); } while (!closingThreads.isEmpty()) { try { Thread t = closingThreads.get(0); switch (t.getState()) { case NEW: t.start(); break; case TERMINATED: closingThreads.remove(0); break; default: // Use limited time join to indicate progress frequently t.join(10000); } // Indicate progress. Useful if closing a single cell takes a long time if (progressable != null) progressable.progress(); } catch (InterruptedException e) { e.printStackTrace(); } } if (masterFile != null) masterFile.close(); }
3-9:判断是否有未关闭的cell输出流,并关闭
12-28:清理cell输出流关闭线程。
30:关闭masterFile
- spatialhadoop2.3源码阅读(十二) GridOutputFormat & GridRecordWriter[Grid Index MapReuce]
- spatialhadoop2.3源码阅读(十一) ShapeRecordReader & SpatialRecordReader[Grid Index MapReuce]
- spatialhadoop2.3源码阅读(十三) RTreeGridOutputFormat & RTreeGridRecordWriter & RTree[RTree Index MapReuce]
- spatialhadoop2.3源码阅读(五) grid 索引生成方法(一)
- spatialhadoop2.3源码阅读(六) grid 索引生成方法(二)
- spatialhadoop2.3源码阅读(四) FileMBR类
- spatialhadoop2.3源码阅读(七) Sampler类
- spatialhadoop2.3源码阅读(八) RTree索引生成方法(一)
- spatialhadoop2.3源码阅读(八) RTree索引生成方法(二)
- spatialhadoop2.3源码阅读(九) ShapeLineInputFormat & ShapeLineRecordReader & SpatialRecordReader[FileMBR]
- spatialhadoop2.3源码阅读(十) TextOutputFormat & LineRecordWriter[FileMBR]
- spatialhadoop2.1源码阅读(一) shadoop脚本文件
- spatialhadoop2.1源码阅读(二) bin/shadoop generate命令
- spatialhadoop2.1源码阅读(三) 自定义InputFormat(SpatialInputFormat & ShapeInputFormat)
- SpatialHadoop2.x源码编译
- tomcat源码阅读步骤十二
- Mapreuce
- Spring源码阅读(十二)—事务
- cocos2d-x windows下用vs开发
- 1
- Overdue resource check-out stack trace
- web中文乱码问题总结
- hammer.js
- spatialhadoop2.3源码阅读(十二) GridOutputFormat & GridRecordWriter[Grid Index MapReuce]
- github 的简单使用
- 引用 Foxmail常用的25个技巧
- 二分查找分析
- 设计模式(二十二)---享元模式
- 欢迎使用CSDN-markdown编辑器
- Maximo 密码加密传送
- NSFileHandle处理文件
- java操作mysql中文乱码解决