边学边记（五） lucene索引结构二（segments_N）

lucene中索引文件的存储用段segments来描述的，通过segments.gen来维护gen信息

和segments.gen一样头信息是一个表示此索引中段的格式化版本 即format值 根据format值判断此段信息中的信息存储格式和存储内容

一个segment_N文件中保存的信息和信息的类型 如下图所示

 

 

  信息存储的格式是这样的

Format, Version, NameCounter, SegCount, <SegName, SegSize, DelGen, DocStoreOffset, [DocStoreSegment, DocStoreIsCompoundFile], HasSingleNormFile, NumField, NormGen^NumField, IsCompoundFile, DeletionCount, HasProx, Diagnostics>^SegCount, CommitUserData, Checksum

 

Format 是一个int32位的整数一般为-9 SegmentInfos.FORMAT_DIAGNOSTICS

Version 值保存操作此索引的时间的long型时间 System.currentTimeMillis()

NameCounter 用来生成新的segment的文件名

SegCount 记录索引中有多少个段 有多少个段后面红色标记的属性就会重复多少次

 

每个段的信息描述：

SegName, SegSize, DelGen, DocStoreOffset, [DocStoreSegment, DocStoreIsCompoundFile], HasSingleNormFile, NumField, NormGen^NumField, IsCompoundFile, DeletionCount, HasProx, Diagnostics

 

SegName -- 和字段有关的索引文件的前缀 上篇博文中生成的索引中 存储的应该是"_0"

SegSize   -- 记录的此段中 document的个数

DelGen  -- 记录了删除操作过后产生的*.del文件的数量 如果为-1 则没有此文件 如果为0 就在索引中查找_X.del文件 如果 >0就查找_X_N.del文件

 

DocStoreOffset  -- 记录了当前段中的document的存储的位置偏移量 如果为-1 说明 此段使用独立的文件来存储doc，只有在偏移量部位-1的时候[DocStoreSegment, DocStoreIsCompoundFile] 信息才会存储

 

DocStoreSegment  -- 存储和此段共享store的segment的name 

DocStoreIsCompoundFile --用来标识存储是否是混合文件存储 如果值为1则说明使用了混合模式来存储索引 那么操作的时候就回去找.cfx文件

 

HasSingleNormFile --  如果为1 表明 域（field）的信息存储来后缀为.fnm文件里 如果不是则存在.fN文件中

NumField -- field的个数 即NormGen数组的大小 这个数组中具体存了域的gen信息 long类型 前提是每个field都分开存储的

DeletionCount, HasProx, Diagnostics

DeletionCount -- 记录在此段中被删除的doc的数量

HasProx -- 此段中是否已一些域（field） 省略了term的频率和位置  如果值为1 则不存在这样的field

Diagnostics -- 诊断表 key - value 比如当前索引的version，系统的版本，java的版本等

 

CommitUserData 记录用户自己的一些属性信息 在使用commit preparecommit 等操作的时候产生

 

Checksum 记录了当前Segments_N文件中到此信息之前的byte的数量 这个信息用来在打开索引的时候做一个核对 如果用户非法自己修改过此文件 那么这个文件记录的checksum就和文件的大小产生了冲突

 

这些信息都存储在SegmentInfos bean中 单个segmeng的信息存储在bean SegmentInfo 中 用上篇博客中写好的索引读取类

  检测segments_n中存储的信息 这里仅检测前几个数据 其他的数据都由SegmentInfos 自己构造 读取函数都是从IndexInput类中继承

 

编写CheckSegmentsInfo 类

/**************** * *Create Class:CheckSegmentsInfo.java *Author:a276202460 *Create at:2010-6-2 */package com.rich.lucene.io;import java.io.File;import java.util.Iterator;import java.util.Map;import org.apache.lucene.index.SegmentInfo;import org.apache.lucene.index.SegmentInfos;import org.apache.lucene.store.Directory;import org.apache.lucene.store.FSDirectory;public class CheckSegmentsInfo {/** * @param args * @throws Exception */public static void main(String[] args) throws Exception {String indexdir = "D:/lucenetest/indexs/txtindex/index4";IndexFileInput input = null;try {input = new IndexFileInput(indexdir + "/segments_2");System.out.println("seg format:" + input.readInt());System.currentTimeMillis();System.out.println("seg Version:" + input.readLong());} finally {if (input != null) {input.close();}}Directory directory = FSDirectory.open(new File(indexdir));SegmentInfos infos = new SegmentInfos();infos.read(directory, "segments_2");System.out.println("info Version:" + infos.getVersion());System.out.println("info Counter:" + infos.counter);System.out.println("info Seg Count:" + infos.size());for (int i = 0; i < infos.size(); i++) {SegmentInfo info = infos.get(i);System.out.println("****************** segment [" + i + "]");System.out.println("segment name:" + info.name);System.out.println("the doc count in segment:" + info.docCount);System.out.println("del doc count in segment:" + info.getDelCount());System.out.println("segment doc store offset:"+ info.getDocStoreOffset());if (info.getDocStoreOffset() != -1) {System.out.println("segment's DocStoreSegment:"+ info.getDocStoreSegment());System.out.println("segment's DocStoreIsCompoundFile:"+ info.getDocStoreIsCompoundFile());}System.out.println("segment IsCompoundFile :"+ info.getDocStoreIsCompoundFile());System.out.println("segment's delcount:" + info.getDelCount());System.out.println("segment's is hasprox:" + info.getHasProx());Map infodiag = info.getDiagnostics();Iterator keyit = infodiag.keySet().iterator();while (keyit.hasNext()) {String key = keyit.next().toString();System.out.println("Diagnostic key:" + key+ " Diagnostic value:" + infodiag.get(key));}Map userdatas = infos.getUserData();Iterator datait = userdatas.keySet().iterator();while(datait.hasNext()){String key = datait.next().toString();System.out.println("user data key:"+key+" value:"+userdatas.get(key));}}}}

 

 

运行结果如下：

seg format:-9
seg Version:1275404730705
info Version:1275404730705
info Counter:1
info Seg Count:1
****************** segment [0]
segment name:_0
the doc count in segment:2
del doc count in segment:0
segment doc store offset:0
segment's DocStoreSegment:_0
segment's DocStoreIsCompoundFile:false
segment  IsCompoundFile :false
segment's delcount:0
segment's is hasprox:true
Diagnostic key:os.version Diagnostic value:5.1
Diagnostic key:os Diagnostic value:Windows XP
Diagnostic key:lucene.version Diagnostic value:3.0.0 883080 - 2009-11-22 15:43:58
Diagnostic key:source Diagnostic value:flush
Diagnostic key:os.arch Diagnostic value:x86
Diagnostic key:java.version Diagnostic value:1.6.0
Diagnostic key:java.vendor Diagnostic value:Sun Microsystems Inc.

 

根据提到的个个byte的值表示的信息可以检测文件中都存的什么值，下面着重分信息lucene对string类型的存储和读取

luncene 写字符串的代码如下：

/** Writes a string. * @see IndexInput#readString() */ public void writeString(String s) throws IOException { UnicodeUtil.UTF16toUTF8(s, 0, s.length(), utf8Result); writeVInt(utf8Result.length); writeBytes(utf8Result.result, 0, utf8Result.length); }

 

/** Encode characters from this String, starting at offset * for length characters. Returns the number of bytes * written to bytesOut. */ public static void UTF16toUTF8(final String s, final int offset, final int length, UTF8Result result) { final int end = offset + length; byte[] out = result.result; int upto = 0; for(int i=offset;i<end;i++) { final int code = (int) s.charAt(i); if (upto+4 > out.length) { byte[] newOut = new byte[2*out.length]; assert newOut.length >= upto+4; System.arraycopy(out, 0, newOut, 0, upto); result.result = out = newOut; } if (code < 0x80) out[upto++] = (byte) code; else if (code < 0x800) { out[upto++] = (byte) (0xC0 | (code >> 6)); out[upto++] = (byte)(0x80 | (code & 0x3F)); } else if (code < 0xD800 || code > 0xDFFF) { out[upto++] = (byte)(0xE0 | (code >> 12)); out[upto++] = (byte)(0x80 | ((code >> 6) & 0x3F)); out[upto++] = (byte)(0x80 | (code & 0x3F)); } else { // surrogate pair // confirm valid high surrogate if (code < 0xDC00 && (i < end-1)) { int utf32 = (int) s.charAt(i+1); // confirm valid low surrogate and write pair if (utf32 >= 0xDC00 && utf32 <= 0xDFFF) { utf32 = ((code - 0xD7C0) << 10) + (utf32 & 0x3FF); i++; out[upto++] = (byte)(0xF0 | (utf32 >> 18)); out[upto++] = (byte)(0x80 | ((utf32 >> 12) & 0x3F)); out[upto++] = (byte)(0x80 | ((utf32 >> 6) & 0x3F)); out[upto++] = (byte)(0x80 | (utf32 & 0x3F)); continue; } } // replace unpaired surrogate or out-of-order low surrogate // with substitution character out[upto++] = (byte) 0xEF; out[upto++] = (byte) 0xBF; out[upto++] = (byte) 0xBD; } } //assert matches(s, offset, length, out, upto); result.length = upto; }

 

 

 

lucene写入字符串的时候使用的是UTF-8 编码格式，java本身的字符串是用unicode表示的 也就是UTF-16 1-2个字节表示 这里说下UTF-16是unicode的一种实现。

UTF-8编码的优点这里就不说了 和UTF-16不一样 UTF-8是用1-3个字节来存储的

 

转换为UTF-8后将所得的byte数组的长度作为前缀存为一个VInt类型整数后面跟字符串，例如：字符串：’“我” 在lucene中的存储为：

3 -26 -120 -111

3表示字符串"我"的utf8 的byte的长度  后面接的三位就是字符串的字节码

 

这样也是为了方便读取。具体代码可以看IndexInput类 。很简单

至于map<String,String>也就是一个循环

mapsize，<String,String>^N

 

lucene读取segment_n文件的代码：

/** * Read a particular segmentFileName. Note that this may * throw an IOException if a commit is in process. * * @param directory -- directory containing the segments file * @param segmentFileName -- segment file to load * @throws CorruptIndexException if the index is corrupt * @throws IOException if there is a low-level IO error */ public final void read(Directory directory, String segmentFileName) throws CorruptIndexException, IOException { boolean success = false; // Clear any previous segments: clear(); ChecksumIndexInput input = new ChecksumIndexInput(directory.openInput(segmentFileName)); generation = generationFromSegmentsFileName(segmentFileName); lastGeneration = generation; try { int format = input.readInt(); if(format < 0){ // file contains explicit format info // check that it is a format we can understand if (format < CURRENT_FORMAT) throw new CorruptIndexException("Unknown format version: " + format); version = input.readLong(); // read version counter = input.readInt(); // read counter } else{ // file is in old format without explicit format info counter = format; } for (int i = input.readInt(); i > 0; i--) { // read segmentInfos add(new SegmentInfo(directory, format, input)); } if(format >= 0){ // in old format the version number may be at the end of the file if (input.getFilePointer() >= input.length()) version = System.currentTimeMillis(); // old file format without version number else version = input.readLong(); // read version } if (format <= FORMAT_USER_DATA) { if (format <= FORMAT_DIAGNOSTICS) { userData = input.readStringStringMap(); } else if (0 != input.readByte()) { userData = Collections.singletonMap("userData", input.readString()); } else { userData = Collections.<String,String>emptyMap(); } } else { userData = Collections.<String,String>emptyMap(); } if (format <= FORMAT_CHECKSUM) { final long checksumNow = input.getChecksum(); final long checksumThen = input.readLong(); if (checksumNow != checksumThen) throw new CorruptIndexException("checksum mismatch in segments file"); } success = true; } finally { input.close(); if (!success) { // Clear any segment infos we had loaded so we // have a clean slate on retry: clear(); } } }

 

/** * Construct a new SegmentInfo instance by reading a * previously saved SegmentInfo from input. * * @param dir directory to load from * @param format format of the segments info file * @param input input handle to read segment info from */ SegmentInfo(Directory dir, int format, IndexInput input) throws IOException { this.dir = dir; name = input.readString(); docCount = input.readInt(); if (format <= SegmentInfos.FORMAT_LOCKLESS) { delGen = input.readLong(); if (format <= SegmentInfos.FORMAT_SHARED_DOC_STORE) { docStoreOffset = input.readInt(); if (docStoreOffset != -1) { docStoreSegment = input.readString(); docStoreIsCompoundFile = (1 == input.readByte()); } else { docStoreSegment = name; docStoreIsCompoundFile = false; } } else { docStoreOffset = -1; docStoreSegment = name; docStoreIsCompoundFile = false; } if (format <= SegmentInfos.FORMAT_SINGLE_NORM_FILE) { hasSingleNormFile = (1 == input.readByte()); } else { hasSingleNormFile = false; } int numNormGen = input.readInt(); if (numNormGen == NO) { normGen = null; } else { normGen = new long[numNormGen]; for(int j=0;j<numNormGen;j++) { normGen[j] = input.readLong(); } } isCompoundFile = input.readByte(); preLockless = (isCompoundFile == CHECK_DIR); if (format <= SegmentInfos.FORMAT_DEL_COUNT) { delCount = input.readInt(); assert delCount <= docCount; } else delCount = -1; if (format <= SegmentInfos.FORMAT_HAS_PROX) hasProx = input.readByte() == 1; else hasProx = true; if (format <= SegmentInfos.FORMAT_DIAGNOSTICS) { diagnostics = input.readStringStringMap(); } else { diagnostics = Collections.<String,String>emptyMap(); } } else { delGen = CHECK_DIR; normGen = null; isCompoundFile = CHECK_DIR; preLockless = true; hasSingleNormFile = false; docStoreOffset = -1; docStoreIsCompoundFile = false; docStoreSegment = null; delCount = -1; hasProx = true; diagnostics = Collections.<String,String>emptyMap(); } }