基于lucene的案例开发:实时索引管理类IndexManager
来源:互联网 发布:flink.apache.org 1.3 编辑:程序博客网 时间:2024/05/14 07:36
转载请注明出处:http://blog.csdn.net/xiaojimanman/article/details/44015983
http://www.llwjy.com/blogdetail/5757ce8c007754704b563dd6a47ca1ca.html
个人的博客小站也搭建成功,网址:www.llwjy.com ,欢迎大家来吐槽~
在前一篇博客中,对实时索引的实现原理做了一些简单的介绍,这里就介绍下,如何利用Lucene来实现索引的管理(Lucene中已经实现了大部分的功能,我们只需要对其再次封装即可)。
逐个击破
在Lucene4.3.1中,实现实时索引时,需要将IndexWrite的相关操作委托给TrackingIndexWriter来处理,具体代码实现如下:
ps:关于如何创建索引这里就不再介绍了,可以参照之前的博客或者该博客后面的完整代码。
this.trackingIndexWriter = new TrackingIndexWriter(this.indexWriter);
同时初始化索引管理对象,代码如下:
this.nrtManager = new NRTManager(this.trackingIndexWriter, new SearcherFactory());
到这里还需要开启两个守护线程:内存索引重读线程和内存数据commit线程。内存索引重读线程执行的频率也就是实时索引的时差,由于内存中的数据不会太多,所以这个延时一般也就是在十几毫秒左右;内存数据commit线程是将内存中的数据写到磁盘上,不至于数据丢失,如果研究过Lucene源码的童鞋也许会发现,即使你不执行commit操作,到内存中的数据达到一定的程度,也会将一部分数据写到磁盘上,只不过重启服务这部分数据就丢失了同时还会造成一系列的问题,http://bbs.csdn.net/topics/390677902 这个地址下就是commit线程死掉之后造成的一系列问题,感兴趣的童鞋可以了解下。
内存重读线程我们只需要配置下参数启动即可,代码如下:
this.nrtManagerReopenThread = new NRTManagerReopenThread(this.nrtManager, indexReopenMaxStaleSec, indexReopenMinStaleSec);this.nrtManagerReopenThread.setName("NRTManager Reopen Thread");this.nrtManagerReopenThread.setPriority(Math.min(Thread.currentThread().getPriority()+2, Thread.MAX_PRIORITY));this.nrtManagerReopenThread.setDaemon(true);this.nrtManagerReopenThread.start();
内存数据commit线程需要自己写代码实现,然后启动该线程即可,代码如下:
private class IndexCommitThread extends Thread{private boolean flag;public IndexCommitThread(String name){super(name);}@SuppressWarnings("deprecation")public void run(){flag = true;while(flag) {try {indexWriter.commit();if (bprint) {System.out.println(new Date().toLocaleString() + "\t" + IndexManagerName + "\tcommit");}TimeUnit.SECONDS.sleep(indexCommitSeconds);} catch (IOException e) {e.printStackTrace();} catch (InterruptedException e1) {e1.printStackTrace();}}}}
this.indexCommitThread = new IndexCommitThread(IndexManagerName + "Index Commit Thread");this.indexCommitThread.setDaemon(true);this.indexCommitThread.start();
那又如何像普通的索引那样使用IndexSearcher呢?当然NrtManager类也提供了相关的方法,可以获取最新可用的IndexSearcher,代码如下:
public IndexSearcher getIndexSearcher(){try {return this.nrtManager.acquire();} catch (IOException e) {e.printStackTrace();return null;}}
当然在使用之后别忘记释放,代码如下:
public void release(IndexSearcher searcher){try {nrtManager.release(searcher);} catch (IOException e) {// TODO Auto-generated catch block e.printStackTrace();}}
另类单例模式
在之前的博客中,我也多次提到,加载索引是一个相当消耗资源的事情,所以我们不可能每一次索引操作都加载一次索引,所以我们就必须使用单例模式来实现IndexManager类。这里的单例模式又和我们常见的单例模式有所区别,普通的单例模式该类只有一个对象,这里的单例模式是该类有多个对象,下面就简单的介绍下此处另类的单例模式。
通过前一篇博客最后,你也许会注意到,系统中关于索引的配置信息是存在HashSet对象中,这也就是说这里IndexManager类会实例化多少次取决于HashSet对象,也就是你配置文件让他实例化多少次就会实例化多少次。既然这样,怎么还能叫单例模式呢?这里的单例是索引的单例,也就是说一个索引只有一个IndexManager对象,不会存在两个IndexManager对象去操作同一个索引的情况。具体代码实现如下:
/** * Initialization on Demand Holder式初始化IndexManager */private static class LazyLoadIndexManager {private static final HashMap<String, IndexManager> indexManager = new HashMap<String, IndexManager>();static {for (ConfigBean configBean : IndexConfig.getConfigBean()) {indexManager.put(configBean.getIndexName(), new IndexManager(configBean));}}}/** *@Description: IndexManager私有构造方法 *@Author: lulei *@Version: 1.1.0 */private IndexManager(ConfigBean configBean){//...}public static IndexManager getIndexManager(String indexName){return LazyLoadIndexManager.indexManager.get(indexName);}
这样我们就可以通过索引名获取到该索引的IndexManager对象。
庐山真面目
说了这么多,下面就把IndexManager的源码附在最后,感兴趣的童鞋可以试试(里面还有一些其他的方法,相信不用介绍也都可以看的懂)
/** *@Description: 索引管理类 */ package com.lulei.lucene.index.manager; import java.io.File;import java.io.IOException;import java.util.Date;import java.util.HashMap;import java.util.concurrent.TimeUnit;import org.apache.lucene.analysis.Analyzer;import org.apache.lucene.index.IndexWriter;import org.apache.lucene.index.IndexWriterConfig;import org.apache.lucene.index.IndexWriterConfig.OpenMode;import org.apache.lucene.search.IndexSearcher;import org.apache.lucene.search.NRTManager;import org.apache.lucene.search.NRTManager.TrackingIndexWriter;import org.apache.lucene.search.NRTManagerReopenThread;import org.apache.lucene.search.SearcherFactory;import org.apache.lucene.store.Directory;import org.apache.lucene.store.NIOFSDirectory;import org.apache.lucene.util.Version;import com.lulei.lucene.index.model.ConfigBean;import com.lulei.lucene.index.model.IndexConfig;public class IndexManager {private IndexWriter indexWriter;//更新索引文件的IndexWriterprivate TrackingIndexWriter trackingIndexWriter;//索引文件采用的分词器private Analyzer analyzer;//索引管理对象private NRTManager nrtManager;//索引重读线程private NRTManagerReopenThread nrtManagerReopenThread;//索引写入磁盘线程private IndexCommitThread indexCommitThread;//索引地址private String indexPath;//索引重读最大、最小时间间隔private double indexReopenMaxStaleSec;private double indexReopenMinStaleSec;//索引commit时间private int indexCommitSeconds;//索引名private String IndexManagerName;//commit时是否输出相关信息private boolean bprint = true;/** * Initialization on Demand Holder式初始化IndexManager */private static class LazyLoadIndexManager {private static final HashMap<String, IndexManager> indexManager = new HashMap<String, IndexManager>();static {for (ConfigBean configBean : IndexConfig.getConfigBean()) {indexManager.put(configBean.getIndexName(), new IndexManager(configBean));}}}/** *@Description: IndexManager私有构造方法 *@Author: lulei *@Version: 1.1.0 */private IndexManager(ConfigBean configBean){//设置相关属性analyzer = configBean.getAnalyzer();indexPath = configBean.getIndexPath();IndexManagerName = configBean.getIndexName();indexReopenMaxStaleSec = configBean.getIndexReopenMaxStaleSec();indexReopenMinStaleSec = configBean.getIndexReopenMinStaleSec();indexCommitSeconds = configBean.getIndexCommitSeconds();bprint = configBean.isBprint();String indexFile = indexPath + IndexManagerName + "/";//创建或打开索引IndexWriterConfig indexWriterConfig = new IndexWriterConfig(Version.LUCENE_43, analyzer);indexWriterConfig.setOpenMode(OpenMode.CREATE_OR_APPEND);Directory directory = null;try {directory = NIOFSDirectory.open(new File(indexFile));if (IndexWriter.isLocked(directory)){IndexWriter.unlock(directory);}this.indexWriter = new IndexWriter(directory, indexWriterConfig);this.trackingIndexWriter = new TrackingIndexWriter(this.indexWriter);this.nrtManager = new NRTManager(this.trackingIndexWriter, new SearcherFactory());} catch(IOException e){e.printStackTrace();}//开启守护进程this.setThread();}/** * @Author: lulei * @Description: 创建索引管理线程 */private void setThread(){this.nrtManagerReopenThread = new NRTManagerReopenThread(this.nrtManager, indexReopenMaxStaleSec, indexReopenMinStaleSec);this.nrtManagerReopenThread.setName("NRTManager Reopen Thread");this.nrtManagerReopenThread.setPriority(Math.min(Thread.currentThread().getPriority()+2, Thread.MAX_PRIORITY));this.nrtManagerReopenThread.setDaemon(true);this.nrtManagerReopenThread.start();this.indexCommitThread = new IndexCommitThread(IndexManagerName + "Index Commit Thread");this.indexCommitThread.setDaemon(true);this.indexCommitThread.start();}/** * @return * @Author:lulei * @Description: 重启索引commit线程 */public String setCommitThread() {try {if (this.indexCommitThread.isAlive()){return "is alive";}this.indexCommitThread = new IndexCommitThread(IndexManagerName + "Index Commit Thread");this.indexCommitThread.setDaemon(true);this.indexCommitThread.start();} catch (Exception e) {e.printStackTrace();return "failed";}return "reload";}/** *@Description: 索引commit线程 *@Author: lulei *@Version: 1.1.0 */private class IndexCommitThread extends Thread{private boolean flag;public IndexCommitThread(String name){super(name);}@SuppressWarnings("deprecation")public void run(){flag = true;while(flag) {try {indexWriter.commit();if (bprint) {System.out.println(new Date().toLocaleString() + "\t" + IndexManagerName + "\tcommit");}TimeUnit.SECONDS.sleep(indexCommitSeconds);} catch (IOException e) {e.printStackTrace();} catch (InterruptedException e1) {e1.printStackTrace();}}}}/** * @return IndexManager * @Author: lulei * @Description: 获取索引管理类 */public static IndexManager getIndexManager(String indexName){return LazyLoadIndexManager.indexManager.get(indexName);}/** * @@Description:释放IndexSearcher资源 * @param searcher */public void release(IndexSearcher searcher){try {nrtManager.release(searcher);} catch (IOException e) {// TODO Auto-generated catch block e.printStackTrace();}}/** * @return IndexSearcher * @Author: lulei * @Description: 返回IndexSearcher对象,使用完之后,调用release方法进行释放 */public IndexSearcher getIndexSearcher(){try {return this.nrtManager.acquire();} catch (IOException e) {e.printStackTrace();return null;}}public NRTManager getNRTManager(){return this.nrtManager;}public IndexWriter getIndexWriter(){return this.indexWriter;}public TrackingIndexWriter getTrackingIndexWriter(){return this.trackingIndexWriter;}public Analyzer getAnalyzer(){return analyzer;}/** * @return * @Author: lulei * @Description: 获取索引中的记录条数 */public int getIndexNum(){return indexWriter.numDocs();}}
ps:最近发现其他网站可能会对博客转载,上面并没有源链接,如想查看更多关于 基于lucene的案例开发 请点击这里。或访问网址http://blog.csdn.net/xiaojimanman/article/category/2841877 或 www.llwjy.com
- 基于lucene的案例开发:实时索引管理类IndexManager
- 基于lucene的案例开发:实时索引的检索
- 基于lucene的案例开发:实时索引的修改
- 基于lucene的案例开发:实现实时索引基本原理
- 基于lucene的案例开发:索引数学模型
- 基于lucene的案例开发:创建索引
- 基于lucene的案例开发:搜索索引
- 基于lucene的案例开发:索引文件结构
- 基于lucene的案例开发:案例初识
- 基于lucene的案例开发:Query查询
- 基于lucene的案例开发:更新说明
- 基于lucene的案例开发:JsonUtil & XmlUtil
- 基于lucene的案例开发:ClassUtil & CharsetUtil
- 基于lucene的案例开发:ParseUtil & ParseRequest
- 基于lucene的案例开发:数据库连接池
- 基于lucene的案例开发:lucene初始认知
- 基于lucene的案例开发:写在开始之前
- 基于lucene的案例开发:分词器介绍
- <s:if> 用法示例即注意事项
- PowerDesigner连接Oracle10g数据库
- tableVIew删除时的delete按钮被挡住时重写的方法
- VS2010 UML 生成代码
- linux安装swftools
- 基于lucene的案例开发:实时索引管理类IndexManager
- zip4j--处理带密码的zip包
- Linux 指令详解 top 系统资源检测
- 常用类第三十一课,DateFormat和SimpleDateFormat,时间和字符串的互相转换
- IOS数据库操作SQLite3使用详解
- JMeter 入门(一) 网页请求录制
- 浏览器中在线浏览pdf
- BOM和DOM
- Git 管理公钥