开源一个文本分析项目

Github

https://github.com/sea-boat/TextAnalyzer

TextAnalyzer

a text analizer that can analyze text. so far, it can extract hot words in a text segment by using tf-idf algorithm,at the same time using a score factor to optimize the final score.

also it provides machine learning to make a classification.

Features

extracting hot words from a text.
1. to gather statistics via frequence.
2. to gather statistics via by tf-idf algorithm
3. to gather statistics via a score factor additionally.

synonym can be recognized

SVM Classificator

this analyzer supports to classify text by svm. it involves vectoring the text. we can train the samples and then make a classification by the model.

for convenience,the model,tfidf and vector will be stored.

kmeans clustering && xmeans clustering

this analyzer supports to clustering text by kmeans and xmeans.

vsm clustering

this analyzer supports to clustering text by vsm.

Dependence

https://github.com/sea-boat/IKAnalyzer-Mirror.git

TODO

• other ml algorithms.
• emotion analization.

How to use

just simple like this

extracting hot words

1. indexing a document and get a docId.

long docId = TextIndexer.index(text);

1. extracting by docId.

 HotWordExtractor extractor = new HotWordExtractor(); List<Result> list = extractor.extract(0, 20, false); if (list != null) for (Result s : list)    System.out.println(s.getTerm() + " : " + s.getFrequency() + " : " + s.getScore());

a result contains term,frequency and score.

失业证 : 1 : 0.31436604户口 : 1 : 0.30099702单位 : 1 : 0.29152703提取 : 1 : 0.27927202领取 : 1 : 0.27581802职工 : 1 : 0.27381304劳动 : 1 : 0.27370203关系 : 1 : 0.27080503本市 : 1 : 0.27080503终止 : 1 : 0.27080503

SVM classificator

1. training the samples.

SVMTrainer trainer = new SVMTrainer();trainer.train();

1. predicting text classification.

double[] data = trainer.getWordVector(text);trainer.predict(data);

kmeans clustering && xmeans clustering

List<String> list = DataReader.readContent(KMeansCluster.DATA_FILE);int[] labels = new KMeansCluster().learn(list);

vsm clustering

List<String> list = DataReader.readContent(VSMCluster.DATA_FILE);List<String> labels = new VSMCluster().learn(list);

==========广告时间==========

公众号的菜单已分为“分布式”、“机器学习”、“深度学习”、“NLP”、“Java深度”、“Java并发核心”、“JDK源码”、“Tomcat内核”等，可能有一款适合你的胃口。

鄙人的新书《Tomcat内核设计剖析》已经在京东销售了，有需要的朋友可以购买。感谢各位朋友。

为什么写《Tomcat内核设计剖析》

=========================

欢迎关注：