文本建模常用的预处理方法

最近看文本建模，给一大段文本，如何建模？？？

以MeTa代码为例：

[[analyzers]] method = "ngram-word" ngram = 1 [[analyzers.filter]] type = "whitespace-tokenizer" [[analyzers.filter]] type = "lowercase" [[analyzers.filter]] type = "alpha" [[analyzers.filter]] type = "length" min = 2 max = 35 [[analyzers.filter]] type = "list" file = "lemur-stopwords.txt" [[analyzers.filter]] type = "porter2-stemmer"

This tells MeTA how to process the text before indexing the documents. “ngram=1” configures MeTA to use unigrams (single words). Each “[[analyzers.filter]]” tag defines a text filter that applies a special function on the text. These filters are being “chained” together; text will first be processed by a whitespace tokenizer which separates words based on white spaces, then all the tokenized words will be converted to lowercase. This is followed by a couple of filters that end up with stopword removal and stemming. These filters can be usually changed depending on the application. For more information on how to use and configure the filters in MeTA see MeTA's Analyzers and Filters documentation.

1、分词：使用空格（type = "whitespace-tokenizer"）作为分割符号，方法是1-gram，即一个空格分一个词，如果是2-gram，则是两个空格分一个词；汉语有专门的分词工具。

2、字母转换为小写：type = "lowercase" ，便于将Me和me看做同一个词。

3、可以根据不同的应用，选择其他过滤器，如type = "length" 、type = "alpha"等。

4、去停顿词和功能词：file = "lemur-stopwords.txt"，简单说是没有任何含义的词汇（a、about、above、上午、下午、中午），中英文都有相应的词集。

5、词干分析：type = "porter2-stemmer"，看下面就懂了。

abandon                       abandon
abandoned                     abandon
abandoning                    abandon
abandonment                   abandon
abandons                      abandon

（2-5对应于前段时间做的微博分析，则是转换所有图片为picture、转换所有超链接为http等处理，一定要做适合自己的过滤）

6、筛选有用词汇：通过信息增益、CHI-test或其他TF-IDF等方法

文本建模系列会不断更新。。。。