使用scikit-learn进行文本分类

1. 数据来源

所用的数据是分类好的数据，详细描述见SMS Spam Collection v. 1，可以从github下载，数据在第4章。每一行数据包括包括两列，使用逗号隔开， 第1列是分类（lable），第2列是文本。

sms = pd.read_csv(filename, sep=',', header=0, names=['label','text'])sms.headOut[5]: <bound method DataFrame.head of      label                                               text0      ham  Go until jurong point, crazy.. Available only ...1      ham                      Ok lar... Joking wif u oni...2     spam  Free entry in 2 a wkly comp to win FA Cup fina...3      ham  U dun say so early hor... U c already then say...4      ham  Nah I don't think he goes to usf, he lives aro...5     spam  FreeMsg Hey there darling it's been 3 week's n...6      ham  Even my brother is not like to speak with me. ...7      ham  As per your request 'Melle Melle (Oru Minnamin...8     spam  WINNER!! As a valued network customer you have...9     spam  Had your mobile 11 months or more? U R entitle...10     ham  I'm gonna be home soon and i don't want to tal...

2. 数据准备

总共有5574行数据，随机从中抽取500行作为测试数据集，其它的作为训练数据集，为此定义了一个函数。运行后发现这个函数有一点小问题，它取不到500个数据，会少几个，分析原因，应该是产生的随机数有重复导致的。n为抽取的数据行数，size是整个数据集的行数。

def randomSequence(n, size):    result = [0 for i in range(size)]    for i in range(n):        x = random.randrange(0, size-1, 1)        result[x] = 1    return result

3. 特征提取

进行文本分类，在调用算法之前需要将文本内容转换成特征。 scikit-learn提供的CountVectorizer, TfidfTransformer两个类可以完成特征的提取。测试数据集共用了训练数据集产生的词汇表。

4.完整的代码

# -*- coding: utf-8 -*-import randomimport pandas as pdfrom sklearn.feature_extraction.text import CountVectorizer, TfidfTransformerfrom sklearn.naive_bayes import MultinomialNB #生成选择训练数据和测试数据的随机序列def randomSequence(n, size):    result = [0 for i in range(size)]    for i in range(n):        x = random.randrange(0, size-1, 1)        result[x] = 1    return result    if __name__ == '__main__':    #读数据    filename = 'data/sms_spam.csv'      sms = pd.read_csv(filename, sep=',', header=0, names=['label','text'])        #拆分训练数据集和测试数据集    size = len(sms)    sequence = randomSequence(500, size)    sms_train_mask = [sequence[i]==0 for i in range(size)]    sms_train = sms[sms_train_mask]      sms_test_mask = [sequence[i]==1 for i in range(size)]       sms_test = sms[sms_test_mask]        #文本转换成TF-IDF向量       train_labels = sms_train['label'].values    train_features = sms_train['text'].values    count_v1= CountVectorizer(stop_words = 'english', max_df = 0.5, decode_error = 'ignore')     counts_train = count_v1.fit_transform(train_features)    #print(count_v1.get_feature_names())    #repr(counts_train.shape)     tfidftransformer = TfidfTransformer()    tfidf_train = tfidftransformer.fit(counts_train).transform(counts_train)        test_labels = sms_test['label'].values    test_features = sms_test['text'].values    count_v2 = CountVectorizer(vocabulary=count_v1.vocabulary_,stop_words = 'english', max_df = 0.5, decode_error = 'ignore')    counts_test = count_v2.fit_transform(test_features)    tfidf_test = tfidftransformer.fit(counts_test).transform(counts_test)        #训练    clf = MultinomialNB(alpha = 0.01)    clf.fit(tfidf_train, train_labels)        #预测    predict_result = clf.predict(tfidf_test)    #print(predict_result)        #正确率    correct = [test_labels[i]==predict_result[i] for i in range(len(predict_result))]    r = len(predict_result)    t = correct.count(True)    f = correct.count(False)    print(r, t, f, t/float(r) )

以上用的是贝叶斯分类算法，也可以换其他算法。

运行结果

runfile('E:/MyProject/_python/ScikitLearn/NaiveBayes.py', wdir='E:/MyProject/_python/ScikitLearn')(476, 468, 8, 0.9831932773109243)