机器学习实战-朴素贝叶斯

前两章我们要求分类器作出艰难的抉择，不过分类器有时候会产生错误，这时会产生错误结果，这是可以要求分类器给出一个最优的类别猜测结果，同事给出这个猜测的概率估计值。
本章会给出一些使用概率论进行分类的方法。首先从一个最简单的概率分类器开始，然后给出一些假设来学习朴素贝叶斯分类器。我们称之为“朴素”，是因为整个形式化过程只做最原始、最简单的假设。

朴素贝叶斯
优点：对于输入数据的准备方式较为敏感
缺点：对于输入数据的准备方式较为敏感
适用数据类型：标称型数据

在文档分类中，整个文档就是实例，而某些元素则构成特征。

# -*- coding:utf-8 -*-#4-1 词表到向量的转化函数def loadDataSet():    postingList = [['my', 'dog', 'has', 'flea', 'problems', 'help', 'please'],                 ['maybe', 'not', 'take', 'him', 'to', 'dog', 'park', 'stupid'],                 ['my', 'dalmation', 'is', 'so', 'cute', 'I', 'love', 'him'],                 ['stop', 'posting', 'stupid', 'worthless', 'garbage'],                 ['mr', 'licks', 'ate', 'my', 'steak', 'how', 'to', 'stop', 'him'],                 ['quit', 'buying', 'worthless', 'dog', 'food', 'stupid']]    classVec = [0,1,0,1,0,1]    return postingList,classVec#词条切分后的集合，类别标签def createVocabList(dataSet):    vocabSet = set([])    for document in dataSet:        vocabSet = vocabSet | set(document) #合集    return list(vocabSet)def setOfwords2Vec(vocabList, inputSet):    returnVec = [0]*len(vocabList) #创建一个其中所有元素都为0的向量    for word in inputSet:        if word in vocabList:            returnVec[vocabList.index(word)] = 1        else: print 'the word: %s is not in my Vocabulary' % word    return returnVec

#bayes-1.pyimport bayeslistOPosts,listClasses = bayes.loadDataSet()myVocabList = bayes.createVocabList(listOPosts)

>>> myVocabList['cute', 'love', 'help', 'garbage', 'quit', 'I', 'problems', 'is', 'park', 'stop', 'flea', 'dalmation', 'licks', 'food', 'not', 'him', 'buying', 'posting', 'has', 'worthless', 'ate', 'to', 'maybe', 'please', 'dog', 'how', 'stupid', 'so', 'take', 'mr', 'steak', 'my']>>> bayes.setOfwords2Vec(myVocabList, listOPosts[0])[0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1]>>> bayes.setOfwords2Vec(myVocabList, listOPosts[3])[0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0]

从词向量计算概率

#4-2 朴素贝叶斯分类器训练函数import numpy as np def trainNB0(trainMatrix, trainCategory):    m = len(trainMatrix)  #numTrainDocs    n = len(trainMatrix[0]) #numWords    p0Num = np.zeros(n); p1Num = np.zeros(n)    p0Denom = 0.0; p1Denom = 0.0#初始化概率    for i in range(m):        if trainCategory[i] == 1:            p1Num += trainMatrix[i]            p1Denom += sum(trainMatrix[i])        else:            p0Num += trainMatrix[i]            p0Denom += sum(trainMatrix[i])#此次因为缩进错误debug了很久...    p1Vect = p1Num/p1Denom    p0Vect = p0Num/p0Denom    pAbusive = sum(trainCategory)/float(m)    return p0Vect,p1Vect,pAbusive

#bayes-1.pyimport bayesfrom numpy import *reload(bayes)listOPosts,listClasses = bayes.loadDataSet()myVocabList = bayes.createVocabList(listOPosts)trainMat=[]for postinDoc in listOPosts:    trainMat.append(bayes.setOfwords2Vec(myVocabList, postinDoc))p0V,p1V,pAb = bayes.trainNB0(trainMat,listClasses)

>>> pAb0.5>>> p0Varray([ 0.04166667,  0.04166667,  0.04166667,  0.        ,  0.        ,        0.04166667,  0.04166667,  0.04166667,  0.        ,  0.04166667,        0.04166667,  0.04166667,  0.04166667,  0.        ,  0.        ,        0.08333333,  0.        ,  0.        ,  0.04166667,  0.        ,        0.04166667,  0.04166667,  0.        ,  0.04166667,  0.04166667,        0.04166667,  0.        ,  0.04166667,  0.        ,  0.04166667,        0.04166667,  0.125     ])>>> p1Varray([ 0.        ,  0.        ,  0.        ,  0.05263158,  0.05263158,        0.        ,  0.        ,  0.        ,  0.05263158,  0.05263158,        0.        ,  0.        ,  0.        ,  0.05263158,  0.05263158,        0.05263158,  0.05263158,  0.05263158,  0.        ,  0.10526316,        0.        ,  0.05263158,  0.05263158,  0.        ,  0.10526316,        0.        ,  0.15789474,  0.        ,  0.05263158,  0.        ,        0.        ,  0.        ])>>> 

利用贝叶斯分类器对文档进行分类时，要计算多个概率的乘积以获得文档属于某个类别的概率。
如果其中一个概率值为0，那么最后乘积也是0。
为了降低这种影响，可以将所有词的出现数改为1，并将分母改为2
另外可以遇到下溢出。由于太多很小的树相乘，可能四舍五入得到0。可以改成log

p0Num = np.ones(n); p1Num = np.ones(n)p0Denom = 2.0; p1Denom = 2.0p1Vect = log(p1Num/p1Denom)p0Vect = log(p0Num/p0Denom)

开始使用numpy向量处理功能

def classifyNB(vec2Classify, p0Vec, p1Vec, pClass1):#要分类的向量    p1 = sum(vec2Classify * p1Vec) + log(pClass1)#向量相乘    p0 = sum(vec2Classify * p0Vec) + log(1.0 - pClass1)    if p1 > p0:        return 1    else:        return 0def testingNB():    listOPosts,listClasses = loadDataSet()    myVocabList = createVocabList(listOPosts)    trainMat=[]    for postinDoc in listOPosts:        trainMat.append(setOfWords2Vec(myVocabList,postinDoc))    p0V,p1V,pAb = trainNB0(array(trainMat),array(listClasses))    testEntry = ['love','my','dalmation']    thisDoc = array(setOfWords2Vec(myVocabList, testEntry))    print testEntry,'classified as:',classifyNB(thisDoc,p0V,p1V,pAb)    testEntry = ['stupid','garbage']    thisDoc = array(setOfWords2Vec(myVocabList, testEntry))    print testEntry,'classified as:', classifyNB(thisDoc,p0V,p1V,pAb)

#bayes-1.pyimport bayesfrom numpy import *reload(bayes)listOPosts,listClasses = bayes.loadDataSet()myVocabList = bayes.createVocabList(listOPosts)trainMat=[]for postinDoc in listOPosts:    trainMat.append(bayes.setOfWords2Vec(myVocabList, postinDoc))p0V,p1V,pAb = bayes.trainNB0(trainMat,listClasses)bayes.testingNB()

测试这两句话

['love', 'my', 'dalmation'] classified as: 0['stupid', 'garbage'] classified as: 1

如果一个词在文档中出现不止一次，这可能意味着包含该词是否出现在文档中所不能表达的某种信息，这种方法被称为词袋模型。

4-4朴素贝叶斯词袋模型def bagOfWords2VecMN(vocabList, inputSet):    returnVec = [0]*len(vocavList)    for word in inputSet:        if word in inputSet:            returnVec[vocabList.index(word)] += 1    return returnVec

下面开始了解贝叶斯的一个著名应用：电子邮件垃圾过滤。
首先准备数据，切分文本

>>> mySent='This book is the best book on Python or M.L. I have ever laid eyes upon.'>>> mySent.split()['This', 'book', 'is', 'the', 'best', 'book', 'on', 'Python', 'or', 'M.L.', 'I', 'have', 'ever', 'laid', 'eyes', 'upon.']

但是标点符号也被当成了词的一部分，可以用正则表达式来切分句子，其中分隔符是除单词、数字外的任意字符串。

>>> import re>>> regEx = re.compile('\\W*')>>> listOfTokens = regEx.split(mySent)>>> listOfTokens['This', 'book', 'is', 'the', 'best', 'book', 'on', 'Python', 'or', 'M', 'L', 'I', 'have', 'ever', 'laid', 'eyes', 'upon', '']

接着我们把里面的空字符串去掉

[tok for tok in listOfTokens if len(tok) > 0]['This', 'book', 'is', 'the', 'best', 'book', 'on', 'Python', 'or', 'M', 'L', 'I', 'have', 'ever', 'laid', 'eyes', 'upon']

然后把所有字幕改成小写

>>> [tok.lower() for tok in listOfTokens if len(tok) > 0]['this', 'book', 'is', 'the', 'best', 'book', 'on', 'python', 'or', 'm', 'l', 'i', 'have', 'ever', 'laid', 'eyes', 'upon']

现在来看数据集中一封完整的电子邮件的实际处理结果。

>>> emailText = open('E:/上学/机器学习实战/4.朴素贝叶斯/email/ham/6.txt').read()>>> listOfTokens=regEx.split(emailText)>>> listOfTokens['Hello', 'Since', 'you', 'are', 'an', 'owner', 'of', 'at', 'least', 'one', 'Google', 'Groups', 'group', 'that', 'uses', 'the', 'customized', 'welcome', 'message', 'pages', 'or', 'files', 'we', 'are', 'writing', 'to', 'inform', 'you', 'that', 'we', 'will', 'no', 'longer', 'be', 'supporting', 'these', 'features', 'starting', 'February', '2011', 'We', 'made', 'this', 'decision', 'so', 'that', 'we', 'can', 'focus', 'on', 'improving', 'the', 'core', 'functionalities', 'of', 'Google', 'Groups', 'mailing', 'lists', 'and', 'forum', 'discussions', 'Instead', 'of', 'these', 'features', 'we', 'encourage', 'you', 'to', 'use', 'products', 'that', 'are', 'designed', 'specifically', 'for', 'file', 'storage', 'and', 'page', 'creation', 'such', 'as', 'Google', 'Docs', 'and', 'Google', 'Sites', 'For', 'example', 'you', 'can', 'easily', 'create', 'your', 'pages', 'on', 'Google', 'Sites', 'and', 'share', 'the', 'site', 'http', 'www', 'google', 'com', 'support', 'sites', 'bin', 'answer', 'py', 'hl', 'en', 'answer', '174623', 'with', 'the', 'members', 'of', 'your', 'group', 'You', 'can', 'also', 'store', 'your', 'files', 'on', 'the', 'site', 'by', 'attaching', 'files', 'to', 'pages', 'http', 'www', 'google', 'com', 'support', 'sites', 'bin', 'answer', 'py', 'hl', 'en', 'answer', '90563', 'on', 'the', 'site', 'If', 'you', 're', 'just', 'looking', 'for', 'a', 'place', 'to', 'upload', 'your', 'files', 'so', 'that', 'your', 'group', 'members', 'can', 'download', 'them', 'we', 'suggest', 'you', 'try', 'Google', 'Docs', 'You', 'can', 'upload', 'files', 'http', 'docs', 'google', 'com', 'support', 'bin', 'answer', 'py', 'hl', 'en', 'answer', '50092', 'and', 'share', 'access', 'with', 'either', 'a', 'group', 'http', 'docs', 'google', 'com', 'support', 'bin', 'answer', 'py', 'hl', 'en', 'answer', '66343', 'or', 'an', 'individual', 'http', 'docs', 'google', 'com', 'support', 'bin', 'answer', 'py', 'hl', 'en', 'answer', '86152', 'assigning', 'either', 'edit', 'or', 'download', 'only', 'access', 'to', 'the', 'files', 'you', 'have', 'received', 'this', 'mandatory', 'email', 'service', 'announcement', 'to', 'update', 'you', 'about', 'important', 'changes', 'to', 'Google', 'Groups', '']

文本解析是个相当复杂的过程，接下来将构建一个极其简单的函数。

def textParse(bigString):    import re    listOfTokens = re.split(r'\W*',bigString)    return [tok.lower() for tok in listOfTokens if len(tok) > 2]def spamTest():    docList=[]; classList = []; fullText = []    for i in range(1,26):        wordList = textParse(open('email/spam/%d.txt'%i).read())        docList.append(wordList)        fullText.extend(wordList)        classList.append(1)        wordList = textParse(open('email/ham/%d.txt'%i).read())        docList.append(wordList)        fullText.extend(wordList)        classList.append(0)    vocabList = createVocabList(docList)    trainingSet = range(50); testSet=[]    for i in range(10):#随机选择10个作为训练集        randIndex = int(random.uniform(0,len(trainingSet)))        testSet.append(trainingSet[randIndex])        del(trainingSet[randIndex])    trainMat=[]; trainClasses = []    for docIndex in trainingSet:        trainMat.append(bagOfWords2VecMN(vocabList,docList[docIndex]))        trainClasses.append(classList[docIndex])    p0V,p1V,pSpam = trainNB0(array(trainMat),array(trainClasses))    errorCount = 0    for docIndex in testSet:        wordVector = bagOfWords2VecMN(vocabList,docList[docIndex])        if classifyNB(array(wordVector),p0V,p1V,pSpam) !=classList[docIndex]:            errorCount += 1            print "classification error",docList[docIndex]    print 'the error rate is: ',float(errorCount)/len(testSet)

因为是随机选择10封电子邮件，所以每次都有差别。如果发现错误，函数会输出错分文档的词表，这样就可以分析是那篇文档发生了错误。

bayes.spamTest()classification error ['yeah', 'ready', 'may', 'not', 'here', 'because', 'jar', 'jar', 'has', 'plane', 'tickets', 'germany', 'for']the error rate is:  0.1bayes.spamTest()the error rate is:  0.0bayes.spamTest()classification error ['benoit', 'mandelbrot', '1924', '2010', 'benoit', 'mandelbrot', '1924', '2010', 'wilmott', 'team', 'benoit', 'mandelbrot', 'the', 'mathematician', 'the', 'father', 'fractal', 'mathematics', 'and', 'advocate', 'more', 'sophisticated', 'modelling', 'quantitative', 'finance', 'died', '14th', 'october', '2010', 'aged', 'wilmott', 'magazine', 'has', 'often', 'featured', 'mandelbrot', 'his', 'ideas', 'and', 'the', 'work', 'others', 'inspired', 'his', 'fundamental', 'insights', 'you', 'must', 'logged', 'view', 'these', 'articles', 'from', 'past', 'issues', 'wilmott', 'magazine']the error rate is:  0.1

下面使用朴素贝叶斯分类器从个人广告中获取区域倾向
本人现在使用的是Anaconda，先安装feedparser包，然后构建一个类似于spamTest()的函数

def calcMostFreq(vocabList,fullText):#计算单词出现的频率    import operator    freqDict = {}    for token in vocabList:        freqDict[token]=fullText.count(token)    sortedFreq = sorted(freqDict.iteritems(),key=operator.itemgetter(1),reverse=True)#排序    return sortedFreq[:30]#选出最多的30个def localWords(feed1,feed0):    import feedparser    docList=[];classList = []; fullText = []    minLen = min(len(feed1['entries']),len(feed0['entries']))    for i in range(minLen):        wordList = textParse(feed1['entries'][i]['summary'])        docList.append(wordList)        fullText.extend(wordList)        classList.append(1)        wordList = textParse(feed0['entries'][i]['summary'])        docList.append(wordList)        fullText.extend(wordList)        classList.append(0)    vocabList = createVocabList(docList)    top30Words = calcMostFreq(vocabList,fullText)#去掉出现次数最高的那些词    for pairW in top30Words:        if pairW[0] in vocabList: vocabList.remove(pairW[0])    trainingSet = range(2*minLen); testSet=[]    for i in range(20):        randIndex = int(random.uniform(0,len(trainingSet)))        testSet.append(trainingSet[randIndex])        del(trainingSet[randIndex])    trainMat=[]; trainClasses = []    for docIndex in trainingSet:        trainMat.append(bagOfWords2VecMN(vocabList,docList[docIndex]))        trainClasses.append(classList[docIndex])    p0V,p1V,pSpam = trainNB0(array(trainMat),array(trainClasses))    errorCount = 0    for docIndex in testSet:        wordVector = bagOfWords2VecMN(vocabList, docList[docIndex])        if classifyNB(array(wordVector),p0V,p1V,pSpam) != classList[docIndex]:            errorCount += 1    print 'the error rate is:',float(errorCount)/len(testSet)    return vocabList,p0V,p1V

在iPython中调试

import bayesny=feedparser.parse('http://newyork.craigslist.org/stp/index.rss')sf=feedparser.parse('http://sfbay.craigslist.org/stp/index.rss')vocabList,pSF,pNY=bayes.localWords(ny,sf)the error rate is: 0.5import bayesny=feedparser.parse('http://newyork.craigslist.org/stp/index.rss')sf=feedparser.parse('http://sfbay.craigslist.org/stp/index.rss')vocabList,pSF,pNY=bayes.localWords(ny,sf)the error rate is: 0.55

为了得到错误率的精确估计，应该多次进行上述实验，最后取平均值
下面开始显示地域相关的用词

def getTopWords(ny,sf):    import operator    vocabList,p0V,p1V=localWords(ny,sf)    topNY=[];topSF=[]    for i in range(len(p0V)):        if p0V[i] > -4.5 : topSF.append((vocabList[i],p0V[i]))        if p1V[i] > -4.5 : topNY.append((vocabList[i],p1V[i]))    sortedSF = sorted(topSF,key=lambda pair:pair[1],reverse=True)    print 'SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**'    for item in sortedSF:        print item[0]    sortedNY = sorted(topNY,key=lambda pair:pair[1], reverse=True)    print "NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**"    for item in sortedNY:        print item[0]

和之前返回排名最高的X个单词不同，这里可以返回大于某个阈值的所有词，这些元组会按照它们的条件概率进行排序。

import bayesny=feedparser.parse('http://newyork.craigslist.org/stp/index.rss')sf=feedparser.parse('http://sfbay.craigslist.org/stp/index.rss')bayes.getTopWords(ny,sf)the error rate is: 0.4SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**SF**meetall61514greatopenanymovieaboutareaNY**NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**NY**showcontactinfoveryguymassagetalknycfirstneedonlyoff