感知机学习

来源：互联网发布：sqlserver 取绝对值编辑：程序博客网时间：2024/04/29 13:35

基本概念：感知机是二类分类的线性分类模型，对应于特征空间中将实例划分为正负两类的分离超平面，属判别模型。感知机学习旨在求出将训练数据进行线性划分的分离超平面。

感知机的定义：

从输入空间Rn到输出空间{+1,-1}的函数映射:f(x)= sign(w*x+b)

模型参数：w----权值向量 b----偏置

wx+b = 0 -----分离超平面方程

数据集{(xi,yi)}with 1<=i<=N称为线性可分的，如果存在分离超平面S:wx+b=0 可以将数据集的正例、负例点正确划分到其两侧

定义经验风险函数（损失函数）：所有误分类点到分离超平面距离之和（这里为了计算方便略去一个比例系数），因此有Loss(w,b)= -sigma{yi*(w*xi+b)}, (xi,yi)属于误分类集合。在训练集上最小化损失函数，求得感知机模型。

感知机学习算法原始形式（使用随机梯度下降方法）

1. 选初值w0,b0

2. 从训练集随机挑选数据(xi,yi)，如果训练集所有数据均以正确分类，那么结束算法，输出结果

3. 如果yi*(w*xi+b)<= 0:

更新 w,b

w = w + r*xi*yi

b = b + r*yi

4. 转2。

程序实现：import osimport sysimport random #This algorithm learns a perceptron model from trainingdataset#note: the trainset is linearly seperable, and different choicesof initial parameters or false-classified points#may lead to different perceptron model, which is reasonableunder this framework.#DATE:2013-7-4, by zhlif __name__ == "__main__":       trainset =[(3,3,1),(4,3,1),(1,1,-1)]       #initialize       w1 = w2 = b = 0       id = 0       # we set learning rate = 1             while True:            id += 1           flag = False           for choice in range(3):              iftrainset[choice][2] * (w1 * trainset[choice][0] + w2 * trainset[choice][1] + b)<= 0:                        flag = True                        break           if flag == False:                  break            #randomlyselect a point from trainset           choice = random.randint(0,2)                 #judge whether it's false-classified           if trainset[choice][2] * (w1 *trainset[choice][0] + w2 * trainset[choice][1] + b) <= 0:                 w1 = w1 + trainset[choice][2] *trainset[choice][0]                 w2 = w2 + trainset[choice][2] *trainset[choice][1]                 b = b + trainset[choice][2]                   #print out current values           print 'Round ',id,':','Flase ClassifiedPoint:',choice + 1,',w1:',w1,',w2:',w2,',b:',b,'\n'             print 'Theperceptron model learned is sign(%d x1 + %d x2 + %d)\n' % (w1,w2,b)

<实验>给定训练集，正例x1=(3,3)x2=(4,3) 负例x3=(1,1)，学习感知机模型

程序运行过程：

Round 1 : FlaseClassified Point: 1 ,w1: 3 ,w2: 3 ,b: 1

Round 2 : FlaseClassified Point: 1 ,w1: 3 ,w2: 3 ,b: 1

Round 3 : FlaseClassified Point: 2 ,w1: 3 ,w2: 3 ,b: 1

Round 4 : FlaseClassified Point: 2 ,w1: 3 ,w2: 3 ,b: 1

Round 5 : FlaseClassified Point: 1 ,w1: 3 ,w2: 3 ,b: 1

Round 6 : FlaseClassified Point: 3 ,w1: 2 ,w2: 2 ,b: 0

Round 7 : FlaseClassified Point: 1 ,w1: 2 ,w2: 2 ,b: 0

Round 8 : Flase ClassifiedPoint: 1 ,w1: 2 ,w2: 2 ,b: 0

Round 9 : FlaseClassified Point: 2 ,w1: 2 ,w2: 2 ,b: 0

Round 10 : FlaseClassified Point: 3 ,w1: 1 ,w2: 1 ,b: -1

Round 11 : FlaseClassified Point: 2 ,w1: 1 ,w2: 1 ,b: -1

Round 12 : FlaseClassified Point: 2 ,w1: 1 ,w2: 1 ,b: -1

Round 13 : FlaseClassified Point: 1 ,w1: 1 ,w2: 1 ,b: -1

Round 14 : FlaseClassified Point: 2 ,w1: 1 ,w2: 1 ,b: -1

Round 15 : FlaseClassified Point: 3 ,w1: 0 ,w2: 0 ,b: -2

Round 16 : FlaseClassified Point: 1 ,w1: 3 ,w2: 3 ,b: -1

Round 17 : FlaseClassified Point: 3 ,w1: 2 ,w2: 2 ,b: -2

Round 18 : FlaseClassified Point: 1 ,w1: 2 ,w2: 2 ,b: -2

Round 19 : FlaseClassified Point: 1 ,w1: 2 ,w2: 2 ,b: -2

Round 20 : FlaseClassified Point: 2 ,w1: 2 ,w2: 2 ,b: -2

Round 21 : FlaseClassified Point: 1 ,w1: 2 ,w2: 2 ,b: -2

Round 22 : FlaseClassified Point: 2 ,w1: 2 ,w2: 2 ,b: -2

Round 23 : FlaseClassified Point: 2 ,w1: 2 ,w2: 2 ,b: -2

Round 24 : FlaseClassified Point: 1 ,w1: 2 ,w2: 2 ,b: -2

Round 25 : FlaseClassified Point: 3 ,w1: 1 ,w2: 1 ,b: -3

The perceptron model learned is sign(1 x1 + 1 x2 + -3)

经过25轮更新，求得分离超平面x1+ x2 -3 = 0

如果选取不同的误分类点，求得的超平面可能不同：

Round 1 : FlaseClassified Point: 1 ,w1: 3 ,w2: 3 ,b: 1

Round 2 : FlaseClassified Point: 3 ,w1: 2 ,w2: 2 ,b: 0

Round 3 : FlaseClassified Point: 2 ,w1: 2 ,w2: 2 ,b: 0

Round 4 : FlaseClassified Point: 1 ,w1: 2 ,w2: 2 ,b: 0

Round 5 : FlaseClassified Point: 1 ,w1: 2 ,w2: 2 ,b: 0

Round 6 : FlaseClassified Point: 3 ,w1: 1 ,w2: 1 ,b: -1

Round 7 : FlaseClassified Point: 3 ,w1: 0 ,w2: 0 ,b: -2

Round 8 : FlaseClassified Point: 2 ,w1: 4 ,w2: 3 ,b: -1

Round 9 : FlaseClassified Point: 2 ,w1: 4 ,w2: 3 ,b: -1

Round 10 : FlaseClassified Point: 2 ,w1: 4 ,w2: 3 ,b: -1

Round 11 : FlaseClassified Point: 2 ,w1: 4 ,w2: 3 ,b: -1

Round 12 : FlaseClassified Point: 3 ,w1: 3 ,w2: 2 ,b: -2

Round 13 : FlaseClassified Point: 1 ,w1: 3 ,w2: 2 ,b: -2

Round 14 : FlaseClassified Point: 2 ,w1: 3 ,w2: 2 ,b: -2

Round 15 : FlaseClassified Point: 3 ,w1: 2 ,w2: 1 ,b: -3

Round 16 : FlaseClassified Point: 2 ,w1: 2 ,w2: 1 ,b: -3

Round 17 : FlaseClassified Point: 3 ,w1: 1 ,w2: 0 ,b: -4

Round 18 : FlaseClassified Point: 1 ,w1: 4 ,w2: 3 ,b: -3

Round 19 : FlaseClassified Point: 2 ,w1: 4 ,w2: 3 ,b: -3

Round 20 : FlaseClassified Point: 3 ,w1: 3 ,w2: 2 ,b: -4

Round 21 : FlaseClassified Point: 2 ,w1: 3 ,w2: 2 ,b: -4

Round 22 : FlaseClassified Point: 1 ,w1: 3 ,w2: 2 ,b: -4

Round 23 : FlaseClassified Point: 1 ,w1: 3 ,w2: 2 ,b: -4

Round 24 : FlaseClassified Point: 3 ,w1: 2 ,w2: 1 ,b: -5

The perceptron model learned is sign(2 x1 + 1 x2 + -5)

此次经过24轮求得超平面2x1+x2-5=0

再次实验，

Round 1 : FlaseClassified Point: 2 ,w1: 4 ,w2: 3 ,b: 1

Round 2 : FlaseClassified Point: 3 ,w1: 3 ,w2: 2 ,b: 0

Round 3 : FlaseClassified Point: 1 ,w1: 3 ,w2: 2 ,b: 0

Round 4 : FlaseClassified Point: 1 ,w1: 3 ,w2: 2 ,b: 0

Round 5 : FlaseClassified Point: 3 ,w1: 2 ,w2: 1 ,b: -1

Round 6 : FlaseClassified Point: 2 ,w1: 2 ,w2: 1 ,b: -1

Round 7 : FlaseClassified Point: 1 ,w1: 2 ,w2: 1 ,b: -1

Round 8 : FlaseClassified Point: 2 ,w1: 2 ,w2: 1 ,b: -1

Round 9 : FlaseClassified Point: 3 ,w1: 1 ,w2: 0 ,b: -2

The perceptron model learned is sign(1 x1 + 0 x2 + -2)

经过9轮求得超平面x1-2=0

感知机学习的对偶形式：

在原始形式的感知机算法中，对于一个误分类点(xi,yi)，每次进行如下参数更新：

w = w + r*yi*xi

b = b + r * yi

我们默认选择w,b的初值均为0. 观察上式，每次更新由(xi,yi)所带来的贡献是等式右边的乘积项，对于固定的一个误分类点(xi,yi),假设由于改点误分类而更新了ni次，那么此点对w参数的累计贡献是r*yi*xi*ni,可以简记为delta_w= ai * xi * yi;同理，对于参数b的累计贡献是r*yi*ni,简记为delta_b= ai * yi.这里，ai= r * ni,r是学习速率(learningrate,在实验中设置为常数1)，ni是实例(xi,yi)由于误分类而引起的更新次数。

从初始值0开始逐步迭代到学习完成，参数w和b的终值为：

w = sigma(i) delta_w

b = sigma(i) delta_b

其中，1<=i<=N，N是训练样本容量。

感知机学习算法的对偶形式：

1. 初始化ai,b = 0

2. 从训练集中随机选取一个实例点，如若不存在误分类点，算法终止输出结果

3. 如果这个实例点是误分类点，即满足yi * {sigma(aj*yj*xj)*xi+ b }<=0:

更新参数ai,b

ai = ai + r

b = b+ r * yi

4. 转2

代码实现：（采用样例中的数据集，学习率=1）

import osimport sysimport random #This algorithm learns a perceptron model from trainingdataset#note: the trainset is linearly seperable, and differentchoices of initial parameters or false-classified points#may lead to different perceptron model, which is reasonableunder this framework.#DATE:2013-7-4, by zhl mx = [[0,0,0],[0,0,0],[0,0,0]] def innerproduct(i,j):    res = 0    for k in range(2):        res += trainset[i][k]* trainset[j][k]    return res def judge(i,b):    tmp = 0    global mx    for j in range(3):        tmp += a[j] *trainset[j][2] * mx[j][i]    if trainset[i][2]* (tmp + b) <= 0:        return True#false-classified    else:        return False#correct-classified if __name__ == "__main__":    global mx    trainset =[(3,3,1),(4,3,1),(1,1,-1)]    #generate Grammatrix    for i in range(3):       for j inrange(3):           mx[i][j] =innerproduct(i,j)       print mx    #initializeparameters    a = [0,0,0]    b = 0    id = 0    # we set learningrate = 1       while True:        id += 1        flag = False        for choice inrange(3):          ifjudge(choice,b) == True:               flag =True               break        if flag ==False:            break            #randomlyselect a point from trainset        choice =random.randint(0,2)           #judge whetherit's false-classified        ifjudge(choice,b) == True:           a[choice] =a[choice] + 1           b = b +trainset[choice][2]             #print outcurrent values        print 'Round',id,':','Flase Classified Point:',choice +1,',a1:',a[0],',a2:',a[1],'a3:',a[2],',b:',b,'\n'       w = [0,0]    for i in range(2):       w[i] = a[0] *trainset[0][2] * trainset[0][0] + a[1] * trainset[1][2] * trainset[1][0] + a[2]* trainset[2][2] * trainset[2][0]       print 'The perceptron modellearned is sign(%d x1 + %d x2 + %d)\n' % (w[0],w[1],b)

可以证明，和原始形式一样，对偶形式的感知机学习算法也是收敛的（训练集线性可分），实验过程如下：

Round 1 : Flase Classified Point: 1 ,a1: 1 ,a2: 0a3: 0 ,b: 1

Round 2 : Flase Classified Point: 1 ,a1: 1 ,a2: 0a3: 0 ,b: 1

Round 3 : Flase Classified Point: 1 ,a1: 1 ,a2: 0a3: 0 ,b: 1

Round 4 : Flase Classified Point: 3 ,a1: 1 ,a2: 0 a3:1 ,b: 0

Round 5 : Flase Classified Point: 2 ,a1: 1 ,a2: 0a3: 1 ,b: 0

Round 6 : Flase Classified Point: 3 ,a1: 1 ,a2: 0a3: 2 ,b: -1

Round 7 : Flase Classified Point: 2 ,a1: 1 ,a2: 0a3: 2 ,b: -1

Round 8 : Flase Classified Point: 3 ,a1: 1 ,a2: 0a3: 3 ,b: -2

Round 9 : Flase Classified Point: 1 ,a1: 2 ,a2: 0a3: 3 ,b: -1

Round 10 : Flase Classified Point: 1 ,a1: 2 ,a2: 0a3: 3 ,b: -1

Round 11 : Flase Classified Point: 2 ,a1: 2 ,a2: 0a3: 3 ,b: -1

Round 12 : Flase Classified Point: 2 ,a1: 2 ,a2: 0a3: 3 ,b: -1

Round 13 : Flase Classified Point: 1 ,a1: 2 ,a2: 0a3: 3 ,b: -1

Round 14 : Flase Classified Point: 3 ,a1: 2 ,a2: 0a3: 4 ,b: -2

Round 15 : Flase Classified Point: 3 ,a1: 2 ,a2: 0a3: 5 ,b: -3

The perceptron model learned issign(1 x1 + 1 x2 + -3)

学习到的模型可能有多个：

Round 1 : Flase Classified Point: 1 ,a1: 1 ,a2: 0a3: 0 ,b: 1

Round 2 : Flase Classified Point: 2 ,a1: 1 ,a2: 0a3: 0 ,b: 1

Round 3 : Flase Classified Point: 3 ,a1: 1 ,a2: 0a3: 1 ,b: 0

Round 4 : Flase Classified Point: 1 ,a1: 1 ,a2: 0a3: 1 ,b: 0

Round 5 : Flase Classified Point: 1 ,a1: 1 ,a2: 0a3: 1 ,b: 0

Round 6 : Flase Classified Point: 2 ,a1: 1 ,a2: 0a3: 1 ,b: 0

Round 7 : Flase Classified Point: 2 ,a1: 1 ,a2: 0a3: 1 ,b: 0

Round 8 : Flase Classified Point: 3 ,a1: 1 ,a2: 0a3: 2 ,b: -1

Round 9 : Flase Classified Point: 3 ,a1: 1 ,a2: 0a3: 3 ,b: -2

Round 10 : Flase Classified Point: 2 ,a1: 1 ,a2: 1a3: 3 ,b: -1

Round 11 : Flase Classified Point: 1 ,a1: 1 ,a2: 1a3: 3 ,b: -1

Round 12 : Flase Classified Point: 2 ,a1: 1 ,a2: 1a3: 3 ,b: -1

Round 13 : Flase Classified Point: 3 ,a1: 1 ,a2: 1a3: 4 ,b: -2

Round 14 : Flase Classified Point: 1 ,a1: 1 ,a2: 1a3: 4 ,b: -2

Round 15 : Flase Classified Point: 1 ,a1: 1 ,a2: 1a3: 4 ,b: -2

Round 16 : Flase Classified Point: 1 ,a1: 1 ,a2: 1a3: 4 ,b: -2

Round 17 : Flase Classified Point: 1 ,a1: 1 ,a2: 1a3: 4 ,b: -2

Round 18 : Flase Classified Point: 1 ,a1: 1 ,a2: 1a3: 4 ,b: -2

Round 19 : Flase Classified Point: 2 ,a1: 1 ,a2: 1a3: 4 ,b: -2

Round 20 : Flase Classified Point: 1 ,a1: 1 ,a2: 1a3: 4 ,b: -2

Round 21 : Flase Classified Point: 3 ,a1: 1 ,a2: 1a3: 5 ,b: -3

Round 22 : Flase Classified Point: 1 ,a1: 1 ,a2: 1a3: 5 ,b: -3

Round 23 : Flase Classified Point: 1 ,a1: 1 ,a2: 1a3: 5 ,b: -3

Round 24 : Flase Classified Point: 3 ,a1: 1 ,a2: 1a3: 6 ,b: -4

Round 25 : Flase Classified Point: 1 ,a1: 2 ,a2: 1a3: 6 ,b: -3

Round 26 : Flase Classified Point: 2 ,a1: 2 ,a2: 1a3: 6 ,b: -3

Round 27 : Flase Classified Point: 1 ,a1: 2 ,a2: 1a3: 6 ,b: -3

Round 28 : Flase Classified Point: 2 ,a1: 2 ,a2: 1a3: 6 ,b: -3

Round 29 : Flase Classified Point: 3 ,a1: 2 ,a2: 1a3: 7 ,b: -4

Round 30 : Flase Classified Point: 3 ,a1: 2 ,a2: 1a3: 8 ,b: -5

The perceptron model learned issign(2 x1 + 2 x2 + -5)

感知机模型算是统计学习中比较简单的一个学习算法，这里引入了原始形式和对偶形式，至于为什么引入对偶形式，书中提到是为了与后面的支持向量机SVM呼应，而且求解最优化问题一般都会分成原始形式和对偶形式。当然要搞清楚这个问题还需要继续往下学习。

我学习的资料是李航老师编写的《统计学习方法》，这本书算是入门Machine Learning,尤其是NLP领域学习问题的一本好的教材。每章内容都比较简短，差不多10-20页，读起来不会像看PRML那种书一样费劲。书里面介绍的算法很多都配有例子可以练手，学习算法还是自己动手实现一下可以领悟的更深刻。。

下一章讲的是k近邻法，之前总是听到这个算法，马上就能揭开它的庐山真面目了~