机器学习简单实验(LMS算法)

LMS为最小均方算法(least mean square)，目标是使得均方误差(MSE）最小，即样本预测输出值与实际输出值之差平方的期望值最小。
下面实现LMS算法学习或运算:

# -*- coding utf-8 -*-# LMS.pyimport numpy as npb = 1a = 0.1x = np.array([[1,1,1],[1,1,0],[1,0,1],[1,0,0]])d = np.array([1,1,1,0])w = np.array([b,0,0])expect_e = 0.005maxtrycount = 20def sgn(v):    if v>0:        return 1    else:        return 0def get_v(myw,myx):    return sgn(np.dot(myw.T,myx))def neww(oldw,myd,myx,a):    mye  = get_e(oldw,myx,myd)    return (oldw + a*mye*myx,mye)def get_e(myw,myx,myd):    return myd - get_v(myw,myx)mycount = 0while True:    mye = 0    i = 0    for xn in x:        w,e = neww(w,d[i],xn,a)        i += 1        mye += pow(e,2)    mye /= float(i)    mycount += 1    print u"After %d times, w: " %mycount    print w    print "error: %f" %mye    if mye<expect_e or mycount>maxtrycount:breakfor xn in x:    print "%d or %d => %d" %(xn[1],xn[2],get_v(w,xn))

结果:

..........................After 11 times, w: [-0.1  0.   0. ]error: 0.250000After 12 times, w: [-0.1  0.1  0.1]error: 0.500000After 13 times, w: [-0.1  0.1  0.1]error: 0.0000001 or 1 => 11 or 0 => 10 or 1 => 10 or 0 => 0

13次迭代之后，error = 0，分类结果正确。
现在实现一个稍微复杂一点的任务，输入矩阵中，若X向量整除结果为6，则为1类，若整除结果为3，为第-1类，在LMS.py的基础上稍作改动:

# -*- coding utf-8 -*-# LMS2.pyimport numpy as npb = 1a = 0.1x = np.array([[1,1,6],[1,2,12],[1,3,9],[1,8,24]])d = np.array([1,1,-1,-1])w = np.array([b,0,0])expect_e = 0.005maxtrycount = 20def sgn(v):    if v>0:        return 1    else:        return -1def get_v(myw,myx):    return sgn(np.dot(myw.T,myx))def neww(oldw,myd,myx,a):    mye  = get_e(oldw,myx,myd)    return (oldw + a*mye*myx,mye)def get_e(myw,myx,myd):    return myd - get_v(myw,myx)mycount = 0while True:    mye = 0    i = 0    for xn in x:        w,e = neww(w,d[i],xn,a)        i += 1        mye += pow(e,2)    mye /= float(i)    mycount += 1    print u"After %d times, w: " %mycount    print w    print "error: %f" %mye    if mye<expect_e or mycount>maxtrycount:breakfor xn in x:    print "%d    %d => %d" %(xn[1],xn[2],get_v(w,xn))test = np.array([1,9,27])print "%d     %d => %d" %(test[1],test[2],get_v(w,xn))test = np.array([1,11,66])print "%d     %d => %d" %(test[1],test[2],get_v(w,xn))

运行结果:

........................After 7 times, w: [ 1.2 -2.8 -1.2]error: 1.000000After 8 times, w: [ 1.4 -2.8  0.6]error: 3.000000After 9 times, w: [ 1.4 -2.8  0.6]error: 0.0000001    6 => 12    12 => 13    9 => -18    24 => -19     27 => -111     66 => -1

分类正确.
但是，如果加入几个不规则的和错的数据，分类器还能不能正确分类呢？
加入数据
x = np.array([[1,1,6],[1,3,12],[1,3,9],[1,3,21],[1,2,16],[1,3,15]])
d = np.array([1,1,-1,-1,1,-1])

After 200 times, w: [ 18.  -17.4  -0.8]error: 2.666667After 201 times, w: [ 18.4 -16.8   1.8]error: 2.6666671    6 => 13    12 => -13    9 => -13    21 => 12    16 => 13    15 => -19     27 => -111     66 => -1

可以看到，现在分类结果出现了错误。是什么原因呢？对于错误的数据，分类器以同样的学习率进行学习，结果当然会出错，所以需要动态改变学习率的大小进行学习，所以需要采用模拟退火算法。