机器学习与深度学习（三） 支持向量机 (Support Vector Machine) SVM

____tz_zs学习笔记

其他学习资料：

支持向量机通俗导论（理解SVM的三层境界）

http://blog.csdn.net/v_july_v/article/details/7624837

支持向量机（SVM）算法

http://www.cnblogs.com/end/p/3848740.html

在深度学习（2012）出现之前，支持向量机被认为机器学习中近十几年来最成功，表现最好的算法。

SVM的主要思想可以概括为两点：

1. 它是针对线性可分情况进行分析，对于线性不可分的情况，通过使用非线性映射算法将低维输入空间线性不可分的样本转化为高维特征空间使其线性可分，从而使得高维特征空间采用线性算法对样本的非线性特征进行线性分析成为可能。
2. 它基于结构风险最小化理论之上再特征空间中构建最优超平面，使得学习器得到全局最优化，并且在整个样本空间的期望以某个概率满足一定上界。

SVM算法特性：

训练好的模型的算法复杂度是由支持向量的个数决定的，而不是由数据的维度决定的。所以SVM不太容易产生过度拟合 

SVM训练出来的模型完全依赖于支持向量（Support Vectors），即使训练集里面所有非支持向量的点都被去除，重复训练过程，结果仍然会得到完全一样的模型。 

一个SVM如果训练得出的支持向量个数比较小，SVM训练出的模型比较容易被泛化。

简单应用案例1：

# -*- coding: utf-8 -*-"""@author: tz_zs"""from sklearn import svmX = [[2, 0], [1, 1], [2,3]]y = [0, 0, 1]clf = svm.SVC(kernel = 'linear')clf.fit(X, y)  print clf# get support vectorsprint clf.support_vectors_# get indices of support vectorsprint clf.support_ # get number of support vectors for each classprint clf.n_support_ 

运行结果：

SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,  decision_function_shape=None, degree=3, gamma='auto', kernel='linear',  max_iter=-1, probability=False, random_state=None, shrinking=True,  tol=0.001, verbose=False)[[ 1.  1.] [ 2.  3.]][1 2][1 1]

简单应用案例2：

# -*- coding: utf-8 -*-"""@author: tz_zs"""print(__doc__)import numpy as npimport pylab as plfrom sklearn import svm# we create 40 separable pointsnp.random.seed(0)X = np.r_[np.random.randn(20, 2) - [2, 2], np.random.randn(20, 2) + [2, 2]]Y = [0] * 20 + [1] * 20# fit the modelclf = svm.SVC(kernel='linear')clf.fit(X, Y)# get the separating hyperplanew = clf.coef_[0]a = -w[0] / w[1]xx = np.linspace(-5, 5)yy = a * xx - (clf.intercept_[0]) / w[1]# plot the parallels to the separating hyperplane that pass through the# support vectorsb = clf.support_vectors_[0]yy_down = a * xx + (b[1] - a * b[0])b = clf.support_vectors_[-1]yy_up = a * xx + (b[1] - a * b[0])print "w: ", wprint "a: ", a# print " xx: ", xx# print " yy: ", yyprint "support_vectors_: ", clf.support_vectors_print "clf.coef_: ", clf.coef_# In scikit-learn coef_ attribute holds the vectors of the separating hyperplanes for linear models. It has shape (n_classes, n_features) if n_classes > 1 (multi-class one-vs-all) and (1, n_features) for binary classification.# # In this toy binary classification example, n_features == 2, hence w = coef_[0] is the vector orthogonal to the hyperplane (the hyperplane is fully defined by it + the intercept).# # To plot this hyperplane in the 2D case (any hyperplane of a 2D plane is a 1D line), we want to find a f as in y = f(x) = a.x + b. In this case a is the slope of the line and can be computed by a = -w[0] / w[1].# plot the line, the points, and the nearest vectors to the planepl.plot(xx, yy, 'k-')pl.plot(xx, yy_down, 'k--')pl.plot(xx, yy_up, 'k--')pl.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],           s=80, facecolors='none')pl.scatter(X[:, 0], X[:, 1], c=Y, cmap=pl.cm.Paired)pl.axis('tight')pl.show()

运行结果：

@author：tz_zsw：[0.90230696 0.64821811]a：-1.39198047626support_vectors_：[[-1.02126202 0.2408932] [-0.46722079 -0.53064123] [0.95144703 0.57998206]]clf.coef_：[[0.90230696 0.64821811]]

线性不可分的情况（linearly inseparable case）

如图1所示，利用一个非线性的映射把原数据集中的向量点转化到一个更高维度的空间中

2，在这个高维度的空间中找一个线性的超平面来根据线性可分的情况处理

简单应用实例（我运行时数据集下载失败）：

# -*- coding: utf-8 -*-"""@author: tz_zs"""from __future__ import print_functionfrom time import timeimport loggingimport matplotlib.pyplot as plt #绘图from sklearn.cross_validation import train_test_splitfrom sklearn.datasets import fetch_lfw_peoplefrom sklearn.grid_search import GridSearchCVfrom sklearn.metrics import classification_reportfrom sklearn.metrics import confusion_matrixfrom sklearn.decomposition import RandomizedPCAfrom sklearn.svm import SVCprint(__doc__)# Display progress logs on stdoutlogging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')################################################## ############################## Download the data, if not already on disk and load it as numpy arrays# 下载名人头像数据集（如果没有的话）lfw_people = fetch_lfw_people(min_faces_per_person=70, resize=0.4)# introspect the images arrays to find the shapes (for plotting)n_samples, h, w = lfw_people.images.shape# for machine learning we use the 2 data directly (as relative pixel# positions info is ignored by this model)X = lfw_people.datan_features = X.shape[1]# the label to predict is the id of the persony = lfw_people.targettarget_names = lfw_people.target_namesn_classes = target_names.shape[0]print("Total dataset size:")print("n_samples: %d" % n_samples)print("n_features: %d" % n_features)print("n_classes: %d" % n_classes)################################################################################ Split into a training set and a test set using a stratified k fold# split into a training and testing setX_train, X_test, y_train, y_test = train_test_split(    X, y, test_size=0.25)################################################################################ Compute a PCA (eigenfaces) on the face dataset (treated as unlabeled# dataset): unsupervised feature extraction / dimensionality reductionn_components = 150print("Extracting the top %d eigenfaces from %d faces"      % (n_components, X_train.shape[0]))t0 = time()pca = RandomizedPCA(n_components=n_components, whiten=True).fit(X_train)print("done in %0.3fs" % (time() - t0))eigenfaces = pca.components_.reshape((n_components, h, w))print("Projecting the input data on the eigenfaces orthonormal basis")t0 = time()X_train_pca = pca.transform(X_train)X_test_pca = pca.transform(X_test)print("done in %0.3fs" % (time() - t0))################################################################################ Train a SVM classification modelprint("Fitting the classifier to the training set")t0 = time()param_grid = {'C': [1e3, 5e3, 1e4, 5e4, 1e5],              'gamma': [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.1], }clf = GridSearchCV(SVC(kernel='rbf', class_weight='auto'), param_grid)clf = clf.fit(X_train_pca, y_train)print("done in %0.3fs" % (time() - t0))print("Best estimator found by grid search:")print(clf.best_estimator_)################################################################################ Quantitative evaluation of the model quality on the test setprint("Predicting people's names on the test set")t0 = time()y_pred = clf.predict(X_test_pca)print("done in %0.3fs" % (time() - t0))print(classification_report(y_test, y_pred, target_names=target_names))print(confusion_matrix(y_test, y_pred, labels=range(n_classes)))################################################################################ Qualitative evaluation of the predictions using matplotlibdef plot_gallery(images, titles, h, w, n_row=3, n_col=4):    """Helper function to plot a gallery of portraits"""    plt.figure(figsize=(1.8 * n_col, 2.4 * n_row))    plt.subplots_adjust(bottom=0, left=.01, right=.99, top=.90, hspace=.35)    for i in range(n_row * n_col):        plt.subplot(n_row, n_col, i + 1)        plt.imshow(images[i].reshape((h, w)), cmap=plt.cm.gray)        plt.title(titles[i], size=12)        plt.xticks(())        plt.yticks(())# plot the result of the prediction on a portion of the test setdef title(y_pred, y_test, target_names, i):    pred_name = target_names[y_pred[i]].rsplit(' ', 1)[-1]    true_name = target_names[y_test[i]].rsplit(' ', 1)[-1]    return 'predicted: %s\ntrue:      %s' % (pred_name, true_name)prediction_titles = [title(y_pred, y_test, target_names, i)                     for i in range(y_pred.shape[0])]plot_gallery(X_test, prediction_titles, h, w)# plot the gallery of the most significative eigenfaceseigenface_titles = ["eigenface %d" % i for i in range(eigenfaces.shape[0])]plot_gallery(eigenfaces, eigenface_titles, h, w)plt.show()