TF/04_Support_Vector_Machines/02_Working_with_Linear_SVMs03_Reduction_to_Linear_Regression

02 Working with Linear SVMs

We introduce a linear SVM on a binary set, which will be a subset of the Iris data. We know for I. setosa, that petal width and sepal length are completely separable. We will create a linear SVM to predict I. setosa based on two features: petal width and sepal length.

It is worth noting that due to the small data set and the randomness of separating into train/test sets, that it may appear that a few points can end up on the wrong side of the line. This is because they are in the test set, and this will result in a lower test accuracy.

Model

We will aim to maximize the margin width, 2/||A||, or minimize ||A||. We allow for a soft margin by having an error term in the loss function which is the max(0, 1-pred*actual).

Graph of Linear SVM

Here is a plot of the linear SVM separator of I. setosa based on petal width and sepal length.

The accuracy is below, plotted over each iteration.

An important observation is that while we achieve the linear separator rather quickly (100% accuracy), the loss function continues to decrease. This is because we are trying to optimize for the maximal linear separator between the two classes.

Ideally, we would have enough data to do a cross validation technique, or even to separate the data into train and test sets before optimization.

#02_linear_svm.py# Linear Support Vector Machine: Soft Margin# ----------------------------------## This function shows how to use TensorFlow to# create a soft margin SVM## We will use the iris data, specifically:#  x1 = Sepal Length#  x2 = Petal Width# Class 1 : I. setosa# Class -1: not I. setosa## We know here that x and y are linearly seperable# for I. setosa classification.import matplotlib.pyplot as pltimport numpy as npimport tensorflow as tffrom sklearn import datasetsfrom tensorflow.python.framework import opsops.reset_default_graph()# Create graphsess = tf.Session()# Load the data# iris.data = [(Sepal Length, Sepal Width, Petal Length, Petal Width)]iris = datasets.load_iris()x_vals = np.array([[x[0], x[3]] for x in iris.data])y_vals = np.array([1 if y == 0 else -1 for y in iris.target])# Split data into train/test setstrain_indices = np.random.choice(len(x_vals),                                 round(len(x_vals)*0.8),                                 replace=False)test_indices = np.array(list(set(range(len(x_vals))) - set(train_indices)))x_vals_train = x_vals[train_indices]x_vals_test = x_vals[test_indices]y_vals_train = y_vals[train_indices]y_vals_test = y_vals[test_indices]# Declare batch sizebatch_size = 100# Initialize placeholdersx_data = tf.placeholder(shape=[None, 2], dtype=tf.float32)y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32)# Create variables for linear regressionA = tf.Variable(tf.random_normal(shape=[2, 1]))b = tf.Variable(tf.random_normal(shape=[1, 1]))# Declare model operationsmodel_output = tf.subtract(tf.matmul(x_data, A), b)# Declare vector L2 'norm' function squaredl2_norm = tf.reduce_sum(tf.square(A))# Declare loss function# Loss = max(0, 1-pred*actual) + alpha * L2_norm(A)^2# L2 regularization parameter, alphaalpha = tf.constant([0.01])# Margin term in lossclassification_term = tf.reduce_mean(tf.maximum(0., tf.subtract(1., tf.multiply(model_output, y_target))))# Put terms togetherloss = tf.add(classification_term, tf.multiply(alpha, l2_norm))# Declare prediction functionprediction = tf.sign(model_output)accuracy = tf.reduce_mean(tf.cast(tf.equal(prediction, y_target), tf.float32))# Declare optimizermy_opt = tf.train.GradientDescentOptimizer(0.01)train_step = my_opt.minimize(loss)# Initialize variablesinit = tf.global_variables_initializer()sess.run(init)# Training looploss_vec = []train_accuracy = []test_accuracy = []for i in range(500):    rand_index = np.random.choice(len(x_vals_train), size=batch_size)    rand_x = x_vals_train[rand_index]    rand_y = np.transpose([y_vals_train[rand_index]])    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})    temp_loss = sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y})    loss_vec.append(temp_loss)    train_acc_temp = sess.run(accuracy, feed_dict={        x_data: x_vals_train,        y_target: np.transpose([y_vals_train])})    train_accuracy.append(train_acc_temp)    test_acc_temp = sess.run(accuracy, feed_dict={        x_data: x_vals_test,        y_target: np.transpose([y_vals_test])})    test_accuracy.append(test_acc_temp)    if (i + 1) % 100 == 0:        print('Step #{} A = {}, b = {}'.format(            str(i+1),            str(sess.run(A)),            str(sess.run(b))        ))        print('Loss = ' + str(temp_loss))# Extract coefficients[[a1], [a2]] = sess.run(A)[[b]] = sess.run(b)slope = -a2/a1y_intercept = b/a1# Extract x1 and x2 valsx1_vals = [d[1] for d in x_vals]# Get best fit linebest_fit = []for i in x1_vals:    best_fit.append(slope*i+y_intercept)# Separate I. setosasetosa_x = [d[1] for i, d in enumerate(x_vals) if y_vals[i] == 1]setosa_y = [d[0] for i, d in enumerate(x_vals) if y_vals[i] == 1]not_setosa_x = [d[1] for i, d in enumerate(x_vals) if y_vals[i] == -1]not_setosa_y = [d[0] for i, d in enumerate(x_vals) if y_vals[i] == -1]# Plot data and lineplt.plot(setosa_x, setosa_y, 'o', label='I. setosa')plt.plot(not_setosa_x, not_setosa_y, 'x', label='Non-setosa')plt.plot(x1_vals, best_fit, 'r-', label='Linear Separator', linewidth=3)plt.ylim([0, 10])plt.legend(loc='lower right')plt.title('Sepal Length vs Pedal Width')plt.xlabel('Pedal Width')plt.ylabel('Sepal Length')plt.show()# Plot train/test accuraciesplt.plot(train_accuracy, 'k-', label='Training Accuracy')plt.plot(test_accuracy, 'r--', label='Test Accuracy')plt.title('Train and Test Set Accuracies')plt.xlabel('Generation')plt.ylabel('Accuracy')plt.legend(loc='lower right')plt.show()# Plot loss over timeplt.plot(loss_vec, 'k-')plt.title('Loss per Generation')plt.xlabel('Generation')plt.ylabel('Loss')plt.show()

Step #100 A = [[-0.47445503] [-0.44861239]], b = [[-2.6099627]]Loss = [ 0.45341077]Step #200 A = [[-0.41855416] [-0.73976141]], b = [[-2.66076303]]Loss = [ 0.27603617]Step #300 A = [[-0.35709888] [-1.00628901]], b = [[-2.71116281]]Loss = [ 0.25569785]Step #400 A = [[-0.3331852 ] [-1.24517143]], b = [[-2.7465632]]Loss = [ 0.18576032]Step #500 A = [[-0.31063056] [-1.41434669]], b = [[-2.77026415]]Loss = [ 0.16047686]

03 SVM Reduction to Linear Regression

Instead of optimizing the maximal linear separator, we change the loss function to maximize the amount of data points we can make fit in our margin. This will give us a linear regression estimation.

# 03_support_vector_regression.py# SVM Regression#----------------------------------## This function shows how to use TensorFlow to# solve support vector regression. We are going# to find the line that has the maximum margin# which INCLUDES as many points as possible## We will use the iris data, specifically:#  y = Sepal Length#  x = Pedal Widthimport matplotlib.pyplot as pltimport numpy as npimport tensorflow as tffrom sklearn import datasetsfrom tensorflow.python.framework import opsops.reset_default_graph()# Create graph#修改位置config = tf.ConfigProto(allow_soft_placement= True, log_device_placement= True)sess = tf.Session(config = config)# Create graph# sess = tf.Session()# Load the data# iris.data = [(Sepal Length, Sepal Width, Petal Length, Petal Width)]iris = datasets.load_iris()x_vals = np.array([x[3] for x in iris.data])y_vals = np.array([y[0] for y in iris.data])# Split data into train/test setstrain_indices = np.random.choice(len(x_vals), round(len(x_vals)*0.8), replace=False)test_indices = np.array(list(set(range(len(x_vals))) - set(train_indices)))x_vals_train = x_vals[train_indices]x_vals_test = x_vals[test_indices]y_vals_train = y_vals[train_indices]y_vals_test = y_vals[test_indices]# Declare batch sizebatch_size = 50# Initialize placeholdersx_data = tf.placeholder(shape=[None, 1], dtype=tf.float32)y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32)# Create variables for linear regressionA = tf.Variable(tf.random_normal(shape=[1,1]))b = tf.Variable(tf.random_normal(shape=[1,1]))# Declare model operationsmodel_output = tf.add(tf.matmul(x_data, A), b)# Declare loss function# = max(0, abs(target - predicted) + epsilon)# 1/2 margin width parameter = epsilonepsilon = tf.constant([0.5])# Margin term in lossloss = tf.reduce_mean(tf.maximum(0., tf.subtract(tf.abs(tf.subtract(model_output, y_target)), epsilon)))# Declare optimizermy_opt = tf.train.GradientDescentOptimizer(0.075)train_step = my_opt.minimize(loss)# Initialize variablesinit = tf.global_variables_initializer()sess.run(init)# Training looptrain_loss = []test_loss = []for i in range(200):    rand_index = np.random.choice(len(x_vals_train), size=batch_size)    rand_x = np.transpose([x_vals_train[rand_index]])    rand_y = np.transpose([y_vals_train[rand_index]])    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})    temp_train_loss = sess.run(loss, feed_dict={x_data: np.transpose([x_vals_train]), y_target: np.transpose([y_vals_train])})    train_loss.append(temp_train_loss)    temp_test_loss = sess.run(loss, feed_dict={x_data: np.transpose([x_vals_test]), y_target: np.transpose([y_vals_test])})    test_loss.append(temp_test_loss)    if (i+1)%50==0:        print('-----------')        print('Generation: ' + str(i+1))        print('A = ' + str(sess.run(A)) + ' b = ' + str(sess.run(b)))        print('Train Loss = ' + str(temp_train_loss))        print('Test Loss = ' + str(temp_test_loss))# Extract Coefficients[[slope]] = sess.run(A)[[y_intercept]] = sess.run(b)[width] = sess.run(epsilon)# Get best fit linebest_fit = []best_fit_upper = []best_fit_lower = []for i in x_vals:  best_fit.append(slope*i+y_intercept)  best_fit_upper.append(slope*i+y_intercept+width)  best_fit_lower.append(slope*i+y_intercept-width)# Plot fit with dataplt.plot(x_vals, y_vals, 'o', label='Data Points')plt.plot(x_vals, best_fit, 'r-', label='SVM Regression Line', linewidth=3)plt.plot(x_vals, best_fit_upper, 'r--', linewidth=2)plt.plot(x_vals, best_fit_lower, 'r--', linewidth=2)plt.ylim([0, 10])plt.legend(loc='lower right')plt.title('Sepal Length vs Pedal Width')plt.xlabel('Pedal Width')plt.ylabel('Sepal Length')plt.show()# Plot loss over timeplt.plot(train_loss, 'k-', label='Train Set Loss')plt.plot(test_loss, 'r--', label='Test Set Loss')plt.title('L2 Loss per Generation')plt.xlabel('Generation')plt.ylabel('L2 Loss')plt.legend(loc='upper right')plt.show()

-----------Generation: 50A = [[ 2.13920832]] b = [[ 2.75946236]]Train Loss = 0.573591Test Loss = 0.494945-----------Generation: 100A = [[ 1.57715869]] b = [[ 3.75246167]]Train Loss = 0.231339Test Loss = 0.199602-----------Generation: 150A = [[ 1.19150889]] b = [[ 4.31496096]]Train Loss = 0.0996593Test Loss = 0.118922-----------Generation: 200A = [[ 1.02305913]] b = [[ 4.54146004]]Train Loss = 0.0792556Test Loss = 0.110936