TF/02_TensorFlow_Way/08 Evaluating_Models

Evaluating Models

Summary

Here we will perform two modeling tasks: regression and classification. We will show how to evaluate the models in the Tensorflow algorithm

Regression Model

The regression model is from a prior section. We will generate input data which will be distributed as Normal(mean=1, sd=0.1) and target data or repeating 10.0 values. The model will optimize a multiplication factor (theoretically = 10) to predict the target data.

Classification Model

The classification model will be half values from Normal(mean=-1, sd=1) and half from Normal(mean=2, sd=1). There will be some overlap in the distributions. The classification model will optimize on predicting a middle point, of which is will classify everything less than it as the first distribution and everything greater than it as the latter distribution. Theoretically, we know that the ideal cutoff is between the two normals at x = 0.5.

Outline

The idea is that we want to split our labeled data set into a training and test set. We then train on the test set, and look at the accuracy on the test set.

Classification Results

Here is the classification results visualized as a histogram:

# 08_evaluating_models.py# Evaluating models in TensorFlow## This code will implement two models.  The first#  is a simple regression model, we will show how to#  call the loss function, MSE during training, and#  output it after for test and training sets.## The second model will be a simple classification#  model.  We will also show how to print percent#  classified correctly during training and after#  for both the test and training sets.import matplotlib.pyplot as pltimport numpy as npimport tensorflow as tffrom tensorflow.python.framework import opsops.reset_default_graph()# Create graphsess = tf.Session()# Regression Example:# We will create sample data as follows:# x-data: 100 random samples from a normal ~ N(1, 0.1)# target: 100 values of the value 10.# We will fit the model:# x-data * A = target# Theoretically, A = 10.# Declare batch sizebatch_size = 25# Create datax_vals = np.random.normal(1, 0.1, 100)y_vals = np.repeat(10., 100)x_data = tf.placeholder(shape=[None, 1], dtype=tf.float32)y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32)# Split data into train/test = 80%/20%train_indices = np.random.choice(len(x_vals), round(len(x_vals)*0.8), replace=False)test_indices = np.array(list(set(range(len(x_vals))) - set(train_indices)))x_vals_train = x_vals[train_indices]x_vals_test = x_vals[test_indices]y_vals_train = y_vals[train_indices]y_vals_test = y_vals[test_indices]# Create variable (one model parameter = A)A = tf.Variable(tf.random_normal(shape=[1,1]))# Add operation to graphmy_output = tf.matmul(x_data, A)# Add L2 loss operation to graphloss = tf.reduce_mean(tf.square(my_output - y_target))# Create Optimizermy_opt = tf.train.GradientDescentOptimizer(0.02)train_step = my_opt.minimize(loss)# Initialize variablesinit = tf.global_variables_initializer()sess.run(init)# Run Loopfor i in range(100):    rand_index = np.random.choice(len(x_vals_train), size=batch_size)    rand_x = np.transpose([x_vals_train[rand_index]])    rand_y = np.transpose([y_vals_train[rand_index]])    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})    if (i+1)%25==0:        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)))        print('Loss = ' + str(sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y})))# Evaluate accuracy (loss) on test setmse_test = sess.run(loss, feed_dict={x_data: np.transpose([x_vals_test]), y_target: np.transpose([y_vals_test])})mse_train = sess.run(loss, feed_dict={x_data: np.transpose([x_vals_train]), y_target: np.transpose([y_vals_train])})print('MSE on test:' + str(np.round(mse_test, 2)))print('MSE on train:' + str(np.round(mse_train, 2)))# Classification Example# We will create sample data as follows:# x-data: sample 50 random values from a normal = N(-1, 1)#         + sample 50 random values from a normal = N(1, 1)# target: 50 values of 0 + 50 values of 1.#         These are essentially 100 values of the corresponding output index# We will fit the binary classification model:# If sigmoid(x+A) < 0.5 -> 0 else 1# Theoretically, A should be -(mean1 + mean2)/2ops.reset_default_graph()# Create graphsess = tf.Session()# Declare batch sizebatch_size = 25# Create datax_vals = np.concatenate((np.random.normal(-1, 1, 50), np.random.normal(2, 1, 50)))y_vals = np.concatenate((np.repeat(0., 50), np.repeat(1., 50)))x_data = tf.placeholder(shape=[1, None], dtype=tf.float32)y_target = tf.placeholder(shape=[1, None], dtype=tf.float32)# Split data into train/test = 80%/20%train_indices = np.random.choice(len(x_vals), round(len(x_vals)*0.8), replace=False)test_indices = np.array(list(set(range(len(x_vals))) - set(train_indices)))x_vals_train = x_vals[train_indices]x_vals_test = x_vals[test_indices]y_vals_train = y_vals[train_indices]y_vals_test = y_vals[test_indices]# Create variable (one model parameter = A)A = tf.Variable(tf.random_normal(mean=10, shape=[1]))# Add operation to graph# Want to create the operstion sigmoid(x + A)# Note, the sigmoid() part is in the loss functionmy_output = tf.add(x_data, A)# Add classification loss (cross entropy)xentropy = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=my_output, labels=y_target))# Create Optimizermy_opt = tf.train.GradientDescentOptimizer(0.05)train_step = my_opt.minimize(xentropy)# Initialize variablesinit = tf.global_variables_initializer()sess.run(init)# Run loopfor i in range(1800):    rand_index = np.random.choice(len(x_vals_train), size=batch_size)    rand_x = [x_vals_train[rand_index]]    rand_y = [y_vals_train[rand_index]]    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})    if (i+1)%200==0:        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)))        print('Loss = ' + str(sess.run(xentropy, feed_dict={x_data: rand_x, y_target: rand_y})))# Evaluate Predictions on test sety_prediction = tf.squeeze(tf.round(tf.nn.sigmoid(tf.add(x_data, A))))correct_prediction = tf.equal(y_prediction, y_target)accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))acc_value_test = sess.run(accuracy, feed_dict={x_data: [x_vals_test], y_target: [y_vals_test]})acc_value_train = sess.run(accuracy, feed_dict={x_data: [x_vals_train], y_target: [y_vals_train]})print('Accuracy on train set: ' + str(acc_value_train))print('Accuracy on test set: ' + str(acc_value_test))# Plot classification resultA_result = -sess.run(A)bins = np.linspace(-5, 5, 50)plt.hist(x_vals[0:50], bins, alpha=0.5, label='N(-1,1)', color='white')plt.hist(x_vals[50:100], bins[0:50], alpha=0.5, label='N(2,1)', color='red')plt.plot((A_result, A_result), (0, 8), 'k--', linewidth=3, label='A = '+ str(np.round(A_result, 2)))plt.legend(loc='upper right')plt.title('Binary Classifier, Accuracy=' + str(np.round(acc_value_test, 2)))plt.show()

Step #25 A = [[ 6.26553822]]Loss = 13.0212Step #50 A = [[ 8.57185745]]Loss = 2.93135Step #75 A = [[ 9.3278532]]Loss = 0.599669Step #100 A = [[ 9.58197975]]Loss = 0.966745MSE on test:1.05MSE on train:0.88Step #200 A = [ 4.30588198]Loss = 1.91649Step #400 A = [ 0.71840709]Loss = 0.464558Step #600 A = [-0.37344712]Loss = 0.228632Step #800 A = [-0.61095661]Loss = 0.201972Step #1000 A = [-0.66159064]Loss = 0.265847Step #1200 A = [-0.69707364]Loss = 0.314225Step #1400 A = [-0.67452353]Loss = 0.225491Step #1600 A = [-0.70979553]Loss = 0.302223Step #1800 A = [-0.65826631]Loss = 0.291439Accuracy on train set: 0.9625Accuracy on test set: 0.95