A GENTLE GUIDE TO USING BATCH NORMALIZATION IN TENSORFLOW

今天看这个源码models/tutorials/image/cifar10_estimator是看到这么一行code:

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS, name_scope)

为了找到什么地方往 UPDATE_OPS 添加了东西。找到了这篇文章，因为暂时我的重心在学会用tensorflow编程，先大概看看，等以后再仔细研究。

---

Related functiones

• tf.nn.moments,
• tf.nn.batch_normalization
• tf.contrib.layers.batch_norm.

Batch Normalization The Easy Way

Perhaps the easiest way to use batch normalization would be to simply use the tf.contrib.layers.batch_norm layer. So let’s give that a go! Let’s get some imports and data loading out of the way first.

import numpy as npimport tensorflow as tffrom tensorflow.examples.tutorials.mnist import input_datafrom utils import show_graphmnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

Next, we define our typical fully-connected + batch normalization + nonlinearity set-up

def dense(x, size, scope):    return tf.contrib.layers.fully_connected(x, size,                                              activation_fn=None,                                             scope=scope)def dense_batch_relu(x, phase, scope):    with tf.variable_scope(scope):        h1 = tf.contrib.layers.fully_connected(x, 100,                                                activation_fn=None,                                               scope='dense')        h2 = tf.contrib.layers.batch_norm(h1,                                           center=True, scale=True,                                           is_training=phase,                                          scope='bn')        return tf.nn.relu(h2, 'relu')

One thing that might stand out is the phase term. We are going to use as a placeholder for a boolean which we will insert into feed_dict. It will serve as a binary indicator for whether we are in training phase=True or testing phase=False mode. Recall that batch normalization has distinct behaviors during training verus test time:

Training

• Normalize layer activations according to mini-batch statistics.
• During the training step, update population statistics approximation via moving average of mini-batch statistics.

Testing

• Normalize layer activations according to estimated population statistics.
• Do not update population statistics according to mini-batch statistcs from test data.

Now we can define our very simple neural network for MNIST classification.

tf.reset_default_graph()x = tf.placeholder('float32', (None, 784), name='x')y = tf.placeholder('float32', (None, 10), name='y')phase = tf.placeholder(tf.bool, name='phase')h1 = dense_batch_relu(x, phase,'layer1')h2 = dense_batch_relu(h1, phase, 'layer2')logits = dense(h2, 10, 'logits')with tf.name_scope('accuracy'):    accuracy = tf.reduce_mean(tf.cast(            tf.equal(tf.argmax(y, 1), tf.argmax(logits, 1)),            'float32'))with tf.name_scope('loss'):    loss = tf.reduce_mean(        tf.nn.softmax_cross_entropy_with_logits(logits, y))

Now that we have defined our data and computational graph (a.k.a. model), we can train the model! Here is where we need to notice a very important note in the tf.contrib.layers.batch_norm documentation:

---

Note: When is_training is True the moving_mean and moving_variance need to be updated, by default the update_ops are placed in tf.GraphKeys.UPDATE_OPS so they need to be added as a dependency to the train_op, example:

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS) if update_ops: updates = tf.group(*update_ops) total_loss = control_flow_ops.with_dependencies([updates], total_loss)

---

If you are comfortable with TensorFlow’s underlying graph/ops mechanism, the note is fairly straight-forward. If not, here’s a simple way to think of it: when you execute an operation (such as train_step), only the subgraph components relevant to train_step will be executed. Unfortunately, the update_moving_averages operation is not a parent of train_step in the computational graph, so we will never update the moving averages! To get around this, we have to explicitly tell the graph:

Hey graph, update the moving averages before you finish the training step!

Unfortunately, the instructions in the documentation are a little out of date. Furthermore, if you think about it a little more, you may conclude that attaching the update ops to total_loss may not be desirable if you wish to compute the total_loss of the test set during test time. Personally, I think it makes more sense to attach the update ops to the train_step itself. So I modified the code a little and created the following training function

def train():    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)    with tf.control_dependencies(update_ops):        # Ensures that we execute the update_ops before performing the train_step        train_step = tf.train.GradientDescentOptimizer(0.01).minimize(loss)    sess = tf.Session()    sess.run(tf.global_variables_initializer())    history = []    iterep = 500    for i in range(iterep * 30):        x_train, y_train = mnist.train.next_batch(100)        sess.run(train_step,                 feed_dict={'x:0': x_train,                             'y:0': y_train,                             'phase:0': 1})        if (i + 1) %  iterep == 0:            epoch = (i + 1)/iterep            tr = sess.run([loss, accuracy],                           feed_dict={'x:0': mnist.train.images,                                     'y:0': mnist.train.labels,                                     'phase:0': 1})            t = sess.run([loss, accuracy],                          feed_dict={'x:0': mnist.test.images,                                    'y:0': mnist.test.labels,                                    'phase:0': 0})            history += [[epoch] + tr + t]            print history[-1]    return history

And we’re done! We can now train our model and see what happens. Below, I provide a comparison of the model without batch normalization, the model with pre-activation batch normalization, and the model with post-activation batch normalization.

As you can see, batch normalization really does help with training (not always, but it certainly did in this simple example).

Additional Remarks

You have the choice of applying batch normalization either before or after the non-linearity, depending on your definition of the “activation distribution of interest” that you wish to normalize. It will probably end up being a hyperparameter that you’ll just have to tinker with.

There is also the question of what it means to share the same batch normalization layer across multiple models. I think this question should be treated with some care, and it really depends on what you think you’ll gain out of sharing the batch normalization layer. In particular, we should be careful about sharing the mini-batch statistics across multiple data streams if we expect the data streams to have distinct distributions, as is the case when using batch normalization in recurrent neural networks. As of the moment, the tf.contrib.layers.batch_norm function does not allow this level of control.