tensorflow cifar_10 代码阅读与理解

前言

Tensorflow 提供cifar_10 benchmark问题的示例代码，并且在中文翻译的官方文档中有专门的一章介绍该卷积神经网络（CNN），作为刚刚接触深度学习与Tensorflow框架的菜鸟，对tf提供的大量库函数与深度学习的trick并不十分熟悉，因此花了两天的时间通读懂了代码，下面具体剖析一下整个程序的过程，作为学习记录。

准备工作

从Github https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10下载到的cifar10程序共包括
1. cifar10.py
2. cifar10_eval.py
3. cifar10_input.py
4. cifar10_multi_gpu_train.py
5. cifar10_train.py
本文只讨论CPU版本，因此可以自动忽略cifar10_multi_gpu_train.py文件。另外，为加快程序调试，避免在程序运行时再去自动下载资源，可以提前去 http://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz下好图片压缩文件，放在默认位置/tmp/cifar10_data 中即可

代码阅读

总体流程

程序入口为cifar10_train中的main方法，其代码其注释如下：

def train():  """Train CIFAR-10 for a number of steps."""  with tf.Graph().as_default():    # use the default graph in the process in the context    global_step = tf.contrib.framework.get_or_create_global_step()  # Returns and create (if necessary) the global step variable. However the method is depressed in V0.8.0    #global_step = tf.Variable(0, name='global_step', trainable=False)    # Get images and labels for CIFAR-10.    images, labels = cifar10.distorted_inputs()    # Build a Graph that computes the logits predictions from the    # inference model.    logits = cifar10.inference(images)    # Calculate loss.    loss = cifar10.loss(logits, labels)    # Build a Graph that trains the model with one batch of examples and    # updates the model parameters.    train_op = cifar10.train(loss, global_step)    class _LoggerHook(tf.train.SessionRunHook):      """Logs loss and runtime."""      def begin(self):        self._step = -1      def before_run(self, run_context):        self._step += 1        self._start_time = time.time()        return tf.train.SessionRunArgs(loss)  # Asks for loss value.      def after_run(self, run_context, run_values):        duration = time.time() - self._start_time        loss_value = run_values.results        if self._step % 10 == 0:          num_examples_per_step = FLAGS.batch_size          examples_per_sec = num_examples_per_step / duration          sec_per_batch = float(duration)          format_str = ('%s: step %d, loss = %.2f (%.1f examples/sec; %.3f '                        'sec/batch)')          print (format_str % (datetime.now(), self._step, loss_value,                               examples_per_sec, sec_per_batch))    #For a chief, this utility sets proper session initializer/restorer. It also creates hooks related to checkpoint and summary saving. For workers, this utility sets proper session creator which waits for the chief to inialize/restore.    with tf.train.MonitoredTrainingSession(        checkpoint_dir=FLAGS.train_dir,        hooks=[tf.train.StopAtStepHook(last_step=FLAGS.max_steps),               tf.train.NanTensorHook(loss),               _LoggerHook()],        config=tf.ConfigProto(            log_device_placement=FLAGS.log_device_placement)) as mon_sess:      while not mon_sess.should_stop():        mon_sess.run(train_op)

其中，专门定义_LoggerHook类，在mon_sess这个对话中注册，代码中最后一句，表示在停止条件达到之前，循环运行train_op，更新网络系数

读取文件

调用cifar10.py中的 distorted_inputs方法，其主要语句是

filenames = [os.path.join(data_dir, 'data_batch_%d.bin' % i)               for i in xrange(1, 6)]  for f in filenames:    if not tf.gfile.Exists(f):      raise ValueError('Failed to find file: ' + f)  # Create a queue that produces the filenames to read.  filename_queue = tf.train.string_input_producer(filenames)  # Pass the list of filenames to the tf.train.string_input_producer function. string_input_producer creates a FIFO queue for holding the filenames until the reader needs them.  # Read examples from files in the filename queue.  read_input = read_cifar10(filename_queue)  reshaped_image = tf.cast(read_input.uint8image, tf.float32)  height = IMAGE_SIZE  width = IMAGE_SIZE  # Image processing for training the network. Note the many random  # distortions applied to the image.  # Randomly crop a [height, width] section of the image.  distorted_image = tf.random_crop(reshaped_image, [height, width, 3])  # Randomly flip the image horizontally.  distorted_image = tf.image.random_flip_left_right(distorted_image)  # Because these operations are not commutative, consider randomizing  # the order their operation.  distorted_image = tf.image.random_brightness(distorted_image,                                               max_delta=63)  distorted_image = tf.image.random_contrast(distorted_image,                                             lower=0.2, upper=1.8)  # Subtract off the mean and divide by the variance of the pixels.  float_image = tf.image.per_image_standardization(distorted_image)  #Linearly scales image to have zero mean and unit norm.  # Set the shapes of tensors.  float_image.set_shape([height, width, 3])  read_input.label.set_shape([1])  # Ensure that the random shuffling has good mixing properties.  min_fraction_of_examples_in_queue = 0.4  min_queue_examples = int(NUM_EXAMPLES_PER_EPOCH_FOR_TRAIN *                           min_fraction_of_examples_in_queue)  #tf.train.shuffle_batch equals 20000.  print ('Filling queue with %d CIFAR images before starting to train. '         'This will take a few minutes.' % min_queue_examples)  # Generate a batch of images and labels by building up a queue of examples.  return _generate_image_and_label_batch(float_image, read_input.label,                                         min_queue_examples, batch_size,                                         shuffle=True)

  # 采用多线程并行读入样本，构成一个训练batch，大小为128，需要注意的是，这里返回的是一个batch，images张量形式为(128,32,32,3)

  return _generate_image_and_label_batch(float_image, read_input.label, min_queue_examples, batch_size, shuffle=True)


其中，read_cifar10方法内容如下:

class CIFAR10Record(object):    pass  result = CIFAR10Record()  # Dimensions of the images in the CIFAR-10 dataset.  # See http://www.cs.toronto.edu/~kriz/cifar.html for a description of the  # input format.  label_bytes = 1  # 2 for CIFAR-100  result.height = 32  result.width = 32  result.depth = 3  image_bytes = result.height * result.width * result.depth  # Every record consists of a label followed by the image, with a  # fixed number of bytes for each.  record_bytes = label_bytes + image_bytes  # Read a record, getting filenames from the filename_queue.  No  # header or footer in the CIFAR-10 format, so we leave header_bytes  # and footer_bytes at their default of 0.  # 下面的FixedLengthRecordReader与reader.read，tf.decode_raw配合起来，是以固定长度读取文件名队列中数据的一个常用方法  reader = tf.FixedLengthRecordReader(record_bytes=record_bytes)  result.key, value = reader.read(filename_queue)  # Convert from a string to a vector of uint8 that is record_bytes long.  record_bytes = tf.decode_raw(value, tf.uint8)  # To read binary files in which each record is a fixed number of bytes, use tf.FixedLengthRecordReader with the tf.decode_raw operation. The decode_raw op converts from a string to a uint8 tensor.  # For example, the CIFAR-10 dataset uses a file format where each record is represented using a fixed number of bytes: 1 byte for the label followed by 3072 bytes of image data. Once you have a uint8 tensor, standard operations can slice out each piece and reformat as needed. For CIFAR-10, you can see how to do the reading and decoding in  # The first bytes represent the label, which we convert from uint8->int32.  # 下面采用tf.strided_slice方法在record_bytes中提取第一个bytes作为标签  result.label = tf.cast(      tf.strided_slice(record_bytes, [0], [label_bytes], [1]), tf.int32) #unfortunately, the method "tf.strided_slice" is deprecated  in this version, What can be subsititued?  # strided_slice Extracts a strided slice from a tensor.  # The remaining bytes after the label represent the image, which we reshape  # from [depth * height * width] to [depth, height, width].   # 下面采用tf.strided_slice方法在record_bytes中的图片数据信息  depth_major = tf.reshape( tf.strided_slice(record_bytes, [label_bytes], [label_bytes + image_bytes], [1]), [result.depth, result.height, result.width]) # Convert from [depth, height, width] to [height, width, depth]. result.uint8image = tf.transpose(depth_major, [1, 2, 0]) return result

如代码所述，read_cifar10其实返回了一个训练样本，包括result.label 和result.uint8image两个数据成员。其中，_generate_image_and_label_batch方法内容如下:

num_preprocess_threads = 16  if shuffle:  # 随机产生一个batch，有16个线程，而读入的缓存大小为20000，capacity为20000+3*128   images, label_batch = tf.train.shuffle_batch(        [image, label],        batch_size=batch_size,        num_threads=num_preprocess_threads,        capacity=min_queue_examples + 3 * batch_size,        min_after_dequeue=min_queue_examples)  else:    images, label_batch = tf.train.batch(        [image, label],        batch_size=batch_size,        num_threads=num_preprocess_threads,        capacity=min_queue_examples + 3 * batch_size)  # Display the training images in the visualizer.  tf.summary.image('images', images)  return images, tf.reshape(label_batch, [batch_size])

 我认为载入的难以理解点就是明明是一个一个样本读取，最后却能返回一个完成的batch，并且还有一个载入的缓存，大小为20000，极客学院网站上解释的图

首先，我们先创建数据流图，这个数据流图由一些流水线的阶段组成，阶段间用队列连接在一起。第一阶段将生成文件名，我们读取这些文件名并且把他们排到文件名队列中。第二阶段从文件中读取数据（使用Reader），产生样本，而且把样本放在一个样本队列中。根据你的设置，实际上也可以拷贝第二阶段的样本，使得他们相互独立，这样就可以从多个文件中并行读取。在第二阶段的最后是一个排队操作，就是入队到队列中去，在下一阶段出队。因为我们是要开始运行这些入队操作的线程，所以我们的训练循环会使得样本队列中的样本不断地出队。
我的理解是FixedLengthRecordReader，reader.read读取的是一个文件名列表中任意一个文件中的一个样本信息，对应上图中dequeue，在这之后可以针对这一个样本进行处理，而最终的tf.train.shuffle_batch，则将16个不同的reader读到的样本组成batch并返回。这些方法必须配套使用，即虽然没有显式的将多线程及batch构成过程编程实现，但tensorflow帮我们实现了上述的机制。

模型定义

反而模型定义部分无需多讲，只要注意以下几点：
1. 虽然与MNIST相比，这里的深度学习网络也是采用了两个卷积层和两个池化层，但中间加入局部响应正则化层和两个全连接层
2. 定义两个全连接层时，通过_variable_with_weight_decay方法将权重的二范数值引入最终的loss计算，其相关代码如下所示:

weights = _variable_with_weight_decay('weights', shape=[384, 192],                                          stddev=0.04, wd=0.004)

def _variable_with_weight_decay(name, shape, stddev, wd):  """Helper to create an initialized Variable with weight decay.  Note that the Variable is initialized with a truncated normal distribution.  A weight decay is added only if one is specified.  Args:    name: name of the variable    shape: list of ints    stddev: standard deviation of a truncated Gaussian    wd: add L2Loss weight decay multiplied by this float. If None, weight        decay is not added for this Variable.  Returns:    Variable Tensor  """  dtype = tf.float16 if FLAGS.use_fp16 else tf.float32  var = _variable_on_cpu(      name,      shape,      tf.truncated_normal_initializer(stddev=stddev, dtype=dtype))  if wd is not None:    weight_decay = tf.mul(tf.nn.l2_loss(var), wd, name='weight_loss')    #Computes half the L2 norm of a tensor without the sqrt    #output = sum(t ** 2) / 2    tf.add_to_collection('losses', weight_decay)  return var

其实，tf.nn.l2_loss就是将所有weights平方和除以2之后，然后weight_decay计算是乘以0.004系数加到losses计算中

训练目标

loss计算方法定义如下：

def loss(logits, labels):  """Add L2Loss to all the trainable variables.  Add summary for "Loss" and "Loss/avg".  Args:    logits: Logits from inference().    labels: Labels from distorted_inputs or inputs(). 1-D tensor            of shape [batch_size]  Returns:    Loss tensor of type float.  """  # Calculate the average cross entropy loss across the batch.  labels = tf.cast(labels, tf.int64)  cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(      logits, labels, name='cross_entropy_per_example')  cross_entropy_mean = tf.reduce_mean(cross_entropy, name='cross_entropy')  tf.add_to_collection('losses', cross_entropy_mean)  # Wrapper for Graph.add_to_collection() using the default graph.  # The total loss is defined as the cross entropy loss plus all of the weight  # decay terms (L2 loss).  return tf.add_n(tf.get_collection('losses'), name='total_loss')  #It seems that the cross_entropy_mean is add_to_collection and added

通过collection中'losses'字段，最后的tf.add_n将通常的熵值与上面所说的weights的二范数值相加作为loss

迭代过程

def train(total_loss, global_step):  """Train CIFAR-10 model.  Create an optimizer and apply to all trainable variables. Add moving  average for all trainable variables.  Args:    total_loss: Total loss from loss().    global_step: Integer Variable counting the number of training steps      processed.  Returns:    train_op: op for training.  """  # Variables that affect learning rate.  num_batches_per_epoch = NUM_EXAMPLES_PER_EPOCH_FOR_TRAIN / FLAGS.batch_size  #50000/128  decay_steps = int(num_batches_per_epoch * NUM_EPOCHS_PER_DECAY)  # Decay the learning rate exponentially based on the number of steps.  lr = tf.train.exponential_decay(INITIAL_LEARNING_RATE,                                  global_step,                                  decay_steps,                                  LEARNING_RATE_DECAY_FACTOR,                                  staircase=True)  #When training a model, it is often recommended to lower the learning rate as the training progresses. This function applies an exponential decay function to a provided initial learning rate. It requires a global_step value to compute the decayed learning rate. You can just pass a TensorFlow variable that you increment at each training step.  tf.summary.scalar('learning_rate', lr)  # Generate moving averages of all losses and associated summaries.  loss_averages_op = _add_loss_summaries(total_loss)  # Compute gradients.  with tf.control_dependencies([loss_averages_op]):  # Returns a context manager that specifies control dependencies.  # Use with the with keyword to specify that all operations constructed within the context should have control dependencies on control_inputs. For example:    opt = tf.train.GradientDescentOptimizer(lr)    grads = opt.compute_gradients(total_loss)  # Apply gradients.  apply_gradient_op = opt.apply_gradients(grads, global_step=global_step)  # Add histograms for trainable variables.  for var in tf.trainable_variables():    tf.summary.histogram(var.op.name, var)  # Add histograms for gradients.  for grad, var in grads:    if grad is not None:      tf.summary.histogram(var.op.name + '/gradients', grad)  # Track the moving averages of all trainable variables.  variable_averages = tf.train.ExponentialMovingAverage(      MOVING_AVERAGE_DECAY, global_step)  variables_averages_op = variable_averages.apply(tf.trainable_variables())  with tf.control_dependencies([apply_gradient_op, variables_averages_op]):    train_op = tf.no_op(name='train')  #Does nothing. Only useful as a placeholder for control edges.  return train_op

训练过程中主要的不同之处是

lr = tf.train.exponential_decay(INITIAL_LEARNING_RATE,                                  global_step,                                  decay_steps,                                  LEARNING_RATE_DECAY_FACTOR,                                  staircase=True)

就是说在每一次迭代过程中，都需要重新计算一次learning rate，而这里初始的INITIAL_LEARNING_RATE为0.1，global_step为当前的迭代次数，decay_steps就是每多少代，learning_rate衰减到到LEARNING_RATE_DECAY_FACTOR×INITIAL_LEARNING_RATE值，比如本程序中LEARNING_RATE_DECAY_FACTOR = 0.1 ，而decay_steps = num_batches_per_epoch * NUM_EPOCHS_PER_DECAY = 50000/128×350 ，也就说每十多万次迭代，lr衰减为原来0.1，然后根据每代的lr，用梯度法计算

  opt = tf.train.GradientDescentOptimizer(lr)   grads = opt.compute_gradients(total_loss)

 而后面的

 variable_averages = tf.train.ExponentialMovingAverage(      MOVING_AVERAGE_DECAY, global_step)

产生一个滑动平均计算对象，MOVING_AVERAGE_DECAY = 0.999，则每一代中的decay值更新如下
           min(decay, (1 + num_updates) / (10 + num_updates))
采用这个计算得到的decay值对上面梯度法更新得到的所有参数进行平滑处理如下：
            shadow_variable = decay * shadow_variable + (1 - decay) * variable

PS

在实际运行过程中，本人采用0.12r版本，出现版本不兼容导致的错误和警告，主要有以下两个问题：
1.cifar10_input中的tf.strided_slice方法原程序中只提供3个参数，而在0.12r中的版本需要提供4个参数，相差一个步长stride的值，这里补充为[1]即可
2.写日志文件大量采用了tf.contrib.deprecated库中的方法，已全部失效, 可直接采用tf.summary.scalar等方法cifar10_train.pycifar10_train