Tensorflow Fine-Tuning 的一些说明

一般训练好的模型可以直接拿来用，或者再原来的基础上增加新的layer。http://tflearn.org/getting_started/#fine-tuning
现在以代码说明:

cifar_10.py

# -*- coding: utf-8 -*-""" Convolutional network applied to CIFAR-10 dataset classification task.References:    Learning Multiple Layers of Features from Tiny Images, A. Krizhevsky, 2009.Links:    [CIFAR-10 Dataset](https://www.cs.toronto.edu/~kriz/cifar.html)"""from __future__ import division, print_function, absolute_importimport tflearnfrom tflearn.data_utils import shuffle, to_categoricalfrom tflearn.layers.core import input_data, dropout, fully_connectedfrom tflearn.layers.conv import conv_2d, max_pool_2dfrom tflearn.layers.estimator import regressionfrom tflearn.data_preprocessing import ImagePreprocessingfrom tflearn.data_augmentation import ImageAugmentation# Data loading and preprocessingfrom tflearn.datasets import cifar10(X, Y), (X_test, Y_test) = cifar10.load_data(dirname="./cifar-10/")X, Y = shuffle(X, Y)Y = to_categorical(Y,10)Y_test = to_categorical(Y_test,10)# Real-time data preprocessingimg_prep = ImagePreprocessing()img_prep.add_featurewise_zero_center()img_prep.add_featurewise_stdnorm()# Real-time data augmentationimg_aug = ImageAugmentation()img_aug.add_random_flip_leftright()img_aug.add_random_rotation(max_angle=25.)# Convolutional network buildingnetwork = input_data(shape=[None, 32, 32, 3],                     data_preprocessing=img_prep,                     data_augmentation=img_aug)network = conv_2d(network, 32, 3, activation='relu')network = max_pool_2d(network, 2)network = conv_2d(network, 64, 3, activation='relu')network = conv_2d(network, 64, 3, activation='relu')network = max_pool_2d(network, 2)network = fully_connected(network, 512, activation='relu')network = dropout(network, 0.5)network = fully_connected(network, 10, activation='softmax')network = regression(network, optimizer='adam',                     loss='categorical_crossentropy',                     learning_rate=0.001)# Train using classifiermodel = tflearn.DNN(network, tensorboard_verbose=0)model.fit(X, Y, n_epoch=50, shuffle=True, validation_set=(X_test, Y_test),          show_metric=True, batch_size=96, run_id='cifar10_cnn')model.save('./cifar10_cnn')

具体的模型可以参照代码中的网址，有pdf文件可以下载。
代码中的卷积层都没有显式的定义name，TFLearn的处理是以layer命名
Conv2D,Conv2D_1,Conv2D_2,FullyConnected,FullyConnected_1,这样的。
训练过程中的输出。

Training Step: 26016  | total loss: 0.38068 | time: 227.672s| Adam | epoch: 050 | loss: 0.38068 - acc: 0.8713 -- iter: 46752/50000Training Step: 26017  | total loss: 0.38373 | time: 228.142s| Adam | epoch: 050 | loss: 0.38373 - acc: 0.8696 -- iter: 46848/50000Training Step: 26018  | total loss: 0.38030 | time: 228.612s| Adam | epoch: 050 | loss: 0.38030 - acc: 0.8680 -- iter: 46944/50000Training Step: 26019  | total loss: 0.38134 | time: 229.079s| Adam | epoch: 050 | loss: 0.38134 - acc: 0.8677 -- iter: 47040/50000Training Step: 26020  | total loss: 0.40366 | time: 229.530s| Adam | epoch: 050 | loss: 0.40366 - acc: 0.8611 -- iter: 47136/50000Training Step: 26021  | total loss: 0.38866 | time: 230.000s| Adam | epoch: 050 | loss: 0.38866 - acc: 0.8656 -- iter: 47232/50000Training Step: 26022  | total loss: 0.40954 | time: 230.477s| Adam | epoch: 050 | loss: 0.40954 - acc: 0.8624 -- iter: 47328/50000Training Step: 26023  | total loss: 0.40813 | time: 230.933s| Adam | epoch: 050 | loss: 0.40813 - acc: 0.8605 -- iter: 47424/50000Training Step: 26024  | total loss: 0.40759 | time: 231.406s| Adam | epoch: 050 | loss: 0.40759 - acc: 0.8630 -- iter: 47520/50000Training Step: 26025  | total loss: 0.40606 | time: 231.858s| Adam | epoch: 050 | loss: 0.40606 - acc: 0.8621 -- iter: 47616/50000Training Step: 26026  | total loss: 0.39454 | time: 232.324s| Adam | epoch: 050 | loss: 0.39454 - acc: 0.8634 -- iter: 47712/50000Training Step: 26027  | total loss: 0.39360 | time: 232.824s| Adam | epoch: 050 | loss: 0.39360 - acc: 0.8615 -- iter: 47808/50000Training Step: 26028  | total loss: 0.37261 | time: 233.300s| Adam | epoch: 050 | loss: 0.37261 - acc: 0.8691 -- iter: 47904/50000Training Step: 26029  | total loss: 0.37792 | time: 233.772s| Adam | epoch: 050 | loss: 0.37792 - acc: 0.8665 -- iter: 48000/50000Training Step: 26030  | total loss: 0.36900 | time: 234.250s| Adam | epoch: 050 | loss: 0.36900 - acc: 0.8674 -- iter: 48096/50000Training Step: 26031  | total loss: 0.35785 | time: 234.717s| Adam | epoch: 050 | loss: 0.35785 - acc: 0.8713 -- iter: 48192/50000Training Step: 26032  | total loss: 0.36420 | time: 235.180s| Adam | epoch: 050 | loss: 0.36420 - acc: 0.8675 -- iter: 48288/50000Training Step: 26033  | total loss: 0.35347 | time: 235.628s| Adam | epoch: 050 | loss: 0.35347 - acc: 0.8693 -- iter: 48384/50000Training Step: 26034  | total loss: 0.38350 | time: 236.098s| Adam | epoch: 050 | loss: 0.38350 - acc: 0.8657 -- iter: 48480/50000Training Step: 26035  | total loss: 0.37958 | time: 236.576s| Adam | epoch: 050 | loss: 0.37958 - acc: 0.8676 -- iter: 48576/50000Training Step: 26036  | total loss: 0.36793 | time: 237.037s| Adam | epoch: 050 | loss: 0.36793 - acc: 0.8705 -- iter: 48672/50000Training Step: 26037  | total loss: 0.35811 | time: 237.501s| Adam | epoch: 050 | loss: 0.35811 - acc: 0.8740 -- iter: 48768/50000Training Step: 26038  | total loss: 0.36710 | time: 237.976s| Adam | epoch: 050 | loss: 0.36710 - acc: 0.8700 -- iter: 48864/50000Training Step: 26039  | total loss: 0.36583 | time: 238.435s| Adam | epoch: 050 | loss: 0.36583 - acc: 0.8715 -- iter: 48960/50000Training Step: 26040  | total loss: 0.37000 | time: 238.890s| Adam | epoch: 050 | loss: 0.37000 - acc: 0.8708 -- iter: 49056/50000Training Step: 26041  | total loss: 0.36465 | time: 239.348s| Adam | epoch: 050 | loss: 0.36465 - acc: 0.8723 -- iter: 49152/50000Training Step: 26042  | total loss: 0.35985 | time: 239.821s| Adam | epoch: 050 | loss: 0.35985 - acc: 0.8767 -- iter: 49248/50000Training Step: 26043  | total loss: 0.36621 | time: 240.288s| Adam | epoch: 050 | loss: 0.36621 - acc: 0.8776 -- iter: 49344/50000Training Step: 26044  | total loss: 0.37555 | time: 240.752s| Adam | epoch: 050 | loss: 0.37555 - acc: 0.8742 -- iter: 49440/50000Training Step: 26045  | total loss: 0.37014 | time: 241.216s| Adam | epoch: 050 | loss: 0.37014 - acc: 0.8753 -- iter: 49536/50000Training Step: 26046  | total loss: 0.37784 | time: 241.675s| Adam | epoch: 050 | loss: 0.37784 - acc: 0.8743 -- iter: 49632/50000Training Step: 26047  | total loss: 0.37061 | time: 242.141s| Adam | epoch: 050 | loss: 0.37061 - acc: 0.8785 -- iter: 49728/50000Training Step: 26048  | total loss: 0.37322 | time: 242.633s| Adam | epoch: 050 | loss: 0.37322 - acc: 0.8792 -- iter: 49824/50000Training Step: 26049  | total loss: 0.37036 | time: 243.098s| Adam | epoch: 050 | loss: 0.37036 - acc: 0.8798 -- iter: 49920/50000Training Step: 26050  | total loss: 0.37210 | time: 260.038s| Adam | epoch: 050 | loss: 0.37210 - acc: 0.8825 | val_loss: 0.67619 - val_acc: 0.8143 -- iter: 50000/50000

现在FineTuning，下面是fineTuning代码

#encoding:utf-8#@Time : 2017/12/8 15:46#@Author : JackNiuimport tflearnfrom tflearn.layers.core import input_data, dropout, fully_connectedfrom tflearn.layers.conv import conv_2d, max_pool_2dfrom tflearn.layers.estimator import regressionfrom tflearn.data_preprocessing import ImagePreprocessingfrom tflearn.data_augmentation import ImageAugmentationfrom tflearn.data_utils import shuffle, to_categorical# Data loading and preprocessingfrom tflearn.datasets import cifar10(X, Y), (X_test, Y_test) = cifar10.load_data(dirname="./cifar-10/")X, Y = shuffle(X, Y)Y = to_categorical(Y,10)Y_test = to_categorical(Y_test,10)# Real-time data preprocessingimg_prep = ImagePreprocessing()img_prep.add_featurewise_zero_center()img_prep.add_featurewise_stdnorm()# Real-time data augmentationimg_aug = ImageAugmentation()img_aug.add_random_flip_leftright()img_aug.add_random_rotation(max_angle=25.)# Redefinition of convnet_cifar10 networknetwork = input_data(shape=[None, 32, 32, 3])network = conv_2d(network, 32, 3, activation='relu')network = max_pool_2d(network, 2)network = dropout(network, 0.75)network = conv_2d(network, 64, 3, activation='relu')network = conv_2d(network, 64, 3, activation='relu')network = max_pool_2d(network, 2)network = dropout(network, 0.5)network = fully_connected(network, 512, activation='relu')network = dropout(network, 0.5)# Finetuning Softmax layer (Setting restore=False to not restore its weights)# 最后一层全连接层softmax = fully_connected(network, 10, activation='softmax', restore=False)regression = regression(softmax, optimizer='adam',                        loss='categorical_crossentropy',                        learning_rate=0.001)model = tflearn.DNN(regression, checkpoint_path='./tmp/fine/model_finetuning',                    max_checkpoints=3, tensorboard_verbose=0)# Load pre-existing model, restoring all weights, except softmax layer ones# 下载之前存在的模型，恢复所有的权重，除了最后一层的权重model.load('cifar10_cnn')# 下载已经训练好的模型，那么这里的之前定义的权重是怎么一一对应到的？ 是根据名字# 获取模型的第一层conv 的w和bprint(tflearn.variables.get_all_trainable_variable())conv1= tflearn.variables.get_layer_variables_by_name("Conv2D")print("Dense1 layer weights:")print(conv1)print(model.get_weights(conv1[0])) # dense1.W# Start finetuningmodel.fit(X, Y, n_epoch=10, validation_set=0.1, shuffle=True,          show_metric=True, batch_size=64, snapshot_step=200,          snapshot_epoch=False, run_id='model_finetuning')model.save('model_finetuning.tflearn')

会加载cifai10_cnn 中的权重以b，但不会加载最后一个全连接层的权重。

[<tf.Variable 'Conv2D/W:0' shape=(3, 3, 3, 32) dtype=float32_ref>, <tf.Variable 'Conv2D/b:0' shape=(32,) dtype=float32_ref>, <tf.Variable 'Conv2D_1/W:0' shape=(3, 3, 32, 64) dtype=float32_ref>, <tf.Variable 'Conv2D_1/b:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'Conv2D_2/W:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'Conv2D_2/b:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'FullyConnected/W:0' shape=(4096, 512) dtype=float32_ref>, <tf.Variable 'FullyConnected/b:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'FullyConnected_1/W:0' shape=(512, 10) dtype=float32_ref>, <tf.Variable 'FullyConnected_1/b:0' shape=(10,) dtype=float32_ref>]Dense1 layer weights:[<tf.Variable 'Conv2D/W:0' shape=(3, 3, 3, 32) dtype=float32_ref>, <tf.Variable 'Conv2D/b:0' shape=(32,) dtype=float32_ref>][[[[  2.27525160e-01  -8.02579746e-02  -1.49709478e-01   1.95076019e-01     -2.86777616e-01  -1.84236169e-01   8.42581168e-02  -3.03720593e-01      2.49673024e-01  -3.37609112e-01   2.24997491e-01   9.04519483e-02     -3.34136724e-01  -2.22282767e-01   4.01710495e-02   1.44447014e-01     -2.03410551e-01  -5.28300963e-02   8.31742585e-02   3.66900163e-03      2.77024060e-01   7.51862824e-02   2.45897129e-01  -2.23820955e-01      1.32494599e-01  -2.48694703e-01   2.52264559e-01   8.77561271e-02     -2.69655794e-01   7.40043744e-02   3.05063482e-02  -8.41584057e-02]...在格式上要和已经训练的模型进行对应， 也不能在新的模型中显式定义name，要不然找不到同一个variable

具体代码参考: http://tflearn.org/getting_started/#fine-tuning
cifar10_cnn 模型文件: http://download.csdn.net/download/u010651137/10153505
可以直接下载。