LeNet Tutorial with Caffe

刚看完Caffe官网example中的LeNet Tutorial

首先，搞清楚LeNet的结构，很简单，就是conv+pool+conv+pool+fc+fc

In general, it consists of a convolutional layer followed by a pooling layer, another convolution layer followed by a pooling layer, and then two fully connected layers

LeNet model定义在文件lenet_train_test.prototxt中，相比原始论文做的一个改动就是，把sigmoid换成了ReLU。

然后开始教怎么定义model了

We assume that you are familiar with Google Protobuf, and assume that you have read the protobuf definitions used by Caffe, which can be found at $CAFFE_ROOT/src/caffe/proto/caffe.proto…看来，Google Protobuf和caffe.proto还是有学习的必要的，有空了要看看。

Step 1: 先写一个Data Layer

直接上图，没什么好说的。

Step 2: 写一个卷积层

几个要注意的地方：

• For the weight filler, we will use the xavier algorithm that automatically determines the scale of initialization based on the number of input and output neurons
• For the bias filler, we will simply initialize it as constant, with the default filling value 0.
• lr_mults are the learning rate adjustments for the layer’s learnable parameters.( In this case, we will set the weight learning rate to be the same as the learning rate given by the solver during runtime, and the bias learning rate to be twice as large as that - this usually leads to better convergence rates.)

Step 3: 定义pooling layer

Step 4: 定义全连接层

Step 5: ReLU

这里有个地方要注意，就是由于经过ReLU非线性转换之后，输出可以直接替换输入（占用原先输入的内存空间），所以这里bottom和top都是用的相同的名字ip1

ReLU之后，再来一层fc

Step 6: 最后来一层loss layer

---

还可以添加一些layer rule（参见caffe.proto）

上图添加的是layer的作用阶段（只在train的时候存在）

Step 7: Solver的定义也很重要（定义在lenet_solver.prototxt中）

---

大功告成！ 然后就可以开始train和test模型了。

输出的信息，根据Solver的定义：

Based on the solver setting, we will print the training loss function every 100 iterations, and test the network every 500 iterations. You will see messages like this:

I1203 solver.cpp:204] Iteration 100, lr = 0.00992565
I1203 solver.cpp:66] Iteration 100, loss = 0.26044
...
I1203 solver.cpp:84] Testing net
I1203 solver.cpp:111] Test score #0: 0.9785
I1203 solver.cpp:111] Test score #1: 0.0606671

其中，For the output of the testing phase, score 0 is the accuracy, and score 1 is the testing loss function.

PS:

MNIST is a small dataset, so training with GPU does not really introduce too much benefit due to communication overheads. On larger datasets with more complex models, such as ImageNet, the computation speed difference will be more significant.