SGD求解器的学习率a和遗忘因子u的设置原则

SGD

Stochastic gradient descent (type: "SGD") updates the weights WW by a linear combination of the negative gradient ∇L(W)∇L(W) and the previous weight update VtVt. The learning rate αα is the weight of the negative gradient. The momentum μμ is the weight of the previous update.

Formally, we have the following formulas to compute the update value Vt+1Vt+1 and the updated weights Wt+1Wt+1 at iteration t+1t+1, given the previous weight update VtVt and current weights WtWt:

Vt+1=μVt−α∇L(Wt)Vt+1=μVt−α∇L(Wt)

Wt+1=Wt+Vt+1Wt+1=Wt+Vt+1

The learning “hyperparameters” (αα and μμ) might require a bit of tuning for best results. If you’re not sure where to start, take a look at the “Rules of thumb” below, and for further information you might refer to Leon Bottou’s Stochastic Gradient Descent Tricks [1].

[1] L. Bottou. Stochastic Gradient Descent Tricks. Neural Networks: Tricks of the Trade: Springer, 2012.

Rules of thumb for setting the learning rate 阿法α

和momentum   miu

A good strategy for deep learning with SGD is to initialize the learning rate αα to a value around α≈0.01=10−2α≈0.01=10−2, and dropping it by a constant factor (e.g., 10) throughout training when the loss begins to reach an apparent “plateau”, repeating this several times. Generally, you probably want to use a momentum μ=0.9μ=0.9 or similar value. By smoothing the weight updates across iterations, momentum tends to make deep learning with SGD both stabler and faster.

This was the strategy used by Krizhevsky et al. [1] in their famously winning CNN entry to the ILSVRC-2012 competition, and Caffe makes this strategy easy to implement in a SolverParameter, as in our reproduction of [1] at ./examples/imagenet/alexnet_solver.prototxt.

To use a learning rate policy like this, you can put the following lines somewhere in your solver prototxt file:

base_lr: 0.01     # begin training at a learning rate of 0.01 = 1e-2lr_policy: "step" # learning rate policy: drop the learning rate in "steps"                  # by a factor of gamma every stepsize iterationsgamma: 0.1        # drop the learning rate by a factor of 10                  # (i.e., multiply it by a factor of gamma = 0.1)stepsize: 100000  # drop the learning rate every 100K iterationsmax_iter: 350000  # train for 350K iterations totalmomentum: 0.9

Under the above settings, we’ll always use momentum μ=0.9μ=0.9. We’ll begin training at a base_lr of α=0.01=10−2α=0.01=10−2 for the first 100,000 iterations, then multiply the learning rate by gamma (γγ) and train at α′=αγ=(0.01)(0.1)=0.001=10−3α′=αγ=(0.01)(0.1)=0.001=10−3 for iterations 100K-200K, then at α′′=10−4α″=10−4 for iterations 200K-300K, and finally train until iteration 350K (since we havemax_iter: 350000) at α′′′=10−5α‴=10−5.

Note that the momentum setting μμ effectively multiplies the size of your updates by a factor of 11−μ11−μ after many iterations of training, so if you increase μμ, it may be a good idea to decrease ααaccordingly (and vice versa).

For example, with μ=0.9μ=0.9, we have an effective update size multiplier of 11−0.9=1011−0.9=10. If we increased the momentum to μ=0.99μ=0.99, we’ve increased our update size multiplier to 100, so we should drop αα (base_lr) by a factor of 10.

Note also that the above settings are merely guidelines, and they’re definitely not guaranteed to be optimal (or even work at all!) in every situation. If learning diverges (e.g., you start to see very large or NaN or inf loss values or outputs), try dropping the base_lr (e.g., base_lr: 0.001) and re-training, repeating this until you find a base_lr value that works.