Techniques for preventing overfitting in training Deep Neural Networks

Data

Data Augmentation

• data cropping
• adding noise to data
• affine warping to data.

Model Regularization

Dropout

Consider deep neural networks that contain a series of hidden layers h = {h1,..., hL}, and predict output y given input x. Dropout is a technique that trains an ensemble of models consisting a subset of variables in h, or even x. On each presentation of a training sample, we train a different sub-model taking a random subset of variable h. Technically speaking, we randomly set hidden variables to zeros according to some probabilities. Thus each sample sees a different model to prevent co-adaptation of features.

Dropout training is similar to bagging, where many different models are trained on different subsets of the data. Dropout differs from bagging in that each model is trained for only one step and all the models share parameters.

• G. E. Hinton. Improving neural networks by preventing co-adaptation of feature detectors

Maxout

Maxout is also a technique for model-averaging, which a natural companion to dropout. Maxout takes the max of a set of inputs. It's particular applicable for convolutional neural networks. In a convolutional neural network , a maxout feature map can be constructed by taking the maximum across k affine feature maps (i.e. pool across channels).

• Ian J. Goodfellow, Yoshua Bengio. Maxout Networks

Drop connection

DropConnect is the generalization of Dropout in which each connection, rather than each output unit, can be dropped with probability. DropConnect is similar to Dropout as it introduces dynamic sparsity within the model, but differs in that the sparsity is on the weights W, rather than the output vectors of a layer. In other words, the fully connected layer with DropConnect becomes a sparsely connected layer in which the connections are chosen at random during the training stage.

• Li Wan. Regularization of Neural Networks using DropConnect