Dropout introduced as a regularisation method for neural networks

Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov introduced dropout, a method that randomly omits units during training to stop neural networks from overfitting. It set new records on several specific speech and object recognition benchmarks at the time of publication.

Background

Large neural networks can be very powerful, but they have a persistent weakness: when trained on a limited amount of data, they tend to overfit. That means the network learns the training examples too well and fails to generalise to new ones. The standard way to fight this was to combine predictions from many different networks at test time, but that is slow and expensive when each network is already large.

The deeper cause is something researchers call co-adaptation. A neuron can become useful only in the presence of several other specific neurons, rather than learning something independently helpful. When neurons lean on each other like that, the network memorises patterns in the training data instead of learning the underlying structure.

What happened

Geoffrey Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov put forward a technique they called dropout. The idea is simple: during training, randomly omit half of the feature detectors on each training case. Each neuron must then learn to detect something generally useful across a large variety of internal contexts, because it cannot count on any particular set of neighbours being present.

At test time there is no need to run many different thinned networks. A single network with smaller weights approximates the average of all the thinned versions, keeping the cost of prediction low.

The approach prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. By breaking up those dependencies, dropout pushes each unit to become more independently useful.

The results were strong across several domains. Dropout gave big improvements on many benchmark tasks and set new records on several specific speech and object recognition benchmarks at the time of publication. The 2014 journal paper in the Journal of Machine Learning Research later confirmed improvements on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on several benchmark data sets at the time of the 2014 paper.

Why it mattered

Overfitting was a serious obstacle for large neural networks trained on limited data, and dropout gave a practical way to address it with low additional cost at test time, though training takes longer. By randomly omitting feature detectors during training, each neuron was forced to learn something useful on its own rather than relying on neighbours. The method produced big improvements on many benchmark tasks and set new records on several specific speech and object recognition benchmarks at the time of publication.

People

Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever

Sources

• Dropout: A Simple Way to Prevent Neural Networks from Overfitting. jmlr.org. Primary source
• Improving neural networks by preventing co-adaptation of feature detectors. arxiv.org. Primary source

Cite this page

AI Achievements. (2012). Dropout introduced as a regularisation method for neural networks. Retrieved 2026-08-29, from https://achievements.ai/milestone/dropout-introduced-as-a-regularisation-method-for-neural

@misc{achievements_dropout_introduced_as_a_regularisation_method_for_neural,
  title  = {Dropout introduced as a regularisation method for neural networks},
  author = {{AI Achievements}},
  year   = {2012},
  url    = {https://achievements.ai/milestone/dropout-introduced-as-a-regularisation-method-for-neural}
}