22 December 2014

Kingma and Ba present the Adam optimiser

Diederik P. Kingma and Jimmy Ba presented Adam, an optimisation algorithm combining and extending existing adaptive gradient methods, adapting learning rates using bias-corrected estimates of the first and second moments of gradients, making it efficient and broadly practical, particularly where heavy hyperparameter tuning is not feasible.

Learning methods Machine learning Deep learning Neural networks Foundational method Independently validated

Background

Training a neural network means adjusting millions of numbers, called weights, to reduce error on some task. The standard way to do this is gradient descent: measure how much each weight contributes to the error, then nudge it in the right direction. When the training data is large, you cannot compute that measurement over all of it at once, so you use a random subset each time. That is stochastic gradient descent, and it is the engine behind nearly all practical machine learning.

The trouble is choosing how big those nudges should be. Too large and the training overshoots; too small and it crawls. Several earlier algorithms tried to set a different step size for each weight automatically, adapting as training progressed. These helped, but each had weaknesses with noisy gradients, non-stationary objectives, or the memory cost of tracking many per-weight statistics.

What happened

On 22 December 2014, Diederik P. Kingma and Jimmy Ba submitted a paper to arXiv presenting Adam, an algorithm combining and extending existing adaptive gradient methods for first-order gradient-based optimisation of stochastic objective functions. The algorithm tracks two running averages for each weight: the mean of recent gradients, and the mean of their squares. Those are the first and second moments. By combining them, Adam sets a separate, adaptive learning rate for every parameter, without needing much memory to do it.

The paper showed the method is invariant to diagonal rescaling of the gradients, meaning it behaves sensibly even when different weights operate on very different scales. The hyper-parameters, the settings a practitioner has to choose before training, have intuitive interpretations and typically require little tuning. That was a practical advantage: getting a new method to work often demands considerable fiddling, and Adam reduced that burden.

Kingma and Ba also provided a theoretical analysis, giving a regret bound on the convergence rate that is comparable to the best known results under the online convex optimisation framework. Alongside the main algorithm, they introduced AdaMax, a variant of Adam based on the infinity norm.

The paper was later published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015. Empirical results in the paper showed that Adam works well in practice and compares favourably to other stochastic optimisation methods.

Why it mattered

Adam was designed to work well where other methods struggled: with noisy or sparse gradients, non-stationary objectives, and problems large in data or parameters. Its computational efficiency and low memory requirements made it practical to use without heavy tuning, and empirical results in the paper showed it compared favourably to other stochastic optimisation methods.

Sources

Adam: A Method for Stochastic Optimization. arxiv.org. Primary source
Adam: A Method for Stochastic Optimization. arxiv.org. Primary source

Cite this page

AI Achievements. (2014). Kingma and Ba present the Adam optimiser. Retrieved 2026-08-29, from https://achievements.ai/milestone/kingma-and-ba-present-adam-optimiser-combining-adaptive

@misc{achievements_kingma_and_ba_present_adam_optimiser_combining_adaptive,
  title  = {Kingma and Ba present the Adam optimiser},
  author = {{AI Achievements}},
  year   = {2014},
  url    = {https://achievements.ai/milestone/kingma-and-ba-present-adam-optimiser-combining-adaptive}
}