Long Short-Term Memory Introduced by Sepp Hochreiter and Jürgen Schmidhuber
In 1997, Sepp Hochreiter at Technische Universität München and Jürgen Schmidhuber at IDSIA published 'Long Short-Term Memory' in Neural Computation, introducing a recurrent neural network architecture with gated memory cells that could learn dependencies across long sequences without suffering from the vanishing gradient problem.
Background
Recurrent neural networks, or RNNs, were the standard tool for learning from sequences in the early 1990s. The idea was straightforward: pass a signal through the network one step at a time, keeping a kind of running memory that gets updated at each step. In principle, that memory could carry information from early in a sequence to the end of it.
In practice, it rarely did. When a network learns, it adjusts its weights by sending an error signal backwards through the steps it just processed, a procedure called backpropagation through time. The further back that signal had to travel, the more it either shrank toward zero or grew out of control. After more than about ten timesteps, gradients that had shrunk to nothing told the weights nothing useful. Sepp Hochreiter had analysed this mathematically in his 1991 diploma thesis, showing that the problem was not a quirk of any particular network but a consequence of the way gradients compound when you multiply them repeatedly through time.
That analysis left the field with a precise description of the problem and no agreed solution. Longer sequences, the kind that appear in real language or speech, remained out of reach.
What happened
Hochreiter, then at Technische Universität München, worked with Jürgen Schmidhuber at IDSIA in Lugano to turn that diagnosis into a fix. Their answer was published in Neural Computation, volume 9, issue 8, in 1997.
The central idea was a memory cell with a controlled internal loop. Instead of letting a signal decay or explode as it passed back through time, the cell kept a constant error flowing inside itself unless a gate explicitly said otherwise. Two gates managed the traffic: an input gate decided what new information could enter the cell, and an output gate decided what the cell would pass on at each step. (The forget gate familiar from later descriptions was not part of this original design; Felix Gers, Schmidhuber, and Fred Cummins introduced that in 1999.) Because the error signal inside a cell could persist without decaying, the network could, in principle, learn which events from hundreds of steps back were worth remembering.
The paper showed this working on synthetic tasks designed to require exactly the kind of long-range memory that standard RNNs could not manage, sequences with gaps of 1,000 timesteps between a relevant signal and the moment the network needed to use it. No previous recurrent architecture had handled gaps on that scale. The architecture was validated further in later work: the 2015 analysis by Klaus Greff and colleagues examined eight LSTM variants across several tasks and found the gated cell design consistently effective, with the input and output gates carrying most of the weight.
Why it mattered
Recurrent neural networks had long struggled to propagate gradient signals across more than a handful of timesteps, a limitation that made learning long-range dependencies in sequential data practically impossible. LSTM's gated cell design allowed gradients to flow over hundreds of timesteps, unlocking practical sequence modelling for speech recognition, machine translation, and language modelling. It became one of the most widely deployed neural architectures of the following two decades, underpinning commercial systems at Google, Apple, and elsewhere before attention-based Transformers began displacing it after 2017.
People
Sepp Hochreiter, Jürgen Schmidhuber
Organisations
Technische Universitaet Muenchen, Dalle Molle Institute for Artificial Intelligence Research
Sources
- Long Short-Term Memory.MIT Press / Neural Computation, Vol. 9 No. 8.Primary source
- Learning Precise Timing with LSTM Recurrent Networks.Journal of Machine Learning Research.Secondary
- An Empirical Evaluation of Recurrent Network Architectures (Greff et al., 2015).arXiv / Journal of Machine Learning Research.Secondary
Cite this page
AI Achievements. (1997). Long Short-Term Memory Introduced by Sepp Hochreiter and Jürgen Schmidhuber. Retrieved 2026-08-22, from https://achievements.ai/milestone/long-short-term-memory-sepp-jurgen
@misc{achievements_long_short_term_memory_sepp_jurgen,
title = {Long Short-Term Memory Introduced by Sepp Hochreiter and Jürgen Schmidhuber},
author = {{AI Achievements}},
year = {1997},
url = {https://achievements.ai/milestone/long-short-term-memory-sepp-jurgen}
}