Neural Turing Machine Introduced by Alex Graves, Greg Wayne, and Ivo Danihelka
In October 2014, Alex Graves, Greg Wayne, and Ivo Danihelka at Google DeepMind published 'Neural Turing Machines', a preprint proposing a neural network architecture augmented with an external memory matrix and differentiable read/write operations, enabling the system to learn algorithms such as sorting and copying from examples alone.
Background
Neural networks in the early 2010s were good at recognising patterns, but they struggled with tasks that require following a sequence of steps. A recurrent neural network, which processes inputs one at a time and passes a summary of what it has seen forward through time, could handle short sequences reasonably well. Ask it to copy a short list or sort a handful of items, though, and it would start to fail. The problem was memory. Everything the network knew had to fit inside its hidden state, a fixed-size vector that got updated at each step. There was no way to write something down and come back to it later.
This is where classical computers have always had an advantage. A Turing machine, the theoretical model that underlies all digital computing, can read from and write to an unbounded tape. It can store intermediate results, loop back, and execute multi-step procedures exactly. Neural networks had nothing equivalent. Researchers could hand-code algorithms, or they could train a network to approximate a behaviour from examples. They could not, at that point, do both at once: train a network from examples so that it learned to run an algorithm by itself.
What happened
In October 2014, Alex Graves, Greg Wayne, and Ivo Danihelka, all working at Google DeepMind, posted a preprint to arXiv describing an architecture they called the Neural Turing Machine. The core idea was to give a neural network access to an external memory matrix, a separate bank of storage that the network could read from and write to during processing. Crucially, those read and write operations were differentiable, meaning gradients could flow back through them during training. The whole system, controller and memory together, could be trained end-to-end using standard gradient descent.
The controller used attention to decide where in memory to look and where to write. Attention here means a mechanism that produces a weighted distribution across memory locations rather than pointing at a single slot. The network learned, from examples alone, how to allocate and retrieve information across those locations. Graves, Wayne, and Danihelka tested the architecture on tasks including copying a sequence, recalling a sequence in sorted order, and associative recall. On the copy task, the NTM generalised to sequences longer than any it had seen during training, something a standard recurrent network could not do.
The paper was a preprint, not a peer-reviewed publication, and the experiments were small in scale by later standards. The point was not to build a production system. It was to show that a single architecture, trained purely from input-output examples, could learn procedures that had previously required a programmer to write them out explicitly. The 2016 Differentiable Neural Computer, published in Nature by a broader DeepMind team, extended this work with a more capable memory-addressing scheme and larger experiments, but the underlying idea of differentiable external memory traces directly back to this October 2014 paper.
Why it mattered
Neural Turing Machines demonstrated that a neural network could be coupled to an addressable external memory and trained end-to-end via gradient descent, moving beyond the fixed-capacity hidden state of standard recurrent networks. This showed in principle that neural architectures could learn to execute multi-step algorithmic procedures, a capability previously associated only with symbolic or hand-programmed systems. The work directly influenced subsequent memory-augmented architectures, including the Differentiable Neural Computer published by DeepMind in 2016.
People
Alex Graves, Greg Wayne, Ivo Danihelka
Organisations
Sources
- Neural Turing Machines.arXiv (Cornell University).Primary source
- Hybrid computing using a neural network with dynamic external memory.Nature.Secondary
Cite this page
AI Achievements. (2014). Neural Turing Machine Introduced by Alex Graves, Greg Wayne, and Ivo Danihelka. Retrieved 2026-08-22, from https://achievements.ai/milestone/neural-turing-machine-alex-ggrave
@misc{achievements_neural_turing_machine_alex_ggrave,
title = {Neural Turing Machine Introduced by Alex Graves, Greg Wayne, and Ivo Danihelka},
author = {{AI Achievements}},
year = {2014},
url = {https://achievements.ai/milestone/neural-turing-machine-alex-ggrave}
}