12 June 2017

Vaswani et al. introduce the Transformer, a sequence model built on self-attention alone

Ashish Vaswani and colleagues proposed the Transformer, a sequence model whose encoding and decoding relied on multi-head self-attention rather than recurrence or convolutions. It set new translation benchmarks while training faster than existing models.

Architectures and models Transformers and attention Neural networks Natural language processing Foundational method Independently validated

Background

Sequence modelling, the task of turning one ordered string of tokens into another, had long been dominated by recurrent neural networks. A recurrent network reads a sequence one step at a time, passing a summary of what it has seen so far into the next step. That design is inherently serial: each position has to wait for the one before it, which slows training and makes it hard to spread work across processors.

The best models of the time placed an attention mechanism on top of this recurrent structure, letting the encoder and decoder look back at relevant parts of the input when producing each output word. Attention helped, but recurrence remained at the core, and with it the speed and memory constraints that came along.

What happened

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin submitted a paper to arXiv on 12 June 2017 proposing a new architecture they called the Transformer. Its sequence-mixing mechanism relied solely on attention, replacing recurrence and convolutions, though it retained position-wise feed-forward layers.

Instead of reading a sequence step by step, the Transformer let every position attend directly to every other position in a single pass. Because no step depended on the output of the previous one, the computation could be parallelised across the whole sequence at once. That made training significantly faster.

The paper tested the model on two standard machine translation tasks. On the WMT 2014 English-to-German task it scored 28.4 BLEU, a measure that compares machine translation output against human reference translations, improving over the existing best results including ensembles by over 2 BLEU. On English-to-French it achieved a BLEU score of 41.8 after training for 3.5 days on eight GPUs, which the authors described as a small fraction of the training costs of the best models from the literature. The team also applied the Transformer to English constituency parsing, where it performed well on both large and limited training data, suggesting the design was not narrowly suited to translation.

Why it mattered

The Transformer showed that neither recurrence nor convolutions were necessary for high-quality sequence modelling. By making computation fully parallelisable, it made large-scale training far more practical. On English-to-German translation it scored 28.4 BLEU, improving over the existing best results including ensembles by over 2 BLEU.

Sources

Attention Is All You Need. arxiv.org. Primary source
Attention is All you Need. papers.nips.cc. Primary source

Cite this page

AI Achievements. (2017). Vaswani et al. introduce the Transformer, a sequence model built on self-attention alone. Retrieved 2026-08-29, from https://achievements.ai/milestone/vaswani-et-al-introduce-the-transformer-a-sequence-model

@misc{achievements_vaswani_et_al_introduce_the_transformer_a_sequence_model,
  title  = {Vaswani et al. introduce the Transformer, a sequence model built on self-attention alone},
  author = {{AI Achievements}},
  year   = {2017},
  url    = {https://achievements.ai/milestone/vaswani-et-al-introduce-the-transformer-a-sequence-model}
}