Bahdanau, Cho and Bengio introduce soft attention for neural machine translation
Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio proposed an attention mechanism letting neural translation models search source sentences dynamically, rather than compressing everything into a single fixed-length vector, achieving performance comparable to phrase-based systems on English-to-French translation.
Background
By 2014, neural machine translation was a live research area but still young. The dominant approach used an encoder-decoder architecture: one network compressed a source sentence into a single fixed-length vector, and a second network read that vector to produce a translation. The compression step was the problem. Every word of a long sentence had to be squeezed into the same amount of space, and information got lost.
Phrase-based statistical translation, which did not work this way, was still the benchmark to beat. Neural approaches were promising but had not yet caught up.
What happened
Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio submitted a paper to arXiv on 1 September 2014 arguing that the fixed-length vector was not just a practical inconvenience but a fundamental bottleneck. Their proposal was to let the decoder look back at the entire source sequence at each step, weighting different source positions according to how relevant they seemed for the word currently being generated. They called this a soft-search: the model did not have to commit to a hard segment of the source, but could spread its attention across positions.
The mechanism works by having the decoder learn, at each output step, which parts of the encoded source matter most. That produces a context vector tailored to each target word rather than one fixed vector shared across the whole translation. The paper describes this as allowing a model to “automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly”.
On English-to-French translation the approach achieved performance comparable to the existing state-of-the-art phrase-based systems. That was a meaningful result: neural translation had been trailing phrase-based methods, and this closed much of the gap. A qualitative inspection of selected examples also showed that the soft-alignments the model produced agreed with human intuition about which source words should correspond to which target words.
The paper was accepted at ICLR 2015 as an oral presentation.
Why it mattered
The paper identified the fixed-length encoder vector as a bottleneck and showed that letting a model soft-search the source sentence while decoding could match phrase-based systems on English-to-French translation. The soft-alignments the model produced agreed with human intuitions about which source words correspond to which target words on inspected examples, suggesting the approach was learning something linguistically real.
People
Sources
- Neural Machine Translation by Jointly Learning to Align and Translate. arxiv.org. Primary source
- Neural Machine Translation by Jointly Learning to Align and Translate. arxiv.org. Primary source
Cite this page
AI Achievements. (2014). Bahdanau, Cho and Bengio introduce soft attention for neural machine translation. Retrieved 2026-08-29, from https://achievements.ai/milestone/bahdanau-cho-and-bengio-introduce-soft-attention-for-neural
@misc{achievements_bahdanau_cho_and_bengio_introduce_soft_attention_for_neural,
title = {Bahdanau, Cho and Bengio introduce soft attention for neural machine translation},
author = {{AI Achievements}},
year = {2014},
url = {https://achievements.ai/milestone/bahdanau-cho-and-bengio-introduce-soft-attention-for-neural}
}