11 October 2018

BERT introduces masked bidirectional pre-training for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova introduced BERT, a language model that pre-trains on both left and right context simultaneously, achieving new best results on eleven natural language processing tasks.

Language understanding Transformers and attention Deep learning Self-supervised learning Foundational method Independently validated

Background

Language models learn to represent words and sentences in a form computers can work with. The standard workflow also demanded a great deal of task-specific engineering. To get a model working well on, say, question answering, researchers typically had to design a specialised architecture for that task from scratch. Pre-training a general model and then adapting it was possible but limited.

What happened

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova introduced BERT, which stands for Bidirectional Encoder Representations from Transformers. The paper was submitted to arXiv on 11 October 2018 and later published at NAACL 2019, where it received the Best Long Paper award.

The key idea is in the name. BERT is designed to pre-train representations that jointly condition on full left and right context at every layer simultaneously, unlike prior bidirectional models which concatenated separate directional passes. Every layer of the model sees the whole sentence at once, in both directions.

Once pre-trained, the model needs only one additional output layer to be fine-tuned for a specific task. Minimal task-specific architecture changes are required compared to prior approaches. That made it unusually general.

The results on eleven natural language processing tasks were striking. BERT pushed the GLUE score, a broad measure of language understanding across several tasks, to 80.5%, a 7.7 point absolute improvement over the previous best. On MultiNLI, accuracy reached 86.7%, up 4.6 points. On the SQuAD v1.1 reading comprehension benchmark, the Test F1 score reached 93.2, a 1.5 point gain, and on the harder SQuAD v2.0 the Test F1 reached 83.1, up 5.1 points.

The paper’s own description is apt: BERT is conceptually simple and empirically powerful. Like prior pre-training methods, BERT does not require labelled data for the pre-training phase, and the fine-tuning step was lightweight enough to make the approach practical across a wide range of tasks.

Why it mattered

BERT's bidirectional approach meant a single pre-trained model could be fine-tuned with just one additional output layer for a wide range of tasks, with minimal task-specific architecture changes compared to prior approaches. It set new records on question answering and language inference benchmarks by margins large enough to be noticed across the field.

Sources

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arxiv.org. Primary source
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. aclanthology.org. Primary source

Cite this page

AI Achievements. (2018). BERT introduces masked bidirectional pre-training for language understanding. Retrieved 2026-08-29, from https://achievements.ai/milestone/bert-introduces-masked-bidirectional-pre-training-for

@misc{achievements_bert_introduces_masked_bidirectional_pre_training_for,
  title  = {BERT introduces masked bidirectional pre-training for language understanding},
  author = {{AI Achievements}},
  year   = {2018},
  url    = {https://achievements.ai/milestone/bert-introduces-masked-bidirectional-pre-training-for}
}