OpenAI Releases GPT-1: Improving Language Understanding by Generative Pre-Training
In June 2018, Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever at OpenAI published 'Improving Language Understanding by Generative Pre-Training', introducing GPT-1, a 117-million-parameter Transformer pretrained on BooksCorpus via unsupervised language modelling and fine-tuned on downstream tasks, outperforming task-specific models on several NLP benchmarks.
Background
Most NLP systems in 2017 and early 2018 were built task by task. If you wanted a model that could answer questions, you trained it on question-answering data. If you wanted one that could recognise whether two sentences contradicted each other, you trained a separate model on examples of that. Each task needed its own labelled dataset, its own architecture tweaks, its own training run. Building and maintaining all of that was expensive, and labelled data was often scarce.
The Transformer, published by Vaswani and colleagues in 2017, had shown that attention-based architectures could outperform recurrent networks on translation. But most researchers were still using Transformers in supervised settings: you had labelled examples, you trained, you evaluated. The idea of first training a large model on raw, unlabelled text and then adapting it to specific tasks had been explored in earlier work, including with word embeddings, but no one had shown it working cleanly at scale with a full Transformer. The assumption, more or less, was that task-specific models trained on task-specific data were the practical way forward.
What happened
In June 2018, Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever at OpenAI published a paper describing what they called generative pre-training. The approach had two stages. First, they trained a 117-million-parameter Transformer on BooksCorpus, a dataset of about 7,000 unpublished books covering a wide range of genres, using a standard language modelling objective: given the words so far, predict the next one. No labels, no human annotation. Just text and the model learning to continue it.
The second stage was fine-tuning. For each downstream task, they took that pre-trained model and trained it a little further on the task’s labelled data, adjusting only the final layer to fit the new output format. The core weights, already shaped by exposure to millions of sentences, stayed largely intact. The paper tested this on twelve NLP benchmarks across four categories: natural language inference (working out whether one sentence follows from, contradicts, or is neutral with respect to another), question answering, semantic similarity, and text classification. GPT-1 outperformed the previous best results on nine of those twelve tasks.
On the GLUE benchmark, a suite of natural language understanding tasks, the model scored 72.8. On the Stanford Question Answering Dataset it reached 59.1% exact match. These were not marginal improvements on some tasks. The model had never seen task-specific training data in the same volume that specialist models had, and it still won. What the paper showed, concretely, was that a single model trained on enough raw text could learn something general enough to be useful almost anywhere, with relatively little additional training on top.
Why it mattered
GPT-1 demonstrated that a single large language model, pretrained on unlabelled text and then lightly fine-tuned, could surpass models trained from scratch on labelled data across a range of language understanding tasks. This validated the pretraining–fine-tuning paradigm as a practical strategy for natural language processing, shifting research attention away from task-specific architectures. It directly seeded the GPT-2 and GPT-3 work that followed, and the broader wave of large language models that defined the field through the early 2020s.
People
Alec Radford, Karthik Rajagopal Narasimhan, Tim Salimans, Ilya Sutskever
Organisations
Sources
- Improving Language Understanding by Generative Pre-Training.OpenAI.Primary source
- Improving Language Understanding by Generative Pre-Training (OpenAI blog announcement).OpenAI.Official
- Improving Language Understanding by Generative Pre-Training (mirror hosted by OpenAI on S3).OpenAI.Secondary
Cite this page
AI Achievements. (2018). OpenAI Releases GPT-1: Improving Language Understanding by Generative Pre-Training. Retrieved 2026-08-22, from https://achievements.ai/milestone/openai-generative-pre-trained-transformer
@misc{achievements_openai_generative_pre_trained_transformer,
title = {OpenAI Releases GPT-1: Improving Language Understanding by Generative Pre-Training},
author = {{AI Achievements}},
year = {2018},
url = {https://achievements.ai/milestone/openai-generative-pre-trained-transformer}
}