OpenAI Released the Full 1.5-Billion-Parameter GPT-2 Model
In November 2019, OpenAI released the full 1.5-billion-parameter version of GPT-2, completing a staged release the organisation had begun in February 2019 with a smaller variant, citing concerns about potential misuse of a model capable of generating coherent long-form text.
Background
By early 2019, the dominant approach to building language models was supervised learning: collect a large labelled dataset, train a model on it, and evaluate on a matching test set. The problem was that labelling text at scale is expensive, and a model trained on one task tended to perform poorly on others.
Alec Radford, Ilya Sutskever and their colleagues at OpenAI had been working on a different approach. Rather than training on curated labels, they trained on raw text from the web, asking the model to do one thing: predict the next word. The idea was that if a model reads enough text, the structure of language itself becomes the training signal. No labels needed.
The February 2019 paper, co-authored by Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Sutskever, described a model built on the Transformer architecture. A Transformer processes text by letting every word attend directly to every other word in a sequence, rather than reading left to right one token at a time. The version they described in full had 1.5 billion parameters, the individual numerical weights that a model learns during training. OpenAI called it GPT-2.
What happened
OpenAI released the paper and a 117-million-parameter version of GPT-2 in February 2019, but held back the full 1.5-billion-parameter model. The stated reason was concern that a model capable of generating convincing long-form text at scale could make it easier to produce misleading content in bulk. Over the following months they released progressively larger versions: 345 million parameters, then 762 million. The full 1.5-billion-parameter weights followed in November 2019.
The model had been trained on a dataset called WebText, assembled from outbound links on Reddit that had received at least three upvotes. That came to about 40 gigabytes of text. On a reading comprehension benchmark called CoQA, GPT-2 scored 55 F1 without any task-specific training, where F1 measures the overlap between the model’s answers and reference answers. Supervised models trained specifically for that task scored higher, but GPT-2 had never seen the task before. On summarisation and translation it showed the same pattern: reasonable results from a model that had only ever been asked to predict the next word.
The November release included a technical note on what independent researchers had found in the months between the staged releases. The model generated fluent text for short passages, but quality declined over longer outputs. Repetition crept in. Factual errors accumulated. OpenAI described the full release as an opportunity for the research community to study these failure modes directly, alongside the capabilities that had prompted the original caution.
Why it mattered
GPT-2's staged release introduced a new precedent in machine-learning practice: withholding a model's full weights on safety grounds, sparking sustained debate about how AI developers should balance openness with harm prevention. The model demonstrated that large-scale unsupervised pre-training on web text could produce general-purpose language generation of a quality that surprised researchers, accelerating investment in scaling laws and successor models. Its eventual full release allowed independent researchers to study both its capabilities and its tendency to degrade over longer passages, grounding subsequent work on controllable generation.
People
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever
Organisations
Sources
- GPT-2: 1.5B Release.OpenAI.Primary source
- Better Language Models and Their Implications.OpenAI.Primary source
- Language Models are Unsupervised Multitask Learners.OpenAI.Primary source
Cite this page
AI Achievements. (2019). OpenAI Released the Full 1.5-Billion-Parameter GPT-2 Model. Retrieved 2026-08-22, from https://achievements.ai/milestone/gpt-2-with-1-5-billion-parameters
@misc{achievements_gpt_2_with_1_5_billion_parameters,
title = {OpenAI Released the Full 1.5-Billion-Parameter GPT-2 Model},
author = {{AI Achievements}},
year = {2019},
url = {https://achievements.ai/milestone/gpt-2-with-1-5-billion-parameters}
}