OpenAI released GPT-3 via private beta API
In May–June 2020, OpenAI published the GPT-3 language model in a paper by Tom B. Brown and colleagues, and began distributing private beta API access. GPT-3's 175 billion parameters made it substantially larger than any publicly described language model at the time, enabling strong few-shot performance across diverse language tasks.
Background
By 2019, language models had already shown they could do useful things if you trained them on enough text and then fine-tuned them on a specific task. OpenAI’s GPT-2, released that year, had 1.5 billion parameters, the adjustable internal values a neural network learns during training. It could produce convincing prose and had drawn attention partly because OpenAI initially held back the full model, citing concerns about misuse. Google’s BERT, from 2018, had pushed further on reading comprehension and classification tasks. These were real advances.
The limitation was awkward, though. To get good performance on a new task, you almost always had to fine-tune: take the pre-trained model, show it hundreds or thousands of labelled examples of the specific thing you wanted it to do, and train it further on those. That meant every new application needed its own dataset and its own training run. It also meant the model’s general ability was less interesting than the task-specific version on top of it.
Some researchers suspected that a much larger model, trained on much more text, might start to sidestep this. The idea was that language, at sufficient scale, carries enough structure that a model might learn to do new things from just a few examples shown in the prompt itself, with no weight updates at all. That idea had not been tested anywhere near the scale needed to know if it was right.
What happened
Tom B. Brown and colleagues at OpenAI posted their findings to arXiv on 28 May 2020. The model they described, GPT-3, used the same basic architecture as its predecessors: a Transformer, which processes text by letting every word attend directly to every other word in a sequence rather than reading one token at a time. What changed was the size. GPT-3 had 175 billion parameters, more than a hundred times as many as GPT-2.
The paper tested a specific idea called few-shot learning, where the model is given a handful of worked examples inside the prompt and then asked to complete a new one, with no changes to its weights. Across dozens of tasks, including translation, arithmetic, reading comprehension and simple question answering, GPT-3 performed competitively with fine-tuned models that had been trained specifically for each task. On the SuperGLUE benchmark, a suite of language understanding tests, GPT-3 in the few-shot setting scored 71.8, still below the fine-tuned human baseline of 89.8, but striking given that it had received no task-specific training at all.
OpenAI did not release the model weights publicly. Instead, in June 2020, they began offering access through a private beta API, letting selected developers send text to the model over the internet and get responses back. That was a deliberate choice, and a departure from how research results had typically been shared. People with access quickly found the model could do things that had not been benchmarked in the paper: drafting code, writing in specific styles, answering questions in ways that looked like genuine comprehension. How much of that was comprehension and how much was pattern-matching at enormous scale was, and remained, an open question.
Why it mattered
GPT-3 demonstrated that scaling a language model to 175 billion parameters, trained on hundreds of billions of tokens of text, produced qualitative improvements in few-shot task performance without any task-specific fine-tuning, challenging assumptions about the limits of unsupervised pre-training. Its release as an API product rather than open weights marked a commercial and safety-policy choice that influenced how subsequent large models were distributed. The breadth of emergent capabilities, from code generation to arithmetic reasoning, set the research agenda that led directly to subsequent work on instruction tuning and reinforcement learning from human feedback.
People
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ari Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric J Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei
Organisations
Sources
- Language Models are Few-Shot Learners.arXiv.Primary source
- OpenAI API (announcement blog post).OpenAI.Official
- OpenAI's new language generator GPT-3 is shockingly good, and completely mindless.MIT Technology Review.Secondary
Cite this page
AI Achievements. (2020). OpenAI released GPT-3 via private beta API. Retrieved 2026-08-22, from https://achievements.ai/milestone/openai-released-gpt-3-beta
@misc{achievements_openai_released_gpt_3_beta,
title = {OpenAI released GPT-3 via private beta API},
author = {{AI Achievements}},
year = {2020},
url = {https://achievements.ai/milestone/openai-released-gpt-3-beta}
}