GPT-3
175-billion-parameter language model that demonstrated strong text generation and reasoning across many tasks with minimal fine-tuning.
5 milestones
LLaMA matches leading models on public data with far fewer parameters
LLaMA, a collection of foundation language models from 7B to 65B parameters trained exclusively on publicly available data, was submitted to arXiv on 27 February 2023. The 13B model outperformed GPT-3 at 175B parameters on most benchmarks, and the weights were released to the research community.
Self-Instruct: language models taught to follow instructions using synthetic self-generated data
Yizhong Wang and colleagues introduced Self-Instruct, a method for training language models to follow instructions using synthetic data the models generate themselves, closing much of the gap with InstructGPT-001 on evaluated tasks while using far less human annotation.
InstructGPT: aligning language models with human feedback at scale
Researchers showed that fine-tuning GPT-3 with human feedback produced a 1.3B parameter model whose outputs labellers preferred over those of the 175B GPT-3, pointing toward a practical method for aligning language models more closely with expressed human preferences.
Instruction tuning lets a 137B model match or beat GPT-3 zero-shot
Jason Wei and colleagues showed that finetuning a 137B language model on over 60 NLP tasks described via natural language instruction templates produced a model, FLAN, that beat zero-shot GPT-3 on 20 of 25 tasks and surpassed few-shot GPT-3 on several benchmarks.
OpenAI released GPT-3 via private beta API
In May–June 2020, OpenAI published the GPT-3 language model in a paper by Tom B. Brown and colleagues, and began distributing private beta API access. GPT-3's 175 billion parameters made it substantially larger than any publicly described language model at the time, enabling strong few-shot performance across diverse language tasks.