InstructGPT: aligning language models with human feedback at scale

Researchers showed that fine-tuning GPT-3 with human feedback produced a 1.3B parameter model whose outputs labellers preferred over those of the 175B GPT-3, pointing toward a practical method for aligning language models more closely with expressed human preferences.

Background

By the early 2020s, the standard assumption in language model research was that bigger meant better. More parameters, more data, more compute. GPT-3, with 175 billion parameters, was the clearest example of this logic. Yet size had a problem that scaling did not fix.

Large language models could produce outputs that were untruthful, toxic, or simply unhelpful. They were optimised to predict text, not to do what a person asking a question actually needed. As the paper put it, these models were “not aligned with their users.”

What happened

Long Ouyang, Jeff Wu, and colleagues at OpenAI built a training pipeline that started with GPT-3 and reshaped it using human judgement rather than raw text prediction. The process had two stages. First, human labellers wrote examples of good responses to prompts, and the model was fine-tuned on those examples using supervised learning. Then labellers ranked sets of model outputs from best to worst, and that ranking data was used to train a separate reward model. GPT-3 was then further fine-tuned using reinforcement learning from human feedback, with the reward model guiding it toward outputs that humans preferred.

The resulting models were called InstructGPT. The striking result was not just that InstructGPT was better in some abstract sense. In direct human evaluations, outputs from the 1.3B parameter InstructGPT were preferred over outputs from the 175B GPT-3. A model with roughly one hundredth the parameters won, because it had been trained to follow intent rather than to predict text.

InstructGPT also showed improvements in truthfulness and produced less toxic output, while performing comparably to GPT-3 on standard NLP benchmarks. The authors were careful to note that the model still made simple mistakes. Even so, the results made a clear case that fine-tuning with human feedback was “a promising direction for aligning language models with human intent.”

Why it mattered

The paper showed that scale alone does not make a language model more useful or safer, and that reinforcement learning from human feedback could close that gap with far fewer parameters. A 1.3B parameter InstructGPT model was preferred by human evaluators over the 175B GPT-3, while also being more truthful and less toxic. The work described a replicable methodology for aligning large language models more closely with expressed labeller preferences, though full reproduction requires proprietary labeller pipelines.

People

Sandhini Agarwal, John Schulman, Amanda Askell

Organisations

OpenAI

Sources

• Training language models to follow instructions with human feedback. arxiv.org. Primary source
• Training language models to follow instructions with human feedback. arxiv.org. Primary source

Cite this page

AI Achievements. (2022). InstructGPT: aligning language models with human feedback at scale. Retrieved 2026-08-29, from https://achievements.ai/milestone/instructgpt-aligning-language-models-with-human-feedback-at

@misc{achievements_instructgpt_aligning_language_models_with_human_feedback_at,
  title  = {InstructGPT: aligning language models with human feedback at scale},
  author = {{AI Achievements}},
  year   = {2022},
  url    = {https://achievements.ai/milestone/instructgpt-aligning-language-models-with-human-feedback-at}
}