Instruction tuning lets a 137B model match or beat GPT-3 zero-shot

Jason Wei and colleagues showed that finetuning a 137B language model on over 60 NLP tasks described via natural language instruction templates produced a model, FLAN, that beat zero-shot GPT-3 on 20 of 25 tasks and surpassed few-shot GPT-3 on several benchmarks.

Background

By 2021, large language models like GPT-3 had shown impressive abilities when given a few worked examples at inference time, a technique called few-shot prompting. Zero-shot performance, where the model gets no examples at all, was considerably weaker. The gap mattered: in real use, you often cannot supply examples, and a model that needs them is harder to deploy broadly.

The standard assumption was that more parameters and more pretraining data were the main levers for improving zero-shot ability. Finetuning on labelled task data was common, but it tended to specialise a model rather than generalise it.

What happened

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le submitted their paper to arXiv on 3 September 2021. They took a 137B parameter pretrained language model and finetuned it on over 60 NLP tasks, each described using natural language instruction templates. The resulting model they called FLAN, short for Finetuned Language Model.

The key idea was that if every task during finetuning is described in words rather than just shown through examples, the model may learn to follow instructions in general, not just solve specific tasks. At evaluation time, FLAN was tested on task types it had never seen during finetuning.

The results were striking. FLAN surpassed zero-shot 175B GPT-3 on 20 of 25 tasks evaluated. On several benchmarks, ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze, it outperformed few-shot GPT-3 by a substantial margin on those specific benchmarks. That is notable because few-shot GPT-3 is a stronger baseline: it gets worked examples at inference time, while FLAN gets none.

Ablation studies, experiments where individual components are removed to see what matters, showed that three things were key to the success of instruction tuning: the number of finetuning datasets, the scale of the model, and the use of natural language instructions.

Why it mattered

The paper provided an early demonstration that instruction tuning can improve zero-shot performance on unseen tasks, without changing the underlying model architecture. FLAN's results showed that describing tasks in natural language instructions during finetuning was a key factor, alongside model scale and the number of finetuning datasets.

People

Andrew M Dai, Quoc V Le

Sources

• Finetuned Language Models Are Zero-Shot Learners. arxiv.org. Primary source
• Finetuned Language Models Are Zero-Shot Learners. arxiv.org. Primary source

Cite this page

AI Achievements. (2021). Instruction tuning lets a 137B model match or beat GPT-3 zero-shot. Retrieved 2026-08-29, from https://achievements.ai/milestone/instruction-tuning-lets-a-137b-model-match-or-beat-gpt-3

@misc{achievements_instruction_tuning_lets_a_137b_model_match_or_beat_gpt_3,
  title  = {Instruction tuning lets a 137B model match or beat GPT-3 zero-shot},
  author = {{AI Achievements}},
  year   = {2021},
  url    = {https://achievements.ai/milestone/instruction-tuning-lets-a-137b-model-match-or-beat-gpt-3}
}