Flan-PaLM: instruction finetuning scaled across tasks, model sizes and families

Researchers showed that finetuning large language models on instruction-phrased datasets improves performance across benchmarks, with Flan-PaLM 540B trained on 1.8K tasks scoring 75.2% on five-shot MMLU and outperforming its base model by 9.4% on average.

Background

By 2022, large language models could be prompted with examples to perform tasks they had not been explicitly trained on. The question was how to make that generalisation more reliable. One approach, instruction finetuning, involved taking a pretrained model and training it further on many datasets that had been rephrased as natural-language instructions. What had not yet been studied was what happened when you scaled this approach: more tasks, bigger models, and richer training signals.

Chain-of-thought prompting, where a model is asked to reason step by step before giving an answer, had also shown gains on difficult problems. Whether it could be combined usefully with instruction finetuning was an open question.

What happened

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Jason Wei and colleagues explored instruction finetuning along three axes: the number of tasks used during finetuning, the size of the underlying model, and whether chain-of-thought data was included in training. They applied this across several model families, including PaLM, T5 and U-PaLM.

The headline result came from Flan-PaLM 540B, finetuned on 1.8K tasks. It outperformed the base PaLM 540B model by 9.4% on average across benchmarks. On five-shot MMLU it scored 75.2%. The gains held across zero-shot, few-shot and chain-of-thought prompting setups, and across benchmarks including BBH, TyDiQA and MGSM.

The team also publicly released Flan-T5 checkpoints. These smaller models achieved strong few-shot performance compared to much larger models such as PaLM 62B.

The paper’s conclusion was direct: instruction finetuning improved performance and usability across the multiple model families and benchmark types studied in this paper.

Why it mattered

The paper showed that instruction finetuning improved performance across multiple model families and benchmark types studied, not a trick tied to one architecture. The team also publicly released Flan-T5 checkpoints, which achieved strong few-shot performance even compared to much larger models such as PaLM 62B.

People

Jeff Dean, Quoc V Le

Sources

• Scaling Instruction-Finetuned Language Models. arxiv.org. Primary source
• Scaling Instruction-Finetuned Language Models. doi.org. Primary source

Cite this page

AI Achievements. (2022). Flan-PaLM: instruction finetuning scaled across tasks, model sizes and families. Retrieved 2026-08-29, from https://achievements.ai/milestone/flan-palm-instruction-finetuning-scaled-across-tasks-model

@misc{achievements_flan_palm_instruction_finetuning_scaled_across_tasks_model,
  title  = {Flan-PaLM: instruction finetuning scaled across tasks, model sizes and families},
  author = {{AI Achievements}},
  year   = {2022},
  url    = {https://achievements.ai/milestone/flan-palm-instruction-finetuning-scaled-across-tasks-model}
}