Process supervision outperforms outcome supervision for LLM reasoning

Researchers showed that giving a model step-by-step feedback on its reasoning, rather than judging only the final answer, raised the solve rate to 78% on a representative subset of the MATH benchmark, and released 800,000 human feedback labels to support further work.

Background

Large language models had become much better at tasks that require several steps of reasoning, but they still made logical mistakes regularly. The core problem was knowing when and how to correct them. Two broad strategies existed: outcome supervision, which scores only the final answer, and process supervision, which scores each intermediate step. Outcome supervision is simpler to collect because you just check whether the answer is right. Process supervision requires a human to judge every step along the way, which takes considerably more effort.

Some earlier work had started comparing the two approaches, but prior comparisons had been limited in scale or had used easier benchmarks. The MATH dataset, a collection of competition-style mathematics problems, was an established test bed for this kind of reasoning, and it was hard enough that even strong models struggled with it.

What happened

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe ran a direct comparison of the two supervision methods, training reward models on human feedback and using those reward models to guide a language model through mathematical problem-solving.

Process supervision outperformed outcome supervision in this evaluation. Their process-supervised model solved 78% of problems from a representative subset of the MATH test set. They also found that active learning, where the model’s own uncertainty guides which examples humans are asked to label next, made process supervision more effective still, improving the return on each labelling session.

To make the findings reproducible, the team released PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train their best reward model.

Why it mattered

The result gave the field a concrete method for catching logical mistakes mid-reasoning rather than only at the end, addressing a known failure mode in even the best models at the time. The team also released PRM800K, the full set of 800,000 step-level human labels, alongside their findings. The finding that active learning improves process supervision further reduced the cost of applying the method.

People

John Schulman, Ilya Sutskever

Sources

• Let's Verify Step by Step. arxiv.org. Primary source
• Let's Verify Step by Step. doi.org. Primary source

Cite this page

AI Achievements. (2023). Process supervision outperforms outcome supervision for LLM reasoning. Retrieved 2026-08-29, from https://achievements.ai/milestone/process-supervision-outperforms-outcome-supervision-for-llm

@misc{achievements_process_supervision_outperforms_outcome_supervision_for_llm,
  title  = {Process supervision outperforms outcome supervision for LLM reasoning},
  author = {{AI Achievements}},
  year   = {2023},
  url    = {https://achievements.ai/milestone/process-supervision-outperforms-outcome-supervision-for-llm}
}