ACHIEVEMENTS.AI

Direct Preference Optimisation reduces need for RL in LM alignment

Rafael Rafailov and colleagues introduced Direct Preference Optimization (DPO), a method that aligns language models with human preferences using only a simple classification loss, bypassing the complex reinforcement learning pipeline that existing approaches required.

Learning methods Language modelsReinforcement learningMachine learning Foundational method Demonstrated

Background

Getting a language model to do what you want is harder than it sounds. Large models trained on text learn a great deal, but their behaviour is shaped entirely by that training data, with no mechanism to steer them toward outputs humans actually prefer. The standard fix, by 2023, was reinforcement learning from human feedback, or RLHF. People would compare pairs of model outputs, label which they preferred, and those labels would train a separate reward model. That reward model would then guide the language model using reinforcement learning.

The procedure worked, but it was cumbersome. Two models had to be trained and kept in balance. The reinforcement learning step was sensitive to hyperparameters and prone to instability, and the need to sample from the language model during fine-tuning added further burden to the process.

What happened

On 29 May 2023, Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning and Chelsea Finn posted a paper to arXiv introducing Direct Preference Optimization. The key insight was mathematical: they found a new way to parameterise the reward model in RLHF that allowed the optimal policy to be extracted in closed form. That meant the whole problem could be recast as a simple classification loss applied directly to the language model, with no reinforcement learning required at all.

The resulting algorithm, which they called DPO, eliminated the need to sample from the language model during fine-tuning and removed the requirement for significant hyperparameter tuning. The authors described it as stable, performant and computationally lightweight.

On the tasks they tested, DPO held up well against the established approach. On the sentiment-control benchmark tested, DPO exceeded PPO-based RLHF in controlling the sentiment of generated text. On summarisation and single-turn dialogue it matched or improved response quality. The method achieved this while being, in the authors’ words, substantially simpler to implement and train.

The paper formalised something that had not been obvious before: that a language model already encodes enough information to act as its own reward model. The title put it plainly: the language model is secretly a reward model.

Why it mattered

RLHF, the dominant alignment method at the time, required fitting a separate reward model and then running reinforcement learning on top of a large language model, a process the authors described as complex and often unstable. DPO showed that the optimal policy can be expressed analytically, recasting the problem as a supervised loss and making alignment substantially simpler to implement and train. In head-to-head tests it matched or exceeded PPO-based RLHF on sentiment control, summarisation and dialogue.

Sources

Cite this page

AI Achievements. (2023). Direct Preference Optimisation reduces need for RL in LM alignment. Retrieved 2026-08-29, from https://achievements.ai/milestone/direct-preference-optimisation-reduces-need-for-rl-in-lm 

@misc{achievements_direct_preference_optimisation_reduces_need_for_rl_in_lm,
  title  = {Direct Preference Optimisation reduces need for RL in LM alignment},
  author = {{AI Achievements}},
  year   = {2023},
  url    = {https://achievements.ai/milestone/direct-preference-optimisation-reduces-need-for-rl-in-lm}
}