12 April 2022

Anthropic paper applies RLHF to train helpful and harmless assistants

Researchers applied preference modelling and reinforcement learning from human feedback to fine-tune language models as assistants, finding that alignment training improved performance on almost all NLP evaluations tested, without measurable loss of specialised skills in the models studied.

Alignment and safety Reinforcement learning Language models Machine learning Foundational method Demonstrated

Background

By early 2022, large language models could produce fluent text across many tasks, but making them behave well was a separate problem. A model trained purely to predict the next word had no particular reason to be helpful or to avoid harmful outputs. Reinforcement learning from human feedback, where human raters compare pairs of model responses and their preferences are used to shape the model’s behaviour, had been explored as one way to bridge that gap. What was less clear was whether imposing such constraints would blunt a model’s general ability.

The question mattered in practice. A model that was safe but useless would not be deployed, and a model that was capable but harmful would be worse than nothing. Researchers wanted to know whether the two goals could be pursued at the same time.

What happened

Yuntao Bai, Andy Jones, and colleagues submitted a paper to arXiv on 12 April 2022 describing how they applied preference modelling and reinforcement learning from human feedback to fine-tune language models to act as helpful and harmless assistants.

The central finding was that this alignment training improved performance on almost all NLP evaluations in the models studied. Far from trading off capability for safety, the fine-tuned models did better across the board. The training also proved compatible with specialised skills: models trained this way retained abilities in areas such as Python coding and summarisation.

The team also explored an iterated online mode of training. Rather than training once and stopping, preference models and RL policies were updated on a weekly cadence with fresh human feedback data. That cycle let them improve both their datasets and their models continuously over time.

On the technical side, they investigated the robustness of RLHF training and identified a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialisation. KL divergence here measures how far the trained model has moved from the version it started as. Finding a clean empirical relationship between reward and that distance provided a diagnostic for how much the model was changing as training progressed, applicable to similar training runs.

Alongside the main results, the paper included analyses on calibration, competing objectives, and out-of-distribution detection, and compared the models directly with human writers.

Why it mattered

The work showed that aligning a model to be helpful and harmless need not come at the cost of capability: in the models studied, alignment training improved performance on almost all NLP evaluations tested and remained compatible with specialised skills such as coding and summarisation. It also identified a concrete mathematical relationship between the RL reward and KL divergence from the model's starting point, providing an empirical relationship between reward and KL divergence that could serve as a diagnostic for policy drift in similar training runs.

People

Amanda Askell, Tom Henighan, Dario Amodei, Jack Clark, Sam McCandlish, Jared Kaplan

Sources

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arxiv.org. Primary source
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arxiv.org. Primary source

Cite this page

AI Achievements. (2022). Anthropic paper applies RLHF to train helpful and harmless assistants. Retrieved 2026-08-29, from https://achievements.ai/milestone/anthropic-paper-applies-rlhf-to-train-helpful-and-harmless

@misc{achievements_anthropic_paper_applies_rlhf_to_train_helpful_and_harmless,
  title  = {Anthropic paper applies RLHF to train helpful and harmless assistants},
  author = {{AI Achievements}},
  year   = {2022},
  url    = {https://achievements.ai/milestone/anthropic-paper-applies-rlhf-to-train-helpful-and-harmless}
}