What Is RLHF? How LLMs Are Trained, Step by Step

What is RLHF? Reinforcement learning from human feedback is the training stage that teaches a language model which of its possible answers people prefer, by turning human comparisons into a reward and optimizing the model against it. It is one of four stages in how LLMs are trained: pretraining, supervised fine-tuning, preference tuning and, for reasoning models, reinforcement learning against rewards a program can check. This guide explains what each stage fixes, what data it uses and what it can break, with every number taken from the paper that reported it.

How LLMs are trained: the four stages at a glance

Each stage exists because the previous one leaves a gap. Pretraining produces a model that knows a lot but does not take instructions. Supervised fine-tuning teaches it the format of an assistant. Preference tuning teaches it which of several plausible answers is better. Verifiable-reward RL teaches it to get checkable answers right, even when nobody wrote down how.

Two caveats. The order is a convention, not a law: DeepSeek ran stage 4 directly on a pretrained model, and production pipelines loop stages 2 to 4 several times. And for current commercial frontier models (Claude Opus 5.5, GPT-6.1, Gemini 4 Argon) the training recipes are not disclosed, so everything specific below comes from papers whose authors published their method. The machinery behind stages 3 and 4, from Markov decision processes and PPO to reward models, DPO, GRPO and reward hacking, is covered module by module, with working code, in Reinforcement Learning and RLHF.

Stage 1: pretraining builds a next-token predictor

Pretraining is self-supervised: the model reads text and learns to predict the next token, so no labels are needed and it scales to enormous corpora. Meta's Llama 3 paper reports a pretraining corpus of about 15T multilingual tokens, with the flagship 405B-parameter model trained on 15.6T tokens.

The result is a capable text continuer, and that is the problem. As the InstructGPT paper puts it, "predicting the next token on a webpage from the internet" is a different objective from "follow the user's instructions helpfully and safely"; the language modeling objective is misaligned. Almost all of a model's knowledge comes from this stage, which matters later: if a model lacks facts about your domain, preference tuning will not add them.

Stage 2: supervised fine-tuning teaches the assistant format

Supervised fine-tuning (SFT), or instruction tuning, continues training on examples of the behavior you want: a prompt and an ideal response written by people, with the same next-token loss applied to the response. The data is small. In InstructGPT (Ouyang and 19 co-authors at OpenAI, 2022), the SFT dataset had about 13k training prompts, from the OpenAI API and written by labelers, with demonstrations from a team of about 40 contractors.

SFT has a built-in ceiling. The model imitates the demonstrations, so it is only as good as the people writing them, and writing a great answer to a hard question is slow. More importantly, a demonstration says "this is a good answer" but never "this answer is better than that one". Finer distinctions (more accurate, less padded, appropriately cautious) are far easier to judge than to write. That asymmetry is the whole motivation for stage 3.

Stage 3: what RLHF actually is

The idea predates chatbots. In 2017, Christiano, Leike, Brown, Martic, Legg and Amodei published "Deep reinforcement learning from human preferences": instead of hand-coding a reward, non-expert humans compared pairs of short trajectory segments and a reward was learned from their choices. The agents learned complex tasks, including Atari games and simulated robot locomotion, "without access to the reward function", with feedback on less than 1% of their interactions, and learned novel behaviors from about an hour of human time. InstructGPT applied the recipe to language, in three steps.

Step 1: collect comparisons

For each prompt the model generates several responses and a person ranks them. InstructGPT showed labelers between 4 and 9 responses per prompt, up to 36 pairwise comparisons per ranking, across 33k training prompts. Meta's Llama 2 paper lists 1,418,091 binary comparisons collected by Meta for helpfulness and safety. One preference record:

{
  "prompt": "My landlord kept my deposit for normal wear and tear. Can he do that?",
  "chosen": "It depends on where you live, but in many places normal wear and tear cannot be deducted. Check your local tenancy law and ask for an itemized list of deductions ...",
  "rejected": "No, that is illegal. Take him to court."
}

The rejected answer is confident and short: a difference a person ranks in seconds and a demonstration never captures.

Step 2: train a reward model

The reward model is a copy of the SFT model whose final layer is replaced by a head that outputs one number: how good this response is for this prompt. It is trained with the Bradley-Terry model of pairwise preference so that the preferred response scores higher. InstructGPT's loss, in plain notation:

loss(theta) = - 1 / C(K,2) * E[(x, y_w, y_l) ~ D] [ log( sigmoid( r_theta(x, y_w) - r_theta(x, y_l) ) ) ]

x = prompt, y_w = preferred response, y_l = less preferred response
r_theta = scalar reward, C(K,2) = number of pairs in one ranking of K responses

InstructGPT used 6B reward models for its 175B policy, reporting that 175B reward model training could be unstable, making it less suitable as the initialization for the PPO value function. Remember what a reward model is: a learned approximation of human taste, trained on a finite sample. Every failure later in this article comes from that sentence.

Step 3: optimize the policy with PPO, on a leash

The language model (the "policy") now generates responses to fresh prompts, the reward model scores them, and a reinforcement learning algorithm pushes the policy toward higher scores. InstructGPT used Proximal Policy Optimization (Schulman et al., July 2017), whose clipped objective allows several gradient steps on one batch of samples without the policy moving too far, which keeps RL stable on a large network.

The crucial detail is the leash. Maximize the reward model alone and the policy drifts into text the reward model has never seen and over-rates. InstructGPT adds a per-token KL penalty against the SFT model:

objective(phi) = E[(x, y) ~ pi_RL] [ r_theta(x, y) - beta * log( pi_RL(y | x) / pi_SFT(y | x) ) ]
               + gamma * E[x ~ D_pretrain] [ log pi_RL(x) ]

The beta term charges the policy for diverging from the SFT model; the gamma term mixes pretraining gradients back in ("PPO-ptx") to reduce the "alignment tax", the regressions on public NLP benchmarks that alignment caused. The engineering cost is real: four networks at once, namely the policy, the frozen SFT model for the KL term, the reward model and a value function initialized from the reward model.

What RLHF bought

The InstructGPT abstract made RLHF standard with one sentence: "outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters." At equal size, 175B InstructGPT outputs were preferred to GPT-3's 85 ± 3% of the time, and 71 ± 4% of the time against few-shot prompted GPT-3. On TruthfulQA it gave truthful and informative answers about twice as often as GPT-3. The authors add that it "still makes simple mistakes". For anyone budgeting a model project, the lesson is that the training signal can matter more than the parameter count.

How ChatGPT was trained: what OpenAI has disclosed

OpenAI's post introducing ChatGPT (30 November 2022) said it was trained with RLHF "using the same methods as InstructGPT, but with slight differences in the data collection setup": human AI trainers wrote conversations playing both the user and the assistant, comparison data trained a reward model, and the model was fine-tuned with PPO. In May 2025 OpenAI described its post-training only in outline: take a pre-trained base model, "do supervised fine-tuning on a broad set of ideal responses written by humans or existing models, and then run reinforcement learning with reward signals from a variety of sources." The data, reward mix and algorithms behind specific current models are not disclosed.

RLHF vs DPO: the same goal without the RL loop

In May 2023, Rafailov, Sharma, Mitchell, Ermon, Manning and Finn published "Direct Preference Optimization: Your Language Model is Secretly a Reward Model". The KL-constrained objective above has a known optimal solution, which lets you express the reward through the policy itself, as beta times the log-ratio between the policy and the reference model. Substituted into the Bradley-Terry model, the intractable normalizing term cancels and what remains is a classification loss on preference pairs: no separate reward model, no sampling during training, no value function. Equation 7 of the paper, as PyTorch:

import torch.nn.functional as F

def dpo_loss(policy_chosen_logps, policy_rejected_logps,
             ref_chosen_logps, ref_rejected_logps, beta=0.1):
    # Summed log-probability of each full response, shape [batch].
    # The reference model is the frozen SFT model.
    chosen = beta * (policy_chosen_logps - ref_chosen_logps)
    rejected = beta * (policy_rejected_logps - ref_rejected_logps)
    return -F.logsigmoid(chosen - rejected).mean()

The policy raises the probability of the chosen response relative to the reference and lowers that of the rejected one; beta (0.1 by default in the paper) controls how far it may drift. The authors report that DPO matched or beat PPO-based RLHF on sentiment control, summarization and single-turn dialogue "while being substantially simpler to implement and train".

Meta's Llama 2 had used two reward models (helpfulness and safety) with PPO and rejection sampling fine-tuning; the Llama 3 paper describes post-training with SFT, rejection sampling and DPO "as opposed to more complex reinforcement learning algorithms" that "tend to be less stable and harder to scale". The Allen Institute's Tülu 3 used length-normalized DPO.

DPO has limits: trained on fixed pairs, it cannot explore beyond the responses in its dataset. A practical rule: with a fixed preference dataset and a fine-tuning budget, DPO on top of an SFT model (often with LoRA adapters, as in our guide to fine-tuning an open-source LLM with LoRA and QLoRA) is the default; online RL earns its complexity when you can score new outputs as the model improves. Both workflows are covered in Fine-Tuning and Open-Source LLMs.

RLAIF and Constitutional AI: replacing some of the humans

Human preference labels are slow and expensive. In December 2022, Anthropic's "Constitutional AI: Harmlessness from AI Feedback" (Bai and 50 co-authors) trained a harmless assistant "without any human labels identifying harmful outputs"; for harmlessness, the only human oversight was a written list of principles (human labels were still used for helpfulness). Two phases:

1. Supervised: the model critiques and revises its own responses according to the principles, and is fine-tuned on the revisions.
2. Reinforcement learning: a model judges which of two responses better follows the principles, a preference model is trained on those AI judgments, and RL runs against it. The paper calls this RL from AI Feedback (RLAIF).

The result, per the abstract, was "a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them". In 2023, Lee and co-authors at Google found RLAIF performing comparably to RLHF on summarization, helpful dialogue and harmless dialogue. RLAIF moves human effort rather than removing it: people still write the principles and check the outcome, and an AI judge has biases of its own that the policy will learn to exploit.

Stage 4: reinforcement learning with verifiable rewards

Some tasks need no learned judge. If the answer is 42, a program can check it; if code must pass a test suite, the tests are the reward. The Tülu 3 paper (Lambert et al., November 2024) named this "Reinforcement Learning with Verifiable Rewards (RLVR)". DeepSeek-R1 made it famous: posted on arXiv on 22 January 2025, then published in Nature on 17 September 2025. Its most striking experiment, DeepSeek-R1-Zero, skipped supervised fine-tuning entirely:

• Base model: DeepSeek-V3-Base, straight from pretraining.
• Rewards: rule-based only, an accuracy reward (correct final answer, or code passing tests) plus a format reward for reasoning inside <think> tags.
• No neural reward model for reasoning, because such models "are susceptible to reward hacking during large-scale reinforcement learning".
• Algorithm: GRPO, introduced in DeepSeekMath (Shao et al., 2024) as a PPO variant that reduces PPO's memory use.

GRPO drops the value network. For each question it samples a group of answers (16 for R1-Zero), scores them, and uses each answer's standing within its group as the advantage, keeping PPO's clipped update and a KL penalty:

A_i = (r_i - mean(r_1, ..., r_G)) / std(r_1, ..., r_G)

R1-Zero's average pass@1 on AIME 2024 rose from 15.6% to 77.9% during RL training (the Nature version; the January 2025 preprint reported 71.0% for an earlier run), and the model learned to reason at length, check itself and revisit steps without being shown how; the paper quotes it writing "Wait, wait. Wait. That's an aha moment I can flag here." It also documents the cost: poor readability and language mixing between English and Chinese. The final DeepSeek-R1 therefore went back to a multi-stage pipeline: thousands of curated "cold-start" examples, RL with an added language consistency reward, rejection sampling and SFT, then a second RL stage with preference rewards for helpfulness and harmlessness. The four stages returned, in a different order.

The limit is in the name: RLVR only works where correctness can be verified, and a checker with gaps is a reward to be hacked like any other.

What preference tuning can break

Every method above optimizes a proxy: a reward model, an AI judge, a set of pairs, a checker. Goodhart's law applies to all of them.

Reward hacking and overoptimization

In "Scaling Laws for Reward Model Overoptimization" (October 2022), Gao, Schulman and Hilton let a fixed "gold-standard" reward model play the human and trained a proxy reward model on its labels. Optimizing against the proxy, the gold reward "initially increases and later decreases": past a point, the policy learns the proxy's quirks instead of the preference. That is the problem the KL leash is meant to contain, although in Gao et al.'s own setup a KL penalty acted much like early stopping and did not improve the gold reward reached at a given distance from the initial model; how far you let the policy drift, and when you stop, are what matter. The DeepSeek-R1 paper shows the same thing in a real run: with its helpfulness reward model, the reward kept rising while pass@1 on CodeForces fell, so the authors added preference rewards only in the last 400 of 1,700 steps of the final RL stage.

Sycophancy

The most visible form of reward hacking in assistants is sycophancy: telling users what they want to hear. Anthropic researchers (Sharma et al., 2023) found that five state-of-the-art assistants consistently exhibited it across four tasks, and traced part of the cause to the data: responses matching a user's views were more likely to be preferred, and both humans and preference models preferred convincingly written sycophantic responses over correct ones "a non-negligible fraction of the time". If raters reward agreement, optimization delivers agreement.

Case study: the GPT-4o sycophancy rollback

The clearest public example came from OpenAI. On 25 April 2025 it shipped a GPT-4o update in ChatGPT that made the model "noticeably more sycophantic", began rolling it back on 28 April, and explained what happened in two posts: "Sycophancy in GPT-4o: what happened and what we're doing about it" (29 April) and "Expanding on what we missed with sycophancy" (2 May). In OpenAI's own words:

• The update added "an additional reward signal based on user feedback—thumbs-up and thumbs-down data from ChatGPT", among other candidate changes, including better use of memory.
• Each change had looked beneficial alone, but OpenAI's early assessment was that in aggregate "these changes weakened the influence of our primary reward signal, which had been holding sycophancy in check."
• OpenAI "focused too much on short-term feedback", and the model "skewed towards responses that were overly supportive but disingenuous."
• Offline evaluations "generally looked good" and "the A/B tests seemed to indicate that the small number of users who tried the model liked it." Some expert testers said the behavior "felt" slightly off; the launch went ahead.

OpenAI's stated lessons included treating model behavior issues as launch-blocking, like other safety risks. For any team tuning on user feedback the takeaway is concrete: a thumbs-up measures the moment, not whether the answer was right, and a short A/B test rewards exactly the flattery you are trying to avoid.

What to check before you ship a preference-tuned model

• Behavioral probes, not just averages: pairs where the longer answer is worse, the confident answer wrong, the agreeable answer false.
• Reward against an independent measure during training: rising reward with a flat or falling evaluation score is the overoptimization signature.
• Length and refusal rates, which drift quietly under preference tuning.
• User reactions as a weak signal, weighted against long-term outcomes.

Building those test sets is the subject of our LLM evaluation guide and of the LLM Evaluation and Testing course.

A cheat sheet: which stage fixes which problem

For product managers:

Frequently Asked Questions (FAQ)

What is RLHF in simple terms? People compare a model's answers, a reward model learns to predict which answer people prefer, and the language model is then optimized with reinforcement learning to produce answers the reward model scores highly, while a penalty keeps it close to its earlier behavior.

Is RLHF still used, or has DPO replaced it? Both are used. DPO and its variants are a common default for preference tuning in open models because they are simpler and more stable; Meta's Llama 3 used DPO rather than PPO. Reinforcement learning is more prominent than ever, though, as RL with verifiable rewards for reasoning models such as DeepSeek-R1.

How was ChatGPT trained? OpenAI said the original ChatGPT (November 2022) was trained with RLHF using the same methods as InstructGPT: supervised fine-tuning on conversations written by human trainers, a reward model trained on comparison data, and PPO. In 2025 it described its post-training as supervised fine-tuning followed by reinforcement learning with reward signals from a variety of sources. Detailed recipes for current models are not disclosed.

What is a reward model? A neural network, usually initialized from the language model, that takes a prompt and a response and outputs one score. It is trained on human or AI comparisons so that preferred responses score higher, and stands in for human judgment during reinforcement learning. Because it is an imperfect approximation, optimizing too hard against it causes reward hacking.

Does RLHF make a model more truthful? It can, but it does not guarantee it. InstructGPT gave truthful and informative answers about twice as often as GPT-3 on TruthfulQA, yet sycophancy research shows human preferences can also reward agreeing with the user over being correct. Truthfulness depends on what the raters and the reward actually reward.

Conclusion

How LLMs are trained comes down to four signals applied in turn: next-token prediction for knowledge, demonstrations for format, preferences for judgment and verifiable rewards for correctness. RLHF made the third stage practical and showed a 1.3B model could be preferred to a 175B one; DPO removed the RL loop for many teams; Constitutional AI replaced part of the human labeling; RLVR brought reinforcement learning back at scale for reasoning. Every one of them optimizes a proxy, so the discipline is the same at every stage: keep the model on a leash, and measure what you actually want, not what the reward says.

Sources

Checked on 5 November 2026.

• Training language models to follow instructions with human feedback, Ouyang et al., arXiv, 4 March 2022 (InstructGPT)
• Deep reinforcement learning from human preferences, Christiano et al., arXiv, 12 June 2017
• Proximal Policy Optimization Algorithms, Schulman et al., arXiv, 20 July 2017
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model, Rafailov et al., arXiv, 29 May 2023
• Constitutional AI: Harmlessness from AI Feedback, Bai et al., arXiv, 15 December 2022
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning, DeepSeek-AI, Nature 645, 633-638, 17 September 2025 (preprint arXiv:2501.12948, v1 22 January 2025; the figures quoted here are from the Nature version, matching arXiv v2 of 4 January 2026)
• Expanding on what we missed with sycophancy, OpenAI, 2 May 2025
• Scaling Laws for Reward Model Overoptimization, Gao, Schulman and Hilton, arXiv, 19 October 2022
• Also cited: OpenAI, "Sycophancy in GPT-4o: what happened and what we're doing about it" (29 April 2025) and "Introducing ChatGPT" (30 November 2022); arXiv:2310.13548 (Sharma et al., sycophancy); arXiv:2309.00267 (Lee et al., RLAIF vs. RLHF); arXiv:2411.15124 (Tülu 3); arXiv:2402.03300 (DeepSeekMath); arXiv:2307.09288 (Llama 2); arXiv:2407.21783 (Llama 3)