Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is RLVR? Reinforcement learning with verifiable rewards

RLVR trains an LLM with rewards from a program that checks the answer, such as a math checker or unit tests, instead of a reward model. How it works, in Python.

What is RLVR? Reinforcement learning with verifiable rewards

RLVR (reinforcement learning with verifiable rewards) is a way to post-train a language model where the reward comes from a program that checks the answer, such as comparing a math result to the known solution or running unit tests on code, instead of from a learned reward model. The model gets rewarded only when its output is verifiably correct, and that simple signal, applied at scale, is how open reasoning models like DeepSeek R1 learned to think step by step.

I ran the demo below on 2026-10-31 with Python 3.13.11 and no packages.

Where does the term RLVR come from?

Allen AI named it. Their Tulu 3 report (Lambert et al., November 2024) introduced “a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR)”. The definition is short: RLVR keeps the RLHF objective and “replaces the reward model with a verification function”. If the answer is verifiably correct, the reward is a constant α, otherwise 0. They set α = 10 “based on pilot experiments”, trained with PPO, and applied it to math (GSM8K, MATH) and to instruction following with checkable constraints (IFEval), reporting gains on those benchmarks.

Two months later, DeepSeek-R1 (January 2025) showed what the idea does at scale. That’s the release I wrote about in DeepSeek R1, and it’s the reason the term spread.

How does RLVR work?

Compare it with RLHF. RLHF has humans rank answers, trains a reward model on the rankings, and optimizes the policy against that model’s score. RLVR deletes the middle step:

  1. Take prompts that have a checkable answer: a math problem with a known result, a coding task with tests, an instruction with a constraint you can test in code (“answer in exactly three bullet points”).
  2. Sample completions from the current model.
  3. Run the verifier. Correct gets the reward; anything else gets 0.
  4. Update the model to make rewarded completions more likely, with a KL penalty against a reference model, as in RLHF.

The objective in the Tulu 3 paper is the RLHF one with the reward model swapped out: maximize v(x, y) - β KL[π(y|x) || π_ref(y|x)], where v is the verifier.

DeepSeek’s version for R1-Zero used two rule-based rewards with equal weight. An accuracy reward checks the final answer, which the model has to put “in a specified format (e.g., within a box)” for math, and uses a compiler and test cases for code. A format reward checks that the reasoning sits between <think> and </think> tags. They chose not to use a neural reward model for reasoning at all, because “neural reward models are susceptible to reward hacking during large-scale reinforcement learning”.

What is GRPO, and why does RLVR use it?

PPO, the classic RL algorithm in RLHF, needs a value model (a critic) to estimate how good a state is, so it can tell whether a completion did better than expected. The DeepSeekMath paper (February 2024) points out that this value function is “typically another model of comparable size as the policy model”, which “brings a substantial memory and computational burden”.

GRPO (Group Relative Policy Optimization), introduced in that paper, “foregoes the critic model, instead estimating the baseline from group scores”. For each question, sample a group of G answers, score each one, and compute every answer’s advantage relative to its own group: A_i = (r_i - mean(r)) / std(r). Answers better than their siblings get pushed up; worse ones get pushed down.

That fits verifiable rewards well. With a 0 or 1 reward, the group mean is the model’s current success rate on that question, which is exactly the baseline you want. For R1-Zero, DeepSeek sampled 16 outputs per question with up to 32,768 tokens each, a learning rate of 3e-6 and a KL coefficient of 0.001.

What does RLVR look like in code?

Here’s the whole loop on a toy. The “model” is a choice between four answer habits for addition problems. Only one of them gets reward: working it out and boxing the correct result. The base model prefers the other habits. Training uses a regex verifier and GRPO advantages over groups of 8.

# rlvr.py: reinforcement learning with a verifiable reward, GRPO-style.
# Python 3 standard library only. The "policy" picks one of four answer habits.
import math, random, re, statistics

random.seed(3)

def verify(completion, answer):
    """The whole reward model: find \\boxed{...}, compare to ground truth. 1 or 0."""
    m = re.search(r"\\boxed\{(-?\d+)\}", completion)
    return 1.0 if m and int(m.group(1)) == answer else 0.0

habits = {  # how a completion gets written for "a + b"
    "boxed, right": lambda a, b: f"<think>{a}+{b}={a + b}</think> \\boxed{{{a + b}}}",
    "right, no box": lambda a, b: f"The answer is {a + b}.",
    "off by one": lambda a, b: f"\\boxed{{{a + b + 1}}}",
    "guess": lambda a, b: f"\\boxed{{{random.randint(0, 200)}}}",
}
names = list(habits)
logits = [0.0, 1.0, 1.0, 0.5]                  # a base model that rarely boxes the right answer
probs = lambda: [math.exp(l) / sum(math.exp(x) for x in logits) for l in logits]

G, LR = 8, 0.5                                 # 8 samples per prompt, as a group
print(f"{'step':>4}" + "".join(f"{n:>14}" for n in names) + "   no-signal groups so far")
dead = 0
for step in range(61):
    if step in (0, 5, 10, 20, 40, 60):
        print(f"{step:>4}" + "".join(f"{p:>14.2f}" for p in probs()) + f"   {dead}")
    a, b = random.randint(1, 99), random.randint(1, 99)
    p = probs()
    picks = random.choices(range(4), weights=p, k=G)
    rewards = [verify(habits[names[i]](a, b), a + b) for i in picks]
    sd = statistics.pstdev(rewards)
    if sd == 0:                                # all right or all wrong: zero advantage
        dead += 1
        continue
    mean = statistics.fmean(rewards)
    adv = [(r - mean) / sd for r in rewards]   # GRPO: the group is the baseline, no critic
    grad = [0.0] * 4
    for i, A in zip(picks, adv):               # d log softmax / d logit_j = 1[j=i] - p_j
        for j in range(4):
            grad[j] += A * ((1.0 if j == i else 0.0) - p[j]) / G
    logits = [l + LR * g for l, g in zip(logits, grad)]

print()
for text in [habits["boxed, right"](2, 2), habits["right, no box"](2, 2), habits["off by one"](2, 2)]:
    print(f"verify({text!r}, 4) = {verify(text, 4)}")
step  boxed, right right, no box    off by one         guess   no-signal groups so far
   0          0.12          0.34          0.34          0.20   0
   5          0.30          0.29          0.26          0.15   1
  10          0.68          0.11          0.13          0.09   1
  20          0.92          0.03          0.03          0.02   5
  40          0.97          0.01          0.01          0.01   20
  60          0.98          0.01          0.01          0.00   39

verify('<think>2+2=4</think> \\boxed{4}', 4) = 1.0
verify('The answer is 4.', 4) = 0.0
verify('\\boxed{5}', 4) = 0.0

The verified habit goes from 12% to 98% in 60 steps, with no reward model and no human labels. Two details are worth reading closely.

First, “right, no box” gets zero reward even though the arithmetic is correct. The verifier can’t parse it. That’s the role of the format requirement: the model learns to put answers where the checker can find them. In real training this is also a trap, because the model optimizes the checker, not your intent.

Second, the last column. A group where all 8 answers are right, or all are wrong, has zero variance and therefore zero advantage: it teaches nothing. Early on, mixed groups are common. By step 60, 39 of 60 groups were wasted because the model almost always gets it right. Real RLVR pipelines hit the same wall, which is why prompt difficulty matters: questions the model always solves or never solves don’t train it.

What did RLVR achieve with DeepSeek R1?

DeepSeek applied RL with rule-based rewards directly to the DeepSeek-V3 base model, with no supervised reasoning data first, to make R1-Zero. The paper (v2, January 2026) reports average pass@1 on AIME 2024 “jumping from an initial 15.6% to 77.9%”. The responses got longer during training and started including verification and reflection, including the often-quoted “aha moment” where the model writes “Wait, wait. Wait. That’s an aha moment I can flag here.”

R1-Zero also showed the limits. The paper lists “poor readability and language mixing, occasionally combining English and Chinese within a single chain-of-thought response”, and weak results outside reasoning, in “writing and open-domain question answering”. The final R1 added cold-start data, a language consistency reward, rejection sampling and SFT around the RL stages. Pure RLVR made a strong reasoner, not a good assistant. For why these models behave as they do at inference, see what are reasoning models.

RLVR vs RLHF: when do you use which?

RLHF RLVR
Reward source Reward model trained on human rankings Program: answer match, tests, constraint checks
Works for Open-ended quality: tone, helpfulness, style Tasks with a checkable answer: math, code, formats
Reward hacking Policy exploits the reward model’s blind spots Policy exploits the checker’s blind spots
Data cost Human comparisons Problems with known answers or tests

They stack. Tulu 3 runs SFT, then DPO for general preferences (DPO vs RLHF covers that step), then RLVR for verifiable skills.

Where does RLVR break?

  • Only where a checker exists. DeepSeek says it plainly: dependable rule-based rewards “are difficult to construct for certain tasks, such as writing”. If the signal is assigned by a model instead, it’s open to exploitation again.
  • Verifiers have blind spots. A math checker that only reads the boxed answer rewards a lucky guess the same as a derivation. Unit tests reward any code that passes them, including code that special-cases the tests. The Tulu 3 authors found “overoptimized outputs” in some higher-KL IFEval runs.
  • Saturated prompts waste compute. As my toy shows, groups with uniform rewards carry no signal.
  • Long rollouts are expensive. 16 samples of up to 32,768 tokens per question adds up fast.

What I would do

I wouldn’t train a reasoning model from scratch as an application team; the open ones exist. Where RLVR’s lesson applies to my work is the verifier itself. When I build agents, the same checks that would make a good RLVR reward, such as “does the SQL run”, “do the tests pass”, “does the JSON validate”, make the best runtime guards and the best evals. I’d write those first.

If I did fine-tune with RL on a narrow task, I’d spend most of the effort on the verifier: adversarial test cases, checks that the answer came from the work shown, and a held-out eval the verifier never sees. I’d log the fraction of no-signal groups, as my toy does, and drop or replace prompts the model has saturated.

FAQ

What does RLVR stand for?

Reinforcement Learning with Verifiable Rewards. The name comes from Allen AI’s Tulu 3 report, November 2024.

Is RLVR the same as GRPO?

No. RLVR describes where the reward comes from: a verifier. GRPO is an RL algorithm that estimates the baseline from a group of samples instead of a critic. DeepSeek R1 used both together; Tulu 3 used RLVR with PPO.

Can RLVR train a model for writing or chat?

Not directly, because writing quality has no checkable answer. DeepSeek notes that for such tasks the reward has to come from a model, which reintroduces reward hacking. Tasks with partially checkable parts, such as format constraints, can use RLVR for those parts.

Why did DeepSeek avoid neural reward models for R1-Zero?

The R1 paper says neural reward models are susceptible to reward hacking in large-scale RL and that retraining them adds compute and complexity, so they used rule-based accuracy and format rewards for reasoning tasks.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.