Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is test-time compute? Scaling LLMs at inference time

Test-time compute is the work a model does while answering: longer reasoning, more samples, a verifier. How it scales and what it costs, with Python.

What is test-time compute? Scaling LLMs at inference time

Test-time compute is the computation a model spends while answering a question, as opposed to the computation spent training it. You can raise it by letting the model write a longer chain of thought, by sampling many answers and voting, or by having a verifier pick the best of several attempts, and on hard problems that extra inference often buys more accuracy than a bigger model would. Reasoning models such as OpenAI’s o-series and DeepSeek R1 are models trained to spend it well.

Why did test-time compute become a scaling axis?

For most of the LLM era, the way to get a better answer was a bigger model trained on more data, the bet captured by scaling laws. Once a model is trained, though, every answer gets roughly the same amount of compute per token whether the question is “what is 2 + 2” or a competition proof. Test-time compute is the second dial: spend more on the questions that need it.

The research case was made in August 2024 by Snell, Lee, Xu and Kumar, “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters”. Two findings from the abstract are the ones I keep:

  • Allocating test-time compute adaptively per prompt, which they call compute-optimal, improved efficiency “by more than 4x compared to a best-of-N baseline.”
  • In a FLOPs-matched comparison, a smaller model with extra test-time compute could outperform “a 14x larger model”, but only on problems where the small model already had a non-trivial success rate.

A month later, OpenAI o1-preview (September 12, 2024) shipped the idea as a product: a model trained to think before answering, with the thinking billed as output tokens. DeepSeek R1 (January 2025) showed in the open that reasoning abilities “can be incentivized through pure reinforcement learning”, with behaviors such as self-reflection and verification emerging during RL training, the recipe I cover in what is RLVR.

What are the ways to spend test-time compute?

There are two families, and real systems mix them.

Sequential: think longer. The model writes more reasoning tokens before the answer, revising and checking its own work. Chain-of-thought prompting is the prompting version; reasoning models are trained to do it, and APIs expose the budget as a reasoning effort or thinking-token setting. It needs no extra infrastructure, but each token depends on the one before, so it adds latency in a straight line.

Parallel: try more times. Sample N independent answers and combine them:

  • Majority vote (self-consistency). Wang et al. (2022) sampled several reasoning paths and kept the most common final answer, and reported gains over plain chain-of-thought of 17.9 points on GSM8K and 11.0 on SVAMP.
  • Best-of-N with a verifier. A separate checker, such as unit tests, a math checker, a reward model or an LLM judge, scores each candidate and you keep the best one.
  • Search. A process reward model scores partial solutions and guides a beam or tree search. This is one of the strategies Snell et al. compare.

Parallel samples can run at the same time, so they cost money more than wall-clock time.

How much does test-time compute help?

It depends on two numbers: how often a single attempt is right, and whether you can check an answer. This simulation makes that concrete. Each simulated sample is right with probability p; wrong samples pick one of four wrong answers. It compares majority vote against best-of-N with a perfect verifier, at 1,500 tokens per sample.

# test_time.py: what extra samples buy you. Python 3, stdlib only, simulated model.
# Each sample is right with probability p; wrong samples pick one of WRONG distinct
# wrong answers. Compare one sample, majority vote, and best-of-N with a perfect checker.
import random
from collections import Counter

random.seed(42)
WRONG, TRIALS, TOKENS_PER_SAMPLE = 4, 20_000, 1_500

def sample(p):
    return "right" if random.random() < p else f"wrong{random.randrange(WRONG)}"

def vote(p, n):
    hits = 0
    for _ in range(TRIALS):
        top = Counter(sample(p) for _ in range(n)).most_common()
        best = [a for a, c in top if c == top[0][1]]
        hits += random.choice(best) == "right"                  # break ties at random
    return hits / TRIALS

print(f"{'p':>4} {'N':>3} {'tokens':>7} {'majority':>9} {'verifier':>9}")
for p in (0.6, 0.35, 0.15):
    for n in (1, 5, 17, 65):
        verifier = 1 - (1 - p) ** n                              # any right sample, found by a checker
        print(f"{p:>4} {n:>3} {n * TOKENS_PER_SAMPLE:>7} {vote(p, n):>9.1%} {verifier:>9.1%}")

Output (Python 3.13.11):

   p   N  tokens  majority  verifier
 0.6   1    1500     60.1%     60.0%
 0.6   5    7500     83.8%     99.0%
 0.6  17   25500     99.1%    100.0%
 0.6  65   97500    100.0%    100.0%
0.35   1    1500     35.5%     35.0%
0.35   5    7500     46.4%     88.4%
0.35  17   25500     68.8%     99.9%
0.35  65   97500     95.1%    100.0%
0.15   1    1500     15.1%     15.0%
0.15   5    7500     12.5%     55.6%
0.15  17   25500      8.3%     93.7%
0.15  65   97500      3.0%    100.0%

Three things stand out. The numbers are from a toy model, but the conditions mirror the ones the papers describe.

Voting helps when the model is usually right. At p = 0.6, five samples take accuracy from 60% to 84%, and seventeen get close to certain.

Voting can make things worse. At p = 0.15, the right answer is rarer than each wrong answer (each wrong one shows up about 21% of the time), so more votes converge on a wrong answer: 15% falls to 3%. This is the same condition Snell et al. hit with the 14x result, which only held where the small model already had a non-trivial success rate.

A verifier changes everything. With a checker that recognizes a right answer, accuracy is the chance that any sample is right, 1 - (1 - p)^N, and even p = 0.15 reaches 94% at seventeen samples. This is why test-time compute works best on code (run the tests) and math (check the answer), and why so much effort goes into reward models for everything else.

The token column is the bill. Seventeen samples at 1,500 tokens is 25,500 output tokens for one question, 17 times the single-shot cost.

What does test-time compute cost a builder?

Every form of it is paid in output tokens, the expensive side of the bill. Reasoning tokens are billed as output even when the API hides them, which I priced in what are reasoning models. Parallel sampling multiplies the whole request. Sequential thinking also adds latency, since the user waits for every reasoning token before seeing the answer.

Vendors now expose the dial directly: OpenAI’s reasoning effort and Anthropic’s effort parameter let you set how much a model spends per request. The engineering question is the one Snell et al. asked: how much compute does this particular prompt need? Easy prompts should get almost none.

Where does test-time compute break?

No verifier, no free lunch. Without a way to check answers, you are limited to voting or to the model’s own judgment, and both fail when the model is confidently wrong.

Overthinking. More reasoning on an easy question adds cost and latency and sometimes talks the model out of a right answer. Adaptive budgets exist for this reason.

Base capability is a floor. Test-time compute amplifies what the model can already sometimes do. It does not create knowledge the model lacks; for missing facts, retrieval is the fix.

Long agent loops multiply it. In an agent, every step can trigger reasoning. A small per-step budget times hundreds of steps is a large bill.

What I would do

I would sort tasks by whether their answers can be checked automatically. For checkable tasks such as code with tests, SQL with expected results or math, I would spend test-time compute freely on parallel attempts with a verifier, since that is where the curve bends fastest. For uncheckable tasks I would use a moderate reasoning effort, measure accuracy against cost on my own eval set at two or three effort levels, and stop at the knee. For search-style questions, I would retrieve first so the model reasons over facts, then let it think; that is the order rSearch uses, with Serper results going to DeepSeek R1. And I would set a per-request token ceiling everywhere, because reasoning spend is the line item that surprises people.

FAQ

What is test-time compute in simple terms?

The amount of computation a model uses while answering, rather than while being trained. Thinking longer, trying several answers and checking them all increase it.

What is the difference between test-time compute and training compute?

Training compute is spent once to produce the model’s weights. Test-time compute is spent on every request. Scaling laws describe how loss falls with training compute; test-time scaling describes how accuracy rises with inference compute on a fixed model.

Is test-time compute the same as a reasoning model?

A reasoning model is one trained to use test-time compute well, by writing long chains of thought before answering. Test-time compute also includes methods that work with any model, such as majority voting and best-of-N with a verifier.

Does more test-time compute always improve accuracy?

No. Majority voting can lower accuracy when the model is usually wrong, and extra reasoning on easy questions adds cost without benefit. Gains are largest when answers can be verified.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.