Test-time compute is the extra computation a model or system spends while answering a request, beyond generating one direct answer. In AI engineering, it is a per-request budget you can trade for accuracy, paid for in tokens, latency, and money.
For example, suppose a model solves a multi-step arithmetic word problem correctly on 50% of single attempts, and its wrong answers rarely agree with each other. Sampling eight answers and taking the majority vote raises accuracy to about 84% in a simple simulation. The cost is roughly eight generations instead of one.
“Test time” means inference time. Training is already finished, and the system is answering a specific request.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why test-time compute matters
Many AI systems now spend variable amounts of computation per request. Reasoning models generate long intermediate steps, and agents retry or verify their work. Each of these is a choice about how much to spend.
Understanding test-time compute helps with:
- Improving hard tasks: Raise accuracy on tasks where a single attempt is unreliable but correct answers are reachable.
- Controlling cost: Decide which requests earn extra compute and which get a single pass.
- Comparing systems fairly: State the inference budget alongside a score, so results are comparable.
- Diagnosing failures: Tell a failure that more samples can fix from one they will only repeat.
Extra compute does not create knowledge or data the system lacks. It helps the system find and select answers it can already reach.
How test-time compute works
There are four common ways to spend test-time compute. Each one can be used alone or combined with the others.
Longer generation
The model produces intermediate reasoning before its final answer. Reasoning models do this, often with a budget you can control.
Repeated sampling with selection
The system generates several independent answers and then picks one. It can pick by majority vote, which is called self-consistency. It can also pick with a scoring model or with an executable check, such as running generated code against tests.
Search
The system explores a tree of partial solutions. It expands promising branches and abandons weak ones.
Verification and repair
A separate pass checks an answer against constraints, sources, or tools. Failures go back for another attempt.
Combining methods
These methods compose. For example, a system might sample eight candidates, verify each one, and return the first that passes. The compute it spends then varies with how hard the request is.
Snell et al., Scaling LLM Test-Time Compute Optimally (arXiv:2408.03314) studies this allocation question. It asks how much inference compute to spend, and on what, for a given problem difficulty. Its lesson is that the best budget depends on the task. It is not the maximum.
Arize’s discussion of sleep-time compute and inference scaling covers related ways to move computation off the critical path of a request.
How to measure the payoff of test-time compute
Measure a budget as accuracy gained per unit of cost and latency, on a fixed task set. A few impressive examples are not evidence.
Simulate majority voting
The following simulation shows why self-consistency helps in some cases and hurts in others. It assumes each sample is independent and uses hypothetical accuracy rates.
In the first case, each sample is correct 50% of the time, and wrong answers are spread across five different values. In the second case, each sample is correct 40% of the time, and every wrong answer is the same value.
import random
from collections import Counter
def majority_vote_accuracy(p_correct, wrong_answers, samples, trials=20_000, seed=0):
rng = random.Random(seed)
wins = 0
for _ in range(trials):
votes = Counter(
"correct" if rng.random() < p_correct else rng.choice(wrong_answers)
for _ in range(samples)
)
top = max(votes.values())
leaders = [answer for answer, count in votes.items() if count == top]
wins += rng.choice(leaders) == "correct"
return wins / trials
scattered = ["w1", "w2", "w3", "w4", "w5"]
systematic = ["w1"]
for samples in (1, 2, 4, 8):
a = majority_vote_accuracy(0.5, scattered, samples)
b = majority_vote_accuracy(0.4, systematic, samples)
print(f"samples={samples} scattered errors: {a:.1%} systematic error: {b:.1%}")
Output:
samples=1 scattered errors: 49.7% systematic error: 40.6%
samples=2 scattered errors: 50.1% systematic error: 40.6%
samples=4 scattered errors: 68.0% systematic error: 35.9%
samples=8 scattered errors: 83.7% systematic error: 28.9%
When errors are scattered, voting helps. The slips disagree with each other, while correct paths converge on one value. Accuracy rises from about 50% to about 84% at eight samples.
When the error is systematic, voting hurts. The same wrong answer keeps winning, and accuracy falls from about 40% to about 29%. More compute made the result worse.
Run a budget sweep on your own tasks
Real tasks rarely match either simulated case exactly. So measure directly. Hold the model, prompts, and task set fixed, and change only the budget.
Budget sweep (run on a fixed set, everything else pinned)
samples per task: 1 -> 2 -> 4 -> 8
report at each step: accuracy, tokens/task, p95 latency, cost/task
decision: smallest budget within noise of the best accuracy
Plot accuracy against samples per task and stop where the curve flattens. Re-run the sweep when the model, prompts, or task mix changes.
If samples run in parallel, latency is set by the slowest sample. Report p95 latency, meaning the latency that 95% of requests stay under, at each budget.
When test-time compute helps and when it does not
Diagnose the failure before you buy more compute.
- Right but inconsistent: If single samples are often right but vary, sampling plus voting or verification has something to select from. Accuracy usually improves with budget.
- Wrong in the same way: If samples share a missing fact, a misread requirement, or a tool that returns bad data, extra samples mostly repeat the error. Voting then selects it confidently.
- Missing evidence: If the model never saw the context it needs, no amount of reasoning over absent evidence fixes the answer.
Verification sets the ceiling
The quality of your selector caps the benefit of sampling. Picking the best of eight candidates with a weak checker can select the most plausible-looking wrong answer.
This failure is familiar from LLM evaluation in general. Prefer executable or source-grounded checks where they exist. Measure the selector itself, not just the generator.
This is why test-time compute belongs inside an evaluation loop with fixed evaluation metrics.
How to control test-time compute in production
Instrument each request with its budget and its outcome. Capture reasoning tokens, sample count, verifier result, total tokens, and latency on the trace for the run.
Then compare quality across budget tiers. If high-budget requests are not measurably better, route fewer requests into that tier.
Decide which requests earn extra compute by difficulty, value, or uncertainty. Do not give every request the maximum.
Set a ceiling for each request type. An open-ended “keep trying until verified” loop turns a hard input into an unbounded bill. Making that cost and reliability risk visible is what AI observability tooling is for.
Limitations of test-time compute
- Diminishing returns: Beyond a point, more samples mostly duplicate existing candidates.
- Wandering reasoning: Longer reasoning can drift or rationalize a wrong premise.
- Inherited bias: Selection methods inherit the biases of their verifier or voting pool.
- Latency limits: Interactive products may not be able to spend seconds of extra compute, whatever the accuracy gain.
- Unfair comparisons: A system with eight samples and a verifier is not fairly compared with a single greedy generation unless the budget difference is stated.
Test-time compute vs. training compute
Training compute is spent once to change model weights. Test-time compute is spent on every request and leaves the weights unchanged.
The two substitute for each other only partly. A stronger model may need less inference spend per task. Heavy inference spend cannot reproduce capabilities that training never produced.
For a serving team, the practical question is allocation. Model size, reasoning budget, and sample count together determine the accuracy and cost of a request. This is why benchmark vs. production evaluation comparisons must state the inference budget alongside the score.
Frequently asked questions
Is test-time compute the same as using a reasoning model?
No. A reasoning model is one way to spend test-time compute, through longer intermediate generation. Sampling, search, and verification are other ways, and you can apply them to ordinary models too.
Does more test-time compute always improve accuracy?
No. It helps when correct answers are reachable by sampling or search and can be recognized by voting or verification. When errors are systematic or evidence is missing, extra compute repeats or entrenches them.
How do I choose a test-time compute budget?
Sweep the budget on a fixed task set and plot accuracy against cost and latency. Pick the smallest budget within noise of the best result, and re-run the sweep when the model, prompts, or task mix changes.
Why do benchmark comparisons need to state test-time compute?
Because the score depends on it. Two systems with the same model but different sample counts or reasoning budgets will score differently. An unstated budget makes the comparison impossible to interpret.
What should I record about test-time compute for each request?
Record the sample count, reasoning tokens or effort setting, verifier and its result, total tokens, and latency. Store them as experiment metadata, because results at different budgets are not comparable.
Putting test-time compute to work
Start by diagnosing why single attempts fail on your tasks. Sampling and verification help with inconsistent errors, not with missing evidence or systematic mistakes.
Run a budget sweep on a fixed task set and choose the smallest budget near the best accuracy. Route only the requests that benefit into higher budget tiers, and set a ceiling for each request type.
Record the budget on every trace and in every experiment. Tools such as Arize AX can segment quality and cost by budget tier, so you can see whether the extra compute is paying for itself.
