A reasoning model is a language model that generates intermediate reasoning tokens before it produces its final answer. In AI engineering, that extra work can improve accuracy on multi-step problems, but it adds latency and token cost that you need to measure and control.
For example, suppose you ask a model for the total on an $80 item with a 25% discount and 8% sales tax. At low reasoning effort, a model may tax the original price and return $66.40. At higher effort, its intermediate steps carry the $60 discounted subtotal into the tax step, and it returns the correct $64.80. The prompt did not change. The inference budget did.
The intermediate tokens are often called thinking tokens or reasoning tokens. These example numbers are hypothetical.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why reasoning models matter for AI engineering
A reasoning model changes how much compute each request uses, not what you send it. You still send a prompt. You get back an answer, plus, depending on the provider, a summary or count of the reasoning that came before it.
That shift affects several everyday tasks:
- Choosing a setting: Decide when a higher reasoning budget is worth its extra latency and cost.
- Comparing prompts or models: Hold reasoning effort constant so you know what caused a quality change.
- Tracking cost: Account for reasoning tokens, which are usually billed even when they are not shown.
- Debugging failures: Tell reasoning errors apart from missing facts or missing context.
Extra reasoning helps most on multi-step arithmetic, code, planning, and constraint checking. It helps least on recall or simple rewriting, where it mainly adds cost and latency.
How reasoning models work
A standard language model generates an answer one token at a time. Each token is conditioned on the prompt and everything generated so far. This is the pattern used by autoregressive models.
A reasoning model inserts a longer generated sequence before the final answer. That sequence can contain intermediate steps, candidate approaches, self-checks, or backtracking.
The sequence comes from the same next-token process. What differs is that training and inference settings encourage the model to spend tokens working through the problem before it commits to an answer.
Spending more compute at inference time like this is a form of test-time compute.
How providers expose reasoning controls
Providers expose this control in different ways:
- Reasoning effort: A setting such as low, medium, or high.
- Thinking budget: An explicit limit on the number of reasoning tokens.
- Adaptive reasoning: The system decides how much to reason for each request.
Google’s Gemini thinking documentation is one public example of a budget-style control. It is the primary source for that provider’s behavior.
Because the controls differ, record the exact setting and the provider’s model version as experiment metadata. Do not assume that “medium” means the same thing at two providers.
Reasoning models vs. ordinary models
The line between a reasoning model and an ordinary model asked to “think step by step” is genuinely blurry.
The practical difference is control and measurement. In a reasoning model, the intermediate computation is a first-class part of inference. You can usually set it, and it is usually metered.
If you cannot set the reasoning, measure it, or see its token count, you are mostly doing prompt design, not configuring a reasoning model.
How to measure and inspect a reasoning model
Start with the trace. Open the span for a reasoning call and look for four things:
- Reasoning-token count.
- Total output tokens.
- Latency.
- The effort or budget setting recorded in the metadata.
Run a controlled comparison
To learn whether more reasoning helps, change only the reasoning setting:
- Hold the prompt, dataset, and evaluator fixed.
- Run the same set at two or more reasoning settings.
- Report task accuracy against tokens and latency per task.
This is the same discipline used in LLM evaluation generally, applied to one extra variable.
Here is an illustrative experiment record. The model name is a placeholder.
Experiment record (illustrative)
- model: <provider model, pinned version>
- reasoning_effort: medium (held constant across prompt variants A/B)
- prompt_variant: B
- result: accuracy, output tokens, reasoning tokens, p95 latency per task
Classify failures before raising the budget
Bucket failures by type before you spend more on reasoning.
Missing facts or missing context: More reasoning rarely helps. The model is reasoning carefully over the wrong evidence. Fix the context first.
Reasoning errors: Arithmetic slips, dropped constraints, or committing too early to a first approach are signs that a larger budget is worth testing.
Change one variable, rerun the same set, and compare. This is the pattern recommended throughout AI observability practice.
Worked example: a discount followed by a tax
The opening example is a classic multi-step slip. The model has to apply the discount first and then tax the discounted subtotal.
The following script computes the correct total and the total you get by taxing the original price.
price = 80.00
discount_rate = 0.25
tax_rate = 0.08
# Correct: apply the discount, then tax the discounted subtotal
subtotal = price * (1 - discount_rate)
correct_total = subtotal * (1 + tax_rate)
# Common slip: tax the original price instead of the subtotal
wrong_total = price * (1 - discount_rate) + price * tax_rate
print(f"discounted subtotal: ${subtotal:.2f}")
print(f"correct total: ${correct_total:.2f}")
print(f"slip (tax on original price): ${wrong_total:.2f}")
Output:
discounted subtotal: $60.00
correct total: $64.80
slip (tax on original price): $66.40
At low effort, a model may produce the plausible but wrong $66.40. At higher effort, the intermediate tokens show the $60 subtotal being carried into the tax step, and the answer matches the hand calculation.
This also shows why comparing two prompts at different effort settings is misleading. The higher-effort run had more computation available, regardless of how the prompt was worded.
Reasoning model vs. chain-of-thought prompting
Chain-of-thought prompting asks any model to show its steps in the visible answer. A reasoning model may use hidden or summarized intermediate tokens that never appear in the answer, under a provider-controlled budget.
The two overlap in spirit but differ in control and accounting:
- Prompted steps: Ordinary output text that you can read and that must appear in the answer.
- Reasoning tokens: Intermediate computation that may be hidden or summarized and is often counted separately.
Do not assume that no reasoning happened just because no steps are visible. Check the token counts in the trace.
Limitations of reasoning models
- More is not always better: On simple tasks, extra reasoning adds latency and cost with no accuracy gain.
- Confident wrong answers: When the task is limited by missing data, longer reasoning produces longer, more confident wrong answers.
- Wandering traces: Long reasoning can drift, contradict itself, or optimize for a derivation that only looks plausible.
- Incomplete visibility: Visible summaries do not expose the full hidden reasoning. They cannot prove why a model answered as it did, so do not use them as the only record for a high-stakes decision.
- Changing behavior: Provider behavior changes over time. An effort setting tuned this quarter needs re-evaluation when the model version changes.
Frequently asked questions
Does a reasoning model always think before answering?
No. Many systems choose reasoning length adaptively or let you set it to a minimal level. Short or factual requests may use few reasoning tokens, and some calls behave much like a direct answer.
Are reasoning tokens the same as output tokens?
For billing, they are often counted as output tokens, so check your provider’s pricing. They play a different role, though. They are intermediate computation, not the answer the user sees. Record them separately so cost analysis can tell answer length from thinking length, as described in token usage.
Can I read the model’s full reasoning?
Generally no. Providers may show a summary, a truncated excerpt, or nothing at all. Treat anything shown as a debugging aid, not as a complete or guaranteed-faithful record of the model’s internal computation.
When should I raise reasoning effort?
Raise it when failure analysis shows multi-step errors, such as dropped constraints, arithmetic slips, or premature answers. Do not raise it for missing facts or missing tools. Treat the change as a controlled experiment, and keep the setting pinned for later comparisons.
Why did my costs go up when I switched to a reasoning model?
Reasoning tokens are usually billed even when you never see them. A short visible answer can sit on top of a long reasoning sequence. Check the reasoning-token count in your traces to see where the spend goes.
Putting reasoning models to work
For every reasoning call, record the model version, the effort or budget setting, reasoning tokens, output tokens, and latency. Keep the setting constant when you compare prompts or models.
Before raising the budget, label your failures. Raise it only for reasoning errors, and measure accuracy against cost and latency on the same dataset.
Arize AX traces reasoning calls with their token counts and metadata, so you can see what each setting costs and whether it changed accuracy. For setup detail, see the Arize AX documentation.
