Token usage is the number of tokens a large language model (LLM) processes and generates for a request or agent run, broken out by category. In AI engineering, token usage is how you price each model call, explain a bill, and find the agent step that suddenly got more expensive.
For example, suppose one step of a support agent reports 12,000 input tokens, of which 9,000 were cached, and 800 output tokens, of which 500 were reasoning tokens. At illustrative prices, that step costs $0.0237. Output is the smallest count but the largest cost, and the 500 reasoning tokens are not charged a second time.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why token usage matters
A single total token count hides most of what you need to know. Two calls with the same total can cost very different amounts, depending on how the tokens split across categories.
Token usage is useful for:
- Estimating cost: Multiply each category by its own price to get the cost of a call or run.
- Debugging spikes: Find the step whose input or output grew before you change the prompt or model.
- Reconciling bills: Match your trace totals to the provider’s invoice, category by category.
- Checking optimizations: Confirm that caching, shorter context, or lower reasoning effort reduced the category you targeted.
- Comparing runs: See whether a cheaper run is really cheaper once retries and failures are counted.
How token usage is counted
A token is a unit of text produced by a tokenizer. One English word is often one or two tokens. Code, numbers, and non-English text tokenize differently, so character counts are not a reliable substitute.
Providers count tokens with their own tokenizer. That provider count, not a local estimate from a different tokenizer, is what appears on the bill. For more on how text becomes tokens, see tokenization.
For a single model call, most providers report some version of the following categories.
Input tokens
Input tokens (also called prompt tokens) cover everything sent to the model on that call. This includes system and developer instructions, conversation history, retrieved documents, tool definitions, tool results, and the current user message.
In an agent, input usually grows at every step. History and tool results are resent with each new call.
Output tokens
Output tokens (also called completion tokens) are the tokens the model generates in reply. They include tool-call arguments.
A short visible answer can still have substantial output. The model may have produced long tool calls or structured data that the user never sees.
Cached input tokens
Cached input tokens are the part of the input that the provider served from a prompt cache instead of processing as new input. They are usually billed at a lower rate.
Arize’s prompt caching benchmark shows how this shows up in practice. For the mechanism itself, see prompt caching.
Reasoning tokens
On a reasoning model, reasoning tokens are generated for internal reasoning before or alongside the answer. Providers handle them differently.
Some bill reasoning tokens as part of output. Some report them as a separate sub-count of output. Some do not expose them at all. They are rarely an independent total that you add on top of output.
The most common accounting error
Consider a response that reports 800 output tokens, of which 500 were reasoning tokens. The billable output is 800, not 1,300.
Treat any sub-count as a breakdown of a larger category until the provider documentation says otherwise. The same rule applies to cached tokens, which are usually a portion of input.
How to calculate cost from token usage
For one call, multiply the tokens in each category by that category’s price, then add the results.
Formula:
call_cost = sum over categories of (tokens_in_category × price_per_token_for_category)
run_cost = sum of call_cost over every model call in the run
Where:
- tokens_in_category is the billable count for one category, such as uncached input, cached input, or output.
- price_per_token_for_category is the provider’s current price for that category.
- call_cost is the cost of one model call.
- run_cost is the cost of a whole agent run.
For an agent run, include every call: the planner, each tool-selection call, retries, summarization calls, and any evaluator calls your application makes. A run-level number is only meaningful if you state what it includes.
Calculate the cost of one agent step in Python
The following script prices the support agent step from the opening example. It also shows what two common accounting mistakes would report. The prices are illustrative only, so use the current price list for your provider and model.
PRICE_PER_MILLION = {
"uncached_input": 3.00,
"cached_input": 0.30,
"output": 15.00,
}
usage = {
"input_tokens": 12_000,
"cached_input_tokens": 9_000,
"output_tokens": 800,
"reasoning_tokens": 500, # already inside output_tokens
}
billable = {
"uncached_input": usage["input_tokens"] - usage["cached_input_tokens"],
"cached_input": usage["cached_input_tokens"],
"output": usage["output_tokens"],
}
total = 0.0
for category, tokens in billable.items():
cost = tokens * PRICE_PER_MILLION[category] / 1_000_000
total += cost
print(f"{category:<16}{tokens:>7,} tokens ${cost:.4f}")
print(f"{'step total':<30}${total:.4f}")
# Two common accounting mistakes
no_cache_discount = usage["input_tokens"] * 3.00 / 1_000_000 + 800 * 15.00 / 1_000_000
double_reasoning = total + usage["reasoning_tokens"] * 15.00 / 1_000_000
print(f"cached billed as uncached: ${no_cache_discount:.4f}")
print(f"reasoning counted twice: ${double_reasoning:.4f}")
Output:
uncached_input 3,000 tokens $0.0090
cached_input 9,000 tokens $0.0027
output 800 tokens $0.0120
step total $0.0237
cached billed as uncached: $0.0480
reasoning counted twice: $0.0312
Worked example: reading one step’s usage
Here is the same step as a table. The prices are hypothetical, chosen only to show the calculation.
| Category | Tokens | Price per million | Cost |
|---|---|---|---|
| Uncached input | 3,000 | $3.00 | $0.0090 |
| Cached input | 9,000 | $0.30 | $0.0027 |
| Output (includes 500 reasoning) | 800 | $15.00 | $0.0120 |
| Step total | 12,800 | $0.0237 |
Output dominated the cost despite being the smallest count. Reporting only “12,800 total tokens” would mislead any cost review.
Reasoning did not add a separate charge. Its 500 tokens are already inside the 800 output tokens. Counting them again would report $0.0312.
Counting the cached tokens at the full input rate would report $0.0480. That overstates this step by about $0.024, which becomes a large error across thousands of runs.
How to debug token usage
Start from the trace, not the invoice. In the Arize Phoenix documentation and Arize AX documentation, each LLM span carries token counts. You can sort the spans in a run by input tokens, output tokens, or cost and find the step that changed.
These patterns are worth checking:
- Input grows across steps: History, tool results, or retrieved text is usually being resent in full. Count how much of the input is new at each step.
- High cached share with no saving: Confirm the provider actually discounted those tokens. Then check that your accounting did not also count them as uncached.
- Output spikes on a few calls: Look for retries, verbose tool arguments, or a model that started writing long explanations. Compare against the step’s visible answer.
- Run total disagrees with the bill: Reconcile category definitions first, especially reasoning and cached sub-counts. Only then assume your instrumentation is missing calls.
At the aggregate level, track median and tail tokens per run alongside task success. A cheaper run that fails and retries can cost more per successful task than a longer run that succeeds once. For more on that measure, see cost per successful task.
Limitations of token usage
Token usage is a consumption measure. Several things fall outside it.
- Cross-provider comparisons: Tokenizers differ, category names differ, and some providers bundle or hide categories that others expose. Comparisons need a documented mapping, not assumed equivalence.
- Hidden reasoning: Reasoning-token visibility is uneven. You may pay for tokens you cannot inspect.
- Untraced calls: If a call bypasses your tracing, or a response omits a usage field, your trace total will understate the bill.
- Value: Token usage cannot tell you whether the tokens improved the answer. That requires evaluation.
This is why token usage belongs next to quality measures such as those in evaluation metrics, not on its own.
Token usage vs. token count, context window, and cost
These terms are often used as if they were interchangeable. They are not.
- Token count: Usually one number for one piece of text or one call. Token usage is the categorized accounting across a request or run.
- Context window: A capacity limit, the maximum number of tokens a model accepts at once. Usage tells you how much of that capacity, and of your budget, you consumed.
- Cost: Usage multiplied by category-specific prices. Two runs with identical totals can cost different amounts if one used more output or fewer cached tokens.
Keeping the categories separate also clarifies optimization. Shortening retrieved context reduces uncached input. Caching a stable system prompt moves input into the cached category. Limiting reasoning effort reduces output. Each change acts on a different term in the sum.
Frequently asked questions
Does token usage include failed or retried calls?
Usually yes for billing. If the provider processed the call, its tokens were consumed whether or not your application used the result. Your run-level accounting should do the same, and it should count evaluator or summarizer calls your system makes as part of the task.
Are cached tokens free?
No. Cached input is typically billed at a reduced rate rather than at zero, and only the matched prefix counts. Check the cached share and the rate actually applied instead of assuming a saving from the cache label alone.
Why does my local token count differ from the provider’s count?
Tokenizers differ by model and provider. Providers may also include special tokens, tool definitions, or formatting you did not count locally. Use provider-reported usage for billing and keep local counts for relative comparisons only.
Do I add reasoning tokens to output tokens?
Usually not. Reasoning tokens are rarely an independent total. Where providers report them, they are typically a breakdown of output. Check your provider’s documentation before adding any sub-count to a total.
Should I optimize total tokens or cost per successful task?
Optimize cost per successful task. Reducing tokens on a step that then fails more often can raise total spend once retries are included. Pair token usage with a success measure on the same runs before deciding that a change saved money.
Putting token usage to work
Record token usage by category for every model call, not as one total. Note which categories are sub-counts of others for each provider you use.
Sum usage across every call in a run, including retries, summarizers, and evaluators. Price each category separately with your provider’s current rates.
When a run gets more expensive, find the span whose usage changed before you edit the prompt. Tools such as Arize Phoenix record token counts on each LLM span, which makes that step easy to locate. For the broader picture of what to monitor alongside cost, see AI observability.
