Cost per successful task is the total cost of running an AI system divided by the number of tasks it completes successfully. In agent work, it shows what you actually spend to get one verified good outcome, including the cost of every failed attempt along the way.
For example, suppose an agent attempts 200 support tasks in one week. Model, tool, and evaluation charges total $160, including failed and retried runs. Review confirms that 80 tasks were resolved correctly, so the cost per successful task is $160 / 80 = $2.00.
The cost per attempted task in that example is only $0.80. Reporting $0.80 would hide the 120 tasks that failed. That is why the $2.00 figure should always appear next to the 40% success rate.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why cost per successful task matters for agents
Token prices and per-request costs describe usage. They don’t tell you what a useful result costs. Agents fail, retry, and call tools, and all of that spend sits behind the tasks that do succeed.
Cost per successful task is useful for:
- Choosing a model: Compare a low-priced model that fails often with a higher-priced model that succeeds more.
- Evaluating prompt or harness changes: Check whether a change lowered cost by succeeding more or by quietly doing less.
- Budgeting production workloads: Estimate what a target number of resolved tickets, code changes, or bookings will cost.
- Spotting waste: Find task types where retries and failed attempts consume most of the budget.
A cheaper model can have a higher cost per successful task if it fails and retries more often. This metric makes that visible.
For broader benchmark context, see the Arize cost per successful task benchmark.
How cost per successful task works
The metric has three parts: a task unit, a success criterion, and a cost boundary. You need to define all three before you calculate anything.
Define the task
Start with a task unit that matches the product. Examples include resolving a support ticket, producing a correct code change, completing a booking, or answering a question with the required evidence.
Define success
Success must be something you can check. A deterministic test passes. A transaction reaches a stated final state. A human or a calibrated evaluator confirms that the stated criteria are met.
Vague success, such as “the model returned an answer,” makes the denominator meaningless. Version the criterion and apply it the same way every time.
Define the cost boundary
Add up the costs incurred to attempt those tasks. In most setups, that includes:
- Model calls: Input and output tokens for every call, including retries and self-correction.
- Tools and retrieval: Search, database, and application programming interface (API) charges.
- Evaluation and review: Automated evaluator runs and human review, when they are part of producing the outcome.
State whether infrastructure, storage, and staff time are included. No single accounting boundary is universal. What matters is that you state yours and keep it the same across comparisons.
How to calculate cost per successful task
Divide the total attributable cost by the number of verified successes.
Formula:
cost per successful task = C / S
success rate = S / A
cost per attempt = C / A
Where:
- C is the total cost of all attempts in scope, including failed and retried runs.
- S is the number of tasks that met the success criterion.
- A is the number of tasks attempted.
The numerator includes failures. The denominator counts only successes. That asymmetry is the point of the metric.
Calculate it in Python
The following code compares two hypothetical agents on the same 200 tasks. Agent A uses a cheaper model. Agent B uses a more expensive model that succeeds more often.
systems = {
"Agent A (cheaper model)": {"cost": 160.00, "attempted": 200, "succeeded": 80},
"Agent B (pricier model)": {"cost": 300.00, "attempted": 200, "succeeded": 170},
}
for name, s in systems.items():
per_attempt = s["cost"] / s["attempted"]
success_rate = s["succeeded"] / s["attempted"]
per_success = s["cost"] / s["succeeded"]
print(
f"{name}: ${per_attempt:.2f}/attempt, "
f"{success_rate:.0%} success, ${per_success:.2f}/success"
)
Output:
Agent A (cheaper model): $0.80/attempt, 40% success, $2.00/success
Agent B (pricier model): $1.50/attempt, 85% success, $1.76/success
Worked comparison: a cheaper model that costs more
Suppose you are choosing between the two agents above. Both ran on the same fixed set of 200 tasks with the same success checker.
| Metric | Agent A | Agent B |
|---|---|---|
| Tasks attempted | 200 | 200 |
| Total cost | $160.00 | $300.00 |
| Verified successes | 80 | 170 |
| Cost per attempt | $0.80 | $1.50 |
| Success rate | 40% | 85% |
| Cost per successful task | $2.00 | $1.76 |
Agent A looks cheaper per attempt. It spends almost half as much per task.
Agent B is cheaper per good outcome. It costs about $1.76 per successful task compared with $2.00 for Agent A, and it resolves 90 more tasks.
The calculation settles the question, not the token price. It only does so when both agents run on the same tasks with the same success criterion and cost boundary.
How to inspect and debug the metric
A single average can mislead. Break it down before you act on it.
- Segment by task type: A mixed average can improve because the system received easier tasks, not because it got better.
- Use a fixed evaluation set: Compare models or prompts on the same tasks, not on whatever production traffic arrived that week.
- Segment production traffic: Split by intent, input length, and tool path. Split by customer tier only where that is appropriate.
- Explain every improvement: When the figure drops, check whether the system succeeded more, spent less per attempt, skipped necessary work, or narrowed what counts as success.
Trace-level records make the numerator auditable. Each attempt should carry the task identifier, attempt number, model and prompt version, token and tool costs, evaluator result, and final task state.
The agent-native evaluation entry explains why evaluating the full path matters for agents. For more on per-run measurement, see agent run evaluation.
Cost per successful task vs. related measures
Several cost and quality metrics sound similar. They answer different questions.
- Cost per request: Counts every request the same, whether it succeeds or fails.
- Cost per token: Describes usage, not outcomes. See token usage for how that is tracked.
- Accuracy: Asks how often outputs meet a criterion on a test set. It says nothing about what those correct outputs cost.
- Return on investment: Assigns a monetary value to a success and includes revenue. Cost per successful task does neither.
Cost per successful task is stricter than cost per request or per token. It assigns the cost of waste to the outcomes that justify running the system.
It also does not account for the cost of a harmful error beyond the run cost already counted. A system can have an attractive cost per successful task and still be unsuitable if its rare failures are severe. Pair it with success rate, error severity, latency, and human review burden.
For how cost fits alongside other agent measures, see agent evaluation metrics.
Limitations
The metric is easy to improve without improving the system. Watch for these problems.
- Weak success checker: If a weak evaluator judges success, the denominator can include bad outcomes. The metric then looks better as quality falls. Calibrate evaluators against human review on a sample, and audit successes as well as failures.
- Delayed confirmation: Some successes take days to confirm. State the observation window and how pending tasks are treated.
- Gaming the denominator: A team can lower the figure by attempting only easy tasks, abandoning hard tasks early, or weakening the success test. Fix task definitions, report coverage and abandonment, and review changes to the success criterion like code changes.
- Partial success: Decide on an explicit rule. Either define tiers and report each one separately, or count partial success as not successful for the headline metric.
- Incomplete costs: Cached tokens, retries inside a provider call, background evaluation, and human escalation are easy to omit. List exclusions under the chart.
If an exclusion differs between the systems you are comparing, the comparison is not valid.
For more on designing success criteria and evaluators, see evaluation metrics.
Frequently asked questions
Should failed attempts count in cost per successful task?
Yes. Failed attempts consume budget on the way to the successes that follow. Excluding them turns the measure into the cost of a successful run alone, which understates what the system spends per good outcome.
What counts as a successful task?
A successful task meets whatever criterion you define before measuring. That could be passing tests, completing a transaction correctly, or a review confirming the required result. If the criterion changes, restate historical results or mark a break in the series.
Can a more expensive model lower cost per successful task?
Yes. If it succeeds more often, needs fewer retries, or avoids costly tool errors, its cost per verified success can be lower despite a higher price per token. In the comparison above, the pricier agent costs $1.76 per success compared with $2.00.
How often should I report cost per successful task?
Report it over a window with enough tasks to be stable. Show it beside the success rate and total spend. For low-volume or high-value tasks, show the underlying counts so readers can see when a small denominator makes the figure jump.
Is cost per successful task a quality score?
No. It combines cost and success into one number, but it does not measure how good each success was or how bad each failure was. Use it to compare systems on the same tasks, not as a standalone quality score.
Putting cost per successful task to work
Before you calculate the metric, write down the task unit, the success criterion and its version, and the cost boundary. Keep all three fixed for any comparison.
When you report it, include the cost per successful task, the success rate, total cost, task count, and the evaluation method. Add any cost exclusions.
To make the numerator auditable, trace each attempt with its model calls, tool calls, costs, and evaluator result. Arize AX can capture those traces and attach evaluation results, so you can join cost to verified outcomes per task. For the evaluation side, see LLM evaluation.
