End-to-end latency is the wall-clock time between a defined start point, such as a user submitting a request, and a defined completion point. It is the latency a person actually experiences, so it is the headline speed metric for AI applications and agents.
For example, suppose a support agent task takes 15 seconds from request to final answer. The trace shows 1 second of queueing, a 6-second model call, two 3-second tool calls running in parallel, a 2-second retry, and a final 3-second model call. Adding up every span gives 18 seconds, but only 15 seconds actually passed, because the two tool calls overlapped.
The completion point might be the final answer delivered or the task’s success check passing. Every wait in between counts.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why end-to-end latency matters for agents
A user does not experience your fastest component. They experience every serial step and every wait, from the moment they ask to the moment they get a result.
End-to-end latency is useful for:
- Setting user-facing targets: Measure the delay people actually notice, not just model speed.
- Comparing releases: Check whether a change made the whole task faster or only one component.
- Finding the slowest stage: Break the total into stages to see where time goes.
- Weighing trade-offs: Track latency alongside task success and cost when you change models or tools.
Arize’s overview of agent evaluation metrics treats task-level latency this way, as one measure next to task success and cost.
What end-to-end latency includes
The contents depend on what you are timing.
A single model request: The total includes time waiting for capacity, processing the prompt, and generating the answer.
An agent task: The total also includes everything around the model. That means loading context, calling tools and waiting on external systems, running validators, retrying failed steps, and any human approval waits you choose to include.
NVIDIA’s AIPerf metrics reference draws the same boundary at the request level for inference serving.
Start and end points
End-to-end latency means nothing until you state its start and end points. Two teams measuring from different points are not comparing the same thing.
“Response complete” alone is ambiguous. It can mean any of these:
- Last token generated: The model finished producing output.
- Last token delivered: The client received the full response.
- Task marked successful: A success check confirmed the task was done.
These differ, especially with streaming. Pick one and write it down.
How to measure end-to-end latency
Choose endpoints you can timestamp reliably. Then measure the wall-clock time between them.
Formula:
end-to-end latency = t1 - t0
Where:
- t0 is the timestamp when your application accepts the request.
- t1 is the timestamp when the final response is fully delivered or the task’s success is recorded.
Measure the clock, not the spans
Never compute end-to-end latency by adding up span durations. In a trace with parallel tool calls, spans overlap. Summing them counts the same seconds twice.
Subtracting the first start time from the last end time does not have this problem. You can see the overlap in any distributed tracing view of a real agent run.
The following script models the trace from the opening example. Each span has a start and end time in seconds. It compares the sum of span durations with the actual wall-clock time.
# (name, start_s, end_s) relative to request start
spans = [
("queue + context", 0, 1),
("model call 1", 1, 7),
("tool A", 7, 10),
("tool B", 7, 10),
("retry", 10, 12),
("model call 2", 12, 15),
]
summed = sum(end - start for _, start, end in spans)
wall_clock = max(end for *_, end in spans) - min(start for _, start, _ in spans)
model_time = sum(end - start for name, start, end in spans if name.startswith("model"))
print(f"sum of span durations: {summed} s")
print(f"wall-clock latency: {wall_clock} s")
print(f"model calls: {model_time} s ({model_time / wall_clock:.0%} of wall clock)")
Output:
sum of span durations: 18 s
wall-clock latency: 15 s
model calls: 9 s (60% of wall clock)
The 3 extra seconds in the sum come from tool B, which ran at the same time as tool A.
Report a distribution, not an average
Averages hide the slow tail that users remember. Report at least a median and a high percentile, such as p95.
Name the request mix when you report. Long tasks and short tasks averaged together describe neither one, so segment by task or request type.
Worked example: where the 15 seconds go
Here is the breakdown of the illustrative support task from the opening example.
| Stage | Duration | Runs in parallel with |
|---|---|---|
| Queueing and context assembly | 1 s | Nothing |
| Model call 1 | 6 s | Nothing |
| Tool call A | 3 s | Tool call B |
| Tool call B | 3 s | Tool call A |
| Retry of a failed step | 2 s | Nothing |
| Model call 2 | 3 s | Nothing |
| Wall clock | 15 s |
The breakdown shows where to act:
- Model calls: The two calls take 9 of the 15 seconds. Faster LLM inference is the largest lever.
- Retry: The 2-second retry costs more than the 1-second queue. Fixing the flaky tool also removes a reliability problem.
- Tool calls: They already run in parallel, so they add 3 seconds, not 6.
End-to-end latency tells you there is a problem worth attention. The breakdown tells you which problem. A single trace gives you both the total and the breakdown for one task.
For more on investigating slow runs, see debugging agents.
End-to-end latency vs. related latency metrics
Use end-to-end latency as the headline number and the component metrics as diagnostics.
- Time to first token: Measures only the wait until generation starts. A system can feel responsive on this measure and still take a minute to finish.
- Inter-token latency: Measures generation speed after the first token. It says nothing about queueing, tools, or retries.
- Request latency: Usually refers to one model or service request. End-to-end latency for an agent task may contain many requests plus tool time.
- Throughput: Counts work completed per unit time across many requests. Raising throughput can raise individual end-to-end latency under load, and the reverse can also happen.
Optimizing a component that is not on the critical path will not move the headline number. In the worked example, speeding up tool A alone saves nothing while tool B still takes 3 seconds.
Common mistakes
- Ambiguous endpoints: Comparing numbers measured from different start or finish points produces a meaningless comparison.
- Including human waits: Counting an overnight human approval in an agent’s latency makes the metric unusable for tuning the system. Exclude it, and say that you did.
- Hiding retries: A task that succeeds after three attempts has one end-to-end figure and a reliability problem the figure conceals. Pair latency with attempt counts.
- Ignoring load: Latency measured on an idle system does not predict latency at your peak concurrency. State the load when you report a number.
- Summing spans: Overlapping spans make the sum larger than the real elapsed time.
Frequently asked questions
How is end-to-end latency different from LLM latency?
LLM latency usually describes one model request. End-to-end latency covers the whole request or task as the user experiences it. That includes queueing, context assembly, every model call, tool calls, retries, and delivery.
Why can’t I add up span durations to get end-to-end latency?
Because spans overlap. Parallel tool calls and nested operations run at the same time, so adding their durations counts shared seconds twice. Subtract the start timestamp from the end timestamp instead.
Should I report average end-to-end latency?
No, report a distribution instead. A median and a high percentile such as p95, segmented by task or request type, show the tail that users complain about. A single average blends fast and slow work.
What usually dominates end-to-end latency in an agent?
It varies, which is why you measure it. Common contributors are long model generations, slow or retried tool calls, and serial steps that could run in parallel. Inspect the trace breakdown before you optimize.
Should streaming change how I measure end-to-end latency?
It changes which endpoint you should name. With streaming, the last token generated and the last token delivered can differ. Track time to first token alongside end-to-end latency so you see both responsiveness and total time.
Putting end-to-end latency to work
Write down your start point, end point, and the request mix before you report a number. Record the median, a high percentile, and the load at the time of measurement.
Pair each latency figure with attempt counts and task success, so a fast number does not hide retries or failures.
When a task is slow, open its trace and read the breakdown before changing anything. Arize AX records span timings across model, retrieval, and tool steps, so you can see which stage is on the critical path.
