Voice agent evaluation is the practice of measuring whether a speech-driven AI agent completes the user’s task correctly. It joins task checks with audio-specific checks on transcription, response timing, and turn-taking, because a transcript that reads well can still describe a failed conversation.
For example, suppose a caller says: “Book me for Tuesday, no, Wednesday morning, under Patel, P-A-T-E-L.” A good evaluation checks four things. The correction to Wednesday was honored, the name and spelling were captured, a Wednesday morning booking exists in the final system state, and the agent waited for the correction instead of booking Tuesday during the pause.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why voice agent evaluation matters
A voice interaction has failure modes that text chat does not. The agent can mishear a name, answer too slowly, or start talking before the caller finishes.
No single measure describes a voice interaction. You need to evaluate task success, transcription, timing, and turn-taking together.
Voice agent evaluation is useful for:
- Confirming outcomes: Check that the booking, payment, or update in the final system state matches what the caller asked for.
- Catching transcription errors: Find misheard names, dates, amounts, and account numbers before the agent acts on them.
- Tuning responsiveness: Balance response speed against the risk of interrupting callers.
- Validating automated judges: Check that an LLM judge’s verdicts agree with human review of recorded calls.
- Refreshing test sets: Track how caller topics, accents, and audio conditions change over time.
For more on the agents themselves, see voice agent.
How voice agent evaluation works
A voice interaction passes through several stages. Audio is captured and transcribed. The agent reasons and may call tools. A response is generated, then synthesized as speech and played.
Each stage can fail in its own way, and an early error changes everything after it. If a name or amount is transcribed incorrectly, the agent may act fluently on the wrong value.
Evaluation therefore works at three levels.
Conversation level: Did the task complete? Were required confirmations given? Did the user need to repeat or correct the agent?
Turn level: Was each transcription faithful? Did the agent answer the request just made? Did it interrupt or leave long silences?
System level: How long did users wait after they finished speaking? How often were calls handed to a human? Where did callers abandon the interaction?
Arize’s tracing and evaluating audio cookbook shows an implementation path for recording and evaluating audio interactions. Use it for the procedure, and this entry for what to measure and why.
What to measure in a voice agent
Score each dimension separately, then report them together.
Task success: Check the final system state. Was the appointment booked for the time the caller asked for, under the right name?
Transcription quality: Word error rate is the common summary. Also check critical fields directly, because one wrong digit in a phone number matters more than several wrong filler words.
Timing: Measure the gap between the user finishing a turn and the agent starting to respond. Track total call duration as well.
Turn-taking: Count interruptions by each party, responses to half-finished sentences, and long silences.
How a system decides that a caller has finished speaking shapes both timing and interruptions. LiveKit’s turn documentation is a primary source for how one framework handles turn detection. Definitions of endpoints and interruptions vary by provider and framework, so record the definitions your system uses beside its results.
For more on how agents decide when a turn has ended, see turn detection.
How to build a voice evaluation set
Build the set from recorded or carefully constructed conversations that reflect real callers. Include a range of conditions:
- Speakers: Different accents and speaking speeds.
- Environments: Background noise and poor connections.
- Requests: Multi-part requests and corrections mid-sentence.
- Pauses: Callers who stop to think before finishing.
Synthetic voices help with repeatability. A set made only of clean synthetic speech will still miss the failures that real audio causes.
Keep three things with each case: the reference transcript, the expected task outcome, and any required disclosures or confirmations.
The LLM evaluation entry covers general evaluator design. For the review side, see human evaluation.
How to measure timing and interruptions
Timing and interruption counts come from timestamps, not from transcript text. Each speech segment needs a speaker, a start time, and an end time.
The following Python code walks through a hypothetical timeline of the Patel call. For each agent turn, it checks whether the agent started before the caller’s last segment ended. If so, it counts an interruption. Otherwise, it records the response gap.
segments = [
("user", 0.0, 1.9), # "Book me for Tuesday"
("user", 2.6, 4.2), # "no, Wednesday morning"
("agent", 3.8, 5.0),
("user", 5.3, 7.4), # "under Patel, P-A-T-E-L"
("agent", 7.0, 9.5),
("user", 10.1, 10.6), # "Yes"
("agent", 11.3, 12.4),
]
interruptions = 0
gaps = []
last_user_end = None
for speaker, start, end in segments:
if speaker == "user":
last_user_end = end
elif last_user_end is not None:
if start < last_user_end:
interruptions += 1
print(f"agent at {start:.1f}s: interrupted caller "
f"({last_user_end - start:.1f}s overlap)")
else:
gaps.append(start - last_user_end)
print(f"agent at {start:.1f}s: waited {start - last_user_end:.1f}s")
print(f"agent interruptions: {interruptions}")
print(f"mean response gap: {sum(gaps) / len(gaps):.1f}s")
Output:
agent at 3.8s: interrupted caller (0.4s overlap)
agent at 7.0s: interrupted caller (0.4s overlap)
agent at 11.3s: waited 0.7s
agent interruptions: 2
mean response gap: 0.7s
The mean response gap looks fine at 0.7 seconds. That number only covers the one turn where the agent waited. The agent spoke over the caller twice, and a transcript of this call would not show it.
This simple rule only counts the agent starting early. A complete check would also count the caller talking over the agent and whether the agent recovered correctly.
Worked example: evaluating the Patel booking
Return to the caller from the opening. A useful trace links five things: the audio, a timestamped transcript, the agent’s interpretation, the booking tool call, and the final booking record.
| Check | Evidence needed | What it catches |
|---|---|---|
| Correction honored | Transcript and agent interpretation | Agent kept “Tuesday” |
| Name captured | Transcript and tool call arguments | Misspelled “Patel” |
| Booking exists | Final booking record | Wrong day or no booking |
| Agent waited | Timestamps | Booking during the pause |
| No talking over the caller | Timestamps | Interruptions |
Suppose a judge reads only the final transcript. It might mark this conversation correct while missing that the agent spoke over the caller twice. Timing and interruption counts catch what the transcript hides.
Use AI observability practices to keep audio, spans, and evaluator results connected. The trace entry describes the underlying record structure.
If a conversation is escalated, record the reason. Also record whether the human received the transcript and summary.
Common mistakes in voice agent evaluation
- Trusting transcript-only judges: A judge given only a transcript cannot hear tone, overlapping speech, or a caller struggling to be understood. It may also treat a confident wrong transcription as ground truth.
- Relying on word error rate alone: A low word error rate does not guarantee that names, amounts, or requested actions were understood correctly.
- Optimizing one timing number: Responding quickly reduces silence but raises interruptions if the system decides a turn has ended too early. The result feels either sluggish or rude.
- Testing only clean audio: Sets without interruptions, pauses, background noise, accents, and corrections miss common real failures.
- Letting the set age: Caller topics, accents in the user base, telephony conditions, and the agent’s own voice or prompts change over time.
Give automated judges the same evidence a reviewer would need, or restrict them to checks that text can support. Then calibrate them against human-labeled calls.
Human review is slower, and reviewers can disagree. Define the rubric and check agreement on a sample.
Refresh cases from recent production calls under your privacy and consent rules. State retention limits for audio in the evaluation plan.
Voice agent evaluation vs. text evaluation
Text evaluation starts from an exact written input. Voice evaluation starts from audio, so the input itself is uncertain.
Transcription is an interpretation. Two systems can legitimately produce different transcripts of the same clip.
If you evaluate only the agent’s text responses against the reference transcript, you hide errors from the speech-to-text step. The agent acts on those errors. Evaluate against what the system heard as well as what the caller said, and report both.
Timing has no real parallel in text chat. A correct answer delivered after a long silence, or started before the caller finished, damages the interaction in ways a transcript score cannot see.
Voice callers also cannot easily reread an answer. Confirmations of names, dates, and amounts deserve explicit checks. A text interface might handle the same need with a visible summary.
Frequently asked questions
What should I evaluate first in a voice agent?
Start with task success on a representative set of recorded or constructed calls, checked against final system state. Next, add transcription checks for critical fields and basic timing and interruption counts. A polished conversation that books the wrong time is a failure regardless of its other scores.
Is word error rate enough to judge transcription?
No. Word error rate averages all word mistakes, but callers and tasks care about specific fields such as names, dates, amounts, and account numbers. Report word error rate for overall quality, and add field-level checks for the values the agent acts on.
Can an LLM judge evaluate voice conversations?
It can evaluate what is visible in the evidence it receives. For example, it can check whether the response addressed the request or whether a confirmation was given. It cannot hear interruptions, tone, or audio quality from a plain transcript, so pair it with human review and measured timing data.
How do I evaluate interruptions?
Count how often the agent starts speaking while the caller is still talking, and how often the caller talks over the agent. Also check whether the agent recovers correctly when interrupted. Define what counts as an interruption in your framework and report that definition with the counts.
Putting voice agent evaluation to work
For every evaluated call, keep the audio, the timestamped transcript, the agent’s tool calls, and the final task state linked in one trace. Report task success, critical-field accuracy, response gaps, and interruption counts side by side.
State who or what judged success. Check automated judges against human review of recorded calls before you rely on them.
Arize AX can trace audio interactions and attach evaluation results to them, which keeps the evidence for each verdict in one place. For broader background on scoring AI systems, see evaluations.
