A voice agent is an AI system that holds a spoken conversation with a user and completes tasks by acting on what it hears. Voice agents matter in AI engineering because every stage, from listening to speaking, runs on a clock, and a failure in any one stage changes the whole conversation.
For example, suppose a caller says, “Move my delivery to Friday, please.” The voice agent transcribes the request, detects that the caller has finished speaking, and calls a tool to look up Friday slots. It hears back 09:00 and 13:00 and asks, “I can do nine or one o’clock. Which suits you?”
The caller interrupts with “Nine is fine.” The agent stops speaking, books the 09:00 slot, and confirms.
Now suppose the transcript had said “Monday” instead of “Friday.” The agent would book Monday quickly and fluently. Nothing downstream would look wrong. Only the recorded audio would show the mistake.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why voice agents matter
A voice agent does more than transcribe speech or read text aloud. It has to listen, decide when the user has finished, act, and reply, all fast enough to feel like a conversation.
Understanding how voice agents work helps you with:
- Locating failures: Trace a bad call to one stage, such as transcription, turn detection, reasoning, tool use, or speech output.
- Treating latency as quality: Slow or badly timed replies change what users say next, so delay is part of the result.
- Designing confirmations: Decide which spoken details to read back before the agent acts on them.
- Evaluating conversations: Score interruptions, repairs, and abandoned turns, not only the final answer.
How a voice agent works
Most voice agents use one of two designs. A cascaded design chains separate models. A speech-to-speech design uses a model that works with audio more directly.
The cascaded pipeline
In the common cascaded design, a call passes through these stages:
- Audio front end: Captures speech and detects voice activity.
- Speech-to-text (STT): Produces a transcript, often streaming partial results. This stage is also called automatic speech recognition (ASR).
- Language model or agent loop: A large language model (LLM) interprets the transcript, keeps track of dialogue state, and calls tools such as an order lookup or a booking system.
- Text-to-speech (TTS): Speaks the reply. Ideally it streams, so speech starts before the full answer is generated.
The conversational controls
Around that chain sit the controls that make it a conversation:
- Turn detection: Decides when the user has finished speaking.
- Barge-in handling: Stops the agent’s speech when the user interrupts.
- Session state: Carries what has been established so far in the call.
For more on deciding when a user has finished, see turn detection.
Speech-to-speech models
Speech-to-speech models collapse some of those stages by taking in and producing audio more directly. That can reduce hand-off errors between separate STT, LLM, and TTS components.
It does not remove the conversational jobs. The system still has to decide when to listen, when to speak, what action to take, and how to recover when it mishears.
LiveKit’s overview of agent sessions describes the session lifecycle these components share.
A voice agent is still an agent
A voice agent is an agent whose input and output are speech. The agency questions are the same as for a text agent: which tools, how much autonomy, and which actions need confirmation. Voice adds timing to all of them.
How to trace a voice agent
Record each stage as a span so you can see where a conversation went wrong. Arize’s Pipecat tracing integration for Arize AX shows how a cascaded pipeline’s stages are captured as spans.
Here is the delivery example as an illustrative session trace:
Task: caller wants to move a delivery to Friday.
Session trace (illustrative shape):
turn 3 STT: "move it to Friday please" confidence n/a
turn detection: end-of-turn at +0.4s OK
LLM: intent=change_delivery, tool=get_slots(day=Fri)
tool result: slots 09:00, 13:00
TTS: "I can do nine or one o'clock. Which suits?"
turn 4 user barges in: "nine is fine"
agent stops speaking within the frame budget
LLM: tool=book_slot(day=Fri, time=09:00) -> confirmed
Failure variant worth testing: STT hears "move it to Monday",
agent books Monday fluently and quickly. Nothing downstream
looks wrong. Only the audio disagrees with the transcript.
The failure variant is why the record must keep the audio linked to the transcript. A transcript-only trace would show a clean, successful call.
Keep the audio with the spans
Inspection needs the audio, not just the transcript. Tracing and evaluating audio sessions in Arize AX covers recording audio alongside spans, so a reviewer can listen to the disputed turn.
With stages as spans, you can break a bad conversation into questions:
- Was the transcript wrong? Listen to the audio.
- Was the turn cut off early? Check the turn-detection timing.
- Was the decision wrong on a correct transcript? Read the model step.
- Was a correct decision spoken badly or late? Check synthesis and latency.
AI observability across many traces turns individual bad calls into failure rates per stage. That is what makes a fix verifiable.
Where voice agents fail
Mishearing is the signature failure. Names, amounts, addresses, and negations are short, sound alike, and carry consequences.
Confirmation strategy is part of the design. Read back the fields that are expensive to get wrong. Do not read back everything.
Timing failures come next. These include responding over the user, waiting so long that the user repeats or hangs up, and failing to stop when interrupted. Each one teaches users to shout, over-enunciate, or abandon the call.
Reasoning and tool failures look the same as in text agents: wrong intent, wrong arguments, or invented policy. The difference is that the user may not see the error, only hear its result.
How to evaluate a voice agent
Score the whole conversation, not just the final answer. A useful starting set covers four areas:
- Task completion: Did the agent finish real user intents?
- Transcription quality on key fields: Were the names, amounts, and dates that drove actions transcribed correctly?
- Interruption and delay behavior: Did the agent stop when interrupted and reply without long pauses?
- End-to-end latency: How long did each turn take from the user finishing to the agent speaking?
Measure these per stage on recorded calls before you tune anything. For a common transcription measure, see word error rate.
The reasoning and tool-use layers can share cases and judges with text agents. Text evaluations cannot see mishearing, timing, or spoken-output failures, so add audio-level checks on sampled real calls.
For more on scoring full calls, see voice agent evaluation. Judging behavior across audio and text together falls under multimodal evaluation.
Limitations of voice agents
These limits come from the pipeline and its controls. A better language model alone does not solve them.
- No review step: A text user can reread a wrong summary before sending it. A caller often cannot, so consequential spoken actions need explicit confirmation and an undo path.
- Uneven recognition: Noise, accents, overlapping speakers, and low-quality phone audio degrade recognition unevenly across users. Average accuracy can look fine while one group of callers consistently fails.
- Tight latency budgets: Each stage’s delay adds up. A chain of individually fast stages can still answer too slowly.
- Privacy obligations: Recordings are close to biometric data and contain whatever else was audible on the call.
Voice agent vs. IVR, voicebot, and voice-enabled chatbot
Interactive voice response (IVR) systems route callers through recorded menus with keypad or keyword choices. The paths are written in advance.
A voice agent interprets free speech with a model. It can pursue a goal across turns, ask clarifying questions, and recover from ambiguity.
“Voicebot” is used loosely for both, so check which one a given product means.
Adding speech input and output to a text chatbot gives spoken access to that chatbot. Without turn management, barge-in, and spoken confirmations, it will feel like a form being read aloud. It uses voice, but it is not yet a voice agent in the conversational sense.
Frequently asked questions
What components does a voice agent need?
At minimum, a voice agent needs audio capture with voice activity detection, transcription or a speech-capable model, dialogue and tool logic, speech output, and session state. To converse rather than take dictation, it also needs turn detection and barge-in handling. Monitoring each stage is what keeps the system maintainable.
How is a voice agent different from speech-to-text with a chatbot?
Speech-to-text plus a chatbot exchanges messages. A voice agent manages a real-time conversation. It decides when the user has finished, can be interrupted, confirms consequential details aloud, and recovers from mishearing within the call.
What should I measure first in a voice agent?
Start with task completion on real intents and transcription accuracy on the fields that drive actions, such as names, amounts, and dates. Then add interruption behavior, response delay, and end-to-end latency. Measure per stage on recorded calls before tuning anything.
Can a voice agent use the same evaluations as a text agent?
Partly. The reasoning and tool-use layers can share cases and judges. Text evaluations cannot see mishearing, timing, or spoken-output failures, so add audio-level checks on sampled real calls.
Why does my voice agent pass text tests but fail on real calls?
Text tests skip the audio stages. Mishearing, early turn cutoffs, and slow replies only appear when you test with real audio and real timing.
Putting voice agents to work
Instrument every stage as its own span, and keep the audio linked to each turn’s transcript. Without the audio, a mishearing looks like a successful call.
Decide which fields the agent must read back before acting, such as dates, amounts, and names. Test the cases where those fields are misheard.
Review sampled real calls per stage, not just overall success. Tools such as Arize AX can store the audio, transcript, and spans together, so you can listen to the turn that went wrong.
