Turn detection is the decision, in a spoken conversation, that the current speaker has finished their turn and the other party may respond. In voice agents, it controls when the agent stops listening and starts answering, which shapes whether the agent feels natural, rude, or slow.
For example, suppose a caller says, “Move twenty dollars from savings to…” and pauses for 1.8 seconds before saying “checking.” An agent that answers after 1.2 seconds of silence cuts the caller off mid-sentence. An agent that notices “from savings to” is an incomplete phrase keeps listening, hears the full request, and acts once, correctly.
Turn detection is also called end-of-turn detection or endpointing.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why turn detection matters for voice agents
Every turn forces a trade-off. Respond too early and you interrupt the user mid-thought. Respond too late and every exchange feels sluggish.
Turn detection affects your work in several ways:
- Reducing interruptions: Avoid cutting users off while they pause to think, breathe, or read a number.
- Keeping replies fast: Avoid long waits after a user has clearly finished.
- Protecting downstream scores: A turn cut short leaves the transcript missing user content, which skews every later evaluation.
- Tuning per traffic type: Settings that work for short commands often fail on dictation, lists, and thoughtful pauses.
For more on voice systems as a whole, see voice agent.
How turn detection works
Turn detection builds on voice activity detection and then adds more evidence that the turn is complete.
Voice activity detection is not enough
A voice activity detector (VAD) classifies audio frames as speech or silence. It is necessary, but it cannot tell you the speaker is done.
People pause inside a turn constantly. They pause to breathe, to recall a number, or to read the next line of an address.
A system that answers at the first silence of a fixed length will cut users off. A system that always waits for a long silence will feel unresponsive, even after a simple “yes.”
Acoustic signals
Turn detection adds cues from the sound of the speech itself. Common acoustic cues include:
- Pitch: A final fall in pitch often marks the end of a turn.
- Lengthening: Speakers often stretch the last sound before a final pause.
- Energy: Speech often gets quieter as a turn ends.
Linguistic signals
Many systems also use the partial transcript. This is often called model-based endpointing.
Consider two utterances. “My account number is four seven…” is incomplete, even during a two-second pause. “That’s all, thanks” is complete almost immediately.
A model that reads the transcript can tell these apart. A silence timer cannot.
LiveKit’s documentation on turn handling options and its guide to tuning turn behavior show how these signals and thresholds are exposed in one production framework. The concepts generalize even where the settings differ.
Worked example: a mid-sentence pause
Here is the transfer request from the opening, handled two ways.
User: "Move twenty dollars from savings to... [1.8s pause] checking."
VAD-only endpointing, 1.2s silence threshold:
-> declares turn over during the pause
-> agent replies "Moving $20 from savings. To which account?"
-> user was mid-sentence: an interruption (false end-of-turn)
With linguistic check on partial transcript:
"...from savings to" is an incomplete phrase
-> system keeps listening past 1.2s
-> user completes the turn, agent acts once, correctly.
Cost of the fix on a genuinely finished short turn ("Yes."),
assuming (illustratively) the completeness check takes 0.2s:
reply starts at 1.2s + 0.2s = 1.4s instead of 1.2s
-> 0.2s of added delay on every finished turn the check must confirm.
The fix is not free. It prevents the interruption but adds a small delay to turns that were already finished.
No single silence threshold resolves this. The same 1.8-second pause is inside a turn here and would end a turn elsewhere.
How to measure turn detection
Measure both directions of error. Interruptions are false ends of turn. Response delay is how long the user waits after a true end of turn.
Define end-of-turn latency first
End-of-turn latency is typically measured from the true end of the user’s speech to the agent starting its response. Some pipelines measure to the moment the system commits to respond instead.
State which endpoint you use. Streaming speech generation can start before a full reply is composed, so the two endpoints can differ.
Count errors on labeled turns
Label a sample of recorded turns with the true end of each turn. Then count:
- Interruptions: The system responded before the user finished.
- Premature cut-offs: The recording or input stopped before the user finished.
- End-of-turn latency: The delay on turns where the end was detected correctly.
Report interruptions per hundred turns. Report latency as a median plus a high percentile.
Segment both by turn type, such as short answers, dictated numbers, and open-ended speech. A global average hides exactly the segments that fail.
Keep these segment results with your other evaluation metrics for the conversation, not in a single blended score.
See the trade-off in a threshold sweep
The following script uses a small hypothetical set of labeled turns. Each turn lists the pauses that occur inside it. A fixed silence threshold interrupts any turn that contains a pause at least that long.
# Hypothetical labeled turns: pauses (seconds) that occur *inside* each turn
turns = {
"Yes.": [],
"Cancel my order.": [0.4],
"That's all, thanks.": [],
"Account number four seven ... two nine": [0.9, 1.5],
"Move twenty dollars from savings to ... checking": [1.8],
"My address is 12 Oak Street ... apartment 4": [0.7, 1.1],
"I want to change my plan because ... it costs too much": [0.6, 1.3],
"Check my balance.": [],
}
for threshold in (0.8, 1.2, 2.0):
interrupted = [t for t, pauses in turns.items() if any(p >= threshold for p in pauses)]
print(f"silence threshold {threshold:.1f}s: "
f"{len(interrupted)} of {len(turns)} turns interrupted, "
f"{threshold:.1f}s wait after every finished turn")
Output:
silence threshold 0.8s: 4 of 8 turns interrupted, 0.8s wait after every finished turn
silence threshold 1.2s: 3 of 8 turns interrupted, 1.2s wait after every finished turn
silence threshold 2.0s: 0 of 8 turns interrupted, 2.0s wait after every finished turn
Raising the threshold removes interruptions but makes every finished turn wait longer, including “Yes.” The interrupted turns are all numbers, addresses, and longer explanations. That pattern is why segmenting by turn type matters.
Use production signals as a check
Ground truth for “the user had finished” is partly judgment. Conversation outcomes help confirm it.
Turns followed by “as I was saying,” a correction, or the user talking over the agent to finish a sentence are strong interruption signals in production data. A rise in interruption rate often shows up first in a review of agent failure modes, before users report it.
The Arize AX cookbook on tracing and evaluating audio covers capturing audio sessions in traces for this kind of review.
How to tune turn detection
Tune one variable at a time on the same labeled set. Typical variables are:
- The silence threshold.
- The endpointing model.
- The minimum and maximum wait bounds.
Adaptive approaches often work better than a fixed threshold. They shorten the wait when the transcript is clearly complete. They lengthen it for numbers, lists, or trailing words like “and” or “because.” Verify this on your own traffic rather than assuming it.
Tracing each turn’s audio, partial transcripts, and the moment the system committed to respond makes a reported interruption reproducible. Reviewing those records through AI observability shows whether a threshold change helped one segment while hurting another.
Turn detection vs. voice activity detection and barge-in
These three capabilities are often built into one component, but they answer different questions.
- Voice activity detection: Is there speech right now?
- Turn detection: Is the user’s turn over?
- Barge-in: Should the agent stop talking because the user started speaking?
Barge-in depends on VAD while the agent is speaking. It does not involve judging whether the user’s turn is complete.
Teams often confuse these. They tune a silence threshold to fix interruptions that are really barge-in failures, or the reverse. Name the failing decision before you change any settings.
Limitations of turn detection
- No certainty: Humans signal turn ends probabilistically and sometimes change their minds mid-pause.
- Degraded audio: Accents, speech impairments, background speech, crosstalk, and poor microphones weaken acoustic and transcript signals at the same time.
- Users adapting to failure: Callers who speak in unnatural, pause-free runs to avoid being cut off are evidence the system has already failed them.
- A floor on latency: Waiting for more evidence always costs time. The practical goal is an accepted interruption rate at an accepted delay for each turn type, not zero of either.
- Settings drift with context: Revisit thresholds when the user population, language, audio channel, or conversation style changes. A setting tuned on quiet desktop calls may behave badly on car audio.
Frequently asked questions
What silence threshold should I use?
There is no universal value. Start from your framework’s default, label real turns, and measure interruptions against end-of-turn latency as you vary it. Expect different effective waits for complete and incomplete-sounding speech rather than one fixed number.
Is a smarter endpointing model always better than a threshold?
Usually for mixed traffic, because it can use the partial transcript to tell a mid-phrase pause from a finished sentence. It adds its own errors and latency, though. On narrow, command-style traffic, a threshold may perform as well. Decide from labeled data, not from demos.
How do I count an interruption fairly?
Count a turn as interrupted when the system committed to a response before the user’s true turn end. Use recordings and transcripts as evidence. In production, corroborate with user repairs such as “as I was saying,” and report the rate per segment.
Does turn detection affect evaluation of the whole conversation?
Yes, directly. A transcript produced after premature cut-offs is missing user content, so downstream scoring of understanding and task success inherits the error. Fix and measure endpointing before drawing conclusions from conversation-level scores.
What is the difference between turn detection and barge-in?
Turn detection decides when the user has finished so the agent can reply. Barge-in decides when to stop the agent’s own speech because the user started talking. They fail differently and need different fixes.
Putting turn detection to work
Label a sample of real turns and segment them by type. Measure interruptions per hundred turns and end-of-turn latency for each segment.
Record which endpoint your latency uses, the threshold or model in use, and the wait bounds. Change one setting at a time and rerun the same labeled set.
Turn detection is one part of voice agent evaluation, which also scores task success, transcription, and timing across the whole conversation. Arize AX can trace audio sessions so you can replay an interruption and see exactly when the system decided to respond.
