Skip to content

Why Voice AI Keeps Interrupting You

You paused to think. The AI thought you were done. Here’s how voice assistants decide when to speak, when to wait, and when to get out of the way.

How It Thinks4 min readEP 05
Paper collage of a caller and a telephone, with sound waves showing the assistant starting to speak before the caller corrects Friday to Thursday.
In this article

You paused to think. The AI thought you were done.

In a dramatized appointment call, the caller asks about Friday, then pauses. The assistant starts to reply. The caller’s next words change the request: actually, Thursday. Now both are speaking, and the assistant is already responding to the wrong day.

The problem begins before the answer. The assistant has to decide whether silence means “your turn” or “give me a second.” That decision is called turn-taking.

Watch Why Voice AI Keeps Interrupting You on YouTube

A pause tells the system you stopped making sound

A voice assistant receives a stream of audio. Voice activity detection, or VAD, estimates where speech starts and stops in that stream. One way to use it is to wait for a stretch of silence and then treat the caller’s turn as complete. OpenAI’s Realtime documentation calls this approach server VAD and lets developers tune the silence duration.

That works well when a caller gives a short, complete answer. It’s less reliable when they pause to find an account number, think of a date, or correct themselves. Shorten the wait and the agent may talk over the caller. Lengthen it across the board and even a clear “yes” can be followed by awkward dead air.

A waveform followed by a 500-millisecond silence timer in an example voice activity detection setup.
The episode uses 500 milliseconds as an example silence threshold. It is adjustable, not a universal rule.

The words give the system another clue

Compare “Thursday works” with “I can do Friday, unless…” Both can be followed by silence. The first sounds complete; the second signals that something is coming.

A system that uses language context can respond differently to those two pauses. OpenAI calls one version semantic VAD. It estimates whether the speaker has finished and can wait longer when the turn sounds incomplete. ElevenLabs’ guide to turn-taking describes other cues systems can use, including intonation, speaking pace, and filler words.

It’s still an estimate. “Friday” can be a complete answer, and there’s no way to know a correction is coming before the caller gives a clue or starts speaking again. The useful change is that the system has more to go on than a timer.

What if the assistant has already started talking?

There’s a second timing decision: when should the assistant stop speaking?

In some real-time systems, an intentional interruption like “Wait, I meant Thursday” can halt the assistant’s playback and return the floor to the caller. Voice designers call that a barge-in. But a brief “uh-huh” can mean “I’m listening,” not “stop.” If the assistant treats every sound as an interruption, it can cut itself off whenever the caller acknowledges it or there’s background noise. ElevenLabs discusses the difference between interruptions and acknowledgments.

Where the wait comes from in a real-time system

On a call, two moments can feel like the same delay: the assistant waiting to decide whether you've finished, and the assistant taking time to find an answer after it has decided. Those are different problems. A faster model or a faster calendar lookup can shorten the second wait. Neither, by itself, makes the assistant better at noticing that you still have something to say.

In a chained system, speech recognition turns audio into text, a model or workflow decides what to do, and a speech engine turns the response back into audio. A real-time system can stream more of that work as the conversation unfolds. Some systems can also listen while they speak. That matters because they can hear a correction while their reply is still playing. They still need to decide whether it is a correction, a quick acknowledgment, or background noise. OpenAI’s voice agent guide compares these architectures and the control each one gives builders.

Same call, different timing

In the replay, the assistant waits through the pause and hears the correction. Thursday becomes the day to check. The scene is a demonstration: no appointment is actually booked. In a real booking flow, the assistant would still need to check availability before offering or confirming a slot.

In the replay, the assistant waits through a pause and responds after the caller says “Actually, Thursday.”
In the replay, waiting lets the assistant hear the correction. It still needs to check availability.

The point of the replay isn't that an assistant should wait longer every time. It's that the extra time mattered at this moment. Waiting through a correction can prevent a wrong answer; making every clear “yes” wait the same amount can make a call feel sluggish.

What to test before you tune the timer

Try the same task with different speech patterns. Use the results to decide whether the problem is turn detection, interruption handling, or the work the assistant does after the caller finishes.

  • “Friday… actually, Thursday.” The assistant uses Thursday before it checks availability.
  • A short “yes,” with no hesitation. It answers promptly instead of waiting through unnecessary silence.
  • A quiet “uh-huh” while the assistant is speaking. It can finish its thought if the caller is only acknowledging it.
  • “Wait, that’s the wrong day” during the reply. It stops, listens to the correction, and uses the corrected date.
  • A slow calendar lookup. It makes clear that it is checking and doesn’t claim the appointment is confirmed before it has a result.

Track how often the assistant cuts people off, how often it misses real interruptions, and whether it gets the task right. Measure the time from the caller’s actual last word to the first audible reply, too. An assistant that jumps in too early can look artificially fast if you only start the clock when it thinks the caller has finished. ElevenLabs’ conversation-flow documentation separates settings for silence, eagerness, interruptions, and waiting for a response.

A pause isn’t a full stop

The next time a voice assistant jumps in, listen for where it went wrong. Did it mistake a pause for the end of your turn? Did it keep talking through a correction? Or did it hear you correctly but take too long to act? Those failures can feel similar on a call, but they need different fixes.

In the appointment example, hearing “Thursday” is the conversational win. Checking whether Thursday actually works is the task. A good voice assistant does both, in that order.

  • Voice AI
  • Voice AI
  • turn-taking
  • voice activity detection
  • barge-in
Building an agent?We build agents and prove how they think.
Start a build