Chapter 3: How Voice Conversations Work

In Chapter 2 you built a working voice agent. It could listen, think, and speak. But if you tested it with real callers, you probably noticed something: the timing felt off. The agent would cut people off mid-sentence, or wait too long after they finished speaking, or get confused when they said “uh-huh” while it was talking.

These are not bugs in your code. They are the central engineering challenges of voice AI. In a text chat, turns are explicit: the user hits send. In a voice conversation, there is no send button. The system has to figure out, in real time, when someone has finished their thought, when they are just pausing to breathe, and when they are genuinely trying to interrupt.

Three mechanisms make voice conversations work: Voice Activity Detection (VAD), turn detection, and interruption handling. You need to understand all three before building anything more complex.

Voice Activity Detection (VAD)

Voice Activity Detection answers the most basic question in a voice conversation: is someone speaking right now?

VAD is a lightweight neural network that analyzes audio frames and classifies each one as speech or silence. It does not transcribe words. It does not understand language. It simply tells the system: “There is a human voice in this audio” or “There is not.”

This binary signal is the foundation everything else builds on. Without VAD, the system would have to run speech-to-text on every frame of audio, including long stretches of silence, background noise, and hold music. VAD acts as a gatekeeper, activating the more expensive parts of the pipeline only when there is speech to process.

Silero VAD

LiveKit uses the Silero VAD model as its standard. Silero runs locally on the CPU, requires minimal RAM (under 100 MB), and processes audio frames in under a millisecond. You loaded it in Chapter 2:

from livekit.plugins import silero

session = AgentSession(
    vad=silero.VAD.load(),
    # ... stt, llm, tts
)

The key configuration parameters control sensitivity and timing:

vad = silero.VAD.load(
    activation_threshold=0.5,      # How loud/clear speech must be (0-1)
    min_speech_duration=0.05,      # Ignore speech shorter than 50ms
    min_silence_duration=0.55,     # Wait 550ms of silence before declaring "not speaking"
    prefix_padding_duration=0.5,   # Include 500ms of audio before detected speech
)

The activation_threshold is the one you will tune most often. A higher value (0.7-0.8) works better in noisy environments (construction sites, busy streets) but might miss soft-spoken callers. A lower value (0.3-0.4) catches quieter speech but may trigger on background noise. The default of 0.5 is a reasonable starting point.

The prefix_padding_duration is subtle but important. When VAD detects speech, it includes the previous 500ms of audio in the speech chunk. This ensures the STT model receives the beginning of the utterance, not just the part after VAD activated. Without this padding, you would lose the first syllable of every sentence.

What VAD Produces

VAD emits two key signals that drive the rest of the conversation system:

Audio stream:  ----[  speech  ]----[  silence  ]----[  speech  ]----
VAD events:         START      STOP                  START      STOP

These start/stop events feed into turn detection and interruption handling. VAD tells the system that someone is speaking. The next layer figures out what that means.

Turn Detection

Turn detection is the hardest problem in voice AI. It answers the question: has the caller finished their thought, or are they just pausing?

Consider this exchange:

Caller: "I need someone to come out on... let me check... Thursday."

There are two pauses in that sentence. If the agent responds after “on…” it will interrupt the caller mid-thought. If it waits too long after “Thursday,” the conversation feels sluggish. Getting this right is what separates a frustrating voice bot from one that feels natural.

LiveKit supports three turn detection strategies, each with different tradeoffs.

Strategy 1: VAD-Only

The simplest approach. The agent waits for VAD to report silence, then starts responding.

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection="vad",
    ),
    vad=silero.VAD.load(),
)

The timeline looks like this:

Caller:  "I'd like to book an appointment"
         [===========speech===========]
VAD:      ^START                       ^STOP
                                        |--min_delay--|
Agent:                                                 "Sure, what time works for you?"

VAD-only mode is fast and works with any language. But it has no understanding of what was said. It cannot tell the difference between a pause for thought and the end of a sentence. This makes it prone to interrupting callers who speak slowly or think out loud.

Best for: Simple, short-utterance interactions. IVR-style menus. Multilingual agents where the turn detector model is not needed.

Strategy 2: STT Endpointing

Your speech-to-text provider can detect phrase boundaries in the transcript stream. Instead of relying purely on silence, the agent waits for the STT model to signal that a complete phrase has been received.

from livekit.plugins import assemblyai

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection="stt",
    ),
    stt=assemblyai.STT(),
    vad=silero.VAD.load(),  # Still recommended for interruption handling
)

The timeline:

Caller:  "I'd like to book an appointment"
         [===========speech===========]
STT:      "I'd"  "like to"  "book an appointment"
                                       ^ENDPOINT
                                        |--min_delay--|
Agent:                                                 "Sure, what time works for you?"

STT endpointing is smarter than pure silence detection because it understands phrase structure. The STT model knows that “I’d like to book an” is probably not a complete thought, even if there is a brief pause.

Note that VAD is still recommended alongside STT endpointing. VAD handles interruption detection (covered below), while STT handles turn boundaries. They serve complementary roles.

Best for: Agents using STT providers with strong endpointing (AssemblyAI in particular). Situations where you want phrase-level accuracy without the overhead of a turn detector model.

Strategy 3: Turn Detector Model

The recommended approach for production. LiveKit provides an open-weights language model (MultilingualModel, based on Qwen2.5-0.5B) that analyzes conversational context to predict whether the caller has finished their turn.

from livekit.plugins.turn_detector.multilingual import MultilingualModel

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection=MultilingualModel(),
    ),
    vad=silero.VAD.load(),
)

The model takes the conversation transcript as input and predicts the probability that the current utterance is complete. It adds roughly 50-160ms of latency per turn on CPU.

The timeline:

Caller:  "I need to think about that for a moment..."
         [===============speech===============]
VAD:      ^START                               ^STOP
STT:      "I need to think about that for a moment"
Model:    P(turn_complete) = 0.12  --> WAIT
          ...caller resumes...
Caller:  "...yeah, let's do Thursday at 2pm"
         [===========speech===========]
Model:    P(turn_complete) = 0.94  --> RESPOND
Agent:                                         "Thursday at 2, got it."

The model understands that “I need to think about that for a moment” signals a pause, not a completed turn. VAD-only mode would have cut in. The turn detector model waits.

This is the strategy we use in production. It supports over a dozen languages and runs on LiveKit Cloud’s inference infrastructure when deployed there.

Best for: Production voice agents. Conversations with complex or long-form caller input. Any situation where premature interruption would damage the user experience.

Comparison

Endpointing Configuration

Regardless of which turn detection strategy you use, you can configure the delay between when the system thinks the turn is complete and when the agent actually responds:

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection=MultilingualModel(),
        endpointing={
            "min_delay": 0.5,   # Wait at least 500ms after detected end-of-turn
            "max_delay": 3.0,   # Never wait more than 3 seconds
            "mode": "dynamic",  # Adapt based on conversation pace
        },
    ),
    vad=silero.VAD.load(),
)

The min_delay prevents the agent from jumping in too quickly. Even when the model is confident the turn is complete, a brief pause feels more natural than an instant response.

The max_delay is a safety valve. If the model is uncertain, the agent will respond after 3 seconds regardless. This prevents awkward silences where both parties are waiting.

Dynamic endpointing (mode="dynamic") adapts the delay within the min/max range based on how the conversation is flowing. If the caller speaks in quick, short bursts, the agent shortens its delay. If the caller takes long pauses between sentences, the agent waits longer. It tracks an exponential moving average of pause durations throughout the session.

Interruptions

Interruptions occur when the caller speaks while the agent is speaking. This is normal in human conversation. People cut in with corrections, add information, or redirect the conversation. A good voice agent handles this gracefully.

Default Behavior

By default, interruptions are enabled. When VAD detects speech while the agent is talking, the agent stops immediately:

Agent:   "I can schedule you for Monday at--"
         [=========speaking=========]
Caller:                        "No, Tuesday"
                               [===speech===]
VAD:                            ^START
Agent:   [STOPS]
         History truncated to: "I can schedule you for Monday at"
Agent:                                        "Tuesday, got it..."

The framework automatically truncates the conversation history. The LLM only sees the portion of the agent’s response that the caller actually heard before interrupting. This prevents confusion where the LLM thinks it said something the caller never heard.

Controlling Interruptions

You can disable interruptions globally or per-utterance. Per-utterance control is more useful in practice because there are specific moments when you do not want the agent to be interrupted:

# Per-utterance: uninterruptible
session.say(
    "Your reference code is Alpha-Bravo-7-4-2.",
    allow_interruptions=False,
)

# Per-utterance: uninterruptible reply generation
session.generate_reply(
    instructions="Read back the confirmation number clearly.",
    allow_interruptions=False,
)

A common pattern is making critical information uninterruptible (reference codes, confirmation numbers, legal disclaimers) while keeping the rest of the conversation interruptible.

Adaptive Interruption Handling

Standard interruption detection has a problem: it treats all speech the same. When a caller says “uh-huh” or “okay” while the agent is talking, that is not an interruption. It is backchanneling, a signal that they are listening and the agent should keep going.

Adaptive interruption handling uses an acoustic model (not transcript-based) to distinguish true interruptions from backchanneling. Because it analyzes audio directly rather than waiting for transcription, it can make this determination faster.

session = AgentSession(
    turn_handling=TurnHandlingOptions(
        turn_detection=MultilingualModel(),
        interruption={
            "mode": "adaptive",
        },
    ),
    vad=silero.VAD.load(),
)

The timeline with adaptive handling:

Agent:   "I've scheduled the appointment for Thursday at 2pm and--"
         [=====================speaking======================]
Caller:                    "uh-huh"
                           [=short=]
Model:   BACKCHANNEL (ignore)
Agent:   [continues speaking]                                "...sent a confirmation"

vs.

Agent:   "I've scheduled the appointment for Thursday at 2pm and--"
         [=====================speaking======================]
Caller:                    "Wait, no, make it Friday"
                           [========longer speech========]
Model:   TRUE INTERRUPTION
Agent:   [STOPS]           "Friday instead? Let me update that."

Adaptive interruption handling is enabled by default on LiveKit Cloud when using VAD with a non-realtime LLM and an STT provider that supports aligned transcripts. You can also set it explicitly with "mode": "adaptive".

To fall back to standard VAD-based interruption detection, set "mode": "vad".

False Interruptions

Sometimes VAD detects speech, the agent stops talking, but the STT transcript comes back empty. This happens with coughs, background noise that sounds speech-like, or brief vocal sounds that are not words. The agent stopped for nothing.

By default, the agent resumes speaking from where it left off after a false interruption:

Agent:   "Your appointment is confirmed for--"
         [=========speaking=========]
Caller:  *cough*
VAD:      ^START
Agent:   [STOPS]
STT:     (empty transcript)
         |---false_interruption_timeout---|
Agent:   "--confirmed for Thursday at 2pm."

You can configure this behavior:

turn_handling = TurnHandlingOptions(
    interruption={
        "resume_false_interruption": True,     # Resume after false interruption
        "false_interruption_timeout": 2.0,     # Wait 2 seconds before resuming
    },
)

The false_interruption_timeout is how long the system waits after the interruption, with no transcribed words, before concluding it was a false positive. Setting this too low risks resuming while the caller is still mid-sentence. Setting it too high creates an awkward gap.

Events

The AgentSession emits events that let you track the state of the conversation. These are useful for logging, analytics, and building UI indicators.

User and Agent State

@session.on("user_state_changed")
def on_user_state(ev):
    # ev.new_state: "speaking" | "listening" | "away"
    print(f"User is now: {ev.new_state}")

@session.on("agent_state_changed")
def on_agent_state(ev):
    # ev.new_state: "initializing" | "idle" | "listening" | "thinking" | "speaking"
    print(f"Agent is now: {ev.new_state}")

The user states are straightforward: speaking when VAD detects speech, listening when silent, and away when the participant disconnects.

The agent states map to the pipeline stages:

initializing --> idle --> listening --> thinking --> speaking
                  ^                                    |
                  |____________________________________|

• initializing: Agent is loading models and connecting.
• idle: Agent is ready but not actively processing. This is the state before the first interaction.
• listening: Agent is receiving user input and waiting for the turn to complete.
• thinking: Turn is complete. The agent is running LLM inference and generating a response.
• speaking: TTS audio is playing back to the caller.

After speaking, the agent returns to listening (or idle if the conversation is paused).

Interruption Events

@session.on("user_interruption_detected")
def on_interruption(ev):
    print(f"Interrupted at: {ev.timestamp}, probability: {ev.probability}")

@session.on("agent_false_interruption")
def on_false_interruption(ev):
    print("False interruption -- resuming speech")

The user_interruption_detected event fires when the agent is interrupted, with a probability score from the adaptive model (if enabled). The agent_false_interruption event fires when the system determines an interruption was a false positive and the agent resumes speaking.

These events are invaluable for debugging conversation quality. If you see a high rate of false interruptions, your VAD threshold may be too low. If you see many true interruptions, your turn detection may be responding too slowly.

Summary

Voice conversations are governed by three interlocking systems:

1. VAD detects the presence of speech. It is the raw signal.
2. Turn detection interprets that signal in context to determine when the caller has finished their thought.
3. Interruption handling manages the case where both parties try to speak at once.

Getting these right is not about finding perfect settings. It is about understanding the tradeoffs (latency vs. accuracy, sensitivity vs. false positives) and tuning for your specific use case. A receptionist agent taking appointment details needs different settings than an IVR menu system.

In Chapter 4, we move from mechanics to intelligence: designing the agent’s instructions and persona so it knows what to say, not just when to say it.