Chapter 1: The Voice AI Landscape

You have probably built a chatbot before: a text box, a POST request, a streamed response. Voice agents look similar on paper, but the moment audio enters the picture, everything changes. Latency that is invisible in text becomes an awkward silence on a phone call. A misheard word can derail an entire conversation. And unlike a chat window, there is no “scroll up.” The caller’s short-term memory is all you get.

This chapter walks through what a voice agent actually is, how the core pipeline works, and where LiveKit fits into the architecture. It also previews what you will build throughout this book.

What Is a Voice Agent?

A voice agent is software that holds a real-time, spoken conversation with a human, autonomously. That last word matters. Traditional IVR systems (“press 1 for billing”) follow rigid scripts. Chatbots handle text, not speech. A voice agent does something different: it listens, thinks, and speaks, all in real time, adapting to what the caller says.

Here is a simple way to think about the spectrum:

IVR                    Chatbot                Voice Agent
"Press 1 for..."       Text in, text out      Speech in, speech out
Fixed menu tree        Flexible but typed     Flexible and spoken
No understanding       Understands text       Understands speech
Touchtone input        Keyboard input         Microphone input

A voice agent combines speech recognition, language understanding, and speech synthesis into a loop that feels like talking to a person. The caller does not need to type, tap, or navigate menus. They just talk.

Why Voice Agents Matter Now

Three things converged to make voice agents practical:

1. Speech-to-text got fast and accurate. Services like Deepgram can transcribe speech in under 200ms with word error rates below 5%. Five years ago, that number was closer to 15%.

2. LLMs got good at conversation. GPT-4-class models can maintain context, follow instructions, and generate natural responses. They handle the messy reality of human speech: false starts, interruptions, topic changes.

3. Text-to-speech got natural. Modern TTS from providers like Cartesia and ElevenLabs produces speech that sounds human, not robotic. Listeners no longer cringe at the output.

The result: you can now build an agent that a caller genuinely mistakes for a human. Not because it is trying to deceive, but because the conversation flows naturally.

The Voice Pipeline: STT, LLM, TTS

Every voice agent runs on the same core pipeline. Audio comes in, gets converted to text, the text is processed by a language model, and the response is converted back to audio.

                    The Voice Pipeline

  Caller's voice          Agent's voice
       |                       ^
       v                       |
  +---------+            +---------+
  |   STT   |            |   TTS   |
  | (Speech |            | (Text   |
  |  to     |            |  to     |
  |  Text)  |            |  Speech)|
  +---------+            +---------+
       |                       ^
       v                       |
       +-------> LLM  --------+
                (Think)

Let us walk through each stage.

Stage 1: Speech-to-Text (STT)

The STT component receives raw audio (a stream of PCM samples, typically 16kHz mono) and produces text. There are two modes:

• Batch transcription. Wait for the caller to finish speaking, then transcribe the full utterance. Simple, but slow. The agent sits in silence while the model processes.

• Streaming transcription. Transcribe as the caller speaks, emitting partial results that get refined. This is what you want for real-time agents. The LLM can start processing before the caller finishes their sentence.

Most voice agents use streaming STT. Deepgram, Google Cloud Speech, and Azure Speech Services all support it. The key metric is time to first token: how quickly the STT emits its first partial transcript after audio begins.

Voice Activity Detection (VAD) works alongside STT. It detects when the caller starts and stops speaking, so the agent knows when to listen and when to respond. Silero VAD is a popular lightweight model for this. Without good VAD, the agent either interrupts the caller or waits too long to respond.

Stage 2: The Language Model (LLM)

Once you have text, the LLM does the thinking. It receives the transcript (plus conversation history and system instructions) and generates a response. This is the same LLM you would use in a text chatbot, but the constraints are different:

• Responses must be short. Nobody wants to listen to a three-paragraph answer on a phone call. The system prompt should instruct the model to be concise.
• Latency is critical. In text chat, 2 seconds feels fine. On a phone call, 2 seconds of silence feels broken. You need fast models (like GPT-4.1-mini or GPT-4.1-nano) and streaming responses.
• The input is messy. STT output contains errors, filler words (“um”, “uh”), and fragments. The LLM must handle this gracefully.

You will typically stream the LLM response token by token to the TTS, so the agent starts speaking before the full response is generated. This pipelining is what makes sub-second response times possible.

Stage 3: Text-to-Speech (TTS)

The TTS component takes the LLM’s text output and synthesizes audio. Modern neural TTS models produce remarkably natural speech, with control over:

• Voice selection. Choose a voice that fits your agent’s persona.
• Speed and prosody. Adjust speaking rate and intonation.
• Streaming synthesis. Like STT, you want streaming here. Send text chunks to the TTS as they arrive from the LLM, and start playing audio before the full response is synthesized.

The full pipeline, when streaming end to end, looks like this:

  Time -->

  Caller speaks:  "What time do you open on Saturdays?"
                   |
  STT streams:     "What time" ... "do you open" ... "on Saturdays?"
                                                      |
  LLM streams:                                        "We open at" ... "9 AM on Saturdays."
                                                       |
  TTS streams:                                         [audio: "We open at"] ... [audio: "9 AM..."]
                                                       |
  Caller hears:                                        "We open at 9 AM on Saturdays."

Each stage overlaps with the next. The caller finishes speaking, and within 500-800ms, the agent starts responding. That is the target.

Why Voice Is Different from Text Chat

If you have built text-based LLM applications, here is what will surprise you about voice:

The biggest difference is latency sensitivity. In a voice conversation, every millisecond of delay is perceptible. This changes how you architect, what models you choose, and how aggressively you optimize.

How LiveKit Fits In

LiveKit is an open-source platform for real-time audio and video. It started as a WebRTC SFU (Selective Forwarding Unit) and evolved into a full framework for building voice and video applications, including AI agents.

The SFU: Selective Forwarding Unit

At its core, LiveKit runs an SFU: a server that receives media streams from participants and forwards them to other participants. Unlike an MCU (Multipoint Control Unit), it does not mix or transcode the media. It just routes it.

                 LiveKit SFU
                 +----------+
  Caller ------->|          |-------> Agent
  (audio in)     |  Routes  |   (receives caller audio)
                 |  media   |
  Caller <-------|  streams  |<------ Agent
  (hears agent)  |          |   (sends agent audio)
                 +----------+

Why does this matter for voice agents? Because the SFU handles all the hard networking problems (NAT traversal, codec negotiation, packet loss recovery, jitter buffers) so your agent code can focus on the conversation logic.

The Agents Framework

LiveKit’s Agents Framework is a Python SDK that lets you write agent logic as a simple Python program. The framework handles:

• Connecting to rooms. Your agent joins a LiveKit room as a participant, just like a human would.
• Receiving audio. The framework delivers the caller’s audio to your STT pipeline.
• Sending audio. Your TTS output is published back to the room as an audio track.
• Session management. The framework manages the agent’s lifecycle: joining, leaving, error recovery.

You write a function that defines how your agent behaves, and the framework handles the plumbing. Here is the conceptual shape (not runnable code yet; that comes in Chapter 2):

# Conceptual structure of a voice agent
agent = VoiceAgent(
    stt=deepgram.STT(),           # Speech-to-Text provider
    llm=openai.LLM(),            # Language model
    tts=cartesia.TTS(),           # Text-to-Speech provider
    chat_ctx=initial_context,     # System prompt and history
)
agent.start(room)                 # Connect to LiveKit room

That is genuinely close to what the real code looks like. The framework and its plugin system abstract away the complexity of managing audio streams, orchestrating the pipeline, and handling edge cases like interruptions.

LiveKit Cloud vs Self-Hosted

You have two deployment options:

• LiveKit Cloud. Managed infrastructure. You deploy your agent code, and LiveKit handles the SFU servers, scaling, and global distribution. Best for getting started and for production workloads where you do not want to manage infrastructure.

• Self-hosted. Run the LiveKit server on your own infrastructure. More control, more responsibility. You handle scaling, monitoring, and upgrades. A typical self-hosted deployment is a LiveKit instance on a VPS.

Both options use the same Agents SDK. Your agent code does not change based on deployment model.

Key Concepts

Before we start building, let us nail down four terms you will encounter constantly.

Rooms

A room is a communication session. When a caller dials in, a room is created. The caller and the agent both join this room. Think of it as a virtual meeting room with exactly two participants. Rooms have metadata (JSON you can attach), and they are ephemeral, lasting only for the duration of the call.

Participants

Everyone in a room is a participant. In a typical voice agent scenario, there are two: the caller (connected via SIP/phone) and the agent (connected via the Agents SDK). Each participant has an identity, a name, and metadata.

Tracks

A track is a single media stream: one audio track or one video track. The caller publishes an audio track (their voice). The agent publishes an audio track (the synthesized speech). The SFU forwards these tracks between participants. For voice agents, you are dealing exclusively with audio tracks.

Sessions

A session is the lifecycle of an agent handling one call. It starts when the agent joins the room and ends when the call disconnects. During a session, the agent maintains conversation state, tracks what information has been collected, and can trigger actions (like API calls) based on the conversation.

The Ecosystem

LiveKit’s voice agent ecosystem includes several components you will use throughout this book:

• Plugins. Modular integrations for STT (Deepgram, Google, Azure), LLM (OpenAI, Anthropic), and TTS (Cartesia, ElevenLabs, PlayHT). Swap providers by changing one line.

• Telephony integration. LiveKit supports SIP trunking, which connects traditional phone networks to LiveKit rooms. A caller dials a phone number, and the call arrives in a LiveKit room where your agent is waiting. Telnyx, Twilio, and Vonage all work as SIP trunk providers.

• Function calling. Agents can use LLM function calling (tool use) to take actions during a conversation: look up a database, book an appointment, transfer a call. This is how agents go beyond conversation into actual work.

• Deployment. Agents are regular Python processes. Deploy them anywhere you run Python: Fly.io, Railway, AWS ECS, a VPS, your laptop. The Agents SDK handles reconnection and graceful shutdown.

What You Will Build

Throughout this book, you will build a production voice agent from scratch. Here is what the final system looks like:

                        Your Production Voice Agent

  Phone call                                              Backend
  +--------+     +----------+     +---------+     +----------------+
  | Caller  |---->| Telnyx   |---->| LiveKit |---->| Your Agent     |
  | (phone) |     | SIP Trunk|     | SFU     |     | (Python)       |
  +--------+     +----------+     +---------+     +-------+--------+
       ^                                                   |
       |                                                   v
       |                                          +--------+-------+
       +------------------------------------------| STT -> LLM ->  |
            Agent speaks back to caller           | TTS pipeline   |
                                                  +--------+-------+
                                                           |
                                                           v
                                                  +--------+-------+
                                                  | API calls,     |
                                                  | database,      |
                                                  | business logic |
                                                  +----------------+

Part 1 (Chapters 1-3) covers the foundations: how the pipeline works, setting up your environment, and building your first agent that runs locally.

Part 2 (Chapters 4-7) goes deeper: conversation design, function calling, state management, and handling the messy realities of phone conversations (interruptions, silence, background noise).

Part 3 (Chapters 8-10) adds telephony: SIP trunking, phone number provisioning, and handling inbound and outbound calls.

Part 4 (Chapters 11-13) hardens your agent for production: testing, monitoring, error handling, and deployment.

Part 5 (Chapters 14-16) covers advanced topics: multi-agent systems, custom STT/TTS models, and scaling to thousands of concurrent calls.

By the end, you will have a voice agent that answers phone calls, holds natural conversations, takes actions, and runs reliably in production.

Summary

Voice agents are not chatbots with a microphone bolted on. They are real-time systems where latency, turn-taking, and audio quality dominate the engineering challenges. The core pipeline (STT to LLM to TTS) is conceptually simple but demands careful orchestration to achieve the sub-second response times that make conversations feel natural.

LiveKit provides the infrastructure layer: an SFU for media routing, an Agents framework for writing agent logic in Python, and a plugin system for swapping providers. Whether you deploy to LiveKit Cloud or self-host, the same code runs.

In the next chapter, we will set up your development environment and get a minimal agent running on your machine, one that listens to your microphone and talks back.