Chapter 2: Your First Voice Agent

You are going to build a voice agent in about 30 lines of Python. By the end, you will speak into your microphone, an AI will listen, think, and speak back.

Setting Up the Environment

Python 3.11+

Voice agents rely heavily on async/await and streaming protocols. Python 3.11 is the minimum supported version. If you are on macOS, the easiest path is:

brew install [email protected]

Verify your version:

python3 --version
# Python 3.11.x or higher

The uv Package Manager

We use uv throughout this book. It is a fast Python package manager written in Rust that replaces pip, pip-tools, and virtualenv in a single binary.

Install it:

curl -LsSf https://astral.sh/uv/install.sh | sh

Verify:

uv --version

Project Structure

Create your project directory and initialize it:

mkdir my-voice-agent && cd my-voice-agent
uv init

This gives you a pyproject.toml and a basic project structure. Here is the layout we will build toward in this chapter:

my-voice-agent/
├── pyproject.toml
├── .env
└── agent.py

Simple. One file for config, one file for secrets, one file for code.

The pyproject.toml

Replace the generated pyproject.toml with this:

[project]
name = "my-voice-agent"
version = "0.1.0"
description = "A minimal voice AI agent"
requires-python = ">=3.11"
dependencies = [
    "livekit-agents[openai,turn-detector]>=1.5.1",
    "livekit-plugins-cartesia>=1.5.1",
    "livekit-plugins-deepgram>=1.5.1",
    "livekit-plugins-silero>=1.5.1",
    "python-dotenv>=1.0.0",
]

Let us break down what each dependency does:

Install everything:

uv sync

This creates a virtual environment and installs all dependencies. It takes about 10 seconds.

Getting API Keys

You need four sets of credentials. All of them have free tiers sufficient for development.

1. LiveKit

LiveKit is the real-time infrastructure layer. It handles WebRTC rooms, audio routing, and agent dispatch.

Option A: LiveKit Cloud (easiest)

1. Go to cloud.livekit.io
2. Create a project
3. Copy your LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET

Option B: Self-hosted

# Install LiveKit server locally for development
brew install livekit
livekit-server --dev

With --dev, the server runs on ws://localhost:7880 with test credentials:

• LIVEKIT_URL=ws://localhost:7880
• LIVEKIT_API_KEY=devkey
• LIVEKIT_API_SECRET=secret

2. Deepgram (Speech-to-Text)

1. Go to console.deepgram.com
2. Create an API key
3. The free tier gives you $200 in credit, more than enough for development

3. OpenAI (LLM)

1. Go to platform.openai.com
2. Create an API key
3. We use gpt-4.1-nano for speed. It is the fastest model suitable for voice

4. Cartesia (Text-to-Speech)

1. Go to play.cartesia.ai
2. Create an account and generate an API key
3. Cartesia Sonic produces natural-sounding speech with very low latency

The .env File

Create a .env file in your project root:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your-api-key
LIVEKIT_API_SECRET=your-api-secret
OPENAI_API_KEY=sk-...
DEEPGRAM_API_KEY=your-deepgram-key
CARTESIA_API_KEY=your-cartesia-key

Never commit this file. Add it to .gitignore:

echo ".env" >> .gitignore

The Minimal Agent: 30 Lines

Create agent.py:

from dotenv import load_dotenv

load_dotenv()

from livekit.agents import (
    AgentSession,
    AgentServer,
    JobContext,
    cli,
)
from livekit.plugins import cartesia, deepgram, silero
from livekit.plugins.openai import LLM
from livekit.plugins.turn_detector.english import EnglishModel

server = AgentServer()


@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        stt=deepgram.STT(model="nova-3", language="en"),
        llm=LLM(model="gpt-4.1-nano"),
        tts=cartesia.TTS(),
        vad=silero.VAD.load(),
        turn_detection=EnglishModel(),
    )

    await session.start(
        agent=None,
        room=ctx.room,
    )

    await ctx.connect()


if __name__ == "__main__":
    cli.run_app(server)

That is it. Let us walk through every piece.

Understanding the Code

The Entrypoint Pattern

server = AgentServer()

@server.rtc_session()
async def entrypoint(ctx: JobContext):
    ...

if __name__ == "__main__":
    cli.run_app(server)

Three things happen here:

1. AgentServer() creates the agent server instance. This is the top-level object that manages the agent’s lifecycle.

2. @server.rtc_session() decorates your function as the handler for new real-time sessions. When a participant joins a LiveKit room, the framework dispatches a job and calls this function with a JobContext.

3. cli.run_app(server) hands control to the LiveKit CLI. This is what gives you the console and start subcommands. It handles signal handling, graceful shutdown, and worker registration.

The JobContext gives you access to:

• ctx.room is the LiveKit room the agent is joining
• ctx.proc is the worker process (for shared state like prewarmed models)
• ctx.connect() connects the agent to the room (must be called to start receiving audio)

AgentSession: The Voice Pipeline

session = AgentSession(
    stt=deepgram.STT(model="nova-3", language="en"),
    llm=LLM(model="gpt-4.1-nano"),
    tts=cartesia.TTS(),
    vad=silero.VAD.load(),
    turn_detection=EnglishModel(),
)

The AgentSession wires together five components into a pipeline:

The pipeline runs continuously: VAD triggers STT, STT feeds the transcript to the LLM, the LLM streams tokens to TTS, and TTS streams audio back to the room. All of this happens concurrently. The agent starts speaking before the LLM has finished generating.

Starting the Session

await session.start(
    agent=None,
    room=ctx.room,
)

await ctx.connect()

session.start() initializes the pipeline and begins listening. The agent parameter accepts an Agent instance that defines the agent’s instructions and tools. We pass None here for a minimal agent that responds without specific instructions. We will add a proper agent in Chapter 4.

ctx.connect() connects the agent to the LiveKit room. This must be called after session.start() because it is what actually starts the audio flowing.

Running Your Agent

Console Mode: Local Testing

uv run python agent.py console

Console mode is your development inner loop. It:

• Creates a local LiveKit room in-process (no LiveKit server needed)
• Connects your system microphone as the “caller”
• Plays agent audio through your speakers
• Prints transcripts to the terminal

Speak into your microphone. The agent will transcribe your speech, generate a response, and speak it back. You should hear a voice reply within 1-2 seconds.

This is the mode you will use 90% of the time during development. No phone numbers, no SIP trunks, no infrastructure. Just you and the agent.

Start Mode: Production

uv run python agent.py start

Start mode runs the agent as a worker that connects to your LiveKit server and waits for jobs. When a participant joins a room (via phone call, web client, or API), the server dispatches the job to your worker.

In production, the flow looks like this:

Phone call → Telnyx SIP → LiveKit SIP Server → LiveKit Room
                                                     ↓
                                          Agent Worker picks up job
                                                     ↓
                                          entrypoint(ctx) is called

You will not need start mode until Chapter 8 (Telephony). For now, console is all you need.

What Happens When a Call Comes In

Understanding the dispatch model helps you debug issues later. Here is the full sequence:

1. Room creation. A participant connects to a LiveKit room (via SIP, web, or mobile). If the room does not exist, LiveKit creates it.

2. Agent dispatch. LiveKit sees that the room needs an agent (configured via room settings or SIP dispatch rules). It sends a job request to available agent workers.

3. Worker accepts. Your agent worker receives the job. The framework calls your @server.rtc_session() function with a JobContext.

4. Session starts. Your code creates an AgentSession, wires up STT/LLM/TTS, and calls session.start().

5. Connection. ctx.connect() connects the agent to the room. Audio begins flowing.

6. Conversation. The VAD/STT/LLM/TTS pipeline runs continuously. Each user utterance triggers a response cycle.

7. Shutdown. When the participant disconnects (hangs up), the room closes, and the agent shuts down gracefully.

In console mode, steps 1-3 are simulated locally. Steps 4-7 work identically.

Prewarming: Reducing First-Response Latency

The Silero VAD model takes a few hundred milliseconds to load. In production, you do not want this delay on the first call. The setup_fnc pattern loads models once per worker process:

from livekit.agents import JobProcess

server = AgentServer()


def prewarm(proc: JobProcess):
    proc.userdata["vad"] = silero.VAD.load()


server.setup_fnc = prewarm


@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        stt=deepgram.STT(model="nova-3", language="en"),
        llm=LLM(model="gpt-4.1-nano"),
        tts=cartesia.TTS(),
        vad=ctx.proc.userdata["vad"],  # reuse prewarmed VAD
        turn_detection=EnglishModel(),
    )

    await session.start(agent=None, room=ctx.room)
    await ctx.connect()

The prewarm function runs once when the worker process starts. Every subsequent job reuses the already-loaded VAD model. This shaves 200-400ms off the first response.

Adding Turn Handling Options

The minimal agent works, but real conversations need interruption handling. Here is the production-ready configuration:

from livekit.agents import TurnHandlingOptions

session = AgentSession(
    stt=deepgram.STT(model="nova-3", language="en"),
    llm=LLM(model="gpt-4.1-nano"),
    tts=cartesia.TTS(),
    vad=ctx.proc.userdata["vad"],
    turn_handling=TurnHandlingOptions(
        turn_detection=EnglishModel(),
        interruption={
            "resume_false_interruption": True,
            "false_interruption_timeout": 0.7,
            "mode": "vad",
        },
    ),
    preemptive_generation=True,
)

What these options control:

• turn_detection=EnglishModel(): Uses a trained model to predict when the user has finished speaking, rather than relying on silence duration alone. This is significantly better than a simple silence timer.

• resume_false_interruption=True: If the user makes a brief noise (cough, “um”) that triggers an interruption, the agent resumes its response instead of generating a new one.

• false_interruption_timeout=0.7: The window (in seconds) to decide if an interruption was real or false. 0.7 seconds is a good default.

• mode="vad": Interruptions are triggered by voice activity detection rather than transcript content.

• preemptive_generation=True: The agent starts generating its next response while the user is still speaking (based on turn detection predictions). This reduces perceived latency.

Choosing Providers

Every component in the pipeline can be swapped. Here is how the major providers compare:

Speech-to-Text (STT)

Recommendation: Deepgram Nova-3 is the default choice. It has the lowest latency for streaming use cases and excellent accuracy. AssemblyAI is a strong alternative if you need features like speaker diarization built into the STT layer.

Language Model (LLM)

Recommendation: Start with gpt-4.1-nano. It is the fastest and cheapest option that still handles conversational voice well. Move to gpt-4.1-mini if you need better reasoning (tool selection, complex instructions). Use gpt-4.1 or Claude for tasks requiring strong structured output or nuanced understanding.

For voice agents, time to first token (TTFT) matters more than raw quality. A 200ms TTFT feels instant; a 600ms TTFT feels sluggish. The user is waiting in silence.

Text-to-Speech (TTS)

Recommendation: Cartesia Sonic has the lowest latency and sounds natural. ElevenLabs produces marginally more expressive voices but at higher latency and cost. For voice agents where responsiveness is critical, Cartesia is the better default.

Swapping Providers

Changing a provider is a one-line change:

# Switch STT from Deepgram to AssemblyAI
# uv add livekit-plugins-assemblyai
from livekit.plugins import assemblyai
session = AgentSession(
    stt=assemblyai.STT(),
    ...
)

# Switch TTS from Cartesia to ElevenLabs
# uv add livekit-plugins-elevenlabs
from livekit.plugins import elevenlabs
session = AgentSession(
    tts=elevenlabs.TTS(),
    ...
)

# Switch LLM from OpenAI to Anthropic
# uv add livekit-plugins-anthropic
from livekit.plugins import anthropic
session = AgentSession(
    llm=anthropic.LLM(model="claude-sonnet-4-20250514"),
    ...
)

The plugin system means every provider implements the same interface. Your agent code does not change, only the import and constructor.

The Complete Agent

Here is the full, production-ready minimal agent combining everything from this chapter:

"""agent.py -- A complete minimal voice agent."""

from dotenv import load_dotenv

load_dotenv()

from livekit.agents import (
    AgentServer,
    AgentSession,
    JobContext,
    JobProcess,
    TurnHandlingOptions,
    cli,
)
from livekit.plugins import cartesia, deepgram, silero
from livekit.plugins.openai import LLM
from livekit.plugins.turn_detector.english import EnglishModel

server = AgentServer()


def prewarm(proc: JobProcess):
    proc.userdata["vad"] = silero.VAD.load()


server.setup_fnc = prewarm


@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        stt=deepgram.STT(model="nova-3", language="en"),
        llm=LLM(model="gpt-4.1-nano"),
        tts=cartesia.TTS(),
        vad=ctx.proc.userdata["vad"],
        turn_handling=TurnHandlingOptions(
            turn_detection=EnglishModel(),
            interruption={
                "resume_false_interruption": True,
                "false_interruption_timeout": 0.7,
                "mode": "vad",
            },
        ),
        preemptive_generation=True,
    )

    await session.start(agent=None, room=ctx.room)
    await ctx.connect()


if __name__ == "__main__":
    cli.run_app(server)

Run it:

uv run python agent.py console

Speak. Listen. You have a voice agent.

What Comes Next

This agent listens and responds, but it has no personality, no instructions, and no tools. It is a blank slate that generates generic responses.

In Chapter 3, we will explore how voice conversations actually work: VAD, turn detection, interruption handling, and why latency budgets matter.

In Chapter 4, we will give our agent a persona and instructions, transforming it from a generic chatbot into a purpose-built business agent.

Key takeaways:

• A voice agent needs five components: VAD, STT, LLM, TTS, and turn detection
• The AgentServer + @server.rtc_session() + cli.run_app() pattern is the standard entrypoint
• Console mode (python agent.py console) is your development inner loop
• Start mode (python agent.py start) connects to a LiveKit server for production
• Prewarm models in setup_fnc to avoid first-call latency
• Every provider is swappable with a one-line change thanks to the plugin system
• Optimize for time-to-first-token. In voice, speed is quality