Chapter 15: Debugging and Optimization

Voice agents fail in ways that web applications do not. There is no stack trace a user can screenshot and paste into a bug report. The caller hears silence, or nonsense, or gets disconnected. Your only forensic evidence is logs, Sentry exceptions, and recorded audio. What follows are the failure modes you will encounter, how to diagnose them, and how to make your agent faster and cheaper once it works correctly.

Common Failure Modes

After running a production voice agent through thousands of calls, certain errors recur with enough frequency to catalogue. The table below maps symptoms to root causes and fixes. Each entry comes from a real incident.

Debugging Workflow

When a call goes wrong, follow this sequence:

1. Check Sentry for exceptions. Filter by the call’s room ID or participant SID.
2. Pull the LiveKit room logs using lk room list and lk room get. These show participant join/leave times and track publications.
3. Review the chat context from your logging middleware. Look at what the LLM saw and what it generated.
4. Listen to the recording if call recording is enabled. The audio tells you exactly what the caller experienced.
5. Reproduce in console mode using uv run python main.py console. Feed the same inputs and see if the failure recurs.

Latency Optimization

Voice is the most latency-sensitive application you can build. Humans notice delays above 300ms. Above 800ms, the conversation feels broken. Three metrics matter:

Time to First Token (TTFT)

TTFT measures how long after the user stops speaking before the agent begins its response. This is the sum of: VAD tail silence (how long the system waits to confirm the user is done), STT finalization latency, LLM time to first token, and TTS time to first audio byte.

TTFT = VAD_tail + STT_finalize + LLM_first_token + TTS_first_byte

Optimization levers:

• VAD tail: Reduce min_endpointing_delay to 0.4-0.5 seconds. Lower values cause more false endpoints (the agent jumps in while the user is pausing mid-sentence). Find the sweet spot for your use case.
• STT finalization: Use streaming STT providers like Deepgram that return partial results. The LLM can start processing before the final transcript arrives.
• LLM first token: Use smaller models. gpt-4.1-nano has roughly 150ms TTFT compared to ~350ms for gpt-4o. For most voice agent tasks, nano-class models are sufficient.
• TTS first byte: Cartesia Sonic streams audio in under 100ms. ElevenLabs is typically 200-400ms. This difference is audible.

session = AgentSession(
    stt=deepgram.STT(
        model="nova-3",
        language="en",
    ),
    llm=openai.LLM(model="gpt-4.1-nano"),
    tts=cartesia.TTS(
        model="sonic-2",
        voice="informative-calm",
    ),
    turn_detection=turn_detector.EOUModel(
        min_endpointing_delay=0.5,
        max_endpointing_delay=5.0,
    ),
)

End-to-End Latency

Measure the full round trip: caller speaks, agent responds. Target under 800ms total. Log timestamps at each stage of the pipeline and identify which component is the bottleneck. In practice, the LLM is usually the largest contributor.

Connection Pre-Warming

Establish provider connections before the first call arrives. TLS handshakes and WebSocket upgrades add 100-200ms on the first request. LiveKit’s framework handles some of this automatically, but if you are using custom HTTP clients, initialize them at worker startup, not at call time.

@server.rtc_session()
async def my_agent(ctx: JobContext):
    # Use pre-warmed resources from worker startup
    vad = ctx.proc.userdata["vad"]
    session = AgentSession(vad=vad, ...)

Context Window Management

Long calls generate long chat histories. A 20-minute call with frequent tool use can produce 15,000+ tokens of context. This causes three problems: increased latency (more tokens to process), increased cost, and degraded LLM performance (the model “forgets” early instructions buried under pages of conversation).

Strategies

Truncation: Drop older messages beyond a threshold. Keep the system prompt and the last N turns. This is the simplest approach but risks losing important early context like the caller’s name.

class MyAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions="...",
            chat_ctx_max_messages=30,  # Keep last 30 messages
        )

Summarization: Periodically summarize the conversation history and replace older messages with the summary. More complex to implement but preserves key information.

Structured state: Instead of relying on chat history for state, extract key facts into a structured object (a dataclass or dict) and include it in the system prompt. This way, even if old messages are truncated, the agent retains awareness of what has been collected.

@function_tool()
async def record_business_name(self, ctx: RunContext, name: str) -> str:
    """Record the business name provided by the caller."""
    self._collected["business_name"] = name
    # Update instructions with current state
    self.update_instructions(
        self._base_instructions + f"\n\nCollected so far: {self._collected}"
    )
    return f"Recorded business name: {name}"

Provider Fallbacks

No provider has 100% uptime. Build fallback chains so a single provider outage does not take down your agent.

from livekit.agents.llm import LLM

async def get_llm_with_fallback() -> LLM:
    """Try primary LLM, fall back to secondary on failure."""
    try:
        primary = openai.LLM(model="gpt-4.1-nano")
        # Verify connectivity with a lightweight call
        await primary.health_check()
        return primary
    except Exception:
        logger.warning("Primary LLM unavailable, falling back to DeepSeek")
        return openai.LLM(
            model="deepseek-chat",
            base_url="https://api.deepseek.com/v1",
            api_key=os.getenv("DEEPSEEK_API_KEY"),
        )

For STT and TTS, the same pattern applies. Keep a secondary provider configured and switch on failure. Log every fallback event so you can track provider reliability over time.

The key decision is where to detect failure. You can check at startup (health check before each call) or mid-call (catch exceptions during streaming and hot-swap). Startup checks are simpler and sufficient for most outages. Mid-call swaps are complex and risk audio glitches but handle transient failures better.

Cost Optimization

Voice agents are expensive. Every call consumes STT, LLM, and TTS credits simultaneously, plus telephony minutes and compute. Here is where the money goes and how to reduce it.

Model Selection

The single largest cost lever is LLM model choice. For a voice agent that collects structured information, gpt-4.1-nano handles 95% of conversations correctly at roughly 1/50th the cost of gpt-4o. Reserve larger models for complex reasoning tasks or escalation paths.

CI Cost Reduction

Running integration tests against OpenAI models is expensive. Use DeepSeek for CI runs. It is compatible with the OpenAI API format, costs a fraction of the price, and is accurate enough for testing tool calling, prompt adherence, and conversation flow.

# conftest.py
@pytest.fixture
def llm():
    if os.getenv("CI"):
        return openai.LLM(
            model="deepseek-chat",
            base_url="https://api.deepseek.com/v1",
            api_key=os.getenv("DEEPSEEK_API_KEY"),
        )
    return openai.LLM(model="gpt-4.1-nano")

Caching

Cache tool results that do not change between calls. Business hours, service lists, and pricing can be fetched once and reused. Use TTL-based caching so data stays fresh without a database hit on every call.

Audio Optimization

TTS is billed per character. Shorter agent responses cost less and sound more natural on a phone call. Prompt your agent to be concise: “Keep responses under two sentences unless the caller asks for details.” This reduces TTS cost by 30-50% compared to verbose agents.

Monitoring in Production

Once your agent is live, instrument these metrics:

• Call completion rate: Percentage of calls that reach the intended end state (business created, appointment booked) vs. abandoned or errored.
• Average call duration: Shorter is usually better. Long calls indicate the agent is struggling.
• TTFT p50 and p95: Median and tail latency. If p95 exceeds 1.5 seconds, callers will complain.
• Tool success rate: Percentage of tool calls that succeed vs. error. A drop indicates a backend issue.
• Fallback trigger rate: How often the agent falls back to a secondary provider. Spikes indicate primary provider instability.

Push these to your observability stack (Datadog, Grafana, or even a simple PostgreSQL table with a dashboard query). Set alerts on call completion rate drops and TTFT spikes. These are the two metrics that most directly correlate with caller satisfaction.

Debugging voice agents is harder than debugging web APIs because the evidence is transient. Audio disappears, and conversations cannot be replayed from a curl command. The investment in structured logging, call recording, and metric instrumentation pays for itself on the first production incident you diagnose in minutes instead of hours.