Chapter 11: Testing Voice Agents

Voice agents are hard to test. A traditional web API takes JSON in and returns JSON out, and you can write assertions against every field. A voice agent takes messy human speech in and produces natural language out. Between those two endpoints sit an STT model, an LLM, a TTS engine, and a caller who might interrupt, mumble, or go off-topic mid-sentence.

“Hard to test” is not an excuse to skip testing. We run 90+ unit tests in three seconds, 26 integration tests in two and a half minutes, and 7 end-to-end voice scenarios in about five minutes per run. This chapter shows you how to build that test pyramid, what each layer catches, and how to avoid the mistakes that made our first test suites useless.

The Test Pyramid for Voice Agents

The classic test pyramid applies to voice agents, but the layers have different characteristics than a typical web application.

                    +---------------+
                    |   E2E Voice   |  7 scenarios, ~5 min/run
                    |   (Cekura)    |  Real STT -> LLM -> TTS
                    +-------+-------+
                            |
                +-----------+-----------+
                |    Integration        |  26 tests, ~2.5 min
                |    (Real LLM)         |  Requires API key
                +-----------+-----------+
                            |
            +---------------+---------------+
            |          Unit Tests           |  90+ tests, ~3 sec
            |       (Mocked, fast)          |  No API keys needed
            +-------------------------------+

Unit tests verify that your task logic, validation rules, and tool methods behave correctly given controlled inputs. They mock the LLM, the session, and the speech pipeline. They run in three seconds with no API keys, so they belong in every PR check.

Integration tests call a real LLM to verify that your prompts produce the expected behavior. Does the availability extractor correctly parse “Monday to Friday, 9 to 5” into structured slots? Does the LLM follow the tool-calling rules you specified? These take minutes and cost money, but they catch prompt regressions that unit tests cannot.

E2E voice tests call your deployed agent through the full pipeline (real STT, real LLM, real TTS) using a synthetic caller. They verify the entire system works together: audio routing, turn detection, tool execution, and backend integration. They are slow, expensive, and flaky enough that you do not run them on every PR.

Each layer catches different bugs. Unit tests catch logic errors in validation and data flow. Integration tests catch prompt regressions and LLM behavior changes. E2E tests catch audio pipeline issues, latency problems, and interaction patterns that only surface in real conversation.

Unit Testing Tasks

The core of a voice agent is its tasks: the structured units of work that collect information, validate it, and complete with a result. Testing tasks means testing tool methods in isolation.

The patch_task Pattern

Every task in our system extends AgentTask from the LiveKit SDK. To test a task, you need to mock two things: the session context that LiveKit provides, and the complete method that signals the task is done. Here is the fixture pattern we use across all task test files:

# tests/unit/conftest.py
from __future__ import annotations

import pytest
from unittest.mock import AsyncMock, MagicMock, patch


@pytest.fixture
def mock_ctx():
    """Create a mock RunContext for tool calls."""
    ctx = MagicMock()
    ctx.session = MagicMock()
    ctx.session.generate_reply = AsyncMock()

    # speech_handle for two-step confirmation testing
    speech_handle = MagicMock()
    speech_handle.wait_for_playout = AsyncMock()
    ctx.session.generate_reply.return_value = speech_handle

    return ctx


def patch_task(task_instance):
    """Patch a task's complete method so we can assert on it."""
    task_instance.complete = MagicMock()
    return task_instance

The mock_ctx fixture gives every tool method a context object that behaves enough like a real RunContext to avoid crashes, while the patch_task helper lets you assert whether complete was called and what result it received.

Testing Tool Methods Directly

With the fixtures in place, testing a tool method is straightforward. Call the method with controlled inputs and assert on the outcome:

# tests/unit/test_collect_business_identity.py
from __future__ import annotations

import pytest
from unittest.mock import MagicMock

from src.tasks.collect_business_identity import CollectBusinessIdentityTask
from src.schemas.business_detail_schema import BusinessIdentityResult


@pytest.fixture
def task(mock_ctx):
    t = CollectBusinessIdentityTask()
    t.complete = MagicMock()
    return t


class TestRecordBusinessNameAndTrade:
    """Tests for the record_business_name_and_trade tool."""

    @pytest.mark.asyncio
    async def test_records_valid_business(self, task, mock_ctx):
        """Valid business name and trade are stored."""
        result = await task.record_business_name_and_trade(
            ctx=mock_ctx,
            business_name="Maple Leaf Plumbing",
            trade_type="plumber",
        )

        # Tool should return a string prompting confirmation
        assert "Maple Leaf Plumbing" in result
        assert "plumber" in result.lower()

        # Task should NOT be complete yet -- needs confirmation step
        task.complete.assert_not_called()

    @pytest.mark.asyncio
    async def test_rejects_empty_business_name(self, task, mock_ctx):
        """Empty business name raises ToolError."""
        from livekit.agents import ToolError

        with pytest.raises(ToolError):
            await task.record_business_name_and_trade(
                ctx=mock_ctx,
                business_name="",
                trade_type="plumber",
            )

    @pytest.mark.asyncio
    async def test_rejects_filler_text(self, task, mock_ctx):
        """Filler words like 'um' or 'uh' are rejected."""
        from livekit.agents import ToolError

        with pytest.raises(ToolError):
            await task.record_business_name_and_trade(
                ctx=mock_ctx,
                business_name="um",
                trade_type="uh",
            )

The pattern is always the same: create the task, call the tool method, and assert on what happened. Did complete get called? Did it get called with the right result? Did the method raise ToolError for invalid input?

Testing the Two-Step Confirmation Flow

The two-step confirmation pattern (described in Chapter 6) prevents the LLM from skipping user confirmation. The record_* method stores a speech_handle, and the confirm_* method checks that a different speech_handle is now active, proving the caller spoke in between.

Testing this requires simulating the speech handle lifecycle:

class TestConfirmBusinessIdentity:
    """Tests for the confirm_business_name_and_trade tool."""

    @pytest.mark.asyncio
    async def test_confirm_after_record_completes_task(self, task, mock_ctx):
        """Confirming after record (with new speech handle) completes the task."""
        # Step 1: Record
        await task.record_business_name_and_trade(
            ctx=mock_ctx,
            business_name="Maple Leaf Plumbing",
            trade_type="plumber",
        )

        # Simulate caller responding -- new speech handle
        new_handle = MagicMock()
        new_handle.wait_for_playout = AsyncMock()
        mock_ctx.speech_handle = new_handle

        # Step 2: Confirm
        await task.confirm_business_name_and_trade(ctx=mock_ctx)

        # Now the task should be complete
        task.complete.assert_called_once()
        result = task.complete.call_args[0][0]
        assert isinstance(result, BusinessIdentityResult)
        assert result.business_name == "Maple Leaf Plumbing"
        assert result.trade_type == "plumber"

    @pytest.mark.asyncio
    async def test_confirm_without_record_raises_error(self, task, mock_ctx):
        """Confirming before recording raises ToolError."""
        from livekit.agents import ToolError

        with pytest.raises(ToolError):
            await task.confirm_business_name_and_trade(ctx=mock_ctx)

Parametrized Tests for Validation

Phone numbers and emails have format requirements. Rather than writing ten nearly identical tests, use pytest.mark.parametrize:

class TestRecordOwnerContact:
    """Tests for phone and email validation."""

    @pytest.mark.parametrize("phone", [
        "+14165550123",      # Valid Canadian
        "+14165550199",      # Valid Canadian
        "+12125550100",      # Valid US
    ])
    @pytest.mark.asyncio
    async def test_accepts_valid_phones(self, task, mock_ctx, phone):
        result = await task.record_owner_contact(
            ctx=mock_ctx,
            owner_name="John Smith",
            phone_number=phone,
            email="[email protected]",
        )
        assert "John Smith" in result

    @pytest.mark.parametrize("phone,reason", [
        ("4165550123", "missing country code"),
        ("", "empty"),
        ("+1416555012", "too short"),
        ("not-a-number", "not numeric"),
    ])
    @pytest.mark.asyncio
    async def test_rejects_invalid_phones(self, task, mock_ctx, phone, reason):
        from livekit.agents import ToolError

        with pytest.raises(ToolError):
            await task.record_owner_contact(
                ctx=mock_ctx,
                owner_name="John Smith",
                phone_number=phone,
                email="[email protected]",
            )

    @pytest.mark.parametrize("email", [
        "[email protected]",
        "[email protected]",
        "",
    ])
    @pytest.mark.asyncio
    async def test_accepts_valid_or_empty_emails(self, task, mock_ctx, email):
        """Email is optional, so empty string should be accepted."""
        result = await task.record_owner_contact(
            ctx=mock_ctx,
            owner_name="John Smith",
            phone_number="+14165550123",
            email=email,
        )
        assert "John Smith" in result

What to Assert On

Task unit tests should verify four things:

1. complete was called: the task finished when it should have.
2. complete was NOT called: the task did not finish prematurely (e.g., before confirmation).
3. The result dataclass is correct: the right fields have the right values.
4. ToolError is raised: invalid input is rejected before the LLM can proceed.

Do not try to test what the agent says. The LLM generates speech text; your unit tests should focus on the data flow through your tools and tasks.

Integration Tests with Real LLMs

Some behaviors cannot be tested without a real LLM. The availability extractor takes natural language like “Monday to Friday, 9 to 5, closed weekends” and produces structured time slots. The output depends entirely on the LLM’s ability to parse and structure that text.

# tests/integration/test_availability_extraction.py
import pytest
from src.utils.availability_extractor import extract_availability


@pytest.mark.asyncio
async def test_standard_business_hours():
    """Standard 9-5 weekday schedule produces correct slots."""
    result = await extract_availability("Monday to Friday, 9am to 5pm")

    assert len(result.slots) == 5  # Mon-Fri
    for slot in result.slots:
        assert slot.start_time == "09:00"
        assert slot.end_time == "17:00"
        assert slot.day_of_week in range(5)


@pytest.mark.asyncio
async def test_split_schedule():
    """Morning and afternoon with lunch break."""
    result = await extract_availability(
        "Monday to Friday, 8am to noon, then 1pm to 5pm"
    )

    # Should produce 10 slots: 2 per day (morning + afternoon)
    assert len(result.slots) == 10


@pytest.mark.asyncio
async def test_weekend_only():
    """Weekend-only schedule."""
    result = await extract_availability("Saturday and Sunday, 10am to 4pm")

    assert len(result.slots) == 2
    days = {slot.day_of_week for slot in result.slots}
    assert days == {5, 6}  # Sat, Sun

Cost Optimization: DeepSeek for CI

Running integration tests against OpenAI on every PR gets expensive fast. We use DeepSeek as a drop-in replacement for CI. It costs roughly one-tenth the price of GPT-4 and produces comparable structured output for our use case:

# conftest.py for integration tests
import os

def get_test_llm_client():
    """Use DeepSeek in CI, OpenAI locally."""
    if os.getenv("CI") or os.getenv("DEEPSEEK_API_KEY"):
        from openai import AsyncOpenAI
        return AsyncOpenAI(
            api_key=os.environ["DEEPSEEK_API_KEY"],
            base_url="https://api.deepseek.com/v1",
        )
    from openai import AsyncOpenAI
    return AsyncOpenAI()  # Uses OPENAI_API_KEY

This is not a perfect substitute. DeepSeek occasionally produces different structured output than OpenAI, so a few tests need model-specific assertions. But it catches the major regressions (broken schemas, missing fields, prompt formatting errors) at a fraction of the cost.

Timeout Handling

LLM APIs are flaky. Tests that pass locally may time out in CI. Always set explicit timeouts:

@pytest.mark.asyncio
@pytest.mark.timeout(30)  # Fail after 30 seconds
async def test_complex_schedule_extraction():
    result = await extract_availability(
        "Mon, Wed, Fri 7am-3pm. Tue, Thu 10am-6pm. "
        "Every other Saturday 9am-1pm."
    )
    assert len(result.slots) >= 5

E2E Voice Testing with Cekura

Unit and integration tests verify your code works. E2E voice tests verify your agent works as a caller would experience it.

Cekura is a voice testing platform that calls your agent with a synthetic persona, records the conversation, and evaluates the outcome. It replaces the manual process of calling your own agent over and over during development.

Setup

Cekura connects to your agent in two ways:

WebRTC-direct: Cekura joins your LiveKit room as a synthetic participant. This skips the phone network entirely and tests agent logic at lower latency. Use this during development.

SIP-dial: Cekura dials your actual phone number through the PSTN. This tests the full path: Telnyx receives the call, routes it via SIP to your LiveKit server, which creates a room and dispatches the agent. Use this for pre-release validation.

For either mode, you configure the agent in Cekura’s dashboard or via their MCP tools:

# Connect Cekura MCP server to Claude Code
claude mcp add --transport http cekura --scope user \
  https://api.cekura.ai/mcp \
  --header "X-CEKURA-API-KEY:$CEKURA_API_KEY"

Designing Test Scenarios

Each scenario defines a persona, a goal, and expected outcomes. Here are the seven scenarios we run for the onboarding agent:

Test Profiles

Each scenario uses a test profile with realistic caller data. One lesson we learned the hard way: do not use obviously fake data.

• Emails ending in @example.com: some validation logic rejects these as placeholder domains.
• Phone numbers with 555: the 555 prefix is reserved for fictional use, and phone validation libraries flag these as invalid.
• Names like “Test User”: your own prompt may reject obviously fake input if you have anti-hallucination rules.

Use realistic but clearly test-identifiable data instead: a real-looking name, a valid phone format with a test-dedicated number, and an email on a domain you control.

What Cekura Evaluates

After each test call, Cekura provides:

• Transcript: full text of what was said by both sides.
• Metrics: time-to-first-word, total duration, talk ratio (how much the agent talked vs. the caller), number of interruptions.
• Tool call log: which tools were called, in what order, with what arguments.
• Custom metric scores: pass/fail evaluations you define based on the conversation outcome.

Evaluating Conversations

Raw transcripts are not enough. You need structured evaluation criteria that can be scored consistently.

Expected Outcome Prompts

For each scenario, define what should happen:

ONBOARD-001 Expected Outcome:
- Agent collects: business name, trade type, services, location, hours,
  owner name, phone, email
- Agent reads back all details and gets confirmation
- Agent triggers onboarding pipeline
- Agent reads back website URL and reference code
- Total duration: under 4 minutes

Rubric Scoring

Define pass/fail conditions on measurable criteria:

• Required fields captured: Did the agent collect all mandatory information? Check tool call arguments in the log.
• Tool ordering: Were tools called in the correct sequence? collect_business_profile must complete before trigger_onboarding_pipeline.
• Latency: Time-to-first-word under 2 seconds. Total pipeline time under 30 seconds.
• Conversation quality: No hallucinated information. No internal tool names spoken aloud. No more than 2 sentences per turn.

Common False Positives and False Negatives

False positive: the test passes but the agent behaved badly. This happens when your evaluation criteria are too loose. “All fields collected” passes even if the agent hallucinated the email address instead of asking for it. Fix this by checking tool call arguments, not just tool call existence.

False negative: the test fails but the agent behaved correctly. This happens with STT errors. The caller said “Maple Leaf Plumbing” but Deepgram transcribed “maple leaf plumming.” Your exact-match assertion fails even though the agent handled it fine. Fix this by using fuzzy matching or LLM-based evaluation for natural language fields.

CI Pipeline Setup

Here is the pipeline we run on every PR:

# .github/workflows/ci.yml (main-agent)
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v4
      - run: uv sync --frozen
      - run: uv run ruff check src/
      - run: uv run mypy src/

  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v4
      - run: uv sync --frozen
      - run: uv run pytest tests/unit/ -q --cov=src --cov-fail-under=80

  integration-tests:
    runs-on: ubuntu-latest
    needs: [lint, unit-tests] # Only run if fast checks pass
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v4
      - run: uv sync --frozen
      - run: uv run pytest tests/integration/ -v --timeout=60
        env:
          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}

The key decisions:

• Lint and unit tests run in parallel. Both are fast and independent.
• Integration tests depend on lint + unit. No point burning API credits if the code does not even compile.
• Coverage enforcement at 80%, not 100%, because testing LLM prompt text line-by-line adds no value. Focus coverage on validation logic, data transformations, and error handling.
• E2E voice tests are NOT in CI. They take five minutes, cost real money per run, and are inherently flaky due to STT/TTS variability. Run them on a schedule (nightly or pre-release) or trigger them manually before deploying.

When to Run E2E Tests

E2E voice tests belong in three places:

1. Before a release. Run all seven scenarios against the staging deployment. If any fail, investigate before promoting to production.
2. On a nightly schedule. This catches regressions from upstream provider changes (STT model updates, LLM behavior shifts) without blocking developer velocity.
3. After prompt changes. If you modify prompts.py, run at least the happy path scenario to verify the agent still completes a full onboarding.

Do not run them on every PR. The feedback cycle is too slow, the cost adds up, and the flakiness will train your team to ignore failures.

Summary

Testing voice agents requires discipline at every layer. Unit tests give you fast feedback on tool logic and validation. Integration tests verify that your prompts produce correct structured output from real LLMs. E2E voice tests prove the whole system works when a human-like caller picks up the phone.

The test pyramid is not a suggestion. It is a survival strategy. Without unit tests, you will spend hours debugging issues that a three-second test run would have caught. Without integration tests, you will ship prompt regressions that break availability parsing. Without E2E tests, you will deploy an agent that passes every test but cannot hold a conversation.

Start with unit tests. They are free, fast, and catch most bugs. Add integration tests when your prompts stabilize. Add E2E tests when you have a deployed agent and need confidence before releases. Resist the temptation to skip any layer, because each one catches failures the others miss.