Chapter 10: Prompt Engineering for Voice

Prompt engineering for voice agents is a different discipline than prompt engineering for chatbots. You are not writing instructions for a model that renders text on a screen. You are writing instructions for a model whose output is spoken aloud to someone holding a phone to their ear, often while driving a van to their next job. Every word the model generates becomes audio. Every formatting artifact becomes nonsense. Every ambiguity becomes a confused caller who hangs up.

What follows are the patterns we developed and refined while shipping a production voice agent and debugging it against seven Cekura test scenarios: synthetic callers designed to probe every failure mode we could think of.

Why Voice Prompts Differ from Chat Prompts

If you have written prompts for ChatGPT or Claude, you have habits that will hurt you in voice. Here is why.

Everything is spoken aloud. When a chat model outputs **Business Name:** Smith Plumbing, the user sees bold text and a label. When a voice agent outputs the same thing, the caller hears “asterisk asterisk business name colon asterisk asterisk smith plumbing.” Markdown is not formatting in voice. It is garbage. Lists with bullet points become a stream of “dash” sounds. Emojis become silence or their Unicode names. Your prompts must explicitly forbid all of it.

Latency matters. Every token the model generates adds latency before the caller hears the first word. A system prompt with 2,000 tokens of preamble means the model processes all of that context before producing its first response token. In chat, nobody notices an extra 200 milliseconds. On a phone call, 200 milliseconds is the difference between a natural pause and an awkward silence. Shorter prompts produce faster first responses.

The caller cannot scroll back. In chat, the user can re-read the last message. On a phone call, once the words are spoken, they are gone. If the agent rattles off a website URL, a reference code, and a phone number in one breath, the caller will remember at most one of them. Your prompts must instruct the agent to slow down, break information into chunks, and pause between critical pieces.

No visual structure. Chat interfaces use whitespace, headers, and line breaks to organize information. Voice has none of that. The agent must use conversational connectors (“so”, “and then”, “one more thing”) to signal transitions. Your prompt must teach this explicitly, with examples.

In our production prompts, we address all four of these with a dedicated section:

# Voice output rules

You are on a live phone call. Everything you say is spoken aloud
through text-to-speech. Follow these rules strictly:
- Plain text only. No markdown, no lists, no emojis, no formatting.
- Maximum two sentences per response. Ask one question at a time.
- NEVER say tool names, function names, or technical terms.
- Say "website" not "URL". Say "phone number" not "DID".

The two-sentence maximum was not an arbitrary choice. We measured response quality across dozens of test calls and found that responses longer than two sentences caused callers to interrupt, lose track, or disengage. One question at a time forces the model to collect information sequentially rather than dumping three questions and hoping the caller answers all of them.

Strict Tool Ordering and Sequence Enforcement

LLMs are eager to please. Given a set of tools, they will call whichever one seems most helpful right now, regardless of whether prerequisites have been met. In a voice agent with a multi-step pipeline, this eagerness creates chaos: the model skips the data collection step and jumps straight to website creation, or it ends the call before reading back the results.

The fix is explicit, numbered sequencing with imperative language:

# Tools -- strict ordering

You MUST follow this exact sequence. Never skip steps.

Step 1: Call `collect_business_profile` AS SOON AS the caller is ready.
Step 2: IMMEDIATELY after `collect_business_profile` returns, call
        `trigger_onboarding_pipeline`. Do NOT wait, do NOT ask the
        caller if they want to proceed, just call it right away.
Step 3: Read out the website URL and reference code that
        `trigger_onboarding_pipeline` returns.

Note the specific language patterns. “IMMEDIATELY after X returns” prevents the model from chatting between steps. “Do NOT wait, do NOT ask the caller if they want to proceed” blocks a common failure where the model says “Shall I go ahead and set up your website?” and waits for an answer, adding unnecessary latency and risking a confused caller who says “uh, I thought that’s what we were doing.”

The strongest pattern we found for forcing immediate tool calls is the return message from a preceding tool:

return (
    "Business profile confirmed and ready. "
    "You MUST now IMMEDIATELY call `trigger_onboarding_pipeline`. "
    "Do NOT say anything to the caller. Do NOT end the call. "
    "Do NOT say goodbye. "
    "Your ONLY next action must be calling trigger_onboarding_pipeline."
)

This is not subtle. It works because it removes every alternative the model might consider. Without this, we observed the agent saying “Great, everything looks good! Is there anything else?” and waiting (indefinitely) for a response before triggering the pipeline.

Preventing Hallucination in Data Collection

Voice agents that collect structured data from callers face a unique hallucination risk: the model interprets conversational filler as data. When you ask “What’s your business name?” and the caller says “Sure, it’s Smith Plumbing,” a naive prompt lets the model record “Sure” as the business name. We saw this happen in testing.

Filler rejection. We maintain an explicit blocklist in both the prompt and the tool validation code:

fillers = {
    "sure", "yes", "yeah", "ok", "okay", "go ahead",
    "yep", "no", "nah", "trade", "business", "name",
    "hi", "hello",
}
if business_name.lower() in fillers:
    raise ToolError(
        f"'{business_name}' is not a business name. "
        "Ask the caller for their actual business name."
    )

The prompt-side rule is just as important: “Do NOT record casual speech like ‘sure’, ‘yes’, ‘go ahead’ as business data.” Defense in depth: the prompt tries to prevent it, the code catches what the prompt misses.

Prompt-as-data rejection. This is the strangest hallucination we encountered. The model would call record_hours("Please tell me your operating hours"), passing its own question as the data. The model was not confused about what the caller said. It was filling in a required field with whatever text was on its mind, which happened to be the question it was about to ask.

if "please" in hours.lower() and (
    "tell" in hours.lower() or "specify" in hours.lower()
):
    raise ToolError(
        "That looks like a question, not operating hours. "
        "Ask the caller."
    )

The corresponding prompt rule: “NEVER pass a question or prompt as a tool argument. For example, NEVER do record_hours(‘Please tell me your hours’).”

Placeholder detection. In our contact collection task, we validate phone numbers and emails at the tool level. A phone number must match a regex pattern. An email must have a valid format. This prevents the model from recording “555-555-5555” (a common LLM placeholder) or “[email protected]” as real contact data.

The pattern across all of these is the same: state the rule in the prompt with CRITICAL RULES, then enforce it in code with ToolError. Never trust the prompt alone. Never rely on code alone. Use both.

Handling Difficult Caller Personalities

Synthetic testing with Cekura scenarios exposed caller personalities that break naive prompts. Three personality types caused the most failures.

Impatient callers. The synthetic caller says “Look, I’m busy, just get this done” or “Can we skip all these questions?” Without explicit guidance, the model either apologizes profusely (wasting more time) or skips required fields to accommodate the caller.

Our prompt handles this directly:

- You MUST collect services FIRST, then location.
  Do NOT skip services even if the caller is impatient
  or says "hurry up".
- If the caller is impatient, acknowledge briefly
  ("Got it, just a couple more quick questions") and
  continue collecting.

The key insight is that the acknowledgment must be brief. A long empathetic response (“I totally understand you’re busy, and I really appreciate your patience, this will just take a moment…”) makes an impatient caller more impatient. One short sentence, then move on.

Chatty callers. The opposite problem. The caller launches into a two-minute story about how they started their plumbing business in their dad’s garage. The agent needs to extract “plumbing” as the trade type from that story without being rude.

The prompt guidance: “politely redirect to collect specific information.” In practice, the model handles this reasonably well with minimal prompting. The bigger risk is the model getting drawn into the story and forgetting to call the recording tool. The two-sentence maximum response rule helps here because it forces the model to stay focused.

Interruptive callers. Some callers talk over the agent constantly. This is especially destructive when the agent is reading back details for confirmation. The caller interrupts halfway through the readback, the agent stops, and the confirmation happens on incomplete information.

The solution is a combination of prompt engineering and code. For critical moments, we disable interruptions:

async def on_enter(self) -> None:
    summary = self._build_summary()
    self.session.generate_reply(
        instructions=CONFIRM_DETAILS_ENTER.replace("{summary}", summary),
        allow_interruptions=False,
    )

For normal conversation, we keep responses short so there is less to interrupt. A one-sentence question gets interrupted less than a three-sentence preamble.

The Readback Gate Pattern

One of our most persistent bugs was the agent confirming details without reading them back. The model would collect all the business details, then immediately call confirm_details(confirmed=true) without ever speaking the details aloud. The caller never heard their business name, never got a chance to correct a typo, and ended up with a website for “Smith Plumming.”

Prompts alone could not fix this. We tried “You MUST read back all details before confirming.” The model would comply 90% of the time. The other 10% was unacceptable.

The solution is a code-enforced gate. The confirm_details tool checks whether readback_complete was called first:

@function_tool
async def readback_complete(self, ctx: RunContext) -> str:
    """Call this AFTER you have read all business details
    back to the caller."""
    self._readback_done = True
    self._readback_handle = ctx.speech_handle
    return (
        "Good. Now wait for the caller to confirm "
        "or request changes."
    )

@function_tool
async def confirm_details(self, ctx: RunContext, confirmed: bool) -> str | None:
    """Confirm or reject the business details."""
    if not self._readback_done:
        raise ToolError(
            "You must read back all details first. "
            "Call readback_complete after reading them back."
        )
    if ctx.speech_handle == self._readback_handle:
        raise ToolError(
            "Wait for the caller to respond before confirming. "
            "Do not confirm in the same turn as the readback."
        )

There are two guards here. The first (_readback_done) ensures the readback happened at all. The second (speech_handle) ensures the confirmation happens on a different turn than the readback, preventing the model from calling readback_complete and confirm_details in the same response.

The prompt reinforces this with explicit sequencing:

You MUST follow this exact sequence:
1. Read back ALL details conversationally to the caller.
2. After finishing the readback, call `readback_complete`.
3. Wait for the caller to respond.
4. If they say yes, call `confirm_details(confirmed=true)`.

This is the defense-in-depth principle again. The prompt tells the model what to do. The code prevents it from doing anything else. Neither is sufficient alone.

Uninterruptible Speech for Critical Moments

Voice agents should generally be interruptible. If the caller starts talking, the agent should stop and listen. But there are moments where the agent must finish speaking: reading back a website URL, spelling out a reference code, reciting a phone number.

LiveKit provides two mechanisms for this:

# For generated replies (LLM decides what to say)
self.session.generate_reply(
    instructions="Read back all details to the caller.",
    allow_interruptions=False,
)

# For scripted speech (you control the exact words)
ctx.session.say(
    "Your reference code is A B C one two three.",
    allow_interruptions=False,
)

We use allow_interruptions=False in three places: the detail readback, the website URL announcement, and the reference code. Everything else is interruptible.

The temptation is to make more things uninterruptible “just in case.” Resist this. An agent that ignores the caller feels broken. Callers will say “hello?” repeatedly, get frustrated, and hang up. Reserve uninterruptible speech for moments where partial information is worse than no information.

Common Prompt Failures and Fixes

After debugging seven Cekura test scenarios and dozens of real calls, we cataloged the most common prompt failures. Here they are, with the fixes that worked.

Several patterns emerge from this table. First, never assume the model will infer boundaries between tasks. If the operations task does not handle phone numbers, the model might announce that limitation to the caller. You must tell it explicitly what to do with out-of-scope information: ignore it silently.

Second, the model’s default behavior when it runs out of things to do is to end the conversation. You must actively prevent premature endings with strong language in both prompts and tool return values.

Third, conditional logic in prompts is fragile. Do not rely on the model to check whether a field is empty before reading it back. Handle that in code. Build the summary string conditionally so the model never sees an empty field.

Principles for Voice Prompt Engineering

After months of iteration, these are the principles we follow for every prompt we write.

Say it, then enforce it. Every behavioral rule appears in both the prompt and the code. The prompt is the first line of defense; the code is the last.

Show, do not tell. Bad/good example pairs teach tone better than abstract rules. “Sound natural” means nothing. “Bad: ‘What is your business name?’ Good: ‘So, what’s the name of your business?’” teaches exactly what you mean.

One question, one turn. Never stack multiple questions in a single response. The caller will answer one and forget the rest.

Name the failure mode. Do not say “be careful with data.” Say “NEVER pass a question or prompt as a tool argument. For example, NEVER do record_hours(‘Please tell me your hours’).” The more specific the prohibition, the more likely the model will follow it.

Keep it short. Every word in your prompt costs latency on the first response. Every word in the agent’s response costs time on the call. Brevity is not a style preference. It is a performance requirement.

Voice prompt engineering is less about cleverness and more about discipline. You are writing operating procedures for a system that will encounter every edge case a real caller can throw at it. The prompts that survive production are the ones that leave no room for interpretation.