ShipVoice
Get founding access
Primer / Part IV · Hardening for Production

Chapter 12: Deployment and Operations

Your agent works on your laptop. You speak into the microphone, it speaks back, it calls tools, it handles edge cases. Now what? A voice agent sitting on a developer machine is a demo. A voice agent answering real phone calls at 3am while you sleep is a product. The gap between the two is what kills most projects.

We will containerize the agent with Docker, deploy it to Fly.io, self-host a LiveKit server on a Hetzner VPS, set up monitoring with Sentry, and walk through the scaling decisions you will face when real traffic arrives.

Containerization with Docker and uv

Voice agents have an unusual build requirement: they need to download model files before they can start. Silero VAD and the turn detection model are loaded from Hugging Face on first run. If you skip this step in your Docker build, the first call after deployment will hang for 10-30 seconds while the models download. Users will hear silence and hang up.

A multi-stage Dockerfile solves this cleanly. The build stage installs dependencies and downloads models. The runtime stage copies the result and runs the agent as a non-root user.

# syntax=docker/dockerfile:1
ARG PYTHON_VERSION=3.11
FROM ghcr.io/astral-sh/uv:python${PYTHON_VERSION}-bookworm-slim AS base

ENV PYTHONUNBUFFERED=1
ENV HF_HOME=/app/.cache/huggingface

# --- Build stage ---
FROM base AS build

RUN apt-get update && apt-get install -y \
    gcc \
    g++ \
    python3-dev \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY pyproject.toml uv.lock ./
RUN uv sync --locked --no-dev --no-install-project

COPY . .

RUN uv run python main.py download-files

# --- Production stage ---
FROM base

ARG UID=10001
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/app" \
    --shell "/sbin/nologin" \
    --uid "${UID}" \
    appuser

WORKDIR /app

COPY --from=build --chown=appuser:appuser /app /app

USER appuser

EXPOSE 8081

CMD ["uv", "run", "python", "main.py", "start"]

Key details:

uv for dependency resolution. The ghcr.io/astral-sh/uv base image ships with uv pre-installed. uv sync --locked --no-dev --no-install-project installs all production dependencies from the lockfile in seconds, not minutes. The --no-install-project flag means we install dependencies first (cached layer) and copy source code second, so code changes do not trigger a full dependency reinstall.

HF_HOME environment variable. Hugging Face models are cached to $HF_HOME. Setting this explicitly ensures the models land in a known directory that survives the multi-stage copy.

download-files command. LiveKit’s agent CLI provides this command specifically for Docker builds. It triggers the prewarm function, which loads Silero VAD and the turn detection model into the cache. Without this, your container starts fast but the first call is slow.

Non-root user. The appuser with UID 10001 cannot write outside /app. This is a basic security measure that costs nothing and prevents a compromised agent process from touching the host filesystem.

Build tools in the build stage only. gcc, g++, and python3-dev are needed to compile native extensions (some audio processing libraries require them), but they do not belong in the production image. The multi-stage build keeps the final image small, typically around 600MB.

Deploying to Fly.io

Fly.io is a good fit for voice agents because it gives you dedicated VMs (not shared containers), multiple regions, and a deployment model that avoids dropped calls. Here is the fly.toml configuration:

app = 'voiceclaw-agent'
primary_region = 'yyz'
kill_timeout = '5m0s'

[build]
  dockerfile = 'Dockerfile'
  ignorefile = '.dockerignore'

[deploy]
  strategy = 'bluegreen'

[env]
  PYTHONUNBUFFERED = '1'

[checks]
  [checks.healthcheck]
    port = 8081
    type = 'tcp'
    interval = '10s'
    timeout = '2s'
    grace_period = '3m0s'

[[vm]]
  memory = '4gb'
  cpu_kind = 'performance'
  cpus = 2

Bluegreen deployment

The strategy = 'bluegreen' setting is essential for voice agents. In a rolling deployment, the old machine is killed before the new one is verified healthy. If a caller is mid-conversation, their call drops. Bluegreen deploys spin up a completely new machine, wait for it to pass health checks, then shift traffic. The old machine stays alive until its kill_timeout expires, giving active calls time to finish naturally.

The kill_timeout = '5m0s' gives active calls up to five minutes to wrap up before the old machine is forcibly stopped. Most onboarding calls complete in 2-3 minutes, so this is generous but safe.

Machine sizing

A performance-2x machine (2 dedicated CPU cores, 4GB RAM) comfortably handles about 2 concurrent calls. Each call requires a simultaneous STT stream, LLM inference, and TTS synthesis. The LLM inference is remote (OpenAI API), so it does not consume local CPU, but the STT and TTS streams involve real-time audio encoding and decoding that do use CPU. Two concurrent calls on 2 cores leaves enough headroom for the system itself.

If you need more concurrency per machine, performance-4x (4 CPU, 8GB) handles roughly 4 concurrent calls. Beyond that, scale horizontally with more machines rather than bigger ones.

Secrets management

Never put API keys in fly.toml. Use Fly’s encrypted secrets:

fly secrets set OPENAI_API_KEY=sk-...
fly secrets set DEEPGRAM_API_KEY=...
fly secrets set CARTESIA_API_KEY=...
fly secrets set LIVEKIT_URL=wss://livekit.yourdomain.com
fly secrets set LIVEKIT_API_KEY=...
fly secrets set LIVEKIT_API_SECRET=...
fly secrets set SENTRY_DSN=https://[email protected]/...

These are injected as environment variables at runtime. They are encrypted at rest and never appear in build logs.

Deploying

fly deploy

That is the entire deploy command. Fly builds the Docker image (remotely, on their build machines), pushes it to their registry, creates a new machine, waits for health checks, and shifts traffic. A typical deploy takes 3-4 minutes, most of which is the Docker build.

Self-Hosting LiveKit Server

LiveKit Cloud is the easiest path. You sign up, get a URL, and point your agent at it. But if you have cost constraints or want full control over the media path, self-hosting is straightforward. LiveKit server is surprisingly lightweight. It is a Selective Forwarding Unit (SFU) that routes audio and video packets without processing them. The heavy work (STT, LLM, TTS) happens in the agent, not the server.

When to self-host

Self-host when you want predictable costs at scale, when you need the SIP server co-located with the SFU for minimal latency, or when your callers are concentrated in a region where you can get a cheap VPS. LiveKit Cloud is the better choice when you need multi-region presence, do not want to manage infrastructure, or are still validating your product.

Hardware requirements

A Hetzner VPS with 2 CPU cores and 8GB RAM is more than enough for an SFU handling dozens of concurrent rooms. The LiveKit server itself uses very little CPU because it is forwarding packets, not transcoding them. The SIP server (which bridges phone calls into WebRTC rooms) is similarly lightweight.

Docker Compose setup

A production LiveKit deployment needs four services: the LiveKit server, the SIP server, Redis (for signaling), and Caddy (for SSL termination).

services:
  caddy:
    image: caddy:2
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    restart: unless-stopped

  livekit:
    image: livekit/livekit-server:latest
    ports:
      - '7880:7880'
      - '7881:7881'
      - '7882:7882/udp'
    volumes:
      - ./livekit.yaml:/etc/livekit.yaml
    command: --config /etc/livekit.yaml
    depends_on:
      - redis
    restart: unless-stopped

  sip:
    image: livekit/sip:latest
    ports:
      - '5060:5060/udp'
      - '5060:5060/tcp'
    volumes:
      - ./sip.yaml:/etc/sip.yaml
    command: --config /etc/sip.yaml
    depends_on:
      - livekit
    restart: unless-stopped

volumes:
  caddy_data:

Caddy handles SSL certificates automatically via Let’s Encrypt. Point your DNS records (livekit.yourdomain.com and sip.yourdomain.com) at the VPS IP, and Caddy does the rest. No manual certificate management.

The LiveKit server config (livekit.yaml) points at Redis for signaling and defines your API keys. The SIP server config (sip.yaml) connects to LiveKit and listens for inbound SIP trunks from your telephony provider (Telnyx, Twilio, etc.).

Scaling

The bottleneck is always the agent

The LiveKit server can handle hundreds of concurrent rooms on modest hardware. Redis barely breaks a sweat. The bottleneck is the agent machine, because each call ties up real resources: a persistent WebSocket to Deepgram for STT, one or more HTTP requests to OpenAI for LLM inference, and a streaming connection to Cartesia for TTS. These run concurrently for the duration of the call.

Horizontal scaling

The simplest way to handle more calls is to run more agent machines. In Fly.io, set min_machines_running in your fly.toml:

[[vm]]
  memory = '4gb'
  cpu_kind = 'performance'
  cpus = 2
  min_machines_running = 2

Two machines with 2 CPUs each gives you roughly 4 concurrent calls. LiveKit distributes new sessions to the agent worker with the lowest load. You do not need a load balancer. The LiveKit dispatch system handles it.

Vertical scaling

Alternatively, run bigger machines. A performance-4x (4 CPU, 8GB) handles roughly 4 concurrent calls on a single machine. This is simpler to operate but less resilient. If the machine goes down, all active calls drop. With horizontal scaling, losing one machine only drops the calls on that machine.

Rule of thumb

2 concurrent calls per 2-CPU machine. This is conservative and accounts for peak CPU during simultaneous STT + TTS + audio encoding. You can squeeze 3 calls onto a 2-CPU machine if your calls are short and bursty, but sustained concurrent calls will start causing audio glitches at that level.

Capacity Planning

Load threshold

LiveKit agents report their load to the server as a value between 0 and 1. The default threshold is 0.7. When a worker reports load above this, the server stops dispatching new sessions to it. Existing calls continue, but new callers get routed to a different worker (or queued if no workers are available).

”Worker at full capacity”

This warning in your logs means exactly what it says: the agent process has hit its load threshold and will not accept new calls. If you see this consistently, you need more machines or bigger machines. If you see it in brief spikes, your current capacity is adequate but tight.

”No warmed process available”

This is worse. It means LiveKit tried to dispatch a call but no agent worker was ready. The caller hears silence or a connection error. This happens when all workers are at capacity, when a machine is still starting up (downloading models, initializing), or when your min_machines_running is set too low.

The fix is prewarming. The prewarm function in your agent loads VAD and turn detection models before any call arrives. Combined with min_machines_running > 0, this ensures at least one machine is always warm and ready. The 3-minute grace_period in the health check gives the machine time to download models and prewarm before Fly starts routing traffic to it.

Monitoring concurrent calls

Track the number of active rooms via the LiveKit server API or dashboard. Compare this against your machine count times 2 (your per-machine capacity). When active rooms consistently exceed 60% of total capacity, it is time to scale up. Do not wait until you hit 100%. Audio quality degrades before you hit the hard limit.

Monitoring with Sentry

Sentry is the simplest way to catch production errors without building your own monitoring stack. Set the SENTRY_DSN environment variable, initialize the SDK in your entry point, and errors are reported automatically.

Common errors and what they mean

Cartesia TTS “no audio frames pushed.” The TTS provider accepted the request but returned no audio data. This typically happens under load when the Cartesia API times out internally. It is usually transient. If it happens frequently, check Cartesia’s status page or add retry logic to your TTS calls.

OpenAI 400 “could not parse JSON body.” The chat context sent to the LLM contained malformed data. This usually means a tool result was not properly serialized, or the conversation history got corrupted by a race condition. Check your tool return types: they should always return strings or simple dicts, never complex objects.

ReadTimeout. One of your API providers (Deepgram, OpenAI, Cartesia) did not respond in time. This is almost always a provider-side issue. Log the provider name and endpoint so you can correlate with their status pages. Retries with backoff help, but a 5-second timeout on TTS synthesis is a hard limit because the caller is waiting in real time.

EOFError in audio decoder. The WebRTC audio buffer overflowed or the connection dropped mid-stream. This happens when the agent machine is CPU-starved and cannot keep up with incoming audio packets. If you see this regularly, you are running too many concurrent calls on the machine.

“Worker at full capacity.” Covered in the scaling section above. This is not an error per se, but it should trigger an alert because it means callers may be unable to connect.

Filtering noise from real issues

Voice agents generate a lot of transient errors. Network blips, provider hiccups, callers hanging up mid-sentence: these all produce exceptions that are not actionable. Configure Sentry to:

  • Ignore ConnectionResetError and BrokenPipeError (caller hung up)
  • Group TTS timeout errors so they do not flood your inbox
  • Alert on sustained error rates, not individual occurrences
  • Tag errors with call_id so you can correlate with specific conversations

The goal is not zero errors. It is knowing the difference between “the network hiccupped” and “we shipped a bug.”

Health Checks and Graceful Shutdown

TCP health check

The agent opens a TCP listener on port 8081 when it starts. Fly.io pings this port every 10 seconds. If the port is unresponsive for two consecutive checks, Fly marks the machine as unhealthy and stops routing new calls to it.

TCP checks are deliberately simple. You do not want a health check that calls an external API (Deepgram, OpenAI) because if that API is down, your agent will be marked unhealthy even though the agent itself is fine. The health check should answer one question: “Is the process running and able to accept connections?” Nothing more.

Grace period

The grace_period = '3m0s' setting tells Fly to wait up to three minutes before expecting the health check to pass. This accounts for the Docker container startup, uv resolving the virtual environment, and the prewarm function downloading and loading models. On a cold start, this can take 60-90 seconds. The three-minute grace period provides a comfortable buffer.

If your models are pre-downloaded in the Docker image (which they should be, per the Dockerfile above), the grace period mostly covers container scheduling and process initialization. But keep it generous. There is no cost to a longer grace period, and a too-short one causes false-positive health check failures that trigger unnecessary restarts.

Graceful shutdown

When Fly decides to stop a machine (during deploy, scale-down, or maintenance), it sends SIGTERM. The agent process receives this and stops accepting new calls, but active calls continue. The kill_timeout = '5m0s' gives those calls up to five minutes to finish before SIGKILL is sent.

This is where bluegreen deployment pays off. During a deploy:

  1. Fly creates a new machine and waits for it to pass health checks.
  2. New calls are routed to the new machine.
  3. The old machine receives SIGTERM and stops accepting calls.
  4. Active calls on the old machine drain naturally.
  5. After kill_timeout, the old machine is forcibly stopped.

No calls are dropped unless they run longer than five minutes. Combined with the bluegreen strategy, this gives you zero-downtime deploys, the one thing you absolutely need when your product is answering phone calls.

Putting It All Together

A production voice agent deployment is not a single machine. It is a system:

  • Docker image with pre-downloaded models for fast cold starts
  • Fly.io for the agent, with bluegreen deploys and performance VMs
  • Hetzner VPS (or LiveKit Cloud) for the SFU and SIP server
  • Sentry for error tracking, tuned to filter noise
  • Health checks that verify the process, not external dependencies
  • Graceful shutdown that drains calls before killing machines

The goal is mundane reliability. The agent should answer calls at 3am on a Saturday the same way it does during your Tuesday morning test. Everything in this chapter (the multi-stage Docker build, the bluegreen deploy, the kill timeout, the prewarmed models) exists to make that possible.

This primer is the theory. ShipVoice is the code: a LiveKit boilerplate with per-minute billing, auth, telephony, and deploy already wired in.

Get founding access