Deepgram Voice Agent

The Deepgram Voice Agent API runs speech-to-text, the LLM and text-to-speech behind a single WebSocket, with its own turn-taking. Bridge it to a Vobiz <Stream> and you have a phone-callable agent that a caller can interrupt mid-sentence — no separate STT vendor, no TTS vendor, and no turn logic of your own. Source code: vobiz-ai/Vobiz-Deepgram-Voice-Agent — the reference FastAPI bridge used throughout this guide (app.py, call.py, mock_vobiz.py), plus a mock client that exercises the whole protocol without placing a call.

Scope: inbound and outbound. Attach a number to an application whose answer URL points at /answer, or place an outbound call with the same URL — the bridge does not care which direction the call came from.

How it works

Vertical call flow: caller audio travels from the caller over PSTN to Vobiz, over a bidirectional WebSocket as media events to app.py, and into the Deepgram Voice Agent as send_media. Agent audio returns as audio bytes, then playAudio, then PSTN. A dashed barge-in loop runs from Deepgram back to Vobiz, labelled UserStartedSpeaking then clearAudio.

Vobiz opens the WebSocket to you — your wss:// URL is the server. Caller audio arrives as media events, the agent’s speech goes back as playAudio, and app.py is a thin relay: format handling, barge-in, and the Vobiz control protocol. Deepgram owns the conversation.

Vobiz fetches /answer and receives a <Stream> element.
Vobiz opens a WebSocket to /media/<secret> and sends a start event.
Caller audio arrives as media events and is forwarded to Deepgram.
Agent audio comes back as raw bytes and goes to Vobiz as playAudio events.
When the caller interrupts, Deepgram signals it and the app sends clearAudio.
After each turn a checkpoint is sent; Vobiz replies playedStream once the caller has actually heard it.

The bidirectional Stream protocol

Every event and control message on the socket — start, media, dtmf, playedStream, clearedAudio, stop — and what you send back.

Audio profiles

The two directions of a Vobiz bidirectional stream are configured independently. Both profiles below are pure passthrough — the bridge never resamples.

`AUDIO_MODE`	Vobiz → app (`<Stream contentType>`)	app → Vobiz (`playAudio`)	Deepgram agent settings
`mulaw` (default)	`audio/x-mulaw;rate=8000`	mu-law 8000	`mulaw` 8k in / `mulaw` 8k out
`l16`	`audio/x-l16;rate=16000`	L16 24000	`linear16` 16k in / `linear16` 24k out

mulaw matches the PSTN leg exactly and is the right default. l16 trades bandwidth for a wider band — 16 kHz in, 24 kHz out — and Deepgram emits 24 kHz natively, so nothing is degraded on the way to Vobiz.

Never put rate=24000 on <Stream contentType>. That attribute configures the inbound direction, which tops out at 16 kHz. 24 kHz is outbound-only — valid on playAudio and nowhere else.

playAudio accepts L16 at 8/16/24 kHz and mu-law at 8 kHz. See audio formats for the full matrix.

Requirements

Requirement	Detail
Deepgram API key	From console.deepgram.com — it runs STT, the LLM and TTS
Vobiz account	`AUTH_ID`, `AUTH_TOKEN`, and a DID — console.vobiz.ai
Public HTTPS + WSS endpoint	Vobiz connects inbound to your `wss://` URL. ngrok is fine in development.
Runtime	Python 3.11+

No separate LLM key is needed. Deepgram manages the OpenAI connection and bills it through your Deepgram account.

If your WebSocket endpoint is IP-restricted, allow inbound TCP 443 from the Vobiz media fleet. The RTP rule (UDP 5000–65535) does not cover it — see IP whitelisting.

Step 1: Configure the bridge

git clone https://github.com/vobiz-ai/Vobiz-Deepgram-Voice-Agent
cd Vobiz-Deepgram-Voice-Agent
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env       # then fill it in

.env

# ---- Required ----
DEEPGRAM_API_KEY=
# Public host Vobiz reaches, no scheme.
PUBLIC_HOSTNAME=your-host.example.com

# ---- Optional ----
HTTP_PORT=5050
# mulaw (default) or l16 — see Audio profiles above.
AUDIO_MODE=mulaw

# ---- Security ----
# Alphanumeric — it becomes a path segment on the wss:// URL.
#   python -c "import secrets; print(secrets.token_hex(16))"
STREAM_SECRET=
# true to validate the X-Vobiz-Signature-V3 HMAC on /answer.
VERIFY_SIGNATURE=

# ---- Outbound only (used by call.py) ----
VOBIZ_AUTH_ID=
VOBIZ_AUTH_TOKEN=
FROM_NUMBER=+91XXXXXXXXXX
TO_NUMBER=+91XXXXXXXXXX

Dependencies

requirements.txt

fastapi
uvicorn[standard]
deepgram-sdk>=7.7.0
python-dotenv
python-multipart
websockets
requests

Step 2: Run it

python app.py

Expose the server publicly, then confirm what it resolved:

curl -s https://<public>/health | python -m json.tool

{
  "answer_url": "https://<public>/answer",
  "stream_url": "wss://<public>/media/<secret>",
  "audio_mode": "mulaw",
  "vobiz_to_app": "audio/x-mulaw;rate=8000",
  "app_to_vobiz": "audio/x-mulaw;rate=8000",
  "play_frame_bytes": 160,
  "agent": { "listen": "flux-general-en", "think": "gpt-4o-mini", "speak": "flux-alexis-en" },
  "answer_signature_checked": false,
  "stream_secret_checked": true
}

Whichever host you use becomes the answer URL below.

Route	Role
`GET/POST /answer`	The XML Vobiz executes when the call is answered
`WS /media/<secret>`	The bidirectional media stream `<Stream>` connects to
`POST /stream-status`	`<Stream statusCallbackUrl>` — `StartStream`, `PlayedStream`, `StopStream`
`POST /hangup`	The call’s `hangup_url`
`GET /health`	Resolved URLs and audio profile

Step 3: Place a call

Inbound
Outbound

Vobiz decides what to do with an inbound call by looking up the Voice Application attached to the number that was dialled. A number on its own is not enough — create the application first, then attach a number to it.1. Create a Voice ApplicationIn the console, go to Voice Applications → Create application. Set Primary answer URL to https://<public>/answer with method POST. Optionally set the Hangup URL to https://<public>/hangup to receive the call-ended webhook.

Vobiz console Create application dialog with fields for application name, primary answer URL with a POST method selector, hangup URL, and fallback answer URL

2. Attach a numberOpen the application and attach one of your DIDs under Attached Numbers → Attach number. Calls to that number now fetch XML from your answer URL.

Attached Numbers panel on a Vobiz voice application showing no numbers attached yet and an Attach number button

Dial the attached number with a 0 or +91 prefix — 09XXXXXXXXX or +919XXXXXXXXX. You should hear the greeting.See Applications for the full reference.

call.py places the call and points it at this server, so no Voice Application is needed:

python call.py --to +919XXXXXXXXX

It posts to Make a Call:

curl -X POST "https://api.vobiz.ai/api/v1/Account/$AUTH_ID/Call/" \
  -H "X-Auth-ID: $AUTH_ID" \
  -H "X-Auth-Token: $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "from": "+91XXXXXXXXXX",
    "to": "+91XXXXXXXXXX",
    "answer_url": "https://<public>/answer",
    "answer_method": "POST",
    "hangup_url": "https://<public>/hangup",
    "hangup_method": "POST"
  }'

Answer the phone and talk. app.py prints [user] and [assistant] lines as the conversation runs.

The answer XML

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Stream bidirectional="true"
          audioTrack="inbound"
          keepCallAlive="true"
          contentType="audio/x-mulaw;rate=8000"
          statusCallbackUrl="https://example.com/stream-status"
          statusCallbackMethod="POST">wss://example.com/media/&lt;secret&gt;</Stream>
  <Hangup/>
</Response>

Attribute	Why it matters
`bidirectional="true"`	Without it the socket is receive-only and `playAudio` is ignored. It is also what unlocks `clearAudio` and `checkpoint`
`keepCallAlive="true"`	Holds XML execution while the socket is open. Without it the element returns immediately, the document ends, and Vobiz hangs up on End Of XML Instructions
`audioTrack="inbound"`	The only track setting valid alongside `bidirectional` — `both` is rejected. It also stops the agent hearing its own playback
`contentType`	Controls Vobiz → app only. Never the playback format
`statusCallbackUrl`	HTTP mirror of the socket lifecycle: `StartStream`, `PlayedStream`, `ClearedAudio`, `DroppedStream`, `StopStream`

The WebSocket URL is the element’s text content, not a url="" attribute.

Malformed answer XML is not an HTTP error. Vobiz accepts the 200, then drops the call about a second later, and the only trace is the CDR field hangup_cause_name: "Invalid Answer XML". A URL carrying two query parameters contains a bare &, which is enough on its own to invalidate the document — the reference bridge runs every interpolated value through html.escape() for this reason.

Inside the bridge

The agent settings

Both ends of the pipeline are Flux, Deepgram’s conversational speech models, and both live on v2 endpoints — so each provider must pin version: "v2". Omit it and the provider falls back to v1, where the Flux model names are not valid.

AGENT_SETTINGS = {
    "type": "Settings",
    "audio": {"input": PROFILE["agent_input"], "output": PROFILE["agent_output"]},
    "agent": {
        "language": "en",
        "listen": {"provider": {"type": "deepgram", "version": "v2", "model": "flux-general-en"}},
        "think":  {"provider": {"type": "open_ai", "model": "gpt-4o-mini"}, "prompt": PROMPT},
        "speak":  {"provider": {"type": "deepgram", "version": "v2", "model": "flux-alexis-en"}},
        "greeting": GREETING,
    },
}

Sending the settings frame is what starts the conversation, greeting included. To swap the LLM, change the think provider — for example {"type": "anthropic", "model": "claude-sonnet-5"}.

Barge-in

Deepgram detects the caller talking over the agent and emits UserStartedSpeaking. Vobiz may still have seconds of the agent’s reply buffered, so the bridge flushes it:

if isinstance(message, AgentV1UserStartedSpeaking):
    await stream.clear()   # {"event": "clearAudio", "streamId": ...}

Vobiz confirms with a clearedAudio event. Without this the agent keeps talking over the caller. See clearAudio.

Knowing the caller actually heard it

playAudio means sent, not heard. After each turn the bridge sends a checkpoint and Vobiz answers playedStream once the buffered audio has played out:

if isinstance(message, AgentV1AgentAudioDone):
    await stream.checkpoint()   # {"event": "checkpoint", "name": "turn-3", ...}

That signal is what lets an agent say goodbye and then hang up without clipping its own last word.

Frame slicing

Deepgram hands over arbitrarily sized audio chunks. Vobiz is happiest with steady telephony-sized frames, so VobizStream.play() re-slices into 20 ms — 160 bytes mu-law at 8 kHz, 960 bytes L16 at 24 kHz — rather than forwarding blindly.

Format mismatches are silent

Since the bridge never resamples, a contentType that disagrees with AUDIO_MODE is just garbage audio into the agent. read_start() compares the XML against start.mediaFormat and prints [audio] WARNING instead of failing quietly.

Security

Both public endpoints are reachable by anyone who learns the URL, so the bridge ships two opt-in controls:

Control	What it protects	How
`STREAM_SECRET`	The media socket	A random alphanumeric string becomes a path segment on the `wss://` URL and is checked on connect. A mismatch closes the socket with `1008`
`VERIFY_SIGNATURE`	`/answer`	Validates the `X-Vobiz-Signature-V3` (or V2) HMAC, keyed by `VOBIZ_AUTH_TOKEN`

extraHeaders cannot authenticate the media socket. The values never reach the WebSocket — not as an upgrade header, and not in the start frame, whose extra_headers field stays the literal "{}". They surface only in the statusCallbackUrl payload, as X-VH-<key>. That makes extraHeaders status-callback metadata rather than stream credentials, which is why the secret rides in the URL path instead.

The webhook signature covers the URL and a nonce, never the body. Voice webhooks are form-encoded, so any scheme that hashes a JSON body will not verify. Query parameters are stripped first, and behind a tunnel the public URL has to be rebuilt — request.url is the internal address Vobiz never saw. Signature headers are only emitted when the callback URL has auth credentials configured on it, which is why VERIFY_SIGNATURE is opt-in. See Validating callbacks.

Test without placing a call

mock_vobiz.py stands in for Vobiz — it speaks the media-stream protocol against a running app.py, answers checkpoint with playedStream, and reports what came back.

python mock_vobiz.py                        # 4s of silence — transport and greeting
python mock_vobiz.py --wav question.wav     # stream a real question (mono, profile sample rate)

A pass means playAudio frames came back in the format the XML asked for.

Configuration reference

Variable	Required	Default	Notes
`DEEPGRAM_API_KEY`	Yes	—	Runs the whole agent — STT, LLM and TTS
`PUBLIC_HOSTNAME`	Yes	—	Public host Vobiz reaches, no scheme
`HTTP_PORT`	No	`5050`	Server port
`AUDIO_MODE`	No	`mulaw`	`mulaw` or `l16`
`STREAM_SECRET`	No	(none)	Alphanumeric; becomes a path segment on the stream URL
`VERIFY_SIGNATURE`	No	(off)	`true` to validate the HMAC on `/answer`
`VOBIZ_AUTH_ID`	Outbound	—	Only needed by `call.py`
`VOBIZ_AUTH_TOKEN`	Outbound	—	Also the webhook signing key
`FROM_NUMBER`	Outbound	—	A DID this account owns
`TO_NUMBER`	Outbound	—	Default destination for `call.py`

Troubleshooting

Symptom	Cause	Resolution
Call hangs up immediately; log shows End Of XML Instructions	`keepCallAlive="true"` missing, or `audioTrack="both"` with `bidirectional="true"`	Set `keepCallAlive="true"` and `audioTrack="inbound"`
Silence in both directions	`bidirectional="true"` missing — `playAudio` is ignored on a one-way stream	Add the attribute
Garbled or chipmunk audio	`<Stream contentType>` and `AUDIO_MODE` disagree	Match them; the app prints `[audio] WARNING` on the start event
Agent talks over the caller	`clearAudio` is not reaching Vobiz	Check `streamId` is set before the first `playAudio`
Stream connects but no usable media arrives	`rate=24000` set on `<Stream contentType>`	24 kHz is outbound-only — use 8000 or 16000 inbound
WebSocket closes with `1008`	`STREAM_SECRET` does not match the secret in the stream URL path	Re-copy the value, or clear it while testing
`/answer` returns `403`	`VERIFY_SIGNATURE=true` but the callback URL has no auth credentials configured, so no signature headers are sent	Set the credentials in the console first, then enable it
Outbound call returns `401` or `402`	`401` credentials, `402` balance. “from number … not owned” means the DID belongs to another account	Check `VOBIZ_AUTH_ID`/`VOBIZ_AUTH_TOKEN` and the `from` DID
Inbound call is never answered	The number has no Voice Application attached, or the application’s answer URL does not point at this server	See Step 3 → Inbound
Inbound call does not connect	The number was dialled without a prefix	Dial with `0` or `+91`
Hangup webhook reports zero cost	Voice and stream are billed as separate line items that appear only in the CDR	Read `GET /cdr/{call_uuid}`

Next steps

Clone the reference bridge: vobiz-ai/Vobiz-Deepgram-Voice-Agent
Read the <Stream> reference, audio formats, stream events and playAudio
Deepgram’s Voice Agent API and Flux docs
Bridging a different model over the same socket? See WebSockets and Gemini Live

AI Voice Platforms

Official SDKs

Browser & WebRTC

Business Phone Systems

Migrations

Deepgram Voice Agent

How it works

The bidirectional Stream protocol

Audio profiles

Requirements

Step 1: Configure the bridge

Dependencies

Step 2: Run it

Step 3: Place a call

The answer XML

Inside the bridge

The agent settings

Barge-in

Knowing the caller actually heard it

Frame slicing

Format mismatches are silent

Security

Test without placing a call

Configuration reference

Troubleshooting

Next steps

​How it works

The bidirectional Stream protocol

​Audio profiles

​Requirements

​Step 1: Configure the bridge

​Dependencies

​Step 2: Run it

​Step 3: Place a call

​The answer XML

​Inside the bridge

​The agent settings

​Barge-in

​Knowing the caller actually heard it

​Frame slicing

​Format mismatches are silent

​Security

​Test without placing a call

​Configuration reference

​Troubleshooting

​Next steps

How it works

Audio profiles

Requirements

Step 1: Configure the bridge

Dependencies

Step 2: Run it

Step 3: Place a call

The answer XML

Inside the bridge

The agent settings

Barge-in

Knowing the caller actually heard it

Frame slicing

Format mismatches are silent

Security

Test without placing a call

Configuration reference

Troubleshooting

Next steps