Pipecat integration

This integration combines Vobiz telephony with the Pipecat voice agent framework to build intelligent AI-powered phone calls.

Overview

What you’ll build: An outbound calling system with real-time AI conversations powered by OpenAI (STT → LLM → TTS), automatic call recording, and bidirectional audio streaming. Three moving parts do all the work:

Part	What it is	Responsibility
`server.py`	Your FastAPI app	Places the call, serves the answer XML, accepts the WebSocket
`bot.py`	The Pipecat pipeline	Runs STT → LLM → TTS for the duration of one call
`pipecat-vobiz`	`VobizFrameSerializer`	Translates between Vobiz’s JSON/base64 WebSocket protocol and Pipecat audio frames

Call flow

ActorsYouserver.pyVobizbot.pyCustomer

Phase 1 · Placing the call

You→server.py

Trigger the call.

server.py→Vobiz

Server calls the Vobiz Call API, filling answer_url from PUBLIC_URL.

Vobiz→server.py

Call accepted.

Phase 2 · Answer & instructions

Vobiz→Customer

Vobiz dials out over PSTN — the phone rings, then the customer answers.

Vobiz→server.py

On answer, Vobiz asks your server what to do.

server.py→Vobiz

Returns VobizXML: record the session, and stream the caller’s audio to your bot.

Phase 3 · The live conversation

Vobiz→bot.py

WebSocket upgrade on /ws, then a single start event carrying the IDs and the negotiated audio format.

parse_vobiz_start() reads this before the transport is built. mediaFormat is authoritative — prefer it over your own contentType.

Vobiz↔bot.pybidirectional · ~20 ms frames

Audio flows both ways as base64 media events.

VobizFrameSerializer→Silero VAD→OpenAI STT→GPT→OpenAI TTS→back to Vobiz

Phase 4 · Teardown & recording

Customerhangs up →Vobiz→bot.py

With auto_hang_up=True the serializer also issues a REST hangup, so a bot-initiated end tears the call down cleanly.

Vobiz→server.py

Recording callback fires; the helper downloads the file.

Steps 1–3 are only for outbound calls. For inbound calls the flow starts at step 6 — Vobiz calls your /answer URL directly. See Receiving inbound calls.

Features

AI Voice Conversations

Natural conversations powered by OpenAI GPT + TTS/STT

Outbound Calling

Trigger calls via REST API from anywhere

Automatic Recording

All conversations automatically recorded and saved

Real-time Streaming

Bidirectional audio via WebSockets

Prerequisites

Vobiz Account with Auth ID and Auth Token → Sign up

OpenAI API Key for LLM, STT, and TTS → Get API key

Python 3.11+ installed on your system

ngrok for local development → Download ngrok

Python 3.11 is a hard floor. pipecat-ai 1.x declares requires-python >=3.11. On Python 3.10, pip does not fail — it silently backtracks and installs an ancient 0.0.x Pipecat, and then every import in bot.py breaks with confusing ModuleNotFoundErrors. Check with python --version before installing.

Version compatibility

This integration targets the Pipecat 1.x API. Versions are pinned in requirements.txt:

Package	Pin	Why
`pipecat-ai[websocket,openai,silero]`	`>=1.8.1,<1.9`	Pipecat 1.x moved VAD onto `LLMUserAggregatorParams` and introduced `LLMContext`. The 1.8 line is the verified one.
`pipecat-vobiz`	`>=0.0.3,<0.1`	0.0.3 adds L16, multi-rate, `start`-event negotiation and `stop`-event hangup.
Python	`>=3.11`	Required by `pipecat-ai` 1.x.

Pipecat 1.9 and 1.10 are published but are not covered by this pin. Widen the upper bound only after re-testing — the serializer contract has changed once already inside 1.x (FrameSerializer.setup() took a StartFrame before 1.3 and a FrameProcessorSetup after).

Pipecat 1.x deprecated pipecat.pipeline.task in favour of pipecat.pipeline.worker (PipelineTask → PipelineWorker). The old import path still works for all of 1.x and is what this repo uses, but it is scheduled for removal in Pipecat 2.0 — which is why the pin is <2.

Installation

Clone the repository

git clone https://github.com/vobiz-ai/Vobiz-X-Pipecat
cd Vobiz-X-Pipecat

Install dependencies

pip install -r requirements.txt

This installs FastAPI, Pipecat, OpenAI SDK, and other required packages.

Configure environment

The repo ships an env.example file. Copy it and fill in your values:

cp env.example .env

.env

# --- Required ---
OPENAI_API_KEY=sk-...
VOBIZ_AUTH_ID=MA_XXXXXXXX
VOBIZ_AUTH_TOKEN=your-token-here
PUBLIC_URL=https://your-ngrok-url.ngrok-free.app

# --- Caller ID for outbound ---
VOBIZ_PHONE_NUMBER=+918065480214

# --- Wire format (must match the <Stream contentType> server.py emits) ---
VOBIZ_ENCODING=audio/x-mulaw
VOBIZ_SAMPLE_RATE=8000
VOBIZ_L16_ENDIAN=le

Required

Variable	Where to find it
`OPENAI_API_KEY`	OpenAI Platform → API Keys
`VOBIZ_AUTH_ID`	Vobiz Console → Account Settings
`VOBIZ_AUTH_TOKEN`	Vobiz Console → Account Settings
`PUBLIC_URL`	Your ngrok HTTPS URL (set in Step 2 of Usage)

Optional

Variable	Default	Purpose
`VOBIZ_PHONE_NUMBER`	—	Caller ID for `/start`. Required unless you pass `from_number` in the request body.
`VOBIZ_ENCODING`	`audio/x-mulaw`	Wire encoding. `audio/x-mulaw` or `audio/x-l16`.
`VOBIZ_SAMPLE_RATE`	`8000`	Wire sample rate. `8000` or `16000`.
`VOBIZ_L16_ENDIAN`	`le`	L16 byte order. Ignored for μ-law. See the note below.
`ENABLE_RECORDING`	`true`	Set `false` to omit the `<Record>` element from the answer XML.
`MAX_RECORDING_LENGTH`	`3600`	Recording cap in seconds.
`ENV`	`local`	`local` or `production`.
`VOBIZ_PROD_WS_URL`	—	Required when `ENV=production` — the public `wss://` URL hosting your bot.
`TRANSFER_AGENT_NUMBER`	—	Required only if you use the human-transfer endpoints.
`AGENT_NAME` / `ORGANIZATION_NAME`	—	Used to build the `serviceHost` query param when `ENV=production`.
`DEEPGRAM_API_KEY`	—	Not read by `bot.py` as shipped. Set it only if you swap `OpenAISTTService` for `DeepgramSTTService`.

VOBIZ_L16_ENDIAN only matters for audio/x-l16. RFC 2586 specifies big-endian (network byte order), but some Vobiz accounts transport L16 as little-endian. If L16 media frames arrive at the expected size yet STT returns no transcripts, flip this to le. Leave the default μ-law encoding and you never hit this.

Usage

Start the server

python server.py

The server runs on http://0.0.0.0:7860.

Start ngrok

In a new terminal, expose your local server:

ngrok http 7860

Copy the ngrok URL from the output (for example, https://abc123.ngrok-free.app).

Important: Update PUBLIC_URL in your .env file with this ngrok URL, then restart the server.

Make a call

There are two ways to trigger an outbound call - pick whichever fits your stack.

Server's /start helper (easiest)
Direct Vobiz API

The repo exposes POST /start on the local server. It wraps the Vobiz Call API and auto-fills answer_url from your PUBLIC_URL, plus uses VOBIZ_PHONE_NUMBER as from when set.

curl -X POST http://localhost:7860/start \
  -H "Content-Type: application/json" \
  -d '{ "phone_number": "+919148223344" }'

The field is phone_number, not to. server.py returns 400 Missing 'phone_number' in the request body for anything else.

Field	Required	Purpose
`phone_number`	Yes	The number to dial.
`from_number`	No	Caller ID. Falls back to `VOBIZ_PHONE_NUMBER`; if neither is set the call is rejected with `400`.
`body`	No	Arbitrary JSON. Forwarded to `/answer` as a URL-encoded query param, then base64-encoded onto the WebSocket URL so `bot.py` can read per-call context.

Response:

{ "call_uuid": "…", "status": "call_initiated", "phone_number": "+919148223344" }

curl -X POST https://api.vobiz.ai/api/v1/Account/YOUR_AUTH_ID/Call/ \
  -H "X-Auth-ID: YOUR_AUTH_ID" \
  -H "X-Auth-Token: YOUR_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "from": "+918011223344",
    "to": "+919148223344",
    "answer_url": "https://your-ngrok-url.ngrok-free.app/answer",
    "answer_method": "POST"
  }'

What happens next:

Phone rings at the phone_number you passed
When answered, Vobiz requests XML from your server’s /answer endpoint
Server returns <Record> + <Stream> pointing at wss://…/ws
Vobiz opens the WebSocket and sends a start event, then media frames
AI assistant speaks and listens (STT → LLM → TTS)
On hangup, Vobiz posts the recording URL to /recording-ready, which downloads it to recordings/

How the flow works

Everything above is driven by one XML document and one WebSocket protocol. This section is what you need if you are adapting the repo rather than running it as-is.

The answer XML

/answer returns this (with {PUBLIC_URL} and the wire format substituted in):

<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Record fileFormat="wav" maxLength="3600" recordSession="true"
            callbackUrl="https://your-ngrok-url.ngrok-free.app/recording-ready"
            callbackMethod="POST">
    </Record>
    <Stream bidirectional="true" audioTrack="inbound"
            contentType="audio/x-mulaw;rate=8000" keepCallAlive="true">
        wss://your-ngrok-url.ngrok-free.app/ws
    </Stream>
</Response>

Why each attribute matters:

Attribute	Value	Why
`bidirectional`	`true`	Lets the bot play TTS audio back to the caller. Without it the stream is listen-only.
`audioTrack`	`inbound`	The caller’s audio. Must be `inbound` when `bidirectional="true"`.
`contentType`	`audio/x-mulaw;rate=8000`	Must match what `VobizFrameSerializer` is constructed with.
`keepCallAlive`	`true`	Keeps the call up when the stream ends, so `<Record>` can finish and callbacks fire.

audioTrack="both" is incompatible with bidirectional="true". The media server accepts the XML, sends a start event, and then never sends audio — a silent failure that looks like a broken bot. Use audioTrack="inbound" for any bidirectional stream. To capture both legs, use two separate one-way <Stream> elements.

The WebSocket protocol

Vobiz speaks JSON text frames. VobizFrameSerializer handles all of this for you; the shapes are here so you can debug what you see on the wire.

start — sent once, immediately after the upgrade

{
  "event": "start",
  "start": {
    "callId": "…",
    "streamId": "…",
    "mediaFormat": { "encoding": "audio/x-mulaw", "sampleRate": 8000 }
  }
}

bot.py reads this with parse_vobiz_start() before building the transport, because it carries both the IDs needed for REST hangup and the negotiated audio format.

mediaFormat is authoritative. It reflects what the media server actually negotiated. If it differs from the contentType you asked for in your <Stream> XML, trust this event — the XML is a request, the start event is the agreement. The serializer adopts these values automatically and logs a warning on mismatch.

media — the audio frames, both directions

{ "event": "media", "media": { "payload": "<base64-encoded audio>" } }

At 8 kHz μ-law this arrives roughly every 20 ms as 160-byte payloads. The serializer base64-decodes, converts μ-law → linear PCM, and resamples to the pipeline rate; outbound TTS makes the same trip in reverse.

stop — the call is over

{ "event": "stop" }

With auto_hang_up=True the serializer also issues a REST hangup using call_id + your auth credentials, so a bot-initiated end tears the call down properly rather than leaving it hanging.

Sample rates

Rate	μ-law	L16	Notes
8000	Supported	Supported	Default. Matches the PSTN carrier path — no transcode.
16000	Supported	Supported	Vobiz transcodes.
24000	Not reliable	Not reliable	Appears in the protocol, but the media server may accept the XML and then silently never open the WebSocket. Avoid.

The Pipecat pipeline

bot.py builds a standard cascaded pipeline. Two details are specific to Pipecat 1.x and telephony:

bot.py

context_aggregator = LLMContextAggregatorPair(
    context,
    # Pipecat 1.x: VAD is wired HERE, not on the transport.
    user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
)

transport = FastAPIWebsocketTransport(
    websocket=runner_args.websocket,
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        add_wav_header=False,   # CRITICAL for telephony
        serializer=serializer,
    ),
)

Two mistakes that produce a silent bot:

Passing vad_analyzer= to FastAPIWebsocketParams. In Pipecat 1.x the transport no longer has that field, and Pydantic silently discards it — you get no error and no turn detection. It belongs on LLMUserAggregatorParams.
Leaving add_wav_header at its default. A WAV header prepended to every frame is interpreted as audio by the media server, and the caller hears noise.

Pipeline order matters — transport.output() sits before context_aggregator.assistant() so what the bot actually said is what gets recorded into context:

pipeline = Pipeline([
    transport.input(),          # Vobiz → frames
    stt,                        # Speech-to-text
    context_aggregator.user(),
    llm,                        # LLM
    tts,                        # Text-to-speech
    transport.output(),         # frames → Vobiz
    context_aggregator.assistant(),
])

Receiving inbound calls

Configure your Vobiz number to handle incoming calls with your Pipecat agent.

Open Applications

Create an application

Click Create New Application and give it a name (for example, “Pipecat Agent”).

Configure URLs

Set the Answer URL to your ngrok URL (for example, https://.../answer) and select POST method. You can use the same URL for Hangup or leave it blank.

Assign phone number

Go to Phone Numbers, select your number, and assign it to the application you just created.

Success: Calls to your Vobiz number will now be handled by your local Pipecat server!

Quick reference

Server endpoints

Endpoint	Method	Description
`/start`	POST	Trigger an outbound call. Body: `{ "phone_number": "+91..." }`. Wraps the Vobiz Call API.
`/answer`	GET, POST	Called by Vobiz when the call is answered. Returns `<Record>` + `<Stream>` XML pointing at `/ws`.
`/ws`	WebSocket	Bidirectional audio between Vobiz and the Pipecat bot. Also served at `/`, `/voice/ws` and `/stream` for flexibility in what you put in the XML.
`/recording-finished`	GET, POST	Vobiz callback when recording stops. Logs metadata.
`/recording-ready`	GET, POST	Vobiz callback when the recording is ready. Downloads the file via `download_recording.py`.
`/active-calls`	GET	Debug view of the in-memory call registry.
`/initiate-transfer`	POST	Flags a call for human transfer, then calls the Vobiz Transfer API.
`/transfer-to-human`	POST	Returns the `<Speak>` + `<Dial>` XML used by the transfer flow.

The call registry is an in-memory dict. It resets on restart and does not work across multiple workers — use Redis or a database before running more than one process in production.

Project files

File	Purpose
`server.py`	FastAPI app - owns `/start`, `/answer`, `/ws`, and the recording callbacks
`bot.py`	Pipecat pipeline - STT → LLM → TTS, runs per call
`download_recording.py`	Helper used by `/recording-ready` to pull the recording from Vobiz
`requirements.txt`	Python dependencies, with the Pipecat version pins
`env.example`	Template for your `.env` file - copy and fill in

VobizFrameSerializer and parse_vobiz_start are not in this repo — they ship in the separate pipecat-vobiz package, which installs into the pipecat.serializers namespace. That is why bot.py imports them from pipecat.serializers.vobiz rather than from a local file.

Customizing the bot

Edit bot.py to customize your AI assistant:

Change Bot Personality

messages = [
    {
        "role": "system",
        "content": "You are a friendly customer service agent..."
    },
]

Change TTS Voice

tts = OpenAITTSService(
    api_key=os.getenv("OPENAI_API_KEY"),
    voice="ballad",  # alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer
)

The bot as shipped waits for the caller to speak first — on_client_connected only logs. To have it greet the caller, queue an LLMRunFrame in that handler so the LLM produces its opening line from the system prompt.

Troubleshooting

Import errors right after pip install

Almost always Python 3.10. pipecat-ai 1.x requires 3.11+, and on 3.10 pip installs a 0.0.x release instead of failing. Run python --version, then pip show pipecat-ai — if the version starts with 0.0., recreate your virtualenv on 3.11+.

The bot never replies — it hears nothing

Check VAD wiring. In Pipecat 1.x, vad_analyzer belongs on LLMUserAggregatorParams, not on FastAPIWebsocketParams. The transport no longer has that field, and Pydantic discards unknown fields silently, so the mistake produces no error at all — just a bot that never detects the end of a turn.

The caller hears static or noise instead of speech

Set add_wav_header=False on FastAPIWebsocketParams. A WAV header on every frame is played as audio by the media server.

start event arrives, then no media frames

You almost certainly have audioTrack="both" together with bidirectional="true". That combination is not supported and fails silently. Use audioTrack="inbound".

Media frames arrive but STT produces nothing (L16 only)

Byte order. Set VOBIZ_L16_ENDIAN=le. The docs and RFC 2586 specify big-endian, but some accounts transport L16 little-endian. Using the default audio/x-mulaw avoids the problem entirely.

The WebSocket never opens at all

Two common causes: the sample rate is 24000 (not reliably supported — use 8000 or 16000), or PUBLIC_URL is stale. After restarting ngrok the URL changes; update PUBLIC_URL and restart the server, since the answer XML is built from it.

Vobiz cannot reach /answer

If PUBLIC_URL is unset the server falls back to the request Host header, which is localhost in local runs — Vobiz cannot route to that. The server prints a warning when this happens. Set PUBLIC_URL to your public HTTPS URL.

The downloaded recording is named .mp3 but will not play as one

Known quirk. The answer XML requests fileFormat="wav", so Vobiz serves a .wav URL — but download_recording.py unconditionally appends .mp3 to the saved filename. The bytes are a RIFF/WAVE container (8 kHz, 16-bit, stereo); only the extension is wrong. Rename it to .wav, or change the fileFormat in the <Record> element to match the extension you want.

Integration complete!You can now make AI-powered phone calls with Vobiz and Pipecat.

Next steps

Customize your AI assistant’s personality in bot.py
Deploy to production (AWS/GCP/Heroku) instead of ngrok
Add custom business logic and integrations

Resources

Vobiz Documentation

External Resources

Build it with an AI agent

Clone, configure, and run the Vobiz-X-Pipecat repo - your first AI voice agent in ~5 minutes.

Open in Cursor

AI Voice Platforms

Official SDKs

Browser & WebRTC

Business Phone Systems

CRM & Helpdesk

Migrations

Pipecat integration

Overview

Call flow

Features

AI Voice Conversations

Outbound Calling

Automatic Recording

Real-time Streaming

Prerequisites

Version compatibility

Installation

Usage

How the flow works

The answer XML

The WebSocket protocol

Sample rates

The Pipecat pipeline

Receiving inbound calls

Quick reference

Server endpoints

Project files

Customizing the bot

Troubleshooting

Next steps

Resources

Build it with an AI agent

​Overview

​Call flow

​Features

AI Voice Conversations

Outbound Calling

Automatic Recording

Real-time Streaming

​Prerequisites

​Version compatibility

​Installation

​Usage

​How the flow works

​The answer XML

​The WebSocket protocol

​Sample rates

​The Pipecat pipeline

​Receiving inbound calls

​Quick reference

​Server endpoints

​Project files

​Customizing the bot

​Troubleshooting

​Next steps

​Resources

​Build it with an AI agent

Overview

Call flow

Features

Prerequisites

Version compatibility

Installation

Usage

How the flow works

The answer XML

The WebSocket protocol

Sample rates

The Pipecat pipeline

Receiving inbound calls

Quick reference

Server endpoints

Project files

Customizing the bot

Troubleshooting

Next steps

Resources

Build it with an AI agent