The Telephony Layer for AI Voice Agents: Connecting Vapi, Retell, ElevenLabs, LiveKit, Pipecat & More to Phone Numbers
How AI voice agent platforms connect to real phone numbers — SIP trunks, WebSocket streaming, and BYOC, with setup paths for each major platform.

A voice agent that reasons instantly and sounds genuinely human is still an idea sitting inside an API response until something wires it to a phone number a customer can dial. That wiring is the telephony layer: the carrier infrastructure and signalling stack between a platform like Vapi, Retell AI or ElevenLabs and the phone network, handling routing, number provisioning and the audio transport that makes a call sound like a call.
There are two ways to get a number. Every major platform will supply one, and that is the fastest path to a first call. Every major platform also ships a setting — bring your own carrier, custom telephony, direct SIP — that connects a number from a carrier you choose. Neither is clever and neither is a trap. Which you want depends on where you are calling, what else you run, and who is going to ask you for paperwork.
This guide covers connecting a number either way: setup for Vapi, Retell AI, ElevenLabs, LiveKit and Pipecat, the longer list of platforms that follow the same pattern, what to ask a carrier before committing, and what is happening underneath.
What Is the Telephony Layer for AI Voice Agents?
An AI voice agent platform handles the intelligence: transcription, reasoning, and speech generation. It does not, on its own, own a phone number or a connection to the public phone network. That's the job of the telephony layer, sometimes called the carrier layer, and it's made up of a few concrete pieces, such as: a DID (direct inward dial number, the actual 10-digit or toll-free number a caller dials), a SIP trunk (the virtual line that carries the call as data instead of a circuit-switched signal), and the signaling and media servers that route audio between the public switched telephone network (PSTN) and whatever platform is running the AI agent.
Vapi, Retell AI, and ElevenLabs all ship with their own native numbers you can buy directly from them. The telephony layer becomes a separate decision the moment a business wants a number from a specific country, wants to keep an existing number, needs volume pricing a platform's native numbers don't offer, or wants one carrier serving multiple AI platforms and internal tools at once.
How AI Voice Agents Connect to Phone Numbers
The path a phone call takes from a ringing number to an AI-generated response follows the same handful of steps regardless of which platform is listening on the other end.
- A caller dials a DID. The number lives with a telephony carrier, not with the AI platform itself.
- The carrier's SIP trunk answers the call. It converts the incoming PSTN circuit into a SIP session, the protocol that carries call setup, teardown, and routing instructions over the internet.
- The SIP session hands off to the AI platform. For Vapi, Retell AI, ElevenLabs, and LiveKit, this happens directly over a SIP trunk: the carrier sends the call to a SIP address the platform exposes, and RTP (real-time transport protocol) packets carry the actual audio both ways. Pipecat, an open-source framework rather than a hosted platform, typically takes this a step further: the carrier requests instructions from a server the developer controls, and that server opens a WebSocket where raw audio streams bidirectionally to a bot process running locally or in the cloud.
- The platform's pipeline processes the audio. Speech-to-text (engines like Deepgram or OpenAI's Whisper) turns the caller's voice into text, a language model (GPT-4o, Claude, or a fine-tuned equivalent) generates a response, and a text-to-speech engine (ElevenLabs, Cartesia, or similar) turns that response back into audio.
- The generated audio streams back over the same SIP/RTP or WebSocket path the call came in on, and the caller hears a reply.
- Call metadata gets logged on the telephony side: recordings, transcripts, and quality metrics like jitter, packet loss, and mean opinion score, which matter more for diagnosing a bad call than anything happening inside the AI pipeline.
The telephony hop itself is usually the fastest link in that chain. Vobiz, for example, publishes sub-80ms latency on its voice network; the AI processing steps, speech-to-text, reasoning, and speech generation, typically add more time to a response than the network connection that carries it.
Ready to wire your agent? Connect your Vapi, Retell, or custom voice stack to global PSTN numbers with sub-80ms latency. Get started with Vobiz for free.
Connection Methods Compared
Every AI voice agent platform in this guide supports at least one of three connection methods, and most support two.
| Method | How Audio Moves | Setup Effort | Best For |
|---|---|---|---|
| SIP trunking | RTP over a SIP session | Dashboard credentials, no code | Production deployments needing carrier choice, specific country coverage, or volume pricing |
| WebSocket streaming | Raw bidirectional audio over a WSS connection | Requires a server the developer controls | Open-source frameworks (Pipecat) and custom audio pipelines |
| Native platform number | Handled entirely inside the platform | Fastest to start, no external carrier | Prototypes and early testing before a production carrier is chosen |
Two Ways to Get a Number
Every platform in this guide provisions a number for you. The question is which carrier sits behind the number you were going to use anyway.
| Platform’s own number | A carrier you connect | |
|---|---|---|
| Setup | Works at signup | 5–15 minutes: a trunk on the carrier side, credentials in the platform |
| Vendors | One account, one invoice | A second account. One carrier can serve several platforms, so this stays flat as you add more |
| Routing | The platform chooses the path | You choose the carrier, the countries and the authentication method |
| Coverage | Varies by platform, and not every market is offered | Wherever the carrier operates — Vobiz covers 130+ countries for numbers and 190 for outbound |
| Where the number lives | In the platform account, so a new platform often means a new number | In your carrier account, pointed at whichever platform you want |
| Compliance paperwork | Varies by platform and plan tier | Whatever the carrier can produce, on your own contract |
The platform’s number buys speed. Your own carrier buys control, and the price is a few minutes of configuration and one more vendor on the list.
A note on numbers and ownership. Nobody owns a phone number. They are allocated by regulators and assigned by carriers, and both the platform and the carrier are renting them to you. The practical difference is where the number lives: one taken from an agent platform is tied to that platform, while one provisioned through a carrier stays in your account and can be pointed at anything. Taking a number out of a platform account is not always offered, so it is worth asking before you build a brand around one.
Why the Telephony Layer Matters
A voice agent that only works in a browser demo does not answer a customer or book a job. The telephony layer is what turns it into a channel a business runs on:
- Outbound sales and collections dialling, where the same agent needs to call from a local number in whatever market it is reaching.
- Inbound support and appointment lines, where a business keeps a published number and routes it to an agent instead of a human queue.
- IVR replacement, swapping a menu tree for a conversation on a number customers already know.
- Multi-platform routing, where marketing runs campaigns from one number, support answers another and outbound sales uses a third, all from a single carrier account.
It is also where the failures you cannot debug from inside the agent show up. When a call sounds wrong, carrier-side telemetry — jitter, packet loss, mean opinion score, which side hung up — describes what happened to the audio before the agent ever heard it.
Connecting Vapi to a Phone Number
Vapi is a developer-focused platform with a dashboard workflow that covers most setup without code. Connecting a carrier’s SIP trunk is how a Vapi assistant gets a number outside Vapi’s own inventory.
Outbound is dashboard-only: add a SIP trunk credential using the carrier’s SIP domain, username and password, import the number as a bring-your-own-SIP-trunk number, assign it to an assistant and place the call.
Inbound needs one API call. Vapi wants an inbound trunk with a gateway for each of the carrier’s signalling IPs on UDP 5060, and the resulting trunk ID is not shown in the dashboard — it has to be read from Vapi’s API. A matching inbound trunk is then created carrier-side, the number linked to it, and an answering assistant assigned.
What this unlocks: one assistant answering on a toll-free number in one country and dialling out on a local mobile number in another, from a single carrier account rather than several platform number pools. Setup guide: Vobiz × Vapi.
Connecting Retell AI to a Phone Number
Retell AI is built around no-code agents, and its telephony setup follows suit: SIP configuration lives entirely in dashboards on both sides, with no webhooks and no custom answer URL.
The architecture uses two trunks pointing in opposite directions from a single number. Outbound authenticates from Retell into the carrier using credentials entered in the Retell dashboard: the number, the SIP domain as termination URI, and a trunk username and password. Inbound routes the other way, with the carrier console pointed at Retell’s SIP hostname over UDP.
One number can carry a different agent in each direction — an outbound sales agent and an inbound support agent — with no IP allowlist or webhook endpoint on either side. Setup guide: Vobiz × Retell AI.
Connecting ElevenLabs to a Phone Number
ElevenLabs agents connect through a dashboard flow with no code on either side.
Outbound: import the number from the SIP trunk, label it, set the trunk’s transport and SIP address, and place a test call. Inbound: set the SIP URI in the carrier’s trunk settings, import the same number, attach it to an agent and dial it.
Transport is TCP on this path rather than the UDP most SIP integrations default to, which is a common cause of a timeout on first setup. Once connected, ElevenLabs analytics, call logs, transcripts and recordings apply to SIP trunk calls exactly as they do to a number bought inside the platform, so changing carrier costs no visibility. Setup guide: Vobiz × ElevenLabs.
Connecting LiveKit to a Phone Number
LiveKit is open-source, self-hostable real-time infrastructure, which makes it carrier-agnostic by design.
Setup starts in the carrier console: create a trunk and generate credentials, producing a SIP domain, a username and a password. Those map into LiveKit’s trunk configuration alongside your LiveKit URL, API key and secret. Inbound additionally needs the number configured in LiveKit’s inbound trunk settings and a dispatch rule so the call reaches your agent.
The model is worth understanding because it differs from the others: an incoming call becomes a participant in a room your agent is already in, which means a human supervisor or a second agent can join the same call without transferring the caller anywhere. Outbound is a few SDK calls rather than a dashboard form. At the time of writing LiveKit also sells some numbers of its own, so bringing your own sits alongside that. The documented setup runs about 15 minutes. Setup guide: Vobiz × LiveKit.
Connecting Pipecat to a Phone Number
Pipecat is an open-source Python framework with no dashboard, so the whole telephony connection lives in code and environment variables.
An outbound call is a REST request to the carrier, which then asks a server you run what to do. That server returns an instruction opening a WebSocket, and audio streams both ways between the carrier and your Pipecat bot from that point on.
Vobiz publishes a Pipecat serializer as a package that installs into the framework’s own namespace, so the protocol handling — format negotiation, frame slicing, barge-in, clean hangup — is an import rather than a bridge you maintain. Setup needs a model provider key, your carrier credentials and a public URL, typically a tunnel during development. Python 3.11 is a hard floor: on 3.10, pip quietly installs a years-old release instead of failing, and every import then breaks for reasons that look unrelated. Setup guide: Vobiz × Pipecat.
More AI Voice Platforms You Can Connect
The same SIP-trunk-or-WebSocket pattern extends to a longer list of platforms Vobiz supports. Hosted platforms take SIP credentials in a dashboard; open-source and developer-first ones use a WebSocket or REST connection in code.
| Platform | Connection method | Best for |
|---|---|---|
| Bolna | WebSocket streaming | Multilingual agents and workflow automation, with strong Indic language support |
| Sarvam | Bundled telephony, or your own account | India-first agents, with number provisioning and KYC built into the platform |
| OpenAI Realtime | SIP, media terminated by OpenAI | Agents built directly on OpenAI’s speech-to-speech model |
| Ultravox | SIP trunk, credentials in the call request | Speech-native agents, outbound calling |
| Smallest AI | SIP trunk | Agents on Smallest AI’s own speech, language and voice models |
| Dograh | Built-in provider, WebSocket streaming | Teams self-hosting an open-source agent stack |
| Rapida AI | Built-in provider, WebSocket streaming | White-label delivery with isolated client workspaces |
| xAI Grok | Direct SIP | Agents on Grok’s voice model, including markets xAI does not supply numbers in |
| Agora | SIP trunk | Agents on Agora’s real-time network, extended to the phone network |
Each of these follows the same logic: a carrier hands off audio either as a SIP session or as a raw WebSocket stream, and the platform decides which one it expects.
Connecting Existing Phone Systems and Browser-Based Calling
Not every deployment starts from a hosted AI platform. Some businesses need a voice agent to plug into infrastructure they already run, or need a way to test a trunk before wiring it into a live agent.
3CX, a business PBX system, connects to a SIP trunk the same way an AI platform does, which makes it possible to route a subset of extensions on an existing office phone system to an AI agent without replacing the PBX itself. Zoiper, a SIP softphone, serves a narrower but useful purpose: testing trunk credentials and call routing manually before pointing a live number at an AI agent, which catches configuration mistakes before a real caller does.
For products that don't need a traditional phone number at all, WebRTC lets a browser or an app place and receive calls directly, embedding a softphone into a website or product with a phone number attached instantly rather than provisioned through a separate telephony console. WebSockets audio streaming is the same raw, bidirectional audio transport Pipecat uses, available on its own for any custom voice pipeline that doesn't fit a platform's native SIP or WebRTC support.
When the Platform’s Number Is Enough — and When It Stops Being
Stay on the platform’s own number while you are prototyping, while one platform and one country cover the use case, while volume is low enough that per-minute differences never reach the invoice, and while nobody has asked for compliance documents in writing.
Connect your own carrier when you hit one of these:
- A country or number type the platform does not offer. The most common wall and the hardest to work around, because no amount of configuration creates inventory that is not there. It is why xAI ships a Direct SIP option at all — its own number supply does not reach every market, India among them.
- A number you cannot change. Printed, advertised, registered, or simply known to your customers.
- Per-minute cost that varies by destination and starts to matter at volume.
- A compliance document a customer wants before they sign.
- A second platform joining the stack, where one carrier behind both beats two platform number pools.
Regional Rules Land on the Carrier Layer
Whichever route you take, the rules of the market you call into apply to the carrier rather than the agent.
Calls into India require TRAI and DLT entity and number registration, with KYC completed before a number goes live. In the US, caller-ID authentication under STIR/SHAKEN applies, and consent rules under the TCPA apply to outbound calling — the FCC ruled in February 2024 that AI-generated voices count as artificial voices for TCPA purposes, which places AI outbound calling squarely inside those rules. Other markets have their own registration regimes and number formats.
Ask any carrier how it handles each market you actually call into, rather than taking a headline country count at face value.
How to Choose a Telephony Layer for Your AI Voice Agent
Every platform above accepts a SIP trunk or WebSocket connection from any carrier following standard protocols, so carriers separate on a handful of points.
Latency, measured. Ask for a published figure and ask what it covers — one-way media, round trip, or the whole call including the model. Vobiz publishes sub-80ms one-way on the carrier leg. A conversational turn has roughly 500ms before a pause feels unnatural, and speech-to-text, the model, text-to-speech and the network all spend from it. Every extra relay in the media path adds a queue and a jitter buffer, and no tuning removes them afterwards. Then test it on your own traffic.
Compliance, in writing. Ask for the current SOC 2 Type II report and, for healthcare, a signed BAA — the documents themselves, not a badge on a marketing page. That applies to us as much as anyone: ask us for ours.
Coverage and number types. Check the specific countries and number types you call — local, mobile, toll-free, and any regulated series a market requires — rather than the headline country count. Vobiz covers 130+ countries for numbers and 190 for outbound.
Authentication fit. Confirm the carrier supports both SIP digest credentials and static IP allowlisting, because platforms differ in what they expect. Cloud-hosted platforms with changing egress addresses need credentials; a fixed SBC can use an allowlist.
Connecting versus porting. Pointing a carrier’s trunk at a platform is usually quick. Moving an existing number between carriers is a formal port and can take days. Work out which you need before you set a date.
Diagnostics. When a call goes badly, carrier-side telemetry is the fastest way to find out why. Ask what you get per call: mean opinion score, jitter, packet loss, negotiated codec, and which side hung up.
Security baseline. TLS for signalling and SRTP for media should be standard, not an upsell.
Developer tooling. SDKs in the languages a team already uses shorten the distance between a connected trunk and a shipped call flow.
Conclusion
Giving an agent a phone number is a smaller job than it looks: minutes on a dashboard for most platforms, a little configuration for the open-source ones. The decision that takes longer is where the number should live — inside the platform, which is simplest while you have one platform and one market, or in a carrier account of your own, which starts paying back the moment a second country, a second platform or a compliance request appears.
Browse the Vobiz integrations library for the platform you are using, or create an account and connect a number to an agent you have already built.
FAQs
How do I give an AI voice agent a phone number?
Either take a number from the agent platform, which works at signup, or connect one from a carrier you choose using the platform’s bring-your-own-carrier setting. The second route takes five to fifteen minutes for hosted platforms: create a SIP trunk with the carrier, then enter the SIP domain, username and password in the platform.
What is the telephony layer for AI voice agents?
The carrier infrastructure that connects an agent to the public phone network. It supplies the phone numbers, the SIP trunks that carry each call, and the routing between the phone network and the agent platform.
What is the difference between a SIP trunk and a phone number?
A phone number, or DID, is the address a caller dials. A SIP trunk is the connection that carries the call as data between the carrier holding the number and the platform running the agent. You need both.
What is BYOC, and how is it different from number porting?
BYOC means connecting an outside carrier’s SIP trunk to an agent platform while the number stays with that carrier. Porting formally moves a number from one carrier to another, which takes longer and is not offered everywhere.
Can I use my own number or carrier with Vapi, Retell AI or ElevenLabs?
Yes. Each has a bring-your-own-carrier or custom telephony setting that accepts a SIP trunk from an outside carrier, and the number stays with that carrier.
How long does it take to connect a carrier?
Usually under ten minutes for hosted platforms once you have the trunk credentials. Vapi inbound adds one API call. LiveKit and Pipecat are configured in code rather than a dashboard, and the documented Vobiz setups run about fifteen minutes for LiveKit and a few minutes for Pipecat.
Does using my own carrier add operational overhead?
A second account and dashboard, yes. If you run more than one agent platform the overhead stays roughly flat — credentials are entered once per platform, but billing, support and compliance records stay in one place.
Can I use one phone number across multiple AI voice platforms?
Yes, if the number comes from your own carrier: point it at whichever platform you like and move it between them. An incoming call reaches one destination at a time, so use separate numbers where two platforms need to answer in parallel.
Can I use the same number for both inbound and outbound AI voice calls?
Yes. Retell AI and Vapi both support linking one number to separate inbound and outbound trunks, so a single number can run a different agent in each direction.
How much latency does the telephony layer add?
Carrier-grade telephony is usually the smallest contributor in the chain — well below speech-to-text, model reasoning and speech generation combined. Vobiz publishes sub-80ms one-way on the carrier leg. Ask any carrier what their figure measures before comparing.
Is SIP trunking secure enough for regulated industries?
It can be, but the security sits with the carrier, because SIP is only a transport protocol. Look for TLS signalling, SRTP media encryption, and certification reports you can actually read — SOC 2 Type II, ISO 27001 — rather than badges.
More Related Articles
When Does Voice AI Need a Phone Number? A Web SDK and WebRTC Primer
Whether or not your voice AI product needs a phone number depends on one thing: does the interaction have to outlive a single browser session. Here's how WebRTC, SIP, and Web SDKs fit together.
Read full article
Ishani SinghThe Telephony Layer Playbook for Voice AI Teams
Most voice AI failures don't start in the model. They start in the telephony layer. Download our 8-page playbook for voice AI teams.
Read full articleSub-80ms RTT Voice Infrastructure: How Vobiz Delivers Production-Ready Low Latency for AI Agents (2026)
Most voice AI stacks settle for 1500-2500ms. Vobiz targets sub-80ms average RTT on the carrier leg. Here's the exact infrastructure stack that makes it possible - and why it matters at scale.
Read full article
Piyush Sahoo