How to Build a Voice AI Agent: Architecture, Latency and Cost

Building a voice AI agent looks easy in a demo: connect a microphone to a model and let it talk. Learning how to build a voice AI agent that holds up on a real phone line is a different job, because the problems are not in the model. They are in the milliseconds between turns, in what happens when the caller interrupts, in a bill that grows with every minute of silence, and in the legal duty to say "I am an AI" before anything else.

This guide is written for engineers who already know how to call an LLM and now have to ship something people will speak to. It covers the three architectures you can choose from today, the latency budget, turn detection and barge-in, tools and telephony, what drives cost, how to evaluate a voice agent, and the consent and disclosure rules, including the EU AI Act transparency duty that has applied since 2 August 2026.

What a voice agent has to get right

Users judge a voice agent on a few things they feel rather than measure. Does it answer quickly enough to seem alive? Can I interrupt it and be heard? Does it understand what I said, including the product name and the account number? Does it stop talking when I start? Behind each of those feelings sits an engineering decision, and most of them are made before you write a single prompt.

The reference point is human conversation. A cross-cultural study of turn-taking in ten languages, Stivers et al. in PNAS (2009), measured a mean gap of about 208 ms between the end of a question and the start of the answer, with every language showing the same pattern: people avoid talking over each other and keep silences short. No voice agent matches that today once speech recognition, reasoning, synthesis and the network are added up, but it explains why a two-second pause feels broken even when the answer is correct.

So the job has five parts: choose an architecture, budget latency stage by stage, decide when the user has finished and what to do when they interrupt, keep cost proportional to value, and build the disclosure and consent steps into the call flow. The full engineering path, from streaming speech recognition to production phone agents, is the subject of Voice AI and Realtime Multimodal Agents; this article is the map.

Three architectures for a voice AI agent

OpenAI's voice agents guide describes three ways to build one, and the same shapes appear in other vendors' stacks.

1. The cascaded pipeline: STT, LLM, TTS

The classic design chains three components. Speech-to-text (STT) turns the caller's audio into a transcript, a text LLM decides what to say and which tools to call, and text-to-speech (TTS) turns the reply into audio. OpenAI's guide puts the reason to choose it plainly: use it when "each stage needs to be visible or replaceable."

That is its strength. You can swap the STT provider for one that handles your accents better, use any text model you already trust, pick a TTS voice independently, run guardrails on the exact transcript, and log every word for audit. Tool calling works exactly as it does in a text agent, so everything you know about building AI agents for production transfers.

The cost is latency and lost information. Every stage adds delay, and the moment speech becomes text you throw away tone, hesitation and emphasis. The model never hears that the caller sounded annoyed.

2. Speech-to-speech in one realtime session

A speech-native model takes audio in and produces audio out in one persistent session, with no text bottleneck in the middle. OpenAI's Realtime API (its models page lists GPT-Realtime-2.1 and GPT-Realtime-2.1 Mini among the current realtime models) and Google's Gemini Live API are two widely documented examples. OpenAI describes this mode as "speech, reasoning, and tool use in one session."

You gain lower latency, natural prosody and built-in turn detection and interruption handling. You give up component choice and some transparency: you take the provider's voices and behavior, and you depend on the transcript the provider gives you for logging. Session limits also shape the design. Google's documentation states that, without context compression, Gemini Live audio-only sessions are limited to 15 minutes and that a single connection lasts "around 10 minutes," so a long call needs session resumption built in from day one.

3. Delegated: a live voice layer over your existing backend

The third pattern separates talking from thinking. A live voice model handles the conversation itself, the turn-taking and the interruptions, and delegates reasoning and tool use to a backend you control. OpenAI's GPT-Live follows this pattern, described in its guide as "full-duplex conversations with a separate backend," useful when you want to "keep your existing text workflow."

A common hybrid in practice: run a speech-to-speech model for the conversational surface, transcribe in parallel for logs and analytics, and route anything with consequences (payments, account changes, identity checks) through deterministic backend code rather than trusting the voice model's judgment.

How to choose

Treat the choice as reversible. Open-source frameworks such as LiveKit Agents (Apache 2.0) and Pipecat (BSD 2-Clause) let you assemble either pipeline behind the same transport and switch later. Prototype both on your real call recordings and measure before committing.

The latency budget

The number that matters is the time from the moment the caller stops speaking to the moment they hear the first sound of the reply. Everything else is a component of it. In a cascade that time is spent in six places:

1. Endpointing. Deciding the caller has finished. This is frequently the largest single contributor, because a cautious silence timeout makes the agent wait before it even starts thinking.
2. Final transcript. The STT stage turning interim hypotheses into a stable result.
3. Time to first token from the LLM, which grows with prompt length and with any reasoning the model does first.
4. Tool calls, if the turn needs data.
5. Time to first audio from TTS.
6. Transport in both directions: browser or phone network, plus the distance between your servers and the model provider.

Write the budget down per stage, measure each one separately, and track the 95th percentile, not the average. A demo hides the slow tail and real traffic exposes it, and the tail usually has a specific cause: a cold connection, a slow tool, a long prompt on turn twelve.

The levers that consistently work:

• Stream everything. Send audio to STT as it arrives, start the LLM on the final transcript immediately, and feed the LLM's output to TTS sentence by sentence instead of waiting for the full reply.
• Keep replies short. A voice reply should be one or two sentences. Long answers increase time to first audio, cost and the chance of being interrupted.
• Fill tool waits honestly. If a lookup takes time, say so ("Let me check that booking") instead of leaving silence, and put a timeout on every tool.
• Co-locate. Run your orchestration in the same region as the model and telephony endpoints, and keep connections warm.
• Shrink the prompt. Every token of system prompt and history is paid for in time to first token on every turn.

Turn detection and barge-in

Two decisions define how natural an agent feels: when to start talking and when to stop.

Deciding the caller has finished

The naive approach is a silence timeout: if nobody has spoken for N milliseconds, the turn is over. It fails in both directions. Short, and the agent cuts off someone who pauses mid-sentence ("I'd like to book a table for... four"). Long, and every exchange drags.

Modern stacks use two layers:

• Voice activity detection (VAD) answers, frame by frame, "is this speech?" It is cheap and runs locally. Silero VAD, an open model under the MIT license, states that one audio chunk of 30 ms or more "takes less than 1ms to be processed on a single CPU thread" and supports 8 kHz and 16 kHz audio.
• A turn detection model answers "is the person done?", using how something was said, not just whether sound is present. LiveKit's turn detector is an audio model that captures "both what is said and how it's said"; its documentation shows the default wait after speech narrowing from 0.5–3.0 seconds without it to 0.3–2.5 seconds with it. Pipecat's open-source Smart Turn model (v3.2, BSD 2-Clause) works on 16 kHz mono audio, analyzes up to 8 seconds of the current turn, and is designed to run alongside a lightweight VAD such as Silero.

Speech-to-speech APIs build this in. OpenAI's Realtime API offers server_vad, which splits turns on silence with tunable threshold, prefix_padding_ms and silence_duration_ms, and semantic_vad, a classifier that scores whether the user has finished so the model "is less likely to interrupt the user," with an eagerness setting from low to high (the default auto equals medium). Start with semantic turn detection, then tune eagerness on real calls: support lines where callers read out reference numbers usually need a more patient setting than a quick FAQ bot.

Handling barge-in

Barge-in is the caller speaking while the agent is talking. Getting it right takes four steps, and skipping the fourth is the most common bug:

1. Detect speech start while the agent is speaking.
2. Stop playback immediately and clear any audio already queued.
3. Cancel the in-flight response so the model stops generating.
4. Truncate the conversation history to what the caller actually heard. If the agent had generated three sentences but the caller interrupted after one, the model must not believe it said all three.

OpenAI's Realtime documentation spells out the WebSocket version: on input_audio_buffer.speech_started, stop playback, then send conversation.item.truncate with the played duration in audio_end_ms, which "removes the unplayed portion from the conversation." Over WebRTC and SIP, the server tracks the output buffer and truncates automatically. On a Twilio phone stream, the equivalent of clearing the queue is the clear message, which "empties all buffered audio."

A minimal handler for a WebSocket session looks like this:

async def on_speech_started(session, playback):
    # 1-2. Caller started talking: stop audio now and drop the queue
    played_ms = playback.stop_and_clear()
    if session.current_item_id is None:
        return
    # 3. Stop the model from generating further audio
    await session.send({"type": "response.cancel"})
    # 4. Keep only what the caller actually heard in the history
    await session.send({
        "type": "conversation.item.truncate",
        "item_id": session.current_item_id,
        "content_index": 0,
        "audio_end_ms": played_ms,
    })

Then deal with false barge-ins. Backchannels ("mm-hm", "right") and the agent's own voice leaking back through a speakerphone can both trigger an interruption. Echo cancellation on the client and a turn detector that ignores short acknowledgments solve most of it; test with real devices, not studio headsets.

Tools in a voice conversation

Tool calling in voice follows the same rules as in text agents, with three additions.

• Latency is audible. A two-second database call is invisible in a chat window and painful on the phone. Announce the wait, set a timeout, and have a spoken fallback when the tool fails.
• Confirm before acting. Read back the structured values ("That is Thursday the 6th at 3 pm for two people, correct?") before any irreversible action. Speech recognition errors on names, dates and numbers are the most expensive kind.
• Keep sensitive values out of the model. Card numbers and PINs spoken aloud end up in transcripts and logs. Collect them through keypad entry (DTMF) routed straight to the payment processor, so they never enter the AI pipeline.

The orchestration patterns behind reliable tool use, state machines, retries, human approval for risky actions, are covered in AI Agents: Architecting and Automating Autonomous Systems. They matter more in voice, where the user cannot scroll back and check what the agent did.

Telephony: putting the agent on a phone number

Most commercial voice agents live on the phone network, and the phone network has its own constraints.

Audio format. Telephone audio is narrowband. Twilio's Media Streams, which stream a live call over a WebSocket in both directions, require outbound audio as audio/x-mulaw at a sample rate of 8000 Hz, base64-encoded. Speech models usually expect more: the Gemini Live API takes 16-bit PCM at 16 kHz and returns 24 kHz. Your media layer resamples in both directions, and your STT accuracy should be measured on 8 kHz audio, not on clean recordings.

Direct SIP. Some providers now accept calls over SIP without a media server in between. OpenAI's Realtime API lets you point a SIP trunk at sip.api.openai.com (there is also an EU endpoint for European data residency); your server receives a realtime.call.incoming webhook and then accepts, rejects, transfers (refer) or hangs up the call through dedicated endpoints. If you reject without a status code, the API answers 603 Decline.

Handover to humans. Every production phone agent needs a clean path to a person: a spoken offer, a SIP transfer, and the context of the call passed along so the caller does not repeat everything. Designing when to escalate, and what the human agent sees, is as much a service-design question as an engineering one; AI for Customer Support and Service treats it from the support team's side.

What drives the cost of a voice agent

Cost has the same structure as latency: it is a sum of stages, and you can only cut what you measure.

Cascade. You pay STT per minute of audio, the LLM per input and output token, TTS per character or second of audio, telephony per minute if on the phone, and your own compute for each concurrent session.

Speech-to-speech. You pay for audio tokens, and the arithmetic has two surprises. First, OpenAI's voice latency and cost guide states that "audio tokens in user messages are 1 token per 100 ms of audio" while "audio tokens in assistant messages are 1 token per 50ms of audio." A minute of caller speech is about 600 tokens; a minute of agent speech is about 1,200. Second, "the entire conversation is sent to the model for each Response," so every new turn re-reads everything before it. Without caching, a ten-minute call costs far more than ten one-minute calls.

Caching softens this, but it is best-effort and fragile. The same guide warns that truncation "busts the cache near the beginning of the conversation," and that the best way to keep the cache rate high is to keep a session's history static. That creates a real trade-off: trimming history saves tokens but can cost you the cache.

Delegated live models can bill by session time instead. For GPT-Live, OpenAI's guide notes that active session time includes moments when "both are silent, or the backend is working." Idle sessions cost money, so close the voice session during long backend work and resume afterwards.

The practical rules:

• Design for short calls. Resolve the caller's need quickly and end the call cleanly.
• Keep system prompts lean and stable, and prune old turns deliberately rather than on every turn.
• Route simple turns to cheaper models only after you have measured quality on the larger one.
• Report cost per successful task, not cost per minute. A cheaper agent that resolves fewer calls is more expensive.

How to evaluate a voice agent

A voice agent fails in ways a text agent cannot: it mishears, it talks over people, it pauses too long, it sounds wrong. Evaluation has to cover the audio, not just the transcript.

Measure at least these:

Build a test set from real (consented) calls and from scripted scenarios played through TTS with background noise, different accents and the phone codec. Include the unhappy paths on purpose: silence, a caller who changes their mind halfway, a tool that times out, a request outside scope, a spoken prompt-injection attempt. Score transcripts with an LLM judge calibrated against human reviewers, and listen to a sample of calls every week. The methods for building such test sets and calibrating judges are in our guide to LLM evaluation; voice adds the latency and interruption metrics on top.

Run the evaluation on every change to the prompt, the model, the voice or the turn-detection settings; simulated callers and regression suites for voice are built step by step in Voice AI and Realtime Multimodal Agents. A tweak that makes the agent more patient can quietly add latency to every turn.

Consent and disclosure: the rules that shape the call flow

A voice agent records and processes a person's voice and, increasingly, sounds human. Both facts carry legal obligations that change how the first ten seconds of every call are designed.

Tell people they are talking to an AI. Article 50(1) of the EU AI Act (Regulation (EU) 2024/1689) requires providers to ensure that AI systems intended to interact directly with people are designed so that "the natural persons concerned are informed that they are interacting with an AI system, unless this is obvious" to a reasonably well-informed person in context. With a natural-sounding voice on a phone line, it rarely is. Article 50(5) requires the information to be given "in a clear and distinguishable manner at the latest at the time of the first interaction." The AI Act applies from 2 August 2026 under Article 113, and the Digital Omnibus on AI, Regulation (EU) 2026/1744, did not postpone Article 50. Breaches of Article 50 can be fined up to EUR 15 million or 3% of worldwide annual turnover, whichever is higher (Article 99(4)).

Mark synthetic audio. Article 50(2) requires providers of systems that generate synthetic audio to mark the output in a machine-readable format so it is detectable as AI-generated. Regulation (EU) 2026/1744 gives providers whose systems were on the market before 2 August 2026 until 2 December 2026 to comply. If you build on a speech provider's voices, check how they meet this; if you generate audio yourself, it is your obligation. Separately, Article 50(4) requires deployers to disclose audio that is a deep fake, such as a cloned voice of a real person.

Treat voice as personal data, and voiceprints as biometric data. A recording of someone's voice is personal data under the GDPR. If you use the voice to uniquely identify a person, for example voice authentication, it becomes biometric data under Article 4(14), and Article 9 prohibits processing it unless an exception applies, such as explicit consent. Transcribing what someone says is not the same as identifying them by their voice; keep the two features separate and do not build voiceprints unless you need them.

Outbound calls are their own compliance project. In the US, the FCC's Declaratory Ruling FCC 24-17 (released 8 February 2024) confirmed that the Telephone Consumer Protection Act's restrictions on "artificial or prerecorded voice" cover AI technologies that generate human voices, so such calls require the prior express consent of the called party unless an emergency purpose or exemption applies. Recording-consent rules also differ by country and, in the US, by state. Involve counsel before the first outbound campaign.

In the call flow, this becomes:

• a scripted first utterance that says the caller is speaking with an AI assistant, and mentions recording if you record;
• a logged disclosure event for every call, so you can prove it happened;
• an instruction never to claim to be human, tested in your evaluation set;
• cloned or custom voices only with the documented consent of the person whose voice it is.

This section is general information, not legal advice. Obligations depend on your role (provider or deployer), your sector and the countries you operate in.

Sources for this section:

• Regulation (EU) 2024/1689 (AI Act), Articles 50, 99 and 113
• Regulation (EU) 2026/1744 (Digital Omnibus on AI)
• Regulation (EU) 2016/679 (GDPR), Articles 4(14) and 9

A production checklist

Before the first real caller:

• Architecture chosen against written requirements (audit, tools, tone, latency), not against a demo.
• Latency measured per stage, with p95 targets and alerts.
• Turn detection tuned on real recordings; barge-in truncates history to what was heard.
• Every tool has a timeout, a spoken fallback and, for irreversible actions, a read-back confirmation.
• Sensitive digits collected by keypad, kept out of transcripts.
• Phone audio path tested at 8 kHz, with resampling verified.
• Human handover with call context, tested end to end.
• AI disclosure in the first utterance, logged on every call; recording consent handled per jurisdiction.
• Session limits and reconnection handled for long calls.
• Cost per successful task tracked from day one.
• Evaluation set with unhappy paths, run on every change.

Frequently Asked Questions (FAQ)

Should I use a speech-to-speech model or a cascaded STT, LLM and TTS pipeline? Use a cascade when you need an exact, auditable transcript, strictly deterministic tool calls, or the freedom to choose each component. Use speech-to-speech when natural conversation and the lowest latency are the product. Many production systems combine both: a realtime model for the conversation, parallel transcription for logs, and deterministic backend code for anything with consequences.

What latency should a voice AI agent aim for? Human conversation leaves a gap of roughly 200 ms between turns on average, which no agent fully matches today. Set your own target as the time from the end of the caller's speech to the first audio of the reply, measure it per stage at the 95th percentile, and remember that endpointing is often the largest single delay.

How do I stop a voice agent from talking over people? Combine voice activity detection with a turn detection model or a semantic turn detection setting, so the agent waits when an utterance sounds unfinished. When the caller does interrupt, stop playback, clear queued audio, cancel the response and truncate the conversation history to what the caller actually heard.

Does a voice agent have to say it is an AI? In the EU, Article 50 of the AI Act, applicable since 2 August 2026, requires that people be informed they are interacting with an AI system unless that is obvious from the context, at the latest at the first interaction. For a human-sounding voice agent, the safe design is a clear spoken disclosure in the first utterance, logged on every call.

Why does a long voice call cost so much more than a short one? With speech-to-speech models, each new response re-reads the whole conversation, so input tokens grow with every turn, and agent audio is counted at twice the token rate of caller audio in OpenAI's Realtime API. Short, focused calls, stable prompts that keep the cache effective and deliberate history pruning keep the cost under control.

Conclusion

A voice AI agent is a real-time distributed system with a language model inside it. The model matters, but the experience is decided by architecture, by the latency budget, by how the agent handles the moment someone starts talking over it, and by whether the call flow was designed with disclosure and consent in place. Choose the architecture from written requirements, measure every stage, treat barge-in and turn detection as first-class features, track cost per successful task, and evaluate with real audio, including the calls that go wrong.