What a voice AI agent actually is
A voice AI agent is not a single model. It is a small orchestra: a speech recogniser, a language model, a speech synthesiser, and a control layer managing whose turn it is to talk. The interesting engineering is in how those components are wired together so the conversation does not feel broken.
If you have used a good one recently (an airline rebooking line, a clinic reminder, an outbound sales qualifier) you probably noticed it responded quickly and let you interrupt without falling over. If you have used a bad one, you noticed dead air, cross-talk, or the agent finishing a two-sentence reply after you had already moved on. Both experiences come from the same pipeline. The difference is tuning.
The three-stage pipeline and how it works
STT, LLM, TTS: the core chain
The classic voice agent runs four things in sequence, wrapped in a control loop:
- Voice activity detection (VAD) decides when the caller has started and stopped speaking.
- Speech-to-text (STT) transcribes the audio into words.
- A language model (LLM) decides what to say, often calling tools (look up an order, book a slot, check a balance).
- Text-to-speech (TTS) turns the reply into audio and streams it back down the line.
Around this sits the orchestration layer: turn-taking, barge-in, echo cancellation, tool invocation, logging. The telephony side (SIP, PSTN, WebRTC) is a separate concern that the platform hides or exposes depending on how much control you want.
Cascaded, speech-to-speech, and hybrid: which architecture dominates
Three architectures are in real use.
Cascaded is the classic chain above. It is the most controllable: you can swap the STT vendor, log every transcript, evaluate the LLM in isolation, and tune the TTS voice independently. It is what nearly every production deployment we see runs on.
Speech-to-speech (S2S) models take raw audio in and emit audio out, skipping discrete STT and TTS steps entirely. They preserve prosody, laughter, hesitation and other paralinguistic signals that a transcription throws away. OpenAI's Realtime API (generally available since 28 August 2025 with gpt-realtime, SIP calling and remote MCP) is the highest-profile example. S2S is faster and more expressive but harder to steer, harder to log, and less mature around tool use.
Hybrid combines an audio-native LLM on the input side with a discrete TTS on the output side. You get some of the input-side benefits of S2S without giving up voice control on output.
Our default in 2026
For nearly every client build we start with a cascaded pipeline. It is boring, well-understood and lets us debug each stage in isolation. We reach for S2S only when the use case genuinely rewards paralinguistic fidelity, and we keep a cascaded fallback for the calls where the model needs to invoke tools reliably.
Why latency is the central engineering problem
Where the time actually goes
Research in conversational psychology puts the average gap between speakers in natural dialogue at around 200 milliseconds, roughly the time it takes to blink. That timing is hardwired across languages. Anything longer feels like a pause, and long pauses train callers to talk over the agent.
Component-level latency, on paper, looks manageable:
- STT: 100 to 500ms
- LLM: 350ms to over a second
- TTS: 75 to 200ms
But according to Retell AI's 2025 benchmarks, most production voice agents deliver 1,400 to 1,700ms at median end to end. The gap between component numbers and system numbers is where the real work happens. Every handoff between stages, every network hop to a cloud region, every SIP-to-CPaaS bridge, adds time. The single largest contributor, in most teams' measurements, is the LLM hop to a public cloud region, followed by the telephony handoff.
Useful production targets from the same benchmarks: time-to-first-token under 300ms, end-to-end latency under 800ms, barge-in response under 200ms. Hit those and the agent stops feeling like a robot.
Streaming and chunking: the trick that makes it feel fast
The naive approach is a waterfall: wait for the caller to finish, wait for the full transcript, send it to the LLM, wait for the full response, synthesise the audio, play it. Simple to build, unusable in production.
The fix is to overlap every stage. STT emits partial transcripts while the caller is still mid-sentence. The LLM starts generating tokens before the full input arrives. As soon as the LLM emits the first few tokens, TTS starts synthesising. The caller hears the first word of the reply while later words are still being generated. Modern platforms stream audio out in 200 to 400ms chunks. Nothing waits for anything upstream to finish.
Every stage emits output before the previous stage finishes. That is what turns a two-second waterfall into an 800ms conversation.
If you are debugging a slow agent, measure the per-layer breakdown before you touch anything. Total latency is a symptom. The layer breakdown is the diagnosis.
Barge-in and turn-taking: the hardest UX problem to solve
Barge-in is what happens when the caller cuts the agent off. Three things have to happen almost instantly: echo cancellation removes the agent's own voice from the incoming audio stream (otherwise the agent hears itself and thinks the caller is talking), TTS stops playing within about 200ms, and the LLM's in-flight response is discarded so it does not resume speaking a stale reply once the caller finishes.
Turn-taking is the mirror problem: deciding when the caller has actually stopped, not just paused. Endpoint too early and the agent interrupts. Endpoint too late and every turn has half a second of dead air baked in. This is a product decision as much as a technical one. It varies by language, by caller demographic (older callers pause more), and by use case (a booking flow tolerates longer pauses than a fast-paced sales call).
The current best practice is an adaptive turn-detection model rather than a fixed silence threshold. LiveKit Agents shipped one in their 1.0 release in April 2025. Retell's turn-taking updates in 2025 focused on distinguishing natural mid-sentence pauses from actual conversation turns. Neither is perfect. If you are building for a specific caller population, expect to tune this against real recordings.
Get plain-English guides like this in your inbox.
One short email a month. WordPress, Shopify, SEO, no fluff. Unsubscribe in one click.
We never share your email.
The platform landscape: VAPI, Retell, LiveKit and Bland
The market for managed voice orchestrators exploded in 2024 and 2025. What used to take months of stitching together Twilio, a custom WebSocket server, Deepgram, OpenAI and ElevenLabs can now be prototyped in a day. Four platforms lead:
- VAPI is a fully hosted orchestrator. Bring your LLM, bring your voice, plug in a phone number, and Vapi handles the real-time voice loop, turn-taking, interruption handling and telephony plumbing. Good default for teams that want model flexibility without owning the media stack.
- Retell AI is the most opinionated on workflow design and call analytics. Pick it if you value a tightly-coupled builder and analytics view over swap-in-swap-out model flexibility.
- LiveKit Agents is open source and self-hostable. Their 1.0 release in April 2025 included an adaptive turn-detection model, and the Python 1.5.x line ships native MCP tool support. Reach for it when data sovereignty, custom media routing, or an unusual pipeline rules out a hosted service.
- Bland AI targets high-volume outbound, particularly healthcare reminders, insurance renewals and collections, with dedicated infrastructure and pre-built CRM integrations.
Pipecat, from Daily, is LiveKit's closest open-source competitor: a strong Python pipeline framework, but you bring your own WebRTC or SIP layer.
Managed vs self-hosted: the real decision
The build-vs-buy split is cleaner than it looks. Managed teams shortlist VAPI vs Retell. Infrastructure teams shortlist LiveKit vs Pipecat. The criteria that actually matter:
- Data sovereignty. If audio or transcripts cannot leave a specific region or your own VPC, managed is out.
- Model flexibility. If you need to route between multiple LLMs by intent, or use a fine-tuned model, check what the platform allows.
- Volume. High-volume outbound has different economics and infrastructure needs than inbound support.
- Geographic latency. Test from the regions your users actually live in. A platform with excellent US-East numbers can fall apart in Singapore. Do not trust marketing benchmarks here.
- Time to launch. Managed platforms win by weeks, sometimes months.
Our own AI development work splits roughly evenly: managed platforms for teams shipping a first agent quickly, self-hosted when the client has real infrastructure constraints or a use case that needs custom media logic. Neither is superior. They are answers to different questions.
What voice agents can and cannot do right now
What is genuinely solved in 2026: IVR deflection, appointment booking, order status, password resets, balance enquiries, appointment reminders, basic outbound qualification. Healthcare contact centres report roughly 30% reduction in hold times and 85% reduction in agent training time after deploying voice agents for these use cases. Financial services numbers are similar. Gartner expects conversational AI to remove $80 billion of contact-centre labour from the system in 2026.
What is still fragile: complex multi-turn reasoning where context builds across five or six exchanges; calls with heavy real-world disfluency (self-correction, trailing off, code-switching between languages); tool use during interruption, where the model is mid-call to an API when the caller starts speaking again. Most benchmarks still evaluate against clean synthetic audio, so published accuracy numbers overstate real-world performance.
Design for escalation, not perfection
The single most important design decision is the handoff. Every voice agent should have a clear escalation path to a human on any intent it cannot resolve confidently. Trying to make the agent handle everything is how you end up with the frustrating experiences everyone complains about.
The grounding problem is the same as it is in text. If your agent needs to answer from your own knowledge base, you need to think about RAG explained: grounding AI on your own data, and the same disciplines that keep a chat agent honest apply to a voice one. If it is doing customer support, the principles in Building a customer-support AI that does not hallucinate transfer directly, with the added constraint that a caller cannot see a citation.
What to consider before you build
A few practical questions we work through with clients before writing any code:
- What is the highest-value call type, and what does deflection look like? Do not try to automate the whole call tree. Pick one intent, get it right, expand.
- What system does the agent need to talk to? CRM, booking system, order database, telephony routing. Integration is usually 60% of the build.
- What is the escalation path? Warm transfer to a human? Callback ticket? Which conditions trigger it?
- What language, accent and caller population? This drives the STT choice and the turn-taking tuning more than the LLM choice does.
- What does post-call look like? Transcript summarisation, task creation, QA scoring. This is where a lot of the ROI actually lands, and it is easy to forget in a demo.
- How will you test at load? Voice agents behave differently at concurrency. A one-caller demo does not tell you what happens when 200 concurrent calls hit the same LLM endpoint.
The broader picture on where agents fit into a business is worth reading alongside this: AI agents for business: what actually works in 2026 covers the wider strategy without repeating the voice-specific engineering.
Voice is a pipeline you can measure, tune and debug like any other. The teams shipping good agents in 2026 are treating it as an infrastructure problem first and a model problem second. If you want to see what that looks like in practice, see what we have shipped.
