Realtime Voice AI Agents: Cascaded Pipelines or Native Speech-to-Speech

Ask three vendors how their voice agent hits sub-second replies, and you will get three different architectures dressed up in the same marketing language. Realtime voice AI agents actually split into 2 real mechanisms, a cascaded pipeline or a native speech-to-speech model, and picking the wrong one costs you either latency or your ability to debug what the agent just said.

What's the actual difference between a cascaded pipeline and native speech-to-speech?#

A cascaded pipeline runs 3 separate model calls in sequence: speech-to-text turns the caller's audio into a transcript, a language model decides what to say, and text-to-speech turns that reply back into audio. Native speech-to-speech is a single model that takes audio in and produces audio out directly, with no transcript step in between.

OpenAI's Realtime API guide names these two architectures directly, "chained" for the cascaded pattern and "speech-to-speech (multimodal)" for the native one, and recommends chained as the default starting point for a first voice agent. That vendor-documented split, not a taxonomy invented for this post, is the frame the rest of this article uses. Source

If you have already built a phone agent, there is a good chance you built a cascaded pipeline without naming it that way. Our guide to building a voice AI agent with n8n and Twilio walks through a Gather-then-LLM-then-Say pattern that is exactly this shape, turn-based and cascaded, just covered there in Twilio-specific terms instead of the vendor-agnostic ones used here.

The public discussion around this topic is noisier than it needs to be: search results skew toward vendor latency leaderboards and near-identical programmatic pages citing suspiciously precise numbers with no traceable source. Every specific figure in this article was pulled from a primary vendor document, a named independent benchmark, or a peer-reviewed paper, not from one aggregator repeating another aggregator.

Three separate models, three separate calls, one per hop before the caller hears a reply.

Why does chaining three models cost you latency at every hop?#

Each hop in a cascaded pipeline has to finish enough of its job before the next one can start: speech-to-text needs enough audio to transcribe, the language model needs the finished transcript, and text-to-speech needs the finished reply. Streaming narrows that gap at every hop, but it does not remove the floor each step adds.

Streaming mitigates the gap without eliminating it: partial transcripts let the language model start reasoning before speech-to-text fully finishes, and sentence-level text-to-speech starts speaking the first clause before the model finishes the last one. Both shrink the perceived delay. Neither removes the floor, since each hop still needs a minimum slice of finished work from the one before it before it can begin its own job.

The real target either architecture is chasing is not a product benchmark, it is human conversation. Stivers et al. found that across many languages, the single most common gap between one person's turn ending and the next person's turn starting sits at 0-200ms, the modal, most frequent gap in the distribution, not an average. Source A cascaded pipeline's 3 sequential hops make that window brutally hard to hit without careful streaming; a native speech-to-speech model, with 1 forward pass instead of 3, starts with a real structural advantage on latency alone.

Every number here is read from the cited source directly, not estimated.

What do OpenAI, Google, and Amazon's speech-to-speech models actually do?#

Exactly 3 vendors ship a real native speech-to-speech product today: OpenAI's gpt-realtime-2.1, Google's Gemini Live API, and Amazon's Nova 2 Sonic. ElevenLabs is the notable non-member of that group. Despite its Conversational AI branding, its own documentation describes a cascaded architecture with a proprietary turn-taking layer added on top, not a single audio-in, audio-out model.

Each of the 3 native models ships real, vendor-published specifics worth knowing before you price one out. OpenAI's gpt-realtime-2.1, released July 2026, bills $32 per 1M audio input tokens and $64 per 1M audio output tokens, and OpenAI states the July update cut p95 latency by at least 25% across its Realtime voice models through improved caching. Source Gemini's native audio models bill $3 per 1M audio input tokens and $12 per 1M output, metered at 25 tokens per second of audio, with a free tier available. Source Amazon's original Nova Sonic launched March 2025 and is now marked legacy, set to reach end-of-life in September 2026; Nova 2 Sonic, announced December 2025, replaces it with a 1,000,000-token context window, asynchronous tool calling, and an adjustable turn-taking sensitivity dial. Source

ElevenLabs calls its product Conversational AI, but its own architecture documentation is specific: a fine-tuned automatic speech recognition model, a configurable or bring-your-own language model, a low-latency text-to-speech model spanning 5,000+ voices and 70+ languages, and a proprietary turn-taking model, all orchestrated on ElevenLabs' servers. Source That orchestration layer itself adds less than 100ms of overhead, on top of, not instead of, the underlying ASR, LLM, and TTS latency each hop still carries. Source It is a managed cascade, a real product worth using, just not the same mechanism as gpt-realtime-2.1 or Gemini Live.

What does a native speech-to-speech model hear that a transcript never will?#

Text is a lossy compression of speech: a cascaded pipeline's transcript step permanently discards tone, pace, and emphasis before the language model ever reasons over the words. A native speech-to-speech model reads the raw audio directly, so it can act on hesitation or a frustrated tone without an extra classification step most teams never actually build.

OpenAI's Realtime API and Gemini Live are both full-duplex: they listen and generate audio at the same time, so voice-activity-detection-triggered interruption cancels an in-flight reply the instant the caller starts talking again, rather than waiting for a text-based turn boundary a cascaded pipeline has to construct after the fact. Source

A cascaded pipeline can approximate tone-awareness by bolting on a separate step, sentiment analysis on the transcript, or prosody tags from the speech-to-text model, but that is an extra component most teams never actually build. Native speech-to-speech gets it by default, because the model reasoning over the reply is the same model that heard the caller say it.

Picture a caller who says "that's fine" in a flat, clipped tone right after a long wait. A transcript alone renders that as plain agreement. The audio itself carries the actual signal, and a model that hears it directly can choose to acknowledge the frustration and check in, rather than plow ahead as if the caller were satisfied.

What do you give up when you collapse three models into one?#

Collapsing 3 models into 1 removes your default text audit trail, makes it harder to swap the language model mid-call, and turns debugging into listening to raw audio instead of reading a log line. Vendor lock-in gets sharper too: the voice itself is the model, so switching vendors means re-testing conversational behavior, not just re-pointing a text-to-speech call.

That transcript gap is not hypothetical. A cascaded pipeline's text log is what makes a support team's quality review, a compliance audit, or a simple "what did the bot actually say" investigation fast. A native speech-to-speech model gives you none of that by default: you would have to build your own transcription and logging layer alongside it, and few teams actually do.

"Harder to swap" is concrete, not abstract. In a cascaded pipeline, the language model is 1 replaceable component behind a stable speech-to-text and text-to-speech boundary, so testing a cheaper or newer model is close to a config change. In native speech-to-speech, the language behavior and the voice behavior are trained into the same weights, so a vendor swap means re-qualifying the whole conversational experience, not just a reply's wording.

OpenAI's own guidance is blunt about one specific limitation: the Realtime API does not expose voice cloning, and its documentation points builders who need a cloned or custom brand voice back toward a chained pipeline paired with a dedicated text-to-speech vendor such as ElevenLabs or Cartesia. Source That single limitation is often the deciding factor long before latency ever enters the conversation.

Neither column wins outright. Cascaded keeps you in control; native speech-to-speech keeps you fast.

3 questions decide the architecture: transcript needs, a custom voice, and whether sub-second response is the real requirement.

So which architecture should your voice agent actually use?#

Default to a cascaded pipeline: it is what OpenAI's own guidance recommends for a first voice agent, it hands you a transcript for quality review, and it lets you swap the language model without re-testing the whole conversation. Reach for native speech-to-speech only when sub-second, full-duplex response is the actual product requirement, not just an impressive demo.

For the cascaded path itself, the n8n and Twilio build guide covers the concrete wiring: Gather, an LLM step, and TwiML Say. That question is separate from whether the product needs agentic reasoning on top of whichever speech mechanism you pick: tool calls, multi-turn planning, deciding when to escalate to a human. If that is the open question, what an AI agent actually is covers the agent-versus-chain distinction on its own terms, independent of voice.

The signals worth watching for are concrete, not vibes: callers frequently talk over the agent, the reply genuinely needs to change based on tone rather than just words, and nobody on the team expects to swap the underlying model 6 months from now. When several of those hold at once, the latency and tone advantages of native speech-to-speech start to outweigh the transcript you would be giving up.

If 2 or fewer of these are true, a cascaded pipeline is the safer default.

What should you set up this week?#

Prototype both architectures against 1 real conversation this week, using the cheapest access point available: Gemini Live API's free tier or OpenAI's Realtime mini model. Do not decide from a demo video. Every price and model name in this space moves fast, so verify each number in this article against the vendor's current documentation before you act on it.

OpenAI alone shipped a new Realtime generation in July 2026, and both Google and Amazon have revised pricing or launched new model versions within the same year. None of the specific dollar figures, latency numbers, or model names in this piece should be treated as permanent. Re-check the OpenAI, Google, and ElevenLabs pricing pages linked above, and confirm the Amazon Nova 2 Sonic per-token price directly at aws.amazon.com/bedrock/pricing before quoting it to anyone, since it could not be independently verified while researching this article.