Audio-Native, TTS-Optional: PolyAI’s Dialog-RSN-1 Carves Out Third Enterprise Voice AI Architecture

Enterprise voice AI architects have spent the last two years choosing between two bad trades. Cascaded pipelines — the standard three-hop system that converts speech to text, feeds that text to an LLM, and speaks the result back — are precise and cheap to run, but they throw away everything in the caller’s voice that isn’t a word: the hesitation before “yes,” the frustration starting to show, the brand name that isn’t quite the dictionary spelling. Speech-to-speech models like GPT Realtime and Gemini Live keep the audio, but lock the output voice into the model and pin a GPU for the entire call, an economics problem that shows up in pricing around $18 per hour in production deployments.

PolyAI published a technical blog post on July 30, 2026, introducing Dialog-RSN-1, the company’s first audio-native large language model, and arguing that neither of those trades is necessary. The model fuses end-pointing, speech recognition, and response generation into a single audio-aware LLM, then hands speech generation to a separate, externally-configured TTS system — letting it hear the full call while keeping the output voice fully controllable.

In production deployments at existing PolyAI customers, the company says Dialog-RSN-1 responds reliably in under 300 milliseconds — with a tighter latency distribution than GPT Realtime 2.1, which the company’s internal testing found typically responds in between 860 and 1,900 milliseconds in live deployments. To put that gap in context: the natural inter-speaker pause in human conversation is around 200 to 300 milliseconds, a figure that is remarkably consistent across languages, cultures, and geographic regions. Callers notice when that rhythm breaks.

Why Neither Existing Architecture Won

The conventional cascaded voice agent works in three steps. An ASR model converts the caller’s speech to text. That text goes to an LLM. The LLM’s reply goes to a TTS synthesizer. Each step is individually fast, but the system hands the LLM only its best guess at what the ASR model heard. No audio signal. No tone. No understanding of the uncertainty in that transcription. A word the ASR model was 60% confident about arrives at the LLM as if it were a certainty.

The result is a class of errors that cascade systems can patch but cannot structurally fix. Improving the system means tuning its individual components: adjusting end-pointing parameters, biasing the ASR toward a specific domain, adapting it through a conversation. That produces a stack of manually-configured parameters that rarely generalize across different caller types or use cases.

Speech-to-speech models addressed that by eliminating the transcription step entirely. GPT Realtime and Gemini Live process audio directly, removing the text bottleneck. But doing so at the full-duplex scale that makes phone conversation feel natural — listening and speaking simultaneously — requires a model to run continuously. PolyAI describes the GPU economics this way: any general-purpose model that operates at a timescale sufficient to generate natural speech requires substantial GPU resources, and always-on, full-duplex variants pin a GPU for the entire duration of a call. The practical consequence is the pricing gap: production cost analyses have put the OpenAI Realtime API at approximately $18 per hour of voice interaction versus cascaded alternatives closer to $4.50.

A separate limitation applies to voice customizability. In a speech-to-speech model, the output voice is baked into the model itself. Changing the intonation, accent, emotional register, or pronunciation of a brand name requires retraining. PolyAI asks: does GPT Realtime mispronounce your brand name? Tough luck.

How Dialog-RSN-1 Differs From Both

Dialog-RSN-1 is audio-aware on the input side only. One model — a unified LLM — takes in the raw audio signal and handles end-pointing, speech recognition, and response generation. But speech synthesis runs in a separate TTS module, kept entirely external. Switching voices becomes a configuration change rather than a model retrain. PolyAI’s position is that the input side is where most of the benefit of audio-aware processing lives; generation rarely needs to hear the caller’s voice in return.

The model also avoids always-on GPU use. Rather than streaming audio continuously and decoding every frame, Dialog-RSN-1 is probed on demand: a voice activity detector fires on acoustic events and at regular timer intervals, then sends what it has captured to the model. The model’s very first output token decides whether to speak — classified as EMPTY (no speech detected), ONGOING (caller hasn’t finished), or COMPLETE (the caller’s turn is done, time to respond). That first token is fast to generate because the attention cache has already been warmed up while the user was speaking.

PolyAI also makes a more pointed claim: in its internal testing, GPT Realtime 2.1 scores on par with cascaded systems on audio-aware evaluations, which it reads as evidence that GPT Realtime may be “a well-engineered cascaded system” rather than a model that genuinely reasons over raw audio. The company notes that users can probe this themselves by asking the model whether Americans pronounce the last letter of the alphabet as “Zed.” This is PolyAI’s own testing and interpretation — no independent verification exists for this claim as of publication, and OpenAI has not responded. If it holds up to independent audit, it would mean enterprises paying 4x the per-call latency for GPT Realtime are not getting the genuine audio-native processing that justifies that premium.

What the Model Actually Hears — and Why That Changes Call Handling

The practical gap the audio-native design is meant to close is the difference between what a text transcript captures and what a human agent hears. Dialog-RSN-1 can be prompted on what it hears in a caller’s accent, mood, or the background noise on the line. It uses the full conversation context and system prompt to interpret audio that trips up cascaded systems: a mispronounced word that context makes obvious, a caller spelling out “Matthew with two Ts,” a brand name like “Audibel” that isn’t the word “audible,” or the difference between a beep on an answering machine and a pause in live conversation.

Turn-taking — the moment-to-moment judgment of when to speak and when to keep listening — is represented as the model’s first output token. PolyAI trains the model to respond quickly when a caller has finished speaking but not to interrupt during pauses, reducing the tension between snappy response times and talking over the user. Because turn-taking is owned by the LLM rather than by a separate acoustic component, it becomes a configurable behavior. System prompt instructions PolyAI documents as functional include “wait for the user to say all six digits of their customer code before responding,” “do not speak if the user is talking to someone else,” and “if it seems like the user thinks they are talking to a voicemail system, interrupt them so they know they are talking to an interactive system.”

How PolyAI Hit Sub-300ms: Five Specific Engineering Choices

The latency target required more than a base-model selection. PolyAI built Dialog-RSN-1 by post-training open-weight multimodal models with supervised fine-tuning followed by reinforcement fine-tuning on in-house training datasets — a pipeline the company describes as base-model agnostic, having evaluated Gemma, GPT-OSS, Qwen, and Mistral. The target of sub-300-millisecond latency when served on NVIDIA A100 GPUs constrains the candidate range to models in the 8B dense to 30B sparse parameter range.

Five specific techniques drive the latency performance:

First, Dialog-RSN-1 pre-fills the attention cache while the user is still speaking. By the time the caller finishes, the model’s key-value cache for the full conversation context is already warm, and the first response token requires minimal additional computation.

Second, the prompt template is kept append-only throughout a conversation, minimizing cache invalidations. Cache invalidation — when a change in context forces the model to recompute portions of the attention cache — is one of the primary sources of latency variance across turns.

Third, each caller is routed consistently to the same GPU across turns, keeping that caller’s activations already in GPU memory. This eliminates cold-start latency that would otherwise add overhead at each turn.

Fourth, PolyAI fine-tuned a speculative drafter specifically for Dialog-RSN-1’s use case and prompt format, achieving a mean token acceptance rate of 3.9 per probe. In speculative decoding, a smaller draft model proposes candidate output tokens in parallel, and the primary model either accepts or rejects them. An acceptance rate of 3.9 means the model is generating roughly four tokens of output per primary-model evaluation rather than one.

Fifth, the model was trained to reason only when necessary — a behavior PolyAI calls “auto-reasoning,” learned during reinforcement fine-tuning. Forcing the model to run extended reasoning chains on every turn would consume time that the latency budget cannot absorb. Teaching the model to reason selectively preserves response speed for simple turns while making deeper inference available when calls actually require it.

Benchmark Results and Production Numbers

PolyAI’s Dialog-Eval benchmark takes a different approach than most voice AI evaluations. Rather than measuring full conversation rollouts — which depend on a realistic user simulator and can mask which specific behaviors succeeded or failed — Dialog-Eval scores point-in-time decisions. Each example is a call truncated at a single moment, with everything fixed: the system prompt, the conversation history, and the caller’s latest audio. The model’s next single step is scored, either one tool call or one user-facing reply.

On Dialog-Eval, Dialog-RSN-1 achieves the highest quality score among real-time-capable models, including scores on audio comprehension and safety dimensions. The audio sub-score is particularly telling: cascaded systems hit a ceiling of around 77, representing the best score achievable when the audio signal has been bottlenecked into a text representation. Dialog-RSN-1 surpasses that ceiling.

On word error rate, Dialog-RSN-1 produces the most accurate transcriptions of any model PolyAI tested, even when competing dedicated ASR models were given the full conversation context in their prompts to make the comparison fairer. The key architectural reason: the same joint model that generates a response also transcribes, already holding the system prompt, history, and call audio. It responds first, then transcribes — so the transcript reflects the same contextual understanding that produced the response.

The company reports two production results from enterprise customers who switched to Dialog-RSN-1. A national restaurant group saw an 11% relative increase in containment — the share of calls resolved without transferring to a human agent — after switching. A large insurance provider cut response latency by 37%. Both figures are company-disclosed and not independently verified by a third party.

Market Position and What Dialog-RSN-1 Is Not

PolyAI serves more than 200 enterprise customers including Marriott, FedEx, and UniCredit, targeting high-call-volume sectors — financial services, healthcare, hospitality, and retail — where call containment rates translate directly into headcount savings. A Forrester Total Economic Impact study found PolyAI customers achieve a 391% return on investment and an average of $10.3 million in savings per deployment.

Dialog-RSN-1 is available today to existing PolyAI customers. New customers can apply for early access as the public API rollout begins. Existing deployments running Raven 3.5 are not automatically switched. The model is English-only at launch; the underlying base model is multilingual, and PolyAI reports seeing Dialog-RSN-1 pick up non-English speech in production already, but broader language support is identified as the next development priority. For web chat and non-English contact center use cases, Raven 3.5 remains the company’s recommended option.

The model is not available as open weights and has no public API pricing. PolyAI operates on custom enterprise contracts that third-party estimates put in the range of $100,000 to $150,000+ annually, plus per-minute usage fees. This positions Dialog-RSN-1 firmly in the same tier as NICE Cognigy and other managed enterprise voice platforms — not as a developer tool competing with Bland AI, Retell AI, or Vapi on per-minute commodity pricing.

A technical report and a paper introducing the Dialog-Eval benchmark framework as an open-source release are expected from PolyAI in upcoming publications. Neither has a confirmed release date as of July 31, 2026.

The industry median voice AI latency — drawn from an analysis of over 4 million live calls — sits at 1.4 to 1.7 seconds, roughly five times slower than the 300-millisecond human expectation. Dialog-RSN-1’s production figures, if independently verified, would place it well below that median and ahead of most competitors on this metric.

---

Frequently Asked Questions

What is the difference between Dialog-RSN-1 and a speech-to-speech model like GPT Realtime?

A speech-to-speech model processes audio on both input and output — the same model hears the caller and generates the spoken reply. That means the output voice is baked into the model, making it difficult to change intonation, accent, or brand-name pronunciation without retraining. It also means keeping a GPU occupied for the entire duration of the call when running in full-duplex mode. Dialog-RSN-1 is audio-aware on the input side only — one model hears the caller and generates a text response, then a separate, externally-configured TTS system speaks it. The separation keeps the output voice fully controllable and eliminates the always-on GPU cost of full-duplex streaming. PolyAI’s internal benchmarks additionally claim GPT Realtime 2.1 scores on par with cascaded systems on audio-aware evaluations, which the company interprets as possible evidence that GPT Realtime may be routing through a transcription step rather than genuinely processing raw audio — though this claim has not been independently verified.

What does “sub-300ms latency” mean in practice for a phone call?

The natural gap between speakers in human conversation is around 200 to 300 milliseconds — a figure that research has found remarkably consistent across languages and cultures. When an AI agent takes longer to respond, callers notice: the rhythm of conversation breaks, and callers interpret the silence as a dropped call or a failure to understand. Industry analysis of more than 4 million live AI phone calls found the median voice AI latency at 1.4 to 1.7 seconds — roughly five times slower than the human benchmark. Sub-300ms means Dialog-RSN-1 is, in PolyAI’s production measurements, operating within the natural conversational rhythm rather than below it.

How is Dialog-RSN-1 different from PolyAI’s existing Raven 3.5 model?

Raven 3.5 is a cascaded system — it converts audio to text before the LLM processes it. Dialog-RSN-1 processes raw audio directly. The practical differences show up in the categories of errors each model can and cannot handle: Raven 3.5 cannot recover from ASR misrecognitions without additional engineering, while Dialog-RSN-1 can use conversational context to interpret ambiguous audio. Raven 3.5 remains PolyAI’s recommended model for non-English contact center use cases and web chat; Dialog-RSN-1 is the recommended option for English voice agents as of July 2026. Both run on PolyAI’s Agent Studio platform.

If PolyAI’s claim about GPT Realtime is accurate, what does that mean for enterprises using it?

PolyAI’s Dialog-Eval benchmark found GPT Realtime 2.1 scoring on par with cascaded systems on audio-aware evaluations — tasks specifically designed to test whether a model is reasoning over raw audio or working from a transcription. If this finding holds under independent audit, it would suggest that enterprises using GPT Realtime are paying a 4x latency premium (860–1,900ms versus under 300ms) without receiving the genuine audio-native processing that would justify it. This claim has not been independently verified as of publication, and OpenAI has not commented on the benchmark methodology. Enterprise teams evaluating voice AI platforms should request vendors’ scores on audio-specific evaluations and verify whether those evaluations test genuine audio reasoning or downstream text processing.