PolyAI has introduced Dialog-RSN-1, a dialog model that perceives the caller’s audio directly instead of reading a transcript. It fuses turn-taking, speech recognition, function calling and response generation into one audio-native model, and is already handling live production calls.
Key Takeaways
- Dialog-RSN-1 is audio-aware on the input side only; TTS stays separate, so the output voice remains controllable.
- It runs as a request-based LLM probed on demand, not an always-on stream that pins a GPU.
- Turn-taking is the model’s first output token:
EMPTY,ONGOINGorCOMPLETE. - PolyAI reports sub-300ms responses, +11% relative containment at a restaurant group, and −37% latency at an insurer.
- English only at launch, delivered through PolyAI’s platform rather than open weights or a public API.
Is it deployable, and by whom?
Yes, but only through PolyAI: no open weights, no public API yet. Existing customers can enable it today; new customers can request early access.
- Company level: large, high-call-volume enterprises. PolyAI reports 100+ enterprise customers and 2,000+ live deployments at its $86M Series D in December 2025. Self-serve developers and SMBs are not the target.
- Industries: restaurants, insurance, financial services, healthcare, hotels, retail, telecom, travel and utilities.
- Applications: booking and reservations, billing and payments, authentication, call routing, order management and troubleshooting.
The architecture
Two architectures dominate, and each concedes something. A cascaded stack sends only the ASR’s best guess to the LLM, so tone, hesitation and recognition uncertainty are gone before the LLM sees anything. Tuning means hand-adjusting end-pointing parameters and ASR biasing that rarely generalize across use cases. Speech-to-speech models such as GPT Realtime and Gemini Live keep the audio but bake the voice into the model, limiting pronunciation control, and always-on full-duplex variants pin a GPU for the entire call.
Dialog-RSN-1 is audio-aware on input only: one model reasons over raw audio and hands generation to a separate, promptable TTS system. It is probed on demand rather than streamed: a high-recall VAD plus a few timers decide when to run it, and the first token of the reply settles whether the agent should speak. Cheap acoustic cues only choose when to ask; the model, with full context, makes the actual turn-taking call.
How it was built
PolyAI post-trained open-weight multimodal models with supervised and reinforcement finetuning on in-house data. The pipeline is broadly base-model agnostic; PolyAI evaluated Gemma, GPT-OSS, Qwen and Mistral. Targeting sub-300ms on A100 GPUs puts candidates in the 8B dense to 30B sparse range. Latency work includes prefilling the attention cache while the user speaks, an append-only prompt template to minimize cache invalidation, routing each caller to the same GPU, a finetuned speculative drafter with mean acceptance of 3.9 tokens, and auto-reasoning learned during RFT. Transcription runs last, after the response or tool call, in parallel with speech generation.
Results
PolyAI evaluated on Dialog-Eval, an internal benchmark it plans to open-source. Each example is a call truncated at one decision point, scoring a single atomic next step rather than a full rollout. PolyAI reports Dialog-RSN-1 as the highest-scoring real-time capable model, puts the cascaded Audio-score ceiling near 77, and notes GPT Realtime 2.1 scoring on par with cascades on audio-aware examples. On transcription, gpt-4o-transcribe’s WER improved from 7.8% to 6.9% once given the same context, with Dialog-RSN-1 lower still.
For this release PolyAI focused on English; Raven 3.5 remains its recommendation for non-English and rich web chat. A technical report and a Dialog-Eval paper are planned.
Check out the Technical details here. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.