NVIDIA VoiceChat-11B Brings Full-Duplex Speech to AI Agents
Published by AINave Editorial • Reviewed by Ramit
NVIDIA VoiceChat-11B is an 11-billion-parameter full-duplex speech model that streams speech understanding and generation through one architecture. For AI builders, the important change is architectural: instead of waiting for separate ASR, LLM, and TTS services to complete, the model handles audio input and output continuously, targeting about 450 milliseconds of turn-taking latency.
Why full-duplex speech changes voice agent design
A conventional voice agent typically transcribes audio, sends text to a language model, then synthesizes a reply. Each handoff adds serialization, network, and inference overhead. VoiceChat-11B is designed to reduce that accumulated delay while supporting interruption handling, so an agent can yield when a user starts speaking instead of finishing a queued response.
The model accepts 16 kHz audio and produces 22.05 kHz audio. Its architecture combines a Fast Conformer speech encoder, a Nemotron Nano V2 9B language model backbone, and a NVIDIA TTS decoder. That makes it a tightly integrated voice interaction system, rather than a general language model with speech adapters.
Tool calling is included, but reliability is the constraint
VoiceChat-11B can emit tool-calling scripts through a separate output channel while continuing the spoken interaction. A voice agent could use that channel for a database lookup, calculation, or API request, then play an interim message while the operation runs. This is useful for customer support and other workflows where silence feels like a failure.
The benchmark results point to a narrower conclusion than the feature list suggests. Reported accuracy reaches 58.5% for simple tool calls and 62.5% for multiple tool calls, but falls to 42.5% for parallel execution. Argument accuracy is 44.2%, and the reported pass@1 result on Full-Duplex-Bench v3 is 33% in the supplied evaluation summary. Builders should therefore treat spoken tool calls as untrusted inputs that require validation, confirmation, and safe failure handling.
Deployment is built around NVIDIA infrastructure
The documented path is a streaming WebSocket service using the NVIDIA vLLM inference engine on Linux. Supported hardware includes A100, H100, H200, B100, B200, and RTX-6000 GPUs according to the model description. No CPU, ARM, Windows, quantized, or smaller deployment option is documented in the supplied material.
That makes the model most practical for teams already operating NVIDIA GPU infrastructure and evaluating full-duplex interaction. It is less attractive if your product depends on portable inference, edge deployment, or independently swappable ASR and TTS components.
Research access does not equal production readiness
The model is documented as ready for research purposes only. Commercial deployment requires review of the OpenMDW License Agreement v1.1 and may require permission from NVIDIA before integration into a product or service. That restriction matters more than the latency headline for founders planning a commercial voice agent.
The training mix includes roughly 550,000 hours of real and synthetic audio, but the supplied material is English-focused. Multilingual support, code-switching, and performance on difficult acoustic conditions are not documented or benchmarked
