NVIDIA Nemotron 3 Diarization: Open-weight, real-time diarization for up to 8 speakers

NVIDIA released Nemotron 3 Diarization, an open-weight, 100M-parameter speaker diarization model that identifies who spoke when in audio with up to eight overlapping speakers. The model runs in real time or offline, outputs speaker-attributed transcripts, and is released under a permissive commercial license. For builders working on meeting analytics, call pipelines, or voice-agent memory, this is a practical upgrade over the previous 4-speaker baseline.

What Nemotron 3 Diarization is and why it matters

Automatic speech recognition (ASR) transcribes words, but it cannot tell you who said them. Diarization fills that gap. Nemotron 3 Diarization extends NVIDIA's earlier Streaming Sortformer model from 4 to 8 speakers, directly handling overlapping speech. A single checkpoint supports both offline recordings and live streaming.

The model is small enough (100M parameters) to run on consumer GPUs like the RTX PRO 5000, and the OpenMDW License 1.1 allows commercial use. That means you can deploy it without per-call licensing fees, which matters for high-volume audio pipelines.

Architecture and streaming: how it handles 8 speakers

Audio is converted to Mel-spectrogram features at 10 ms steps, stacked by a factor of 8 to produce 80 ms encoder frames. A 31-layer Transformer encoder with rotary positional embeddings (RoPE) processes those frames, followed by a Conv1D upsampling back to 10 ms resolution. The output is a [T, 8] tensor of per-speaker activity probabilities.

Overlap is handled natively: if two people talk at the same time, two channels activate in the same frame. Speakers are ordered by arrival time, keeping labels stable across streaming chunks. Streaming uses an Arrival-Order Speaker Cache (AOSC) and a FIFO queue to maintain context. The recommended input-buffer latency is 0.32 seconds.

Performance: better accuracy and much higher throughput

On Voice Arena's Diarization-Bench, the model ranked first with 14.72% Diarization Error Rate (DER), roughly a 24% relative reduction over the next-best system. Compared to the 4-speaker baseline at 1.04 s latency, DER dropped by a mean of 41.0% across eight evaluation conditions, with improvements ranging from 9.0% to 65.2%. NVIDIA noted one minor regression: on 2-speaker CALLHOME at 30.4 s latency, DER rose from 5.68% to 5.98%.

Throughput also jumped significantly. At 30.4 s latency, the model reached 15,113x real-time factor (RTFx) versus 2,619x for the baseline, tested in BF16 on an NVIDIA RTX PRO 5000 with torch.compile(). These are batched numbers, but the improvement is large enough to matter for batch processing.

Deployment options and limitations

The model runs on Linux through NVIDIA NeMo on Ampere, Ada Lovelace, Hopper, or Blackwell GPUs. For production serving, NVIDIA lists Baseten and DigitalOcean. On-device deployments are possible through the Argmax Pro SDK 3. It is not yet available on Hugging Face Inference Providers.

Key limitations: recordings with more than 8 speakers can produce missed or misassigned speech. Heavy noise, reverberation, and far-field capture also increase error rates. The model outputs anonymous speaker labels (speaker1, speaker2, etc.); mapping those to real identities is left to downstream applications.

Training used roughly 10,000 hours of real conversations and 82,611 hours of simulated multi-talker mixtures, including licensed data from David AI. Adding that data cut compound DER from 11.19% to 10.42%.

For builders: if you need speaker attribution in meeting transcripts, call analytics, or podcast pipelines, Nemotron 3 Diarization is a strong open option. It is not a magic fix for noisy or extremely crowded audio, but it fills a real gap with a permissive license and GPU-friendly size.