NVIDIA releases Nemotron 3.5 ASR, a compact 0.6B speech recognition model that can transcribe 40 languages in real-time on CPU only

NVIDIA’s NeMo speech team quietly released a 600M parameter automatic speech recognition model in early June Nemotron 3.5 ASRThis model features “one checkpoint handles 40 languages,” supports real-time streaming output, and can run without a GPU. For developers building local Agent pipelines, this might be one of the most practical open-source voice options available right now.

What is Nemotron 3.5 ASR

Nemotron 3.5 ASR is NVIDIA’s successor after nemotron-speech-streaming-en-0.6bA multilingual expansion version released after the English version. It also has 600M parameters, but through the “language-ID prompt conditioning” mechanism, a single model can handle speech transcription across 40 languages, without the need for separate models for each language or model switching.

The model architecture uses Cache-Aware FastConformer-RNNTConsisting of 24 layers of FastConformer encoders paired with an RNNT (Recurrent Neural Network Transducer) decoder. The term “Cache-Aware” refers to the model’s ability to cache encoder self-attention and convolution activation states during streaming audio processing, processing only new audio segments each time while reusing previous cache, thus avoiding redundant computation caused by overlapping windows in traditional buffered streaming. The result is that each audio frame is processed only once, with both computational cost and end-to-end latency decreasing simultaneously, while accuracy remains unaffected.

40 languages, three levels

These 40 languages are not all at the same quality level, and NVIDIA divides them into three tiers:

• Ready-to-use (19 languages)Plug-and-play, high-accuracy speech recognition supporting English (American/British), Spanish, French, Italian, Portuguese, Dutch, German, Turkish, Russian, Arabic, Hindi, Japanese, Korean, Vietnamese, Ukrainian, and more.
• Extensive coverage (13 languages): Production-ready speech recognition covering Polish, Swedish, Czech, Norwegian, and other European languages.
• Adaptive type (8 languages)The tokenizer is already supported, but fine-tuning with domain-specific data is required to achieve full transcription quality.

There are two language control modes: set at inference time. target_lang Specify language (such as zh-CN、ja-JP), or set it as auto Let the model auto-detect. In auto-detect mode, the model outputs language labels after punctuation, making it suitable for handling mixed-language audio sources.

Adjustable delay: one knob controls speed and accuracy

Nemotron 3.5 ASR has a very practical design:att_context_size Parameters allow you to freely adjust the trade-off between latency and accuracy during inference, without retraining the model.

• [56, 0]: 80ms Ultra-Low Latency Mode, Perfect for Real-Time Interaction Scenarios
• [56, 1]: 160ms Low Latency
• [56, 3]560ms Balanced Mode (Default)
• [56, 13]1.12s Maximum Accuracy Mode

A single checkpoint covers the full range. Developers can select the optimal operating point at inference time based on the application scenario (real-time captions vs. batch transcription), without needing to prepare different models for different scenarios.

Where is it faster than Whisper?

In speech recognition, OpenAI’s Whisper has long been the open-source benchmark. Nemotron 3.5 ASR isn’t positioned to “replace Whisper,” but to offer a better alternative in specific scenarios.

E2E Networks benchmarked Whisper large-v3-turbo (809M parameters), Parakeet TDT 0.6B, and Nemotron Speech Streaming 0.6B across 58 configurations on NVIDIA L4 GPUs. The results show that Nemotron achieved 258x real-time processing speed in streaming scenarios, with Word Error Rate (WER) remaining consistently stable across all configurations. In contrast, Whisper experienced 3.5% WER degradation on short audio segments (chunk=10s), which is a hidden pitfall that most benchmarks fail to expose.

The specific differences can be understood from several perspectives:

• Streaming capabilityWhisper is an encoder-decoder architecture, inherently suited for batch processing; Nemotron’s Cache-Aware FastConformer-RNNT is designed for streaming, outputting while listening, with end-to-end latency potentially below 100ms.
• parallel processingNVIDIA’s official data indicates that Nemotron ASR on H100 can process more concurrent streams simultaneously than its predecessor Parakeet RNNT, directly reducing the operational cost per stream.
• CPU-friendlyThe 600M parameter model size allows it to run on pure CPU environments and Apple Silicon (MPS), with audio staying on-device and no per-minute billing concerns.

However, Whisper also has its advantages: Whisper large-v3-turbo remains more robust in noisy environments and accented audio, reflecting its training foundation on hundreds of thousands of hours of diverse internet audio.

Additional Features: Word Boosting and Speaker Diarization

Beyond basic speech-to-text, Nemotron 3.5 ASR also comes with several practical built-in features:

• Word BoostingVocabulary can be customized for priority recognition—such as technical terms, brand names, or industry jargon—without retraining the model. This is especially useful for speech recognition in healthcare, legal, and technology applications.
• Speaker DiarizationIdentify and distinguish different speakers, suitable for multi-person dialogue scenarios such as meetings, podcasts, and interviews.
• Automatic Punctuation and CapitalizationOutput text with proper punctuation and correct capitalization; no additional punctuation recovery steps are needed.

Deployment Methods and Licensing

Nemotron 3.5 ASR with OpenMDW-1.1 Open source licensed, model weights are available on HuggingFace (nvidia/nemotron-3.5-asr-streaming-0.6b) and can be directly used for commercial purposes.

There are several options for deployment methods:

• Use NeMo framework directly：pip install nemo_toolkit[asr] Just a few lines of code to load the model and start transcribing.
• OpenAI-compatible HTTP server: Can wrap the model in /v1/audio/transcriptions Behind the endpoint, any OpenAI-compatible client can use it directly.
• NVIDIA NIM:NVIDIA’s cloud-hosted services also offer Nemotron ASR deployment options.
• HuggingFace TransformersSupports the standard HuggingFace loading process.

The LiveKit team also released detailedIntegration GuideDemonstrates how to integrate Nemotron 3.5 ASR into a real-time voice Agent pipeline, including a complete implementation example of a local teleprompter.

What it means for the local Agent ecosystem

Previously, when running speech recognition locally, developers’ options were usually Whisper (accurate but slow, large models require GPU) or various smaller models (fast but limited language support). Nemotron 3.5 ASR fills a middle ground: small enough (600M parameters), capable of real-time inference on CPU alone; broad language coverage (40 languages), no need for separate models per language; modern architecture (Cache-Aware streaming), controllable latency.

For developers building local AI Agents, this means voice input is no longer a step that relies on cloud-based APIs. The model can run entirely offline, keeping audio data on the local machine while maintaining real-time streaming capability and acceptable recognition quality. Combined with NVIDIA’s own Nemotron 3 Nano (language model) and Magpie TTS (text-to-speech), the complete Agent pipeline—from voice input to reasoning to voice output—can all run locally.

Things to pay attention to

This model isn’t perfect. According to Geeky Gadgets’ review, there are still areas for improvement in the following aspects:

• Punctuation accuracy in real-time transcriptionIn streaming mode, punctuation marks are occasionally inserted at incorrect positions.
• Automatic language detection：auto Pattern detection results are inconsistent; specify manually when the language is known. target_lang The results are noticeably better.
• Specific languages require fine-tuningThe 8 languages at level 3 require additional fine-tuning to reach usable quality—they can’t just be downloaded and used directly.

Overall, Nemotron 3.5 ASR is an important strategic move by NVIDIA in the open-source speech recognition space. It’s not designed to beat Whisper, but rather provides a more competitive choice in three key areas: streaming, multilingual capabilities, and local deployment.

Source: KOCPC Chinese