Microsoft Research Asia recently released the open-source speech AI model VibeVoice series, which covers three major areas: automatic speech recognition (ASR), text-to-speech (TTS), and real-time streaming TTS. It focuses on business applications featuring “single-pass long audio processing” and “simultaneous multi-speaker recognition,” breaking through the limitations of traditional models that require splitting audio into short segments and stitching them together. As of now, VibeVoice has GitHub It has accumulated over 48,000 Stars, making it one of the most watched open-source voice projects since the second half of 2025.
Three Major Models, Each Addressing Different Pain Points
VibeVoice is not a single model, but a series of model families, currently containing three members:
- VibeVoice-ASR（7B）A speech-to-text model that can process up to 60 minutes of continuous audio in a single session, outputting structured results that include speaker diarization, timestamps, and content. Supports over 50 languages and allows users to provide custom hotwords to improve recognition accuracy for domain-specific terminology.
- VibeVoice-TTS（1.5B）A text-to-speech model capable of synthesizing up to 90 minutes of speech, supporting up to 4 different characters speaking simultaneously, with each character having an independent voice and speaking style that remains consistent throughout the entire dialogue. Supports multiple languages including Chinese and English. The paper for this model has been accepted as an Oral presentation at ICLR 2026.
- VibeVoice-Realtime（0.5B）A lightweight real-time TTS model with approximately 300ms first-token latency, supporting streaming text input, suitable for embedding in conversational AI applications. Recently added experimental voices for 9 languages (German, French, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Spanish), as well as 11 English style voices.
Core Technology: 7.5Hz Ultra-Low Frame Rate + Autoregressive Diffusion Generation
VibeVoice’s key technological breakthrough lies in the “continuous speech tokenizer.” Traditional speech models typically represent audio at a rate of 50 to 100 frames per second. While this works fine for short audio clips, processing a 90-minute long audio file results in a computational load that skyrockets to hundreds of thousands of tokens, far exceeding the processing capacity of current large language models.
VibeVoice compresses the frame rate to 7.5Hz, generating 90-minute dialogues requires processing only about 64,000 tokens, significantly reducing computational costs while maintaining audio quality without noticeable degradation. The underlying architecture adopts a Qwen2.5-based LatentLM next-token diffusion framework: first, a Variational Autoencoder (VAE) encodes speech waveforms into a sequence of continuous latent vectors, then a causal Transformer progressively predicts the next speech segment, enabling “sentence-by-sentence” autoregressive continuous speech generation.
This approach differs significantly from traditional TTS, which generates entire audio segments in one go. It enables the model to better understand contextual logic, avoiding issues with semantic inconsistency between sentences, and can naturally present non-verbal cues such as breathing, pauses, and accent variations in multi-speaker dialogue scenarios.
ASR Benchmark: Surpassing Closed-Source Multimodal LLMs
In speech recognition, VibeVoice ASR delivers impressive performance. According to Microsoft’s technical report, across five authoritative benchmarks including AISHELL-4, AMI, and AliMeeting, VibeVoice ASR consistently outperforms the previously leading closed-source multimodal large models.
Compared to OpenAI’s Whisper, VibeVoice ASR’s advantages lie in two main areas: First, Whisper requires splitting long audio into short segments for separate processing, which easily loses context at the cut points; Second, Whisper lacks native speaker diarization capability, resulting in higher error rates in multi-speaker scenarios. VibeVoice ASR integrates transcription, speaker recognition, and timestamp marking into a single inference. On Reddit, users conducted medical audio benchmarks testing 31 speech-to-text models, with VibeVoice 9B version also ranking among the top.
TTS Applications: Audiobooks, Podcasts, Meeting Notes
In terms of TTS, VibeVoice has quite clear application scenarios. Traditional AI podcast generation typically only produces a few minutes of two-person dialogue, while VibeVoice supports up to 90 minutes and up to 4 people speaking simultaneously. Users simply need to provide a text script with character annotations, and the model automatically generates conversational audio with natural breathing, pauses, and intonation variations.
The 0.5B Realtime version is better suited for embedding in actual products. With 300 milliseconds of first-token latency plus streaming text input capability, it can be directly integrated into conversational AI agents as a real-time TTS engine, eliminating the need to connect to third-party TTS services.
Open Source Licensing and Deployment
VibeVoice’s full lineup uses MIT open-source licensing, allowing local deployment, which is particularly attractive to enterprises that prioritize data privacy. The model weights are published at Hugging Face Currently, it has been integrated into the Hugging Face Transformers ecosystem and also supports vLLM inference acceleration.
However, VibeVoice was temporarily removed in September 2025 after being discovered to be misused for improper purposes such as deepfake audio. Microsoft stated at the time that “responsible AI use is one of Microsoft’s guiding principles.” The TTS code was later restored, but this incident also highlighted the potential risks of high-quality speech synthesis technology in defending against Deepfakes. Microsoft explicitly warns users in the README that “you must ensure the reliability of generated content and avoid using it in misleading ways.”
Source: KOCPC Chinese