ByteDance’s AI research team Seed officially launched the native audio-video full-duplex large model SeedRealtime on August 5, and simultaneously announced that the model has been fully rolled out on the Doubao App, claiming to be the first in the industry to achieve large-scale deployment of audio-video full-duplex technology. This model uses a unified architecture to natively integrate audio, video, and text, enabling real-time interaction on continuous multimodal information streams and delivering a natural conversation experience of “seeing, listening, and speaking” all at once. After updating the Doubao App to the latest version, users can select “make a call” in the dialog box to enter the video call interface and experience it for free.
From cascade to end-to-end: the technical path of full-duplex interaction
The Seed team is atTech blogIt points out that real-time audio-video interaction has long been stuck between two modes. Traditional cascaded systems rely on multiple modules such as ASR, vision models, and TTS in series, with latency stacking layer by layer and information loss at each stage; although end-to-end models are smoother, many solutions still rely on an external VAD (Voice Activity Detection) to determine turn-taking, remaining close to a question-and-answer half-duplex interaction.
The key difference of SeedRealtime lies in unifying audio, visual, timing, and expression into a single end-to-end model, enabling perception, understanding, decision-making, and expression to proceed simultaneously on continuous audio-video streams, rather than listening first, then watching, and finally responding. On the engineering side, the team reduces response time through continuous audio-video chunked input and streaming output generation, while combining efficient quantization and inference optimization to improve service efficiency. The end-to-end human evaluation results officially released by Seed show that, compared with cascade models, SeedRealtime reduces audio-video dialogue pacing issues by about half, with noticeable declines in interruptions, slow responses, and stalls caused by background noise and bystander chatter being mistakenly triggered. The probability of a smooth, complete single-session dialogue has also improved significantly.
Three core capabilities: joint understanding, proactive intervention, and rhythm control.
SeedRealtime publicly claims three core breakthroughs, which correspond to the three most difficult aspects of real-time audio-video interaction.
First is audio-visual joint understanding. The model natively supports deep fusion of sound, imagery, and temporal information. When a user says “how do I do this,” it can determine what “this” refers to by combining the current screen, gestures, gaze, and past actions; when encountering homophones or ambiguous speech, it also uses the visual scene to disambiguate, so that what is seen, heard, and said all line up.
Next is the proactive interaction capability. The model maintains continuous environmental awareness and can proactively alert users when it detects changes in the visual state of a scene. For example, if a user instructs it to “remind me when you see a certain exhibit,” the model will speak up as soon as the camera pans across the target, and it can deliver real-time correction and feedback based on visual state changes—elevating interaction from passive response to proactive collaboration.
Third is a fluid interaction rhythm. Turn-taking decisions are no longer delegated to external VAD rules; instead, the model continuously decides based on multimodal information—speaking up without hesitation when it should take the floor, and staying silent without interrupting when it shouldn’t. In the official demo at Daxing Airport, a crowded and noisy environment, the model isn’t falsely triggered when companions chat about unrelated topics, yet when they formally ask about flight information, it can answer by drawing on previously seen visual content, and when it spots a ride-hailing sign, it naturally chimes in with route guidance.
Seven real-world use cases: people recognition, translation, page monitoring, and study companion.
The Seed team demonstrated the model’s capabilities through 7 real-world scenario cases. In a multi-person dining scenario, after the user introduced each friend in turn, the model was able to match names to faces based on physical features, continuously associate each person’s voice with their identity in subsequent conversations, and offer a travel plan that balanced everyone’s differing needs:
In a Sichuan restaurant, the model can identify dishes directly from the screen, recommend them to foreign diners in English, and even explain cultural background questions like “why is there no fish in yuxiang shredded pork.”
At the Hebei Museum, the model continuously watches the camera feed. When it scans over the designated gold-and-silver-inlaid bronze tiger-devouring-deer screen base, it proactively alerts, then explains treasures such as the Changxin Palace Lamp and techniques such as casting and gold-and-silver inlay:
Operational examples are equally specific: when the user poured whole coffee beans directly into the portafilter, the model immediately pointed out, “you’ll need to grind them into a fine powder first,” and after extraction, based on the crema color and liquid volume in the cup, it proactively suggested, “shorten the extraction time by 2 to 3 seconds next time.”
While studying the ResNet paper, the model can explain how skip connections alleviate gradient vanishing alongside the network architecture diagrams, and during rapid page-flipping, it proactively stops when it recognizes the “3.4 Implementation” section, reporting training configurations such as learning rate, momentum, and weight decay.
The family scene demonstrates anti-interference capability: while the father talks on the phone in the background, the model can still follow the direction of the girl’s finger to provide real-time pronunciation correction and example sentences, without being led astray by unrelated conversation.
In a crowded airport, a companion casually mentioned “Old Li’s flight” during small talk, but the model wasn’t triggered to respond; when the user formally asked later, the model answered by combining the information from the big screen it had just seen and supplemented it with the baggage carousel location found online.
The battlefield of real-time multimodal dialogue: everyone is rushing in.
SeedRealtime is targeting the real-time multimodal dialogue track, where multiple vendors have already staked their positions in recent years. OpenAI demonstrated voice mode starting with GPT-4o and launched the Realtime API, focusing on low-latency voice-to-voice conversation; Google offers real-time voice conversation capabilities through Gemini Live. By comparison, SeedRealtime differentiates itself by incorporating video understanding into the same full-duplex decision-making system, enabling the model to continuously watch the screen and proactively interject, rather than being limited to voice turn-taking. ByteDance has placed it directly into the Doubao app, which has a massive user base, moving the technology from research demonstration to consumer-facing product deployment.
Conclusion
ByteDance’s launch of SeedRealtime advances real-time AI interaction from “understanding what you say and responding quickly” to “seeing what you see and taking the initiative.” The official roadmap for what’s next includes continuously reducing end-to-end latency, strengthening speaker identification and response judgment in complex multi-person scenarios, and enabling tool calling so the model evolves from “being able to converse” to “being able to take action” — turning real-time multimodal understanding into concrete assistance like searches, bookings, and task completion. For users, the most immediate change has already appeared in the video call interface of the Doubao app (using the “Make a Call” feature). The editor has tested it and confirmed it works in Taiwan as well. If you’re interested, feel free to try it yourself.
Source: KOCPC Chinese