Can you imagine AI producing a complete short drama for you from just a one-sentence description? From a fleeting idea, a novel, to a professional screenplay, AI automatically completes the entire workflow from scriptwriting, storyboard design, and character development to final video generation—this is no longer just science-fiction movie material. The popular open-source project by Professor Huang Chao’s team from the Department of Data Science at the University of Hong Kong ViMax Framework is turning this vision into reality.
HKU open-sources multi-agent video generation framework “ViMax,” enabling AI to write, direct, and act from a single sentence.
ViMax has garnered over 5,500 stars on GitHub and is fully open-sourced under the MIT license. Described as an “Agentic Video Generation” framework, it is essentially an all-in-one AI film crew: an AI screenwriter crafts the script, an AI director sets the pace, an AI cinematographer composes shots and camera movements, and an AI editor polishes the details. They all collaborate automatically throughout the entire process—you just need to provide the idea.
The Leap from Fragmented Generation to Systematic Production
While current mainstream AI video tools like veo 3 and Runway perform impressively at generating short clips, they can only produce segments lasting a few seconds to several tens of seconds, and they face two critical bottlenecks. The first isthe complexity of narrative planning explodes,Long-form videos require coordinating the narrative logic of hundreds of shots, involving multiple dimensions such as character development, plot progression, and thematic resonance. This multi-layered story structure far exceeds the single-turn processing limits of current language models. Second,Cross-camera visual consistency challengeExisting generative models act independently and lack memory of preceding content, causing character appearances and scene styles to frequently “change faces” between shots. This visual memory deficit creates a snowball effect in long-form videos, seriously degrading the viewing experience.
ViMax is specifically designed to address these two bottlenecks. Rather than further refining a single generative model, it completely rethinks the AI video production workflow at the system architecture level.
Four Creative Modes: Complete Coverage from Inspiration to Final Video
ViMax offers four distinct creative entry modes to meet the diverse needs of everyone from general users to professional creators:
- From Idea to Video (Idea2Video)The simplest mode lets users enter just a one-sentence idea (for example, “What happens when cats and dogs are best friends and meet a new kitten?”), and the system automatically completes the entire process from script to video.
- Novel to Video (Novel2Video)An intelligent literary adaptation engine that automatically condenses full-length novels and transforms them into episodic video content, enabling character tracking, narrative compression, and scene-by-scene visual adaptation.
- Script to Video (Script2Video)Designed for professional creators, it supports standard Hollywood-formatted screenplay input, giving users complete control over every detail from visual language to pacing.
- Smart Cameo（AutoCameo）Upload a personal photo, and ViMax can put you or your pet into the video as story characters while maintaining a consistent appearance and natural interactions.
In-Depth Analysis of Multi-Agent Collaborative Architecture
ViMax’s core design philosophy is to break down long-form video production into five coordinated stages, with each stage handled by a dedicated AI agent:
Phase One: Script WritingWhether the user inputs a simple idea, a complete novel, or a script excerpt, the screenwriter agent can reorganize it into a standardized film and television screenplay, including scene settings, character dialogue, and story pacing. This stage adopts RAG (Retrieval-Augmented Generation) technology: the system first performs an in-depth analysis of the raw material and builds a comprehensive index covering character relationships, plot threads, and thematic elements, ensuring that every local planning decision can “see” the overall story landscape.
Phase 2: Storyboard PlanningThe storyboarding agent applies professional film production theory to design precise cinematic language for every scene, covering camera placement, movement trajectories, lighting layout, and character blocking to form a complete shooting blueprint. ViMax adopts a recursive narrative decomposition strategy here, breaking down the full script according to narrative logic intoEvent layer(core narrative node),Scene layer(performable drama unit) andlens layer(Precise Instruction Execution) Three levels, ensuring the language model faces a moderate planning scope at each cognitive level.
Phase 3: Visual Asset GenerationIt adopts a two-step “image-first, video-second” generation strategy. First, core visual elements are created (character designs, scene designs, key frames), and then corresponding dynamic video clips are generated based on them. The system intelligently selects the reference images needed for the first frame of the current shot, including storyboard data from the preceding timeline, so that the longer the video, the more accurate multiple characters and environmental elements become.
Phase 4: Quality ControlFor each generation task, the system produces multiple versions simultaneously. A quality control agent uses a Multimodal Large Language Model / Vision Language Model (MLLM/VLM) to conduct a professional evaluation and select the version that best meets the storyboard requirements. If none of the candidates meet the standards, the system automatically adjusts parameters and regenerates, simulating the workflow of human creators repeatedly selecting the best materials.
Phase 5: Unified CoordinationThe Director Agent serves as the overall commander, monitoring the coordinated operation of the entire workflow, maintaining stylistic consistency across outputs at every stage, and handling task allocation and information synchronization among agents.
Underlying Architecture: Modular Design and Multi-Model Support
ViMax is developed with Python 3.12 and uses uv as the project management tool. Its dependencies include mainstream open-source packages such as the Faiss vector database (for RAG retrieval), MoviePy (video editing), OpenCV (image processing), and LangChain (AI workflow orchestration).
In terms of model support, ViMax features a modular design that lets users freely swap the underlying model:
- dialogue model: Supports various models on Google Gemini and OpenRouter, and also supports MiniMax-M2.7 (1 million token context).
- Image generatorSupports Google Imagen and other APIs.
- Video GeneratorSupports Google Veo and other APIs
This design frees ViMax from vendor lock-in, allowing users to mix and match based on budget and needs. For example, using OpenRouter to access affordable LLMs paired with Google Veo’s high-quality video generation creates the most cost-effective combination.
AutoCameo: Put Yourself in the Video
A highlight of ViMax is its AutoCameo feature. Users upload a photo of themselves, and the system can make the person in the photo appear in videos with a consistent look, playing various roles. This is no longer a simple composite where a head is pasted onto a body—instead, the AI understands the person’s features and maintains consistency in appearance, expression, and movement across dozens of shots.
This is a big deal for content creators: you can “star” in short films without actually filming, turn your pet into the protagonist of a story, or even create personalized birthday greeting videos. Effects that traditionally required green screens, professional studios, and post-production teams can now be achieved with a single open-source framework.
Open Source Ecosystem and Community Response
Since ViMax was open-sourced in November 2025, it has gained over 5,500 stars and 935 forks on GitHub, a growth rate that is quite remarkable among academic open-source projects. In addition, the HKU Chao Huang team (HKUDS) has been very active in the AI open-source domain in recent years. ViMax is already the third major AI breakthrough the team has launched in a short period; previously, they had accumulated extensive open-source achievements in areas such as recommendation systems and neural networks.
Based on the Discord discussion, developers’ attention regarding ViMax is mainly focused on: the potential for integration with existing video generation models (such as CogVideo and Stable Video Diffusion), the actual effectiveness of the AutoCameo feature, and its runtime performance on consumer-grade GPUs.
ViMax Open Source Project (click here)
Summary: The Dawn of the One-Person Film Crew Era
ViMax doesn’t represent the advancement of a single AI model; it’s the key turning point that brings AI video production from the “tool era” into the “system era.” Current AI video tools are like providing high-quality brushes and paints, but creating a complete painting still requires a human artist. ViMax’s positioning is more like providing a complete “automated factory”: from raw materials (creative ideas) to finished products (complete videos), everything is automated.
ViMax is still in early stages. Currently, it relies on external APIs such as Google Veo and Imagen for the actual image and video generation, which means users must pay the corresponding API fees. The configuration file shows that the video generation API has a daily limit of just 10 calls, reflecting that high-quality AI video generation still comes with a significant cost. If you were to use Seedance 2.0, it would be even more expensive; ViMax merely streamlines the entire production workflow.
Source: KOCPC Chinese