Just take a casual video with your phone, and the people in it can be reconstructed into 4D dynamic models. You can view them from any angle or even orbit the camera around them, as if multiple cameras were filming simultaneously from different angles on set. 4DAnyone, published by Ant Research in collaboration with Zhejiang University and the Hong Kong University of Science and Technology, has taken this technology from a research paper to public availability. The project recently gained significant attention after being listed on Hugging Face.
4DAnyone: From a single phone video to a dynamic 4D model
Traditional human 4D reconstruction requires multiple synchronized cameras. Take DNA-Rendering, the industry benchmark database, as an example: it uses 48 synchronized cameras for capture, which is why volumetric human capture has been confined to studios for years. 4DAnyone’s approach fundamentally changes this limitation—the input only requires an uncalibrated phone video. The system first uses a video diffusion model to generate dozens of multi-view consistent new frames, then elevates these frames into 4D Gaussian Splatting (4DGS) representation, ultimately producing a dynamic model that can be freely rotated and viewed from any angle.
AntResearch just dropped 4DAnyone on Hugging Face!
Convert a video of any person into a 4D gaussian splat for that person
▶️ https://t.co/JyRkvXVKc7 pic.twitter.com/Cc6LovWtqF
— Hugging Apps (@HuggingApps) August 21, 2026
This articleThesisAccepted by SIGGRAPH Asia 2026. The code and models have been open-sourced under the Apache-2.0 license. In testing, after uploading a close-up video of a person, the system generates surrounding views while maintaining the person’s appearance consistency across different pitch angles. Users can freely choose the number and layout of output views, from a basic 6-view surround to a full 48-view omni-directional capture, all with one click.
The Perspective Bottleneck of Diffusion Models and Its Solutions
The main challenge in generating multi-view images with previous video diffusion models was capacity limitations. The single-pass feed-forward capacity of the Diffusion Transformer (DiT) is limited, and when the number of target views to generate exceeds the processing range, the views must be split into multiple groups for batch processing. This triggers two cascading problems.
First, as the number of reference viewpoints grows, the conditioning cost for each new viewpoint increases linearly at O(N), diluting cross-view appearance guidance and causing subsequently generated viewpoints to drift further and further from the original character’s true appearance. Second, after grouping, different groups cannot directly exchange information, leading to overall structural drift where different batches of viewpoints each produce inconsistent geometric shapes. This is exactly the main bottleneck of past approaches to 4D reconstruction with diffusion models: each individual frame looks fine when viewed from a single viewpoint, but they all contradict one another when fed into 4DGS reconstruction. The two core techniques proposed by 4DAnyone address these two bottlenecks respectively.
Reference Context Packing (RCP) compresses the growing reference views into a fixed-length mixed-resolution context, reducing the reference context cost from O(N) to O(1). Target Context Routing (TCR), on the other hand, dynamically rotates the grouping of target views during the denoising process: in the high-noise phase, it frequently swaps the contents of each group to determine the global structure, while in the low-noise phase, it fixes the grouping so that adjacent views stabilize details together. Together, these two mechanisms allow 4DAnyone to handle dozens of target views in a single inference without structural collapse.
Training scale and benchmark performance
4DAnyone’s training used 128 H20-3E GPUs, combined with the self-built game engine dataset MVGameHuman, as well as light-stage and real-world video data. On the DNA-Rendering benchmark, 4DAnyone achieves a PSNR of 24.33 dB, while the best comparison method only reaches 21.47 dB. This nearly 3 dB gap brings a clearly visible difference in multi-view consistency and reconstruction quality, especially in the fidelity of human silhouettes and clothing textures, where 4DAnyone significantly outperforms existing diffusion model methods.
The paper also validates on three sequences from the DyMVHumans dataset, achieving a reconstruction result of 21.03 dB. This figure is lower than DNA-Rendering’s score, but it reflects the generalization limits in real uncontrolled scenes. The paper candidly acknowledges that the generated back details are essentially “reasonable speculation consistent with the front,” because the original video never captured the back at all, and the system must fill in the missing information based on human body priors learned during training. This is precisely the fundamental difference between single-view reconstruction and multi-camera capture.
Actual Usage and Hardware Requirements
The project provides complete inference workflows and multiple camera configurations on GitHub, supporting layouts from basic 6-view surround setups to a full 48-view three-tier elevation arrangement. Input videos should have a resolution of at least 720p, with 1080p 9:16 portrait footage being ideal. The subject should be fully visible or framed from the upper body up, with at least 121 frames and only slight camera movement. Users can freely adjust the number of viewpoints, elevation tiers, and coverage range via command-line parameters to suit different downstream application scenarios. For example, the 6-view configuration is suitable for quick previews, 24 views for standard 4DGS reconstruction, and the 48-view three-tier elevation setup achieves free-viewpoint rendering quality.
4DAnyone – GitHub
Minimum hardware requirement is 32 GB VRAM, with optional FlashAttention-3 or SageAttention for acceleration. Installation follows the standard conda and pip workflow, and model weights plus example videos are downloaded automatically on first run—no need to fetch them manually. For researchers familiar with PyTorch environments, the entire process from download to generating the first set of results can be completed in about half an hour. The project also includes a detailed nerfstudio integration guide, allowing users to feed the generated videos into subsequent 3DGS reconstruction.
However, the training-side costs remain substantial—the computing resources required for 128 H20-3E GPUs are far beyond what a typical academic lab can afford. That said, inference can already run on a single high-end graphics card, meaning the technology is not too far from real-world deployment. For content creators or game developers who need to quickly generate dynamic 3D human models, 4DAnyone offers a more flexible path than traditional multi-camera studio capture. The project has already accumulated over 640 GitHub stars and has drawn community attention on Hugging Face, and future community contributions could further lower the barrier to entry.
Source: KOCPC Chinese